Oracle Data Integration Solutions - Oracle Data · PDF fileOracle OpenWorld 2014 Oracle Data Integration Solutions (DIS) Oracle Data Integrator (ODI) Oracle ... •Easy to extend and

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |

Oracle Data Integration Solutions

Joachim Jaensch - Principal Sales Consultant

Short Overview


Safe Harbor Statement

The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.

3


Agenda

4

1

Oracle OpenWorld 2014

Oracle Data Integration Solutions (DIS)

Oracle Data Integrator (ODI)

Oracle GoldenGate (OGG)

Oracle Enterprise Data Quality (OEDQ)

Oracle Enterprise Metadata Management (OEMM)

Oracle Data Service Integrator (ODSI)

Oracle Active Data Guard (OADG)

2

3

4

5

6

7


Oracle Data Integration Solutions (DIS) Introduction

Oracle Confidential – Internal/Restricted/Highly Restricted 5


Oracle Data Integration Solutions 12c

Deliver real-time data integration for Cloud and Big Data

Big Data

Cloud

Apps

Database

• Real-time data replication; optimized for Database 12c and Oracle Exadata

• End-to-end integrated with simplified deployment

• Unified tooling for both structured data sources and Hadoop / NoSQL

• Flexible deployment on-premise or in the Cloud for heterogeneous systems

• Expanded support for 3rd party systems and Oracle Applications in real-time data integration and continuous availability solutions

Oracle Data Integrator

Oracle GoldenGate

Oracle Enterprise Data

Quality

Oracle Data Services

Integrator

Oracle Metadata

Management

Oracle Active

Data Guard



E-LT: Bulk Data Processing and Fast Data Transformation

Big Data

Cloud

Apps

Database

• Certified for leading technologies to deliver fast time to value

• High-performance, low cost of ownership E-LT architecture

• Lightweight deployment

• Flexible, easy to enrich functionality


High Performance E-LT

Declarative Design

Data Services

Extensible Knowledge Modules

Structured & Unstructured Data

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | 8

Industry Leading Performance

Extremely Fast Execution and Reduced Cost

E-LT provides a flexible architecture for

optimized performance on any platform

Benefits

• Leverages set-based transformations

• Improves performance for loading,

no network hop

• Takes advantage of existing infrastructure:

hardware and software

Conventional ETL Architecture

Extract Load

Transform

Next Generation Architecture

“E-LT”

Load Extract

Transform Transform


Declarative Design

Improved Developer Productivity For Faster Time to Solution

9

• Easy and universal for designing simple to complex mappings

• Robust and Reusable: Create Mappings once for any physical design

• Quick to define and refactor maps, high degree of automation

• Extensible model for any data integration mechanism

• Benefits

• Significantly reduces the learning curve

• Shortens implementation times

• Simpler maintenance

ODI Declarative Design

Define How: Knowledge Modules

2 1

Automatically

Generate

Dataflow

Define

What You Want


Reverse Engineer Metadata

Journalize (CDC)

Load from Source to Staging

Check Constraints

Integrate, Transform

Data Service

Knowledge Modules

Simpler Physical Design and Shorter Implementation Time

10

Oracle Sqoop Hive HBase Oracle Merge

SAP ERP

SAP BW Oracle

Datapump Oracle DBLink

JMS External Tables

Teradata

Oracle Spatial

Siebel eBusines

Suite IBM DB2 Netezza SCD

Key Architecture Benefits: • Faster development and simpler

maintenance using templates

• Easy to extend and add new best

practices

• Enforces predictability and reduces cost

of ownership

Pluggable Knowledge Modules Architecture

Sample out-of-the-box Knowledge Modules:


Big Data: Hive, HBase, Sqoop

Best for Oracle: Merge, Spatial, Multi-Table Insert, Optimizer Hints and more

Real-Time: CDC with GoldenGate

Applications: E-Business Suite, Siebel, PeopleSoft, JD Edwards Enterprise One, JD Edwards World, SAP ERP and SAP BW

Heterogeneous: Optimizations for all major RDBMS: IBM DB2, Microsoft SQL Server, Teradata, Netezza, etc.

Applications

Legacy

Databases

Big Data

Cloud

Extended Connectivity

Leverages Existing IT, Faster Implementation


• Key Architecture Benefits: 100% Java, Open APIs, fast E-LT

D A

B

File C

C$_0

C$_1

LKM

LKM

IKM

I$ E$ (Errors)

CKM IKM RKM

JKM

Check-Load Transform Extract-Load

ODI Agent

Application & SOA Business Intelligence & Data Warehouse

ODI Agent may be deployed in any part of the architecture

How ODI Works: E-LT Architecture

12

High Performance, Flexible, Lightweight Architecture


Integration Between GoldenGate and ODI 12c

• Support real-time and bulk from the same

interface

• Deploy GoldenGate processes via ODI Bulk Load Scenario Real Time Scenario

• Enable GoldenGate journalizing on

interfaces without restructuring.

• Manage GoldenGate Capture/Delivery

instances in ODI Topology

• Set common GoldenGate parameters

graphically

Capture Delivery


Oracle GoldenGate

Oracle GoldenGate

Log-based Change Data Capture and Delivery

Active-Active Replication

Reliability and Transaction Integrity

Heterogeneous Source and Targets

Big Data/Java/FlatFile Integration

• High-performance, low-

impact real-time data

integration and

replication

• Timely data for

improved business

insight

• Continuous availability

for 24/7 operation Big Data

Cloud

Apps

Database

Heterogeneous Data Replication

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle OpenWorld 2014 15

More Information later!


Oracle Enterprise Data Quality

Profile, Explore, Audit

Parse, Standardize, Cleanse

Governance

Match, Merge, Enrich

Case Management

• Improves data accuracy,

usability and ‘fitness for

purpose

• Unified interface for

ease-of-use, lower TCO

• Depth of capability in

multiple data domains

reduces project risk

Big Data

Cloud

Apps

Database


Data Quality for Customer and Product Data

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential – Internal/Restricted/Highly Restricted

Data Quality Issues – A simple Customer Table

Name Address City State Zip Phone Email

Bob Williams 36 Jones Avenue Newton MA 02106 617 555 000 [email protected]

Robert Williams 36 Jones Av. MA 02106 617555000

Burkes, Mike and Ilda 38 Jones av. Nweton MA 02106 617-532-9550 [email protected]

Jason Bourne, Bourne & Cie. 76 East 51st Newton MA 617-536-5480 6175541329

… … … … … … …

Mis-fielded data

Matching Records

Typos

Mixed business and contact names

Multiple Names

Non Standard formats

Missing Data

17


• Solution

– Profile business data to find the bad data and assess quality over time

– Cleanse, match and merge data before it gets loaded into the Data Mart or Data Warehouse

– Repair data during batch processing flows

• Benefits

– Stop the garbage-in-garbage-out cycle!

– Improve the trust of data within business marts and data warehouses

– Enable IT to deliver value to the lines of business during every data integration flow

– Improve business efficiency with match and de-duplicate redundant records

Profile, Cleanse and Govern Business Data


Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential – Internal/Restricted/Highly Restricted 19

Profiling – Understand the data first

Interactive exploration of data, identifying distribution and outlying values with drill-

downs

Identify and quantify issues in

data

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential – Internal/Restricted/Highly Restricted

Oracle Enterprise Data Quality – Standardization

• Standardize, Transform and Parse • Split names and name elements • Identify individuals and businesses • Derive additional attributes

Name: Dr Ellen Van Der Heijde

Title: Dr

First: Ellen

Last: Van Der Heijde

Gender: Female

Name: Jalila Abdul-Alim (Do Not Call)

First: Jalila

Last: Abdul-Alim

Gender: Female

Note: Do Not Call

Title: Mr

First: R

Middle: J

Last: MacDonald

Gender: Male

Name: Mr RJ & Mrs FB MacDonald

Title: Mrs

First: F

Middle: B

Last: MacDonald

Gender: Female

20


Oracle Enterprise Data Quality - Matching

•Match & Merge data from disparate sources •Create ‘best’ record based on survivorship rules

Title: Mr First: Robert Last: Fulmar Gender: Male DoB: 12/05/1978 Phone: 555-120-1329 Address: 9405 Main St Fairfax Virginia 22030

First: Bob Last: Fulmar Gender: Male Email: [email protected]

Title: Dr First: R Last: Fulmer DoB: 01/01/1978 Email: [email protected] Address: 9407 Main Street Fairfax VA 22031-4001

Title: Dr First: Robert Last: Fulmar Gender: Male DoB: 12/05/1978 Email: [email protected] Phone: 555-120-1329 Address: 9407 Main St Fairfax VA 22031-4001



Oracle Enterprise Metadata Management

Report to Source Lineage

Impact Analysis, Model Versioning

Supports Metadata Standards

Annotations and Tagging

Big Data Enabled Big Data

Cloud

Apps

Database


Trust your data

• Exploring Metadata

• Sarbanes-Oxley Act (SOX) Compliance

• Stewardship Collaboration

• Lifecycle Change Management

• Data Standardization & Data Governance


Metadata Management

“Metadata Management involves

managing data about other data, whereby this ‘other data’ is generally referred to as content.

- wikipedia


”


Value of Enterprise Metadata Management


ETL

BI Dashboards

App

ETL

ETL

How was sales figure calculated?

What will happen if I change this

table?

What reports use the mainframe

data? Sys Admin

Executive

BI Developer

Where did this data

come from?

Application User

Which reports use this

customer data?

CDC

Hadoop Data Lake

Data Steward

Can I trust the sources of this

customer data?

ETL

Developer

Solves a significant pain point for a wide variety of business consumers and technical staff

I want to design an experiment to measure the

success of a signup page. What data do I have?

Data Scientist

GG


Data Discovery Analytics

Oracle Metadata Management Logical Architecture

Oracle Confidential

25

Actionable Information

Data Lin

eage

Imp

act

An

alys

is

ETL Data Quality

Data Integration

Data Streams Integration Data Enrichment

File Based Cloud

Data Sources

Big Data Relational Applications

Business

Glossary

Metadata

Repository

Semantic

Linking


Exploring Metadata

Text Search and Metadata Browser,

Model/Diagram Visualizers, Birds-eye View, Metadata Reporting

Data Flow and Lineage Analyzer

Sarbanes-Oxley Act (SOX) Compliance

Versioning and Model Compare

URI Based Call back from BI reports or Business applications

Stewardship Collaboration

Feedback Comments and Review Board

Metadata Tagging with Labels

Multimedia attachments (documents, videos, presentation, code)

URL link references for e-mail, blogging, social networking

Detailed Feature List

Lifecycle Change Management

Metadata Harvesting from any DM, DW, ETL/DI, BI Tooling and Big Data/Hadoop

Enterprise Architecture Management (Metadata Configuration Management)

Impact of Change

Data Standardization & Data Governance

Supports Variety of Metadata Standards

Annotations and Tagging

Business Glossary with Semantic Lineage

Oracle Integrated

Oracle Database

Oracle Data Integrator, Oracle GoldenGate

Oracle Business Intelligence Enterprise Edition

Oracle Business Intelligence Applications (ODI and older Informatica based OBIA)


Oracle Data Services Integrator

Federated Data Services

Data Virtualization

Bi-directional Reads & Writes

Integrates with Oracle SOA

Integrates with Oracle Business Analytics

• Easily develop data

services and ad-hoc

queries without

infrastructure changes

• Open and standards

based—SOA, Web

Services, Java, Service

Data Object

Big Data

Cloud

Apps

Database


Data Federation and Virtualization


Oracle Service Oriented Architecture (SOA) – Product Mapping

Oracle® Reference Architecture, SOA Infrastructure, Release 3.2, 4479-03, February 2012


• Consolidation: Move data from distributed sources to central hub; consuming applications query hub database

• Data Virtualization / Federation: Query and update distributed sources in-place and provide combined result

• Hybrid Approach: Federation and Consolidation are combined based on best approach for individual use cases

Different Data Access Techniques

Source A

Consolidated Hub

Source B

ODI/OGG

Consumer

Consolidation only

Source A Source B

ODSI

Consumer

Data Virtualization only

Source A

Consolidated Hub

Source B

ODSI

ODI/OGG

Consumer

Source C

Hybrid Consolidation & Data Virtualization


When to Use Which Integration Technology

Data Virtualization Service Bus / Business Process

• Combine & normalize data access services

(Create, Read, Update, Delete)

• Orchestrate services to business processes

• Service Routing / Mediation

Data Virtualization Consolidation

• Data cannot be replicated (security, ownership)

• Consolidation is not efficient for access pattern

• Real-time access is required and replication is not efficient

• Transactional updates

• Need to change model frequently

• Performance is priority

• Data needs to be offloaded from original source

• Replication is used for redundancy


Data Virtualization Use Cases

• Data Services Layer – Provide a clean data abstraction for SOA and application developers

– Shield consumers from changes in data topology

• Augmenting Business Intelligence – Extend access to sources outside the data warehouse

• Rapid Prototyping of Data Integration – Quick creation of an interface to integrated data without building a consolidated store

• Cloud Data Integration – Combine onsite and offsite data resources in a unified interface


• Native SOA Data Integration Native XML type system allows processing and passing of hierarchical & SOA data.

Based on open standards (XML, WS, JDBC/SQL, XQuery)

Tight integration in Oracle SOA: Fast-path with OSB, Governance with OER

• Normalization and Reuse through Data Services Multi-layer approach to normalizing and transforming source data

• Highly Scalable, Industrial-Strength Performance Linear scalability through WLS Clusters

Performance through caching, streaming and query optimization

• Fine-Grained Security and Auditing Row and column-based data security

• Comprehensive Update Capabilities Automated CRUD operations on data services

Distributed XA transactions through WLS Transaction Manager

ODSI Core Strengths


Oracle Active Data Guard

Snapshot Database

Open for Read-Only Operations

Continuous Recovery

Using for GoldenGate Extract

Automatic Block Repair Big Data

Cloud

Apps

Database

Oracle Active Data Guard

High Availability of the Database

• Main Component of

Maximum Availability

Architecture (MAA)

• Role Transition between

Source and Standby

Database in Case of

Failure


Note: A single DR copy may be multi-purposed for different combinations of the use cases described

SYNC

or

ASYNC

Data Guard &

Active Data Guard

Use Cases: Data Guard and Active Data Guard Real-Time Data Protection and Availability for Oracle Database

New DB

Version Standby First Patching,

Database Rolling Maintenance

Exact copy

of primary Query & Report Offload

Open Read-Only

Snapshot

Standby

Convert to Test Database (open read-write)

Single Command Refresh

Exact copy

of primary Offload RMAN Backups

Exact copy

of primary Disaster Recovery

Manual or Automatic Failover

Redo Far Sync,

GoldenGate Downstream

Exact copy

of primary Source for thin snaps/clones

Exact copy

of primary Extract offload, source for

GoldenGate ALO mode


Production

Database

Continuous redo

shipping, validation & apply

Real-time Reporting Fast Incremental

Backups

Physical Standby

Database

Read-write

Workload

Data Guard

Standby Database: Failover Target


Production

Database

Continuous redo

shipping, validation & apply

Real-time Reporting Fast Incremental

Backups

Active Standby Database

(physical standby open read-only)

Read-write

Workload

Active Data Guard

Standby Database: Offload Production + Failover Target


Best Data Protection, Auto-Repair, and High ROI

Active Data Guard

Zero data loss at any distance

Primary Data Center DR Data Center

Automatic Block Repair

Enterprise Manager Cloud Control

Offload Fast Incremental

Backups

Offload read-only workload to open standby database

37


Data Guard and Active Data Guard Feature Sets

Data Guard (included with EE) Data Protection with HA

• Zero or near-zero data loss protection

• Transparent – all datatypes, workloads

• Continuous data validation

• Detect silent corruption

• Automatic database failover

• Dual-purpose DR as test system

• Simple migrations and upgrades

• Oracle Enterprise Manager integration

Active Data Guard (Option) Advanced Protection with High ROI

• Zero data loss at any distance

• Automatic corruption repair

• Auto-replay of inflight transactions

• Offload transport compression*

• Offload read-only workload

• Offload read-mostly workload

• Offload fast incremental backups

• Automation for rolling upgrades

* Also requires Advanced Compression Option


Active Data Guard: Disaster Recovery and Data Protection

High-performance, simple, drop-in physical replication for HA and DR

Zero data loss, unique data corruption protection, automatic failover, offload read-only workloads

Disaster recovery for all data types, applications and workloads

GoldenGate: Flexible HA, Maintenance and Migrations, Real-Time Data Integration

Logical replication for advanced replication requirements, target open read-write

Enable minimal/zero downtime upgrades and application migration using 2nd database copy

Heterogeneous platforms, bi-directional replication, transformations, subsetting, multiple topologies

Oracle Replication Solutions for Business Continuity


Active Data Guard

Oracle Products for Data Movement

• Optimized Set-based transformation • Highest latency for real-time feeds • Heterogeneous • Supports highly complex transformation

• Not invasive on source • Fastest read and write • Oracle Database only • No transformation; no partial sets • Unidirectional

• Not Invasive on source • Very fast read and write • Heterogeneous or homogeneous • Basic transformation and partial sets • Uni- and Bi-Directional

GoldenGate

Data

Integrator

Comparing Tradeoffs and Benefits Driven by Different Architectures


Resources

41

Oracle Data Integration Oracle Data Integration OracleGoldenGate ORCL DataIntegration blogs.oracle.com/dataintegration


Oracle GoldenGate




http://www.oracle.com/us/products/middleware/data-integration/overview/index.html

Data Integration

Oracle OpenWorld 2014


Questions and Answers

42 Oracle OpenWorld 2014


Documents

Oracle Data Integration Solutions - Oracle Data · PDF fileOracle OpenWorld 2014 Oracle Data Integration Solutions (DIS) Oracle Data Integrator (ODI) Oracle ... •Easy to extend and