<Insert Picture Here>
Oracle Data Integrator – Technical Deck
Mark PareSr. Sales ConsultantOracle Higher Education
3
<Insert Picture Here>• 4 Key Differentiators• Architecture• 6 Steps to Production• ODI or ESB?• Popular Usage Scenarios
Agenda
4
Oracle Data Integrator 4 Key Differentiators
5
Data IntegrationData IntegrationData
WarehousingMaster Data Management
Real Time Messaging
FederationMigration
Data in Disparate SourcesData in Disparate Sources
ERP
------------
------
CRM
- - -
Legacy
------------
------ ---------
---
------
Best-of-breed Applications
Information How and Where you Want ItInformation How and Where you Want It
Business IntelligenceCorporate Performance
ManagementBusiness Activity
MonitoringBusiness Process
Management
HAVE…
NEED…
- - -- - - - - -- - -
Data Synchronization
Why Data Integration?
6
Challenges & Emerging SolutionsIn Data Integration
CHALLENGE EMERGING SOLUTION
1. Increasing data volumes; decreasing batch windows
2. Non-integrated integration
3. Complexity, manual effort of conventional ETL design
4. Lack of knowledge capture
Shift from E-T-L to E-LT
Convergence of integration solutions
Shift from custom coding to declarative design
Shift to pattern-driven development
7
Oracle Data Integrator
• Data Movement and Transformation from Multiple Sources to Heterogeneous Targets
1. Performance: Heterogeneous “E-LT”
2. Flexibility: Active Integration Platform
3. Productivity: Declarative Design
4. Hot-Pluggable: Knowledge Modules
BENEFITS KEY DIFFERENTIATED FEATURES
8
Differentiator: E-LT ArchitectureHigh Performance
Conventional ETL Architecture
Extract LoadTransform
Next Generation Architecture
“E-LT”“E-LT”LoadExtract
Transform Transform
Transform in Separate ETL Server• Proprietary Engine• Poor Performance• High Costs• IBM & Informatica’s approach
Transform in Existing RDBMS• Leverage Resources• Efficient• High Performance
BenefitsOptimal Performance & ScalabilityEasier to Manage & Lower Cost
11
9
Enables real-time data warehousing and operational data hubs Services plug into Oracle SOA Suite for comprehensive integration
Oracle Data Integrator
Data-oriented Integration
Event ConductorEvent Conductor
Event-orientedIntegration
Service ConductorService Conductor
Service-orientedIntegration
Declarative Design
Metadata
Data ConductorData Conductor
Differentiator: Active IntegrationBatch, Event-based, and Service-oriented Integration
• Evolve from Batch to Near Real-time Warehousing on Common Platform
• Unify the Silos of Data Integration
• Data Integrity on the Fly• Services Plug into Oracle
SOA Suite
• Benefits
22
10
Differentiator: Declarative DesignDeveloper Productivity
Conventional ETL DesignSpecify ETL Data Flow Graph• Developer must define every step of
Complex ETL Flow Logic• Traditional approach requires specialized
ETL skills• And significant development and
maintenance efforts
Declarative Set-based Design• Simplifies the number of steps• Automatically generates the Data Flow
whatever the sources and target DB• Example: [SALAH]
Benefits Significantly reduce the learning curve Shorter implementation times Streamline access to non-IT pros
ODI Declarative Design
Define How: Built-in Templates
Define What You Want
Automatically GenerateDataflow
11 22
33
11
JournalizeRead from CDC
Source
LoadFrom Sources to
Staging
CheckConstraints before
Load
IntegrateTransform and Move
to Targets
ServiceExpose Data and Transformation
Services
ReverseEngineer Metadata
Tailor to existing best practices Ease administration work Reduce cost of ownership
Reverse
Journalize
Load
Check
IntegrateServices
Pluggable Knowledge Modules Architecture
CDC
Sources
Staging Tables
Error Tables
Target Tables
WS
WS W
S
SAP/R3
Siebel
Log Miner
DB2 Journals
SQL Server Triggers
Oracle DBLink
DB2 Exp/Imp
JMS Queues Check MS Excel
Check Sybase
Oracle SQL*Loader
TPump/ Multiload
Type II SCD
Oracle Merge
Siebel EIM Schema
Oracle Web Services
DB2 Web Services
Sample out-of-the-box Knowledge Modules
Benefits
Differentiator: Knowledge ModulesHot-Pluggable: Modular, Flexible, Extensible
44
12
Oracle Data IntegratorArchitecture
13
• Java design-time environment
• Runs on any platform• Thin client for browsing
Metadata
• Java runtime environment• Runs on any platform• Orchestrates the execution of
data flows
• Metadata repository• Pluggable on many RDBMS• Ready for deployment• Modular and extensible
metadata
Design-Time
Metadata Management
Runtime
Agent
Data Flow Conductor
Service Interfaces and Developer APIs
UserInterfaces
Thin Client
Data Flow Generator
Knowledge Module Interpreter
Knowledge Modules
Master Repository
Work Repositories
Runtime Repositories
Data Flow Generator
Runtime Session
Interpreter
Data Flow
Operator
Designer
Architecture: Conceptual View
14
Architecture: Component View
ODI Design-Time Environment Development Servers and Applications
Design-time Repositories
Code Execution
Execution Log Return Codes
AgentData Flow Conductor
CRM
Legacy
ERP
Data Warehouse
Files / XML
User Interfaces
Administrators
Designers
Topology/Security
Metadata/Rules
Development
ESB
Production Servers and Applications
ODI Runtime Environment
Runtime Repository Return Codes
Code
Execution Log
Execution
Metadata Navigator
Production
CRM
Legacy
ERP
Data Warehouse
Files / XML
ESB
User Interfaces
Administrators
Operators
Thin Client
Data Stewarts
Topology/Security
Execution Log
Metadata Lineage
AgentData Flow Conductor
Scenarios and ProjectsReleases
15
Oracle Data Integrator6 steps to Production
16
ODI Design-Time Environment ODI Runtime Environment
User Interfaces
Overview: 6 steps to Production
1. Retrieve/Enrich metadata
2. Design transformations
3. Orchestrate data flows
4. Generate/Deploy data flows
5. Monitor executions
6. Analyze impact / data lineage
Development
Development Servers and Applications
AgentData Flow Conductor
CRM
Legacy
ERP
Data Warehouse
Files / XML
User Interfaces
Administrators
Designers
ESB
Design-time RepositoriesDesign-time
Repositories
Production
Production Servers and Applications
AgentData Flow Conductor
CRM
Legacy
ERP
Data Warehouse
Files / XML
Operator
Metadata Navigator
ESB
Runtime Repository
17
1. Reverse-engineer Metadata• Automatic• Customizable• 40+ technologies supported
2. Enrich Metadata• Documentation• Declarative rules for Data
Integrity• Cross-technologies references
Design-Time Environment
ODI Designer
Design-time Repositories
Development Servers and Applications
CRM
Legacy
ERP Data Warehouse
Files / XML
ESB
Retrieve/Enrich Metadata11
18
Oracle Data Integrator “Interface”
Declarative Design
11 Define What You Want 33 Automatically GenerateData flows
22 Define How to Do It: Select Template
Bulk Load • Changed Data Capture • Incremental Update • Slowly Changing Dimension
Design Transformations22
19
1. Sequence Transformations
2. Leverage OracleDI Tools• Data Quality Processes• Files/Archives Management• Send/Receive Emails• Web Services Invokation• Event Detection• Create your Own Tools
3. Use Control Structures• Loops• Conditions• Error Handling
33 Orchestrate Data Flows
20
1. Create Scenarios• Compile Data Flows for Run-time
2. Version the Data Flows• Advanced Version Management
3. Deploy to ProductionDesign-time Repositories
Scenarios and ProjectsReleases
Runtime Repository
Generate and Deploy Data Flows44
21
• View sessions running in real-time
• Review generated code• Detailed run-time statistics• Restart failed sessions
55 Monitor Executions
22
Analyze impact / data lineage
• Maintain a large number of data flows in a complex environment
• Web-based end-to-end data lineage1. Understand your data flows2. Follow the path of data3. Drill-down to transformations
?
66
23
Oracle Data IntegratorODI or ESB?
24
What tool is best suited for task X?
Requirement ESB
ODI Recommended
Latency / VolumeSynchronous Integration ESB
Asynchronous Integration with routing and transformation ESB
Asynchronous Integration for Active Data Warehousing (mini-batch) ODI
Batch Integration with High Volume ODI
TransformationsIn-memory XSLT Transformations (XML to XML) ESB
Transformations in App Server ESB
Transformations in Database (E-LT) ODI
Integration TopologyData Warehouse Loading (E-LT) ODI
JMS to JMS ESB
JMS to DB/App with routing and transformation (real-time or synchronous) ESB
JMS to DB/App with bulk transformation (mini-batch) ODI
DB/App to DB/App (batch or mini-batch with CDC) ODI
DB/App to DB/App (synchronous or real-time with CDC Adapters) ESB
…
25
Oracle Data Integrator
ESB and ODI in real-life scenarios
Data Volume Processing
Data Latency
Message by Message
Mini Batches Large Volume(over 1M)
Synchronous(immediate)
Asynchronous
Batch(over 2 hours)
Oracle Enterprise Service Bus
Real-life
Scenarios
26
Oracle Data IntegratorExtended Capabilities
27
Extended Capabilities
• Master Data Management enabled• Common Format Designer• Automated generation of canonical format and transformations• Built-in Data Integrity
• Real-time enabled• Changed Data Capture• Message Oriented Integration (JMS)
• SOA enabled• Generation of Data Services• Generation of Transformation Services
• Extensibility• Knowledge Modules Framework• Scripting Languages• Open Tools
28
Oracle Data IntegratorMaster Data Management Enabled
29
MDM Enabled: Canonical Format Design
• Use in conjunction with packaged MDM solution
• Design and Populate Canonical Format1. Use existing metadata artifacts
to design MDM application (entities, fields, relationships)
2. Generate and maintain Master Data structure
3. Generate and deploy transformations using metadata artifacts
CRM
Enterprise Service Bus
SCM Legacy ERP
Master Data
30
MDM Enabled: Built-in Data Integrity
• Data Integrity Firewall• Auditing, cleansing and
recycling
1. Declare constraints at table level2. Design mappings and check
flow integrity3. Audit, cleanse or recycle
rejected records
Message Id Name City
Duplicated Record 001 John Doe New York
Duplicated Record 022 John Doe Boston
Invalid City Reference 230 Albert Fresh Maris
31
Oracle Data IntegratorReal-time Enabled
32
Real-time enabled: Changed Data Capture
• Publish and Subscribe CDC Framework• Database logs• Triggers• Third-tier solutions
• Ensures “read” transaction integrity across multiple tables
1. Design or generate Mappings
2. Select Journalized Data Only
3. Start Journals
CDC
33
Real-time enabled: Message Oriented Integration
• Connect to Publish and Subscribe JMS Message Providers
• Ensure messages delivery with transaction integrity
• High-volume bulk transformations1. Design complex bulk
transformations mixing Queues, Databases and Applications
2. Use JMS Queues and topics as sources or targets
JMS Provider(MOM, ESB)
CDC
Subscribe
Publish
34
Oracle Data IntegratorSOA Enabled
35
SOA Enabled: Data Access Services
• Generate and share data access services
1. Generate and deploy data services2. Test data services3. Leverage data services in your
SOA infrastructure
SOA Infrastructure
Services
Data Access
Transform
Business
Business Processes
ESB
36
SOA Enabled: Data Flow Services
• Expose transformations as Web Services
1. Orchestrate data flows2. Publish data flows as web services
in your SOA infrastructure
SOA Infrastructure
Services
Data Access
Transform
Business
Business Processes
ESB
Bulk Transf
.
37
Oracle Data IntegratorExtensible Framework
38
JournalizeRead from CDC
Source
LoadFrom Sources to
Staging
CheckConstraints before
Load
IntegrateTransform and Move
to Targets
ServiceExpose Data and Transformation
Services
ReverseEngineer Metadata
Reverse
Journalize
Load
Check
IntegrateServices
Pluggable Knowledge Modules Architecture
CDC
Sources
Staging Tables
Error Tables
Target Tables
WS
WS W
S
Extensibility: Knowledge Modules
• 120+ KMs out-of-the-box Tailor to existing best practices Ease administration work Reduce cost of ownership
• Customizable and extensible
KMInterpreter
KM’s Meta Code
Metadata
Executed Code
39
Extensibility: Scripting Framework
• Extend data flows with scripting procedures
• Leverage all database languages
• SQL, PL/SQL, Transact SQL, etc.
• Use Operating Systems shell scripts
• Win32 DOS, sh, ksh, csh, OS400 commands, JCL, etc.
• Choose from compatible Bean Scripting Framework languages
• Java, JavaScript, Jython (Java Python), Perl, etc.
40
Extensibility: Open Tools
• Extend ODI tools• Add your own tools to the
Design Palette1. Implement OdiOpenToolAbstract
Java Interface2. Register Open Tool in ODI Designer3. Use Open Tool in your design
environment
41
Popular Usage Scenarios
42
E-LT for Data WarehouseCreate Data Warehouse for Business Intelligence Populate Warehouse with High Performance ODI
Heterogeneous sources and targets
Incremental load Slowly changing
dimensions Data integrity and
consistency Changed data capture Data lineage
Data Warehouse
Cube
Cube
Cube
----------------
Op
era
tion
al
An
aly
tics
Metadata
LoadTransformCapture Changes
Incremental UpdateData Integrity
AggregateExport
43
SOA InitiativeEstablish Messaging Architecture for Integration Incorporate Efficient Bulk Data Processing with ODI
Invoke external services for data integration
Deploy data services Deploy transformation
services Integrate data and
transformation services in your SOA infrastructure
Services
Data Access
Transformation
Others
----------------
Op
era
tion
al
Metadata
Generate DataServicesExpose Transformation Services
Deploy and reuse Services
Business Processes
44
Master Data ManagementCreate Single View of the TruthSynchronize Data with ODI
Use in conjunction with packaged MDM solution
Use as infrastructure for designing your own hub
Create declarative data flows Capture changes (CDC) Reconcile and cleanse the
data Publish and share master data Extend metadata definitions
Master Data
----------------
Metadata
Change Data CaptureMaster Data Load
Canonical Format DesignCleansing and Reconciliation
Master Data Publishing
----------------
CDCCDC
CDC
45
MigrationUpgrade Applications or Migrate to New Schema Move Bulk Data Once and Keep in Sync with ODI
Bulk-load historical data to new application
Transform source format to target
Synchronize new and old applications during overlap time
Capture changes in a bi-directional way (CDC)
Old
Ap
plic
atio
ns
Ne
w A
pp
lica
tion
Metadata
Initial bulk loadCDC for synchronization
Transformation to new application format
CDC for loop-back synchronization
CDC
CDC----------------
46
ODI Enhances Oracle BIPopulate Warehouse with High Performance ODI
Oracle Business Intelligence Suite EE:
Simplified Business Model View Advanced Calculation &
Integration Engine Intelligent Request Generation Optimized Data Access
Oracle Data Integrator: Populate Enterprise Data
Warehouse Optimized Performance for Load
and Transform Extensible Pre-packaged E-LT
Content
Siebel CRM
Oracle EBSPeopleSoftSAP/R3
Other Sources
Oracle Data Integrator
E-LT Metadata
E-LT Agent
Oracle BI Enterprise Data
Warehouse
Oracle BI Suite EE
Oracle BI Server
Oracle BI Presentation Server
Answers Interactive Dashboards Publisher Delivers
Bulk E-LT
47
ODI Enhances Oracle SOA SuiteAdd Bulk Data Transformation to BPEL Process
Oracle SOA Suite: BPEL Process Manager for
Business Process Orchestration
Oracle Data Integrator: Efficient Bulk Data Processing
as Part of Business Process Interact via Data Services and
Transformation Services
Oracle SOA Suite
Business Activity Monitoring
Web Services Manager
Business Rules Engine
Enterprise Service Bus
BPEL Process Manager
Bulk Data Processing
Oracle Data Integrator
E-LT Metadata
E-LT Agent
48
ODI Enhances Oracle SOA SuitePopulate BAM Active Data Cache Efficiently
Oracle SOA Suite: Business Activity Monitoring
for Real-time Insight
Oracle Data Integrator: High Performance Loading of
BAM’s Active Data Cache Pre-built and Integrated
Data Warehouse
Oracle SOA Suite
BPEL Process Manager
Web Services Manager
Business Rules Engine
Enterprise Service Bus
Bulk and Real-Time
Data Processing
SAP/R3PeopleSoft
Message Queues
CDC
Business Activity Monitoring
Active Data Cache
Event Engine Report Cache
Event Monitoring Web Applications
Oracle Data Integrator
MetadataAgent
49
Roadmap and Direction
50
Oracle Data Integrator: Roadmap
• Focus Areas for Next Major Release• Deep Integration with Fusion Middleware
• Runtime, Design time, Security, Administration, Events
• Functional Integration with Oracle Warehouse Builder• Runtime Integration, Metadata Sharing, Knowledge Module Sharing
• Deployment of ODI for Embedded Data Integration• OracleBI Enterprise Edition, Data Hubs, Application Migrations
• Enhanced Usability and Debuggability• Wizards, New Views, User-definable Debugging
• Improved Support for Native Oracle Database Features• Oracle OLAP
51
ODI Statement of Direction
• Statement of Direction• http://www.oracle.com/technology/products/odi/statement-of-direction.pdf
• Key Points of Direction• Commitment to heterogeneous systems support
• Including: DB2, Teradata, Netezza, Hyperion, etc.
• Commitment to Fusion design principles• Including: J2EE compliance, container portability
• Commitment to best-of-class E-LT performance• Across platforms, batch & realtime, high complexity