UKOUG Tech 15 - Migration from Oracle Warehouse Builder to Oracle Data Integrator - Eurocontrol Case Study

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)

E : [email protected] : www.rittmanmead.com

Rittman Mead ConsultingEurocontrol - Migration from OWB to ODI

UKOUG Tech 2015 - BirminghamJérôme Françoisse



Jérôme Françoisse

•Consultant for Rittman Mead‣Oracle BI/DW Architect/Analyst/Developer

•ODI Trainer•Providing ODI support on OTN Forums•ODI 12c Beta Program Member•Blogger at http://www.rittmanmead.com/blog/•Email : [email protected]•Twitter : @JeromeFr

http://www.rittmanmead.com/blog/

mailto:[email protected]



Rittman Mead

•Oracle Gold partner with offices in Europe, US (Atlanta)

•UKOUG Partner of the year in 4 categories in 2015

•Provide consulting, training, global managed services for customers around the world

•80+ consultants including 1 Oracle ACE Director and 2 Oracle ACEs

•All expert in Oracle BI, DW, EPM and Analytics tech

•Skills in broad range of supporting Oracle tools: OBIEE, OBIA, ODIEE, Essbase, Oracle OLAP, GoldenGate, Exadata, Endeca

•Blog : http://www.rittmanmead.com/blog/•Twitter : @rittmanmead

http://www.rittmanmead.com/blog/



Agenda

•OWB & ODI•Migration Utility•Migration steps•Migration execution•Fixing•Opportunities



History of OWB & ODI

•2000 - First release of OWB•2006‣Oracle buys Sunopsis - ODI 10g‣OWB Standard and Enterprise Editions

•2008 - Roadmap to merge the products•2009 - Oracle Data Integrator Enterprise Edition•2010 - ODI 11g•2013‣ODI 12c‣OWB is not shipped with Oracle DB 12c ‣Last OWB release - 11.2.0.4

•2015 - End of standard support for OWB



Similarities

•Leverage database capabilities as data transformation engine•Objects are stored in the database•Flow-based development•Can use the ELT paradigm



Flow-based



ETL vs ELT

•Reduce network transfer•Reduce hardware costs•Set-based operations

•Leverage target platform capabilities•Referential Integrity Check



ODI Differences

•Heterogeneous (Oracle, IBM DB/2, SQL Server, Flat files, Webservices, Hadoop, Cloud Apps, …)

•No deployment, the code is generated at run-time•Jobs are executed by agents•Repository stored on any RDBMS (not only Oracle)•Separation between Physical and Logical implementation•Customisation and Reusability thanks to Knowledge Modules•Parallelism•Real-time loading

create table<%=odiRef.getTable("L", "INT_NAME", "A")%>(<%=odiRef.getColList("", "[COL_NAME]\t[DEST_WRI_DT] " + odiRef.getInfo("DEST_DDL_NULL"), ",\n\t", "", "INS or UPD")%>)



ODI Differences

•Heterogeneous (Oracle, IBM DB/2, SQL Server, Flat files, Webservices, Hadoop, ...)

•No deployment, the code is generated at run-time•Jobs are executed by agents•Repository stored on any RDBMS (not only Oracle)•Separation between Physical and Logical implementation•Customisation and Reusability thanks to Knowledge Modules•Parallelism•Real-time loading



Migration Utility

•Needs OWB 11.2.0.3+•Patch OWB 17830453or 21977765 (12.2.1)

•ODI 12.1.2 : Patch 17053768•ODI 12.1.3+ : No Patch•Based on ODI SDK•Command Line Interface•3 modes :‣Fast Check‣Dry Run‣Run



Migration Utility

•Supported‣Module / Location Model and Topology‣All data objects (except sequences) Datastores‣Mapping (Classic, Code-template and Pluggable) Mapping

•Unsupported‣Process flows‣PL/SQL objects (procs, packages, ...)‣Oracle Discoverer / OBIEE metadata‣Queues, Streams, CDC, User Defined Types‣Dimension and cube, cursors‣Data quality, data profiles, data auditors‣Configuration details (security, OMB scripting, ...)



Migration steps• Installing Oracle DB 12c• Installing ODI 12.1.2 and patch 17053768•Plug the homebrew scheduler in ODI•Upgrading a clone OWB repository to 11.2.0.4 and apply patch 17830453•Execute migration utility in Dry Run mode•Fix issues on OWB side• Installing ODI 12.1.3•Build Knowledge Modules for missing loading strategies.•Execute migration utility in Run mode•Clean up topology•Synchronise Test environment and Dev environment•For each mapping : Change knowledge modules, remove unnecessary objects, unit test

•Creation of Load Plans•Testing



OWB Architecture

DevelopmentRepository

Deployment

DEV

PROD

DDR_SRC

DDR_STG

DDR

SWH_SRC

SWH_STG

DDR

OWB_REPO

PE21 - Dev

PE5 - Prod

PE7

Sources and Targets



ODI Architecture

Repositories ExecutionContext

DEV

PROD

DDR_SRC

DDR_STG

DDR

PE21 - Dev

PE16 - Test

PE9ODI_DEV

Sources and Targets

MASTER

DEV

PE9ODI_TEST

EXEC

PE9ODI_PROD

EXEC

SWH_SRC

SWH_STG

DDR

PE5 - Prod

TEST

HF_TEST

MASTER

MASTER

DDR_SRC

DDR_STG

DDR

DDR_SRC

DDR_STG

DDR

HF_PROD



Migration UtilityODI_MASTER_USER=DEV_ODI_REPOODI_MASTER_URL=jdbc:oracle:thin:@porape9.ops.cfmu.eurocontrol.be:1542/odi_devODI_MASTER_DRIVER=oracle.jdbc.OracleDriverODI_USERNAME=SUPERVISORODI_WORK_REPOSITORY_NAME=WORKREP_DEVOWB_WORKSPACE_OWNER=OWB_REPOSOWB_URL=porape9.ops.cfmu.eurocontrol.be:1542:owb2odiOWB_WORKSPACE_NAME=OWB_REPOS

MIGRATION_MODE=DRY_RUN

MIGRATION_OBJECTS=PROJECT.SWH.MODULE.DDR;PROJECT.SWH.MODULE.EUACA.MAPPING.UWB_LDIM_IMATCH_METHOD_010_INS;

MIGRATION_STRATEGY=CREATE



Migration UtilityODI_MASTER_USER=DEV_ODI_REPOODI_MASTER_URL=jdbc:oracle:thin:@porape9.ops.cfmu.eurocontrol.be:1542/odi_devODI_MASTER_DRIVER=oracle.jdbc.OracleDriverODI_USERNAME=SUPERVISORODI_WORK_REPOSITORY_NAME=WORKREP_DEVOWB_WORKSPACE_OWNER=OWB_REPOSOWB_URL=porape9.ops.cfmu.eurocontrol.be:1542:owb2odiOWB_WORKSPACE_NAME=OWB_REPOS

MIGRATION_MODE=DRY_RUN

MIGRATION_OBJECTS=PROJECT.SWH.MODULE.DDR;PROJECT.SWH.MODULE.EUACA.MAPPING.UWB_LDIM_IMATCH_METHOD_010_INS;

MIGRATION_STRATEGY=CREATE

oracle@prisme111: cd /opt/oracle/11.2.0.4/owb/bin/unixoracle@prisme111: ./migration.sh ODI_MASTER_REP_PASSWD ODI_SUPERVISOR_PASSWD OWB_WORKSPACE_PASSWD migration.config



Migration Utility: Report



Migration Utility: Result



Mappings not migrated

•Errors in input connectors - Fixed with ODI 12.1.3•Procedure inside mappings - Wrapping into functions•No “Delete” loading strategy - Development of new IKM Knowledge Module

•No “Update” only mode with the Merge syntax - Edit of existing IKM



PSM - Scheduler



PSM - Scheduler



PSM - Scheduler



PSM - Monitoring



PSM - Monitoring



PSM - Parameter



PSM - Scheduler



Cleanup Topology



Migration is a good opportunity for...

•Re-assess the business rules•Define Data Stewardship•Refactoring•Delete unused batch•Complete missing documentation



Execution OWB jobs from ODI

•Register in Topology•ODI Tool : OdiStartOwbJob•Monitor in Operator



Further reading

•Blog Post by David Allan (@im_dave) and Julien Testut (@JulienTestut) on ODI12c migration : https://blogs.oracle.com/dataintegration/entry/planning_the_journey_from_oracle

•OTN article by Stewart Bryson (@stewartbryson) : Making the Move from Oracle Warehouse Builder to Oracle Data Integrator 12c : http://www.oracle.com/technetwork/articles/datawarehouse/bryson-owb-to-odi-2130001.html

• “OWB – Making the Move to Oracle Data Integrator” by Michael Rainey (@mRainey) at UKOUG 2012:http://www.rittmanmead.com/files/ukoug2012_rainey_odi.pdf

•Blog post series by Peter Scott (@dw_pete) on Rittman Mead blog:http://www.rittmanmead.com/2011/10/oracle-warehouse-builder-and-data-integrator/

https://blogs.oracle.com/dataintegration/entry/planning_the_journey_from_oracle

https://blogs.oracle.com/dataintegration/entry/planning_the_journey_from_oracle

http://www.oracle.com/technetwork/articles/datawarehouse/bryson-owb-to-odi-2130001.html

http://www.oracle.com/technetwork/articles/datawarehouse/bryson-owb-to-odi-2130001.html

http://www.rittmanmead.com/files/ukoug2012_rainey_odi.pdf

http://www.rittmanmead.com/2011/10/oracle-warehouse-builder-and-data-integrator/

http://www.rittmanmead.com/2011/10/oracle-warehouse-builder-and-data-integrator/



Data & Analytics

UKOUG Tech 15 - Migration from Oracle Warehouse Builder to Oracle Data Integrator - Eurocontrol Case Study