Scripting and Automating ODI 11g using Groovy and the ODI SDK

T : +44 (0) 8446 697 995 E : [email protected] W: www.rittmanmead.com

Scripting and Automating ODI 11g using Groovy and the ODI SDKMichael Rainey, Principal Consultant, Rittman MeadRittman Mead BI Forum 2013 Master Class, May 2013

mailto:[email protected]


http://www.rittmanmead.com



•Introduction to the ODI SDK•Introduction to Groovy scripting language•Demonstrations‣Developer’s Quickstart‣Metadata Setup‣Initial Load: Interfaces, Scenarios, and Load Plans‣ETL Migration to ODI

About this presentation...






Intro to the ODI SDK

•Java based API‣Develop using Java, Jython, or Groovy

•Perform nearly every action that can be done in ODI Studio‣Generate metadata‣Create / Modify Interfaces, Packages, Load Plans, etc‣Manage execution sessions

•Resources‣Data Integration bloghttps://blogs.oracle.com/dataintegration/tags/sdk‣Rittman Mead bloghttp://www.rittmanmead.com/tag/odi-11g-sdk/‣Oracle Data Integrator Java API Referencehttp://docs.oracle.com/cd/E28280_01/apirefs.1111/e17060/index.html





https://blogs.oracle.com/dataintegration/tags/sdk

https://blogs.oracle.com/dataintegration/tags/sdk

http://www.rittmanmead.com/tag/odi-11g-sdk/

http://www.rittmanmead.com/tag/odi-11g-sdk/

http://docs.oracle.com/cd/E28280_01/apirefs.1111/e17060/index.html

http://docs.oracle.com/cd/E28280_01/apirefs.1111/e17060/index.html


Intro to Groovy

•Agile, dynamic object oriented programming language for the Java Virtual Machine

•Like Java...but simplified‣Most Java syntax is valid in Groovy‣Additional features inspired by Python, Ruby, and Smalltalk‣Seamless integration with existing Java classes and libraries

•SwingBuilder ‣Create Swing GUIs using Groovy “builders”‣Easy to develop and maintain‣Hierarchy of components

•http://groovy.codehaus.org/





http://groovy.codehaus.org/

http://groovy.codehaus.org/


Groovy and ODI Integration

•Oracle Data Integrator is delivered with a Groovy editor‣Create and save .groovy script files‣Integrates with the ODI instance and security manager

•Transaction management simplified‣Create transaction‣Persist‣Commit transaction‣On exception...Rollback transaction






Develop Groovy in ODI Procedures

•ODI Procedure and Knowledge Module steps‣Groovy as a Technology

•Less integrated than .groovy scripts‣Must define

-Connection to repository-ODI username & password-ODI Instance for security and transaction management






Use Case - Developer’s Quickstart

•Case:‣Many find it difficult to use Oracle Data Integrator “right out of the box”‣Other ETL tools provide a quick and easy way to begin development

•Solution:‣Determine which tasks need to be automated and develop Groovy script

-Create Data Server, Physical Schema, and Logical Schema-Map Logical Schema/Physical Schema using the default Global context-Create new Model-Build and execute Package to reverse engineer tables into ODI Datastores‣Initial code developed by Oracle’s David Allan “ODI 11g – Expert Accelerator for Model Creation”https://blogs.oracle.com/dataintegration/entry/odi_11g_expert_accelerator_for





https://blogs.oracle.com/dataintegration/entry/odi_11g_expert_accelerator_for

https://blogs.oracle.com/dataintegration/entry/odi_11g_expert_accelerator_for


Developer’s Quickstart - Capture User Input






Demonstration

Developer’s Quickstart - ODI SDK and Groovy






Use Case - Metadata Setup

•Case:‣Setup of Staging and Foundation metadata ‣Add data warehouse audit columns

•Solution:‣Reverse engineer Source tables‣Copy tables to Staging and Foundation Models‣Add data warehouse audit columns to Foundation Datastores via Groovy script






Demonstration

Metadata Setup - ODI SDK and Groovy






Use Case - Initial Load Setup: Interfaces, Scenarios, and Load Plans

•Case:‣Setup objects for Source to Foundation initial load

•Solution:‣Groovy scripts

-Create Source to Foundation Interfaces-Generate a Scenario for each Interface -Build a Load Plan to execute all Scenarios






Initial Load Setup - InteractiveInterfaceHelperWithActions






DemonstrationInitial Load Setup - Interfaces, Scenarios, & Load Plans - ODI SDK and Groovy






Use Case - Migration to Oracle Data Integrator

•Case:‣Automate the migration of mappings from an “old” ETL tool to Oracle Data Integrator‣Oracle Warehouse Builder is the ETL tool in this example

•Solution:‣Place the “old” ETL tool metadata in a database schema‣Build Groovy scripts against the metadata to create ODI objects

-Topology-Project-Metadata- Interfaces






Migration to ODI - OWB Metadata






Migration to ODI - OWB Metadata






Demonstration

Migration to Oracle Data Integrator - ODI SDK and Groovy






•More information can be found at http://www.rittmanmead.com•Contact us at [email protected] or [email protected]•Follow-us on Twitter (@rittmanmead & @mRainey) or Facebook (facebook.com/rittmanmead)

In summary...

•ODI SDK can perform nearly every action as the ODI Studio•Groovy script is the language of choice for developing against the ODI SDK‣Java-based, but simple to develop‣Developed in Groovy script files or Procedure steps

•Automation of large, batch tasks in ODI can save loads of time










Scripting and Automating ODI 11g using Groovy and the ODI SDKMichael Rainey, Principal Consultant, Rittman MeadRittman Mead BI Forum 2013 Master Class, May 2013



