Creating Data Marts from COBOL Files (ISAM to RDBMS)
Creating Data Marts from COBOL FilesDoug Evans / Chris Martin
GDT 2008 International User ConferenceAugust 10 – 13 Lake Las Vegas, Nevada
GDT ETL
What is ETL? Extract Transform Load
Extract data from your legacy system. Transform (Convert or cleanse). Load the data into the relational database, data warehouse,
data mart or Operational Data Store. Why GDT-ETL?
Built for Legacy Systems. Native access to ISAM data. Transformation from COBOL data types to more general
types. Allows you to focus on where the data is going not how you
get access to it.
Creating Data Marts from COBOL FilesDoug Evans / Chris Martin
GDT 2008 International User ConferenceAugust 10 – 13 Lake Las Vegas, Nevada
GDT ETL
Data flow for a system running GDT MF file system to a relational database
MF FilesMF Files
GDT
RDMSGDT-ETLGDT-ETL
RDMS
Creating Data Marts from COBOL FilesDoug Evans / Chris Martin
GDT 2008 International User ConferenceAugust 10 – 13 Lake Las Vegas, Nevada
GDT ETL
ETL Features Command line tool and Graphical User Interface
(ETLPortal) Comprehensive error logging and process
monitoring. Will work with existing enterprise scheduling
facilities.
ETL Requirements Windows
GDT V4.1 or greater .NET Framework 2.0
Creating Data Marts from COBOL FilesDoug Evans / Chris Martin
GDT 2008 International User ConferenceAugust 10 – 13 Lake Las Vegas, Nevada
PLUS
PLUS
FILE DEFINITION
FILE COBOL Select statement
Multi 01 RecordDefinitions
-------------------------------------------------------------------
SQL1GEN
Creates GDTSQL01.DLL
Using the ORDERS fileSQL1GEN process
SQLCONV.PAR
Creating Data Marts from COBOL FilesDoug Evans / Chris Martin
GDT 2008 International User ConferenceAugust 10 – 13 Lake Las Vegas, Nevada
SQL1GEN process
GDT process that generates a File Access Component (GDTSQL01.DLL) To be used by GDT ETL to generically access all
Micro Focus ISAM files Configuration
SQLCONV.PAR FILE DEFINITION(S) fd’s FILE SELECT STATEMENTS MULTIPLE 01 RECORD DEFINITION PARAMETER FILES
Requires GDT V4.1 GDTSQL01.DLL contains all necessary file info.
Key information, data types, record types, fields etc..
Creating Data Marts from COBOL FilesDoug Evans / Chris Martin
GDT 2008 International User ConferenceAugust 10 – 13 Lake Las Vegas, Nevada
The ETL processGDT_ETL
GDTSQL01DLL
OrderInfo.
File Access Component – Found via the GDTSYS.PAR SQL Setting
ETLCONFIG.XML – Configuration for ETL to find / access FAC and DB
Creating Data Marts from COBOL FilesDoug Evans / Chris Martin
GDT 2008 International User ConferenceAugust 10 – 13 Lake Las Vegas, Nevada
GDT_ETLCommand Line Control
[-v | -version] Displays the programs build version. No other commands will
be executed after this one.
[-c | --createobj] <outputfile> Will create the necessary database objects. <outputfile> is
the flat file that will contain all of the SQL that would have been used to create the objects. Default is no value which will load everything into the database
[-l | --loaddata] <reportfreq> Will load the data into the database. <reportfreq> is an integer
representing how often GDT-ETL will report the status of the load. [DEFAULT <reportfreq> = 0]
Creating Data Marts from COBOL FilesDoug Evans / Chris Martin
GDT 2008 International User ConferenceAugust 10 – 13 Lake Las Vegas, Nevada
GDT_ETLCommand Line Control
[-r | --record] <recordname> Will only execute the given commands [-lDdcf] when it finds
the record that matches the name given in <recordname>. If used with [-i] it will limit the command for the <recordname> found in <ISAMFile>.
[-i | --isamfile] <ISAMfile> Will only execute the given commands [-lDdcf] when it finds
the ISAM file that matches <ISAMfile> Can be used to limit the [-r] to a specific record.
[-D | --delobj] Will delete all objects in the database. Limit using [-ri]
commands. Do not use at the same time as [-d]. [-d | --deldata]
Will delete all data in the database NOT the objects. Limit using the [-ri] commands. Do not use at the same time as [-D].
Creating Data Marts from COBOL FilesDoug Evans / Chris Martin
GDT 2008 International User ConferenceAugust 10 – 13 Lake Las Vegas, Nevada
GDT_ETLCommand Line Control
[-f | --filelayout] Will display the layout of the file(s) in the file access
component. [-t | --dbtype] Will display the DB type defined in the XML config
file. [-T | --testload]
Will test GDT_ETL to make sure it will process the file access component properly before attempting any action against the database.
[-? | --help] Short help usage message will be displayed by [-?],
verbose use [--help]
Creating Data Marts from COBOL FilesDoug Evans / Chris Martin
GDT 2008 International User ConferenceAugust 10 – 13 Lake Las Vegas, Nevada
Data is
in th
e data
base via the E
TL Load
Results of the ETL process
Creating Data Marts from COBOL FilesDoug Evans / Chris Martin
GDT 2008 International User ConferenceAugust 10 – 13 Lake Las Vegas, Nevada
ETL in a nutshellHear from the Chief Engineer
Build the tables and the relationships in SQL Each record denotes a new table. Each sub record denotes a new table with a 1 to
many relationship with the parent record. Nested OCCURS within a file produces more relationships.
Each field produces a field in a SQL table and ETL maps COBOL data types to the SQL Servers data types.
Creating Data Marts from COBOL FilesDoug Evans / Chris Martin
GDT 2008 International User ConferenceAugust 10 – 13 Lake Las Vegas, Nevada
ETL in a nutshell Hear from the Chief Engineer
Optional Modify the table schema and stored procedures to
suit your needs. Build transformations as needed.
Creating Data Marts from COBOL FilesDoug Evans / Chris Martin
GDT 2008 International User ConferenceAugust 10 – 13 Lake Las Vegas, Nevada
ETL in a nutshell Hear from the Chief Engineer
How does ETL build database schemas? When Generating a table schema GDT-ETL makes
the following assumptions. Each file consists of 1 or more record types. Each record in a file consists of 1 or more fields and it may
contain hierarchy of sub records as denoted by the OCCURS clause.
Each field has a location in a record as well as an associated COBOL data type.
Group name and redefines are ignored.
Creating Data Marts from COBOL FilesDoug Evans / Chris Martin
GDT 2008 International User ConferenceAugust 10 – 13 Lake Las Vegas, Nevada
ETL in a nutshell Hear from the Chief Engineer
Create the database schema and stored procedures. Build the tables and relationships Three basic data type transformations:
PIC X(NN) to char(NN) PIC 9(NN) to INT PIC S9(NN)V9(NN) to double
Setup default stored procedures for each table: For each table there are 3 separate stored procedures
Insert Update Delete
The tables can be modified but the stored procedures must reflect any schema modifications.
The store procedures can be modified but the parameters must stay the same.
Creating Data Marts from COBOL FilesDoug Evans / Chris Martin
GDT 2008 International User ConferenceAugust 10 – 13 Lake Las Vegas, Nevada
ETL in a nutshell Hear from the Chief Engineer
Build the default database objects Setup the ETL Configuration.
Set database properties. Reference the location of the file access component.
Build the database objects.
Creating Data Marts from COBOL FilesDoug Evans / Chris Martin
GDT 2008 International User ConferenceAugust 10 – 13 Lake Las Vegas, Nevada
ETL in a nutshell Hear from the Chief Engineer
What value do transformations add? Allows for cleansing of data before it enters the
database. Determine if it is accurate. Avoid data corruption.
Create more interesting data transformation Example: Date time
Creating Data Marts from COBOL FilesDoug Evans / Chris Martin
GDT 2008 International User ConferenceAugust 10 – 13 Lake Las Vegas, Nevada
ETL in a nutshell Hear from the Chief Engineer
Use Data Integration tools BusinessObjects Data Integrator.
Develop the transformations by hand. Modify the table layout if desired. Modify the default stored procedures to setup desired
transformations.
Creating Data Marts from COBOL FilesDoug Evans / Chris Martin
GDT 2008 International User ConferenceAugust 10 – 13 Lake Las Vegas, Nevada
ETL Reference Guides
Installation, Configuration and Operations Guide
ETL Portal User Guide Command Line HELP ETL Portal HELP
Creating Data Marts from COBOL FilesDoug Evans / Chris Martin
GDT 2008 International User ConferenceAugust 10 – 13 Lake Las Vegas, Nevada
ETLPortal
Setup for SQL1GEN process. All of the pieces that we talked about when processing via the Command Interface.
Creating Data Marts from COBOL FilesDoug Evans / Chris Martin
GDT 2008 International User ConferenceAugust 10 – 13 Lake Las Vegas, Nevada
ETLPortal
Selection of Files via the SQLCONV.PAR file. By selecting the Disabled boxprevents that file from being processed into the GDTSQL01.DLL. (like commenting
it out in the SQLCONV.PAR file).
Creating Data Marts from COBOL FilesDoug Evans / Chris Martin
GDT 2008 International User ConferenceAugust 10 – 13 Lake Las Vegas, Nevada
ETLPortal
Creating Data Marts from COBOL FilesDoug Evans / Chris Martin
GDT 2008 International User ConferenceAugust 10 – 13 Lake Las Vegas, Nevada
ETLPortalRunning the SQL1GEN Process
Creating Data Marts from COBOL FilesDoug Evans / Chris Martin
GDT 2008 International User ConferenceAugust 10 – 13 Lake Las Vegas, Nevada
ETLPortal
ETL Command Selection.
Creating Data Marts from COBOL FilesDoug Evans / Chris Martin
GDT 2008 International User ConferenceAugust 10 – 13 Lake Las Vegas, Nevada
ETLPortal
Create Objects is selected. This will create the tables and stored procedures.Filtering by File and / or Record is possible.
Force ILLDEC recovery is checked to allow ETL to recover from Illegal Decimal.
Creating Data Marts from COBOL FilesDoug Evans / Chris Martin
GDT 2008 International User ConferenceAugust 10 – 13 Lake Las Vegas, Nevada
ETLPortal
ETL create is successful!
Creating Data Marts from COBOL FilesDoug Evans / Chris Martin
GDT 2008 International User ConferenceAugust 10 – 13 Lake Las Vegas, Nevada
ETLPortal
Viewing the TABLES produced under SQL Server Manager.
Creating Data Marts from COBOL FilesDoug Evans / Chris Martin
GDT 2008 International User ConferenceAugust 10 – 13 Lake Las Vegas, Nevada
ETLPortal
Viewing the Stored Procedures produced under SQL Server Manager.
Creating Data Marts from COBOL FilesDoug Evans / Chris Martin
GDT 2008 International User ConferenceAugust 10 – 13 Lake Las Vegas, Nevada
ETLPortal
Viewing the Stored Procedures produced under SQL Server Manager.
Creating Data Marts from COBOL FilesDoug Evans / Chris Martin
GDT 2008 International User ConferenceAugust 10 – 13 Lake Las Vegas, Nevada
ETLPortal
Load Data is selected. This will load the tables by processing the stored procedures.
Creating Data Marts from COBOL FilesDoug Evans / Chris Martin
GDT 2008 International User ConferenceAugust 10 – 13 Lake Las Vegas, Nevada
ETLPortal
ETL Load is successful!
Creating Data Marts from COBOL FilesDoug Evans / Chris Martin
GDT 2008 International User ConferenceAugust 10 – 13 Lake Las Vegas, Nevada
ETLPortal
ETL output window shows issues with the data.
Creating Data Marts from COBOL FilesDoug Evans / Chris Martin
GDT 2008 International User ConferenceAugust 10 – 13 Lake Las Vegas, Nevada
ETLPortal
Viewing the Table contents under SQL Server Manager
Creating Data Marts from COBOL FilesDoug Evans / Chris Martin
GDT 2008 International User ConferenceAugust 10 – 13 Lake Las Vegas, Nevada
ETLPortal
A look at the Insert stored procedure for the PRODUCT_REC table.As is from the creation by ETL.
Creating Data Marts from COBOL FilesDoug Evans / Chris Martin
GDT 2008 International User ConferenceAugust 10 – 13 Lake Las Vegas, Nevada
ETLPortal
A look at the Update stored procedure for the PRODUCT_REC table.As is from the creation by ETL.
Creating Data Marts from COBOL FilesDoug Evans / Chris Martin
GDT 2008 International User ConferenceAugust 10 – 13 Lake Las Vegas, Nevada
ETLPortal
A look at the Delete stored procedure for the PRODUCT_REC table.As is from the creation by ETL.
Creating Data Marts from COBOL FilesDoug Evans / Chris Martin
GDT 2008 International User ConferenceAugust 10 – 13 Lake Las Vegas, Nevada
GDT-ETL
THE DEMONSTRATION! and Questions?