OLTP to OLAP COnversion

Teradata 13.0Normalization Transformation using Fast load Utility

Description: BISP is committed to provide BEST learning material to the beginners and advance learners. In the same series, we have prepared a complete end-to end Hands-on Guide for building financial data model in Teradata. The document focuses on how the real world requirement should be interpreted. The document focuses on converting the transaction model to OLAP model. Join our professional training program and learn from experts.

History:Version Description Change Author Publish Date0.1 Initial Draft Anil Tekam 12th Aug 20110.1 Review#1 Amit Sharma 18th Aug 2011

www.bispsolutions.com | www.hyperionguru.com | www.bisptrainings.com | Page 1

ContentsContents.......................................................................................................................... 2

Source Data Model ..........................................................................................................3

Target Data Model...........................................................................................................3


Summary: In the below, we are going to convert the transitional model with multiple tables spread across with one subject area. i/e supplier has Suppliers, Suppier_Type etc. As part of normalization transformation it, the below source data mode will be de-normalized to start schema.

Source Data Model

Target Data Model


Customer Source Model and Mapping Sheet



Supplier Source Model and Mapping Sheet


Product Source Model and Mapping Sheet


Employee Source Model and Mapping Sheet


Store Source Model and Mapping Sheet


Normalize Transformation Using Fast Load UtilityDIM_CUSTOMER:- Step #1:-First we perform the load operation from multiple tables to single target table with the help of fast load utility. All the multiple table is normalize table which has placed in single database and then we create the denormalize table with the help of normalize table in other database.

Step #2:- First we use the fast export utility because this utility help for searching the data from source tables. 1.) BEGIN EXPORT STATEMENT. 2.) DEFINE THE OUTPUT FILE NAME.3.) THE SELECT PULLS DATA FROM TABLE (REGION,TERRITORIES,CUSTOMER_TYPE,CUSTOMER).4.) END THE JOB AND LOGOFF TERADATA.


/* ---------------------------------------------------------------*/ ALWAYS GOOD TO IDENTIFY THE /* @(#) FASTEXPORT SCRIPT */ SCRIPT AND AUTHOR IN/* @(#) Version 1.1 */ COMMENTS/* @(#) Created by BISP Team *//* ---------------------------------------------------------------*/

Step #3:- Now the we execute this fast export script in command prompt. After successful execution it will generate the output log file and system generated data file or output data file.

Step #4:- Now the fast export script has successfully executed and it has generated output log file and system generated output data file.

Step #5:- Now we create the fast load flat file script because fast load utility help for loading the data into the target table. We use the system generated output data file for loading purpose in empty target table.

1.) LOGIN ONTO TERADATA.2.) DEFINING INPUT DATA FILE3.) DEFINING THE ERROR FILES


4.) FAST LOAD PERFORM END LOADING.5.) DEFINE LOGOFF CONNECTION

1.) LOGIN ONTO TERADATA:- we logon into the teradata fast load utility.2.) DEFINING INPUT DATA FILE:- Now we comes the DEFINE statement. Fast load must know the structure and the name of the flat file to be used as the input flat file or source file for loading the data.3.) DEFINING THE ERROR FILES:- In this script, there is no create table statement for the error tables. This error files work as error table. Fast load automatically generate both error table. In this instance, they are named err2_cust and UV_cust table. First error table (err2_cust) contains only constraints based related error and second error table (UV_cust) contains only unique violation error.4.) FAST LOAD PERFORM END LOADING:- The target table must be empty at the loading time and if the loading has completed, then we use the END LOADING statement. If we don't use the END LOADING statement then we get error i.e. Target table being loaded and that time we can't access the loading data.5.) DEFINE LOGOFF CONNECTION:- Now all the sessions has logoff. This is the last statement of this fast load script. Now the table lock has released and if no rows in the error table, then that error table automatically dropped and if there is any single row in error table then we check it, then we take appropriate action and drop that error table manually.

Step #6:- Now we execute this fast load flat file script in command prompt. It will take few second for successful execution.


/* ++++++++++++++++++++++++++++ */ Always good to identify the /* FASTLOAD SCRIPT TO LOAD THE */ script and author in comment. /* OLTP.CUSTOMER,CUSTOMER_TYPE,REGION, *//* TERRITORIES TABLE Version 1.1 *//* Created by BISP Team *//* ++++++++++++++++++++++++++++ */

Step #7:- This fast load script has successfully executed in command prompt. After the execution fast load generated the output log file.

Step #8:- Now we show the output log file of fast load utility. This output log file generated with three phase's, which is following-1.) Logon/connection2.) Insert Phase3.) End Loading Phase


1.)DEFINING LOGON CONNECTION:- In this logon statement, we have successfully created the logon connection of fast load utility.

2.)INSERT PHASE:- In this insert phase, we have performed the DML operation i.e. insert operation. In this insert operation, we are sending all 3 records from system generated output data file (fexp_tp_data.txt) to target table i.e. TRG_CREDIT table.

3.)END LOADING PHASE:- In this END LOADING PHASE, we have completed the loading process into the target table. System has read total 3 records from output data file and now applied the insert operation in all 3 records. There is no error in error table 1 and error table 2. We can manually drop both the error table from the database, where the error table place and we are not loading any duplicate rows then total duplicate rows is zero.

Step #9:- Now we show the target table and that target table has successfully loaded with the help of fast load utility. This target table is denormalize table and it has loaded. We can check the target table with the help of following sql query-SELECT * FROM OLAP.DIM_CUSTOMER;This sql query help for loading the data into target table.


DIM_SUPPLIERStep #10:-First we perform the load operation from multiple source tables to single target table with the help of fast load utility. All the multiple source table is normalize table which has placed in different database and after that we create the denormalize table with the help of normalize table and that denormalize table place in different database.

Step #11:- Now we use the fast export utility because this utility help for searching the data from source tables. We create the fast export script. This script help for searching the data.1.) BEGIN EXPORT STATEMENT.


2.) DEFINE THE OUTPUT FILE NAME.3.) THE SELECT PULLS DATA FROM TABLE (SUPPLIER_TYPE,SUPPLIERS).4.) END THE JOB AND LOGOFF TERADATA.

Step #12:- Now we execute this fast export script in command prompt. This fast export script generate the output data file after the successful execution.

Step #13:- This fast export script has successfully executed and it has generated output log file and system generated output data file. This data file hold the data which has come with the help of sql select query.



Step #14:- Now we show the system generated output data and this data file come with the help of fast export utility. This file is unreadable form and only system can read this output data file.

Step #15:- Now we use the fast load utility for loading the data into target table. We load the data from system generated output data file to empty target table. Now fast load script are as follow-

1.) LOGIN ONTO TERADATA.2.) DEFINING INPUT DATA FILE3.) DEFINING THE ERROR FILES4.) FAST LOAD PERFORM END LOADING.5.) DEFINE LOGOFF CONNECTION


/* ++++++++++++++++++++++++++++ */ Always good to identify the /* FASTLOAD SCRIPT TO LOAD THE */ script and author in comment. /* OLTP.SUPPLIER,SUPPLIER_TYPE *//* TABLE Version 1.1 *//* Created by BISP Team *//* ++++++++++++++++++++++++++++ */

1.) LOGIN ONTO TERADATA:- we logon into the teradata fast load utility.2.) DEFINING INPUT DATA FILE:- Now we comes the DEFINE statement. Fast load must know the structure and the name of the flat file to be used as the input flat file or source file for loading the data.3.) DEFINING THE ERROR FILES:- In this script, there is no create table statement for the error tables. This error files work as error table. Fast load automatically generate both error table. In this instance, they are named err2_cust and UV_cust table. First error table (err2_cust) contains only constraints based related error and second error table (UV_cust) contains only unique violation error.4.) FAST LOAD PERFORM END LOADING:- The target table must be empty at the loading time and if the loading has completed, then we use the END LOADING statement. If we don't use the END LOADING statement then we get error i.e. Target table being loaded and that time we can't access the loading data.5.) DEFINE LOGOFF CONNECTION:- Now all the sessions has logoff. This is the last statement of this fast load script. Now the table lock has released and if no rows in the error table, then that error table automatically dropped and if there is any single row in error table then we check it, then we take appropriate action and drop that error table manually.

Step #16:- Now we execute the fast load flat file script in command prompt. This fast load script generate the output log file after the successful execution.

Step #17:- Now this fast load script has successfully executed and it has generated output log file.





2.)INSERT PHASE:- In this insert phase, we have performed the DML operation i.e. insert operation. In this insert operation, we are sending all 12 records from system generated output data file (sup_data.txt) to target table i.e. DIM_SUPPLIER table.


Step #19:- Now the data has successfully loaded into the target table with the help of fast load utility. This target table has placed in different database and this target table


is denormalize table. We can access the record from target table using the following sql query-SELECT * FROM OLAP.DIM_SUPPLIER;This sql select query help for showing the records from target table.

DIM_PRODUCT

Step #20:- we perform the load operation from multiple source tables to single target table with the help of fast load utility. All the multiple source table is normalize table which has placed in different database and after that we create the denormalize table with the help of normalize table and that denormalize table place in different database.1.) BEGIN EXPORT STATEMENT. 2.) DEFINE THE OUTPUT FILE NAME.


3.) THE SELECT PULLS DATA FROM TABLE (PRODUCTS,BRAND,CATEGORY,PACKAGING).4.) END THE JOB AND LOGOFF TERADATA.


Step #22:- Fast export script has successfully executed and it has generated output log file and system generated data file.

Step #23:- Now we are showing the data of system generated data file. This data has in unreadable form and only system understand this data. We use this data file for loading purpose in target table.





1.) LOGIN ONTO TERADATA:- we logon into the teradata fast load utility.2.) DEFINING INPUT DATA FILE:- Now we comes the DEFINE statement. Fast load must know the structure and the name of the flat file to be used as the input flat file or source file for loading the data.3.) DEFINING THE ERROR FILES:- In this script, there is no create table statement for the error tables. This error files work as error table. Fast load automatically generate both error table. In this instance, they are named err2_prod and UV_prod table. First error table (err2_prod) contains only constraints based related error and second error table (UV_prod) contains only unique violation error.


/* ++++++++++++++++++++++++++++ */ Always good to identify the /* FASTLOAD SCRIPT TO LOAD THE */ script and author in comment. /* OLTP.PRODUCTS *//* TABLE Version 1.1 *//* Created by BISP Team *//* ++++++++++++++++++++++++++++ */

4.) FAST LOAD PERFORM END LOADING:- The target table must be empty at the loading time and if the loading has completed, then we use the END LOADING statement. If we don't use the END LOADING statement then we get error i.e. Target table being loaded and that time we can't access the loading data.5.) DEFINE LOGOFF CONNECTION:- Now all the sessions has logoff. This is the last statement of this fast load script. Now the table lock has released and if no rows in the error table, then that error table automatically dropped and if there is any single row in error table then we check it, then we take appropriate action and drop that error table manually.


Step #26:- Fast load flat file script has successfully executed and it has generated output log file.





2.)INSERT PHASE:- In this insert phase, we have performed the DML operation i.e. insert operation. In this insert operation, we are sending all 6 records from system generated output data file (PROD_data.txt) to target table i.e. DIM_PRODUCT table.


Step #28:- Now we are showing the target table records. All the records has successfully loaded into the denormalize target table. We are using following sql query


for showing the result of target table- SELECT * FROM OLAP.DIM_PRODUCT;This sql query help for showing the result of table.

DIM_EMPLOYEE:-Step #29:-First we perform the load operation from multiple tables to single target table with the help of fast load utility. All the multiple table is normalize table which has placed in single database and then we create the denormalize table with the help of normalize table in other database.

Step #30:- First we use the fast export utility because this utility help for searching the data from source tables. 1.) BEGIN EXPORT STATEMENT. 2.) DEFINE THE OUTPUT FILE NAME.3.) THE SELECT PULLS DATA FROM TABLE (DEPARTMENT,EMPLOYEE).4.) END THE JOB AND LOGOFF TERADATA.









1.) LOGIN ONTO TERADATA:- we logon into the teradata fast load utility.2.) DEFINING INPUT DATA FILE:- Now we comes the DEFINE statement. Fast load must know the structure and the name of the flat file to be used as the input flat file or source file for loading the data.3.) DEFINING THE ERROR FILES:- In this script, there is no create table statement for the error tables. This error files work as error table. Fast load automatically generate both error table. In this instance, they are named err2_emp and UV_emp


/* ++++++++++++++++++++++++++++ */ Always good to identify the /* FASTLOAD SCRIPT TO LOAD THE */ script and author in comment. /* OLTP.PRODUCTS *//* TABLE Version 1.1 *//* Created by BISP Team *//* ++++++++++++++++++++++++++++ */

table. First error table (err2_emp) contains only constraints based related error and second error table (UV_emp) contains only unique violation error.4.) FAST LOAD PERFORM END LOADING:- The target table must be empty at the loading time and if the loading has completed, then we use the END LOADING statement. If we don't use the END LOADING statement then we get error i.e. Target table being loaded and that time we can't access the loading data.5.) DEFINE LOGOFF CONNECTION:- Now all the sessions has logoff. This is the last statement of this fast load script. Now the table lock has released and if no rows in the error table, then that error table automatically dropped and if there is any single row in error table then we check it, then we take appropriate action and drop that error table manually.







2.)INSERT PHASE:- In this insert phase, we have performed the DML operation i.e. insert operation. In this insert operation, we are sending all 15 records from system generated output data file (emp_data.txt) to target table i.e. DIM_EMPLOYEE table.


Step #38:- Now we are showing the target table records. All the records has successfully loaded into the denormalize target table. We are using following sql query


for showing the result of target table- SELECT * FROM OLAP.DIM_EMPLOYEE;This sql query help for showing the result of table.

DIM_STORE:-Step #39:-First we perform the load operation from multiple source tables to single target table with the help of fast load utility. All the multiple source table is normalize table which has placed in different database and after that we create the denormalize table with the help of normalize table and that denormalize table place in different database.

Step #40:- Now we use the fast export utility because this utility help for searching the data from source tables. We create the fast export script. This script help for searching the data.1.) BEGIN EXPORT STATEMENT. 2.) DEFINE THE OUTPUT FILE NAME.


3.) THE SELECT PULLS DATA FROM TABLE (STORE,STORE_REGION).4.) END THE JOB AND LOGOFF TERADATA.








1.) LOGIN ONTO TERADATA:- we logon into the teradata fast load utility.2.) DEFINING INPUT DATA FILE:- Now we comes the DEFINE statement. Fast load must know the structure and the name of the flat file to be used as the input flat file or source file for loading the data.3.) DEFINING THE ERROR FILES:- In this script, there is no create table statement for the error tables. This error files work as error table. Fast load automatically


/* ++++++++++++++++++++++++++++ */ Always good to identify the /* FASTLOAD SCRIPT TO LOAD THE */ script and author in comment. /* OLTP.STORE,STORE_REGION *//* TABLE Version 1.1 *//* Created by BISP Team *//* ++++++++++++++++++++++++++++ */

generate both error table. In this instance, they are named err2_str and UV_str table. First error table (err2_str) contains only constraints based related error and second error table (UV_str) contains only unique violation error.4.) FAST LOAD PERFORM END LOADING:- The target table must be empty at the loading time and if the loading has completed, then we use the END LOADING statement. If we don't use the END LOADING statement then we get error i.e. Target table being loaded and that time we can't access the loading data.5.) DEFINE LOGOFF CONNECTION:- Now all the sessions has logoff. This is the last statement of this fast load script. Now the table lock has released and if no rows in the error table, then that error table automatically dropped and if there is any single row in error table then we check it, then we take appropriate action and drop that error table manually.







2.)INSERT PHASE:- In this insert phase, we have performed the DML operation i.e. insert operation. In this insert operation, we are sending all 28 records from system generated output data file (str_data.txt) to target table i.e. DIM_STORE table.

3.)END LOADING PHASE:- In this END LOADING PHASE, we have completed the loading process into the target table. System has read total 28 records from output data file and now applied the insert operation in all 28 records. There is no error in error table 1 and error table 2. We can manually drop both the error table from the database, where the error table place and we are not loading any duplicate rows then total duplicate rows is zero. Step #48:- Now we are showing the target table records. All the records has successfully loaded into the denormalize target table. We are using following sql query for showing the result of target table- SELECT * FROM OLAP.DIM_STORE;This sql query help for showing the result of table.



Documents

OLTP to OLAP COnversion