OLTP to OLAP COnversion

Teradata 13.0Normalization Transformation using Fast load Utility

Description: BISP is committed to provide BEST learning material to the beginners and advance learners. In the same series, we have prepared a complete end-to end Hands-on Guide for building financial data model in Teradata. The document focuses on how the real world requirement should be interpreted. The document focuses on converting the transaction model to OLAP model. Join our professional training program and learn from experts.

History:Version Description Change Author Publish Date0.1 Initial Draft Anil Tekam 12th Aug 20110.1 Review#1 Amit Sharma 18th Aug 2011

www.bispsolutions.com | www.hyperionguru.com | www.bisptrainings.com | Page 1

ContentsContents.......................................................................................................................... 2

Source Data Model ..........................................................................................................3

Target Data Model...........................................................................................................3

Summary: In the below, we are going to convert the transitional model with multiple tables spread across with one subject area. i/e supplier has Suppliers, Suppier_Type etc. As part of normalization transformation it, the below source data mode will be de-normalized to start schema.

Source Data Model

Target Data Model

Customer Source Model and Mapping Sheet

Supplier Source Model and Mapping Sheet

Product Source Model and Mapping Sheet

Employee Source Model and Mapping Sheet

Store Source Model and Mapping Sheet

Normalize Transformation Using Fast Load UtilityDIM_CUSTOMER:- Step #1:-First we perform the load operation from multiple tables to single target table with the help of fast load utility. All the multiple table is normalize table which has placed in single database and then we create the denormalize table with the help of normalize table in other database.

Step #2:- First we use the fast export utility because this utility help for searching the data from source tables. 1.) BEGIN EXPORT STATEMENT. 2.) DEFINE THE OUTPUT FILE NAME.3.) THE SELECT PULLS DATA FROM TABLE (REGION,TERRITORIES,CUSTOMER_TYPE,CUSTOMER).4.) END THE JOB AND LOGOFF TERADATA.

/* ---------------------------------------------------------------*/ ALWAYS GOOD TO IDENTIFY THE /* @(#) FASTEXPORT SCRIPT */ SCRIPT AND AUTHOR IN/* @(#) Version 1.1 */ COMMENTS/* @(#) Created by BISP Team *//* ---------------------------------------------------------------*/

Step #3:- Now the we execute this fast export script in command prompt. After successful execution it will generate the output log file and system generated data file or output data file.

Step #4:- Now the fast export script has successfully executed and it has generated output log file and system generated output data file.

Step #5:- Now we create the fast load flat file script because fast load utility help for loading the data into the target table. We use the system generated output data file for loading purpose in empty target table.

1.) LOGIN ONTO TERADATA.2.) DEFINING INPUT DATA FILE3.) DEFINING THE ERROR FILES

4.) FAST LOAD PERFORM END LOADING.5.) DEFINE LOGOFF CONNECTION

1.) LOGIN ONTO TERADATA:- we logon into the teradata fast load utility.2.) DEFINING INPUT DATA FILE:- Now we comes the DEFINE statement. Fast load must know the structure and the name of the flat file to be used as the input flat file or source file for loading the data.3.) DEFINING THE ERROR FILES:- In this script, there is no create table statement for the error tables. This error files work as error table. Fast load automatically generate both error table. In this instance, they are named err2_cust and UV_cust table. First error table (err2_cust) contains only constraints based related error and second error table (UV_cust) contains only unique violation error.4.) FAST LOAD PERFORM END LOADING:- The target table must be empty at the loading time and if the loading has completed, then we use the END LOADING statement. If we don't use the END LOADING statement then we get error i.e. Target table being loaded and that time we can't access the loading data.5.) DEFINE LOGOFF CONNECTION:- Now all the sessions has logoff. This is the last statement of this fast load script. Now the table lock has released and if no rows in the error table, then that error table automatically dropped and if there is any single row in error table then we check it, then we take appropriate action and drop that error table manually.

Step #6:- Now we execute this fast load flat file script in command prompt. It will take few second for successful execution.

/* ++++++++++++++++++++++++++++ */ Always good to identify the /* FASTLOAD SCRIPT TO LOAD THE */ script and author in comment. /* OLTP.CUSTOMER,CUSTOMER_TYPE,REGION, *//* TERRITORIES TABLE Version 1.1 *//* Created by BISP Team *//* ++++++++++++++++++++++++++++ */

Step #7:- This fast load script has successfully executed in command prompt. After the execution fast load generated the output log file.

Step #8:- Now we show the output log file of fast load utility. This output log file generated with three phase's, which is following-1.) Logon/connection2.) Insert Phase3.) End Loading Phase

1.)DEFINING LOGON CONNECTION:- In this logon statement, we have successfully created the logon connection of fast load utility.

2.)INSERT PHASE:- In this insert phase, we have performed the DML operation i.e. insert operation. In this insert operation, we are sending all 3 records from system generated output data file (fexp_tp_data.txt) to target table i.e. TRG_CREDIT table.

3.)END LOADING PHASE:- In this END LOADING PHASE, we have completed the loading process into the target table. System has read total 3 records from output data file and now applied the insert operation in all 3 records. There is no error in error table 1 and error table 2. We can manually drop both the error table from the database, where the error table place and we are not loading any duplicate rows then total duplicate rows is zero.

Step #9:- Now we show the target table and that target table has successfully loaded with the help of fast load utility. This target table is denormalize table and it has loaded. We can check the target table with the help of following sql query-SELECT * FROM OLAP.DIM_CUSTOMER;This sql query help for loading the data into target table.

DIM_SUPPLIERStep #10:-First we perform the load operation from multiple source tables to single target table with the help of fast load utility. All the multiple source table is normalize table which has placed in different database and after that we create the denormalize table with the help of normalize table and that denormalize table place in different database.

Step #11:- Now we use the fast export utility because this utility help for searching the data from source tables. We create the fast export script. This script help for searching the data.1.) BEGIN EXPORT STATEMENT.

2.) DEFINE THE OUTPUT FILE NAME.3.) THE SELECT PULLS DATA FROM TABLE (SUPPLIER_TYPE,SUPPLIERS).4.) END THE JOB AND LOGOFF TERADATA.

Step #12:- Now we execute this fast export script in command prompt. This fast export script generate the output data file after the successful execution.

Step #13:- This fast export script has successfully executed and it has generated output log file and system generated output data file. This data file hold the data which has come with the help of sql select query.

Step #14:- Now we show the system generated output data and this data file come with the help of fast export utility. This file is unreadable form and only system can read this output data file.

Step #15:- Now we use the fast load utility for loading the data into target table. We load the data from system generated output data file to empty target table. Now fast load script are as follow-

1.) LOGIN ONTO TERADATA.2.) DEFINING INPUT DATA FILE3.) DEFINING THE ERROR FILES4.) FAST LOAD PERFORM END LOADING.5.) DEFINE LOGOFF CONNECTION

/* ++++++++++++++++++++++++++++ */ Always good to identify the /* FASTLOAD SCRIPT TO LOAD THE */ script and author in comment. /* OLTP.SUPPLIER,SUPPLIER_TYPE *//* TABLE Version 1.1 *//* Created by BISP Team *//* ++++++++++++++++++++++++++++ */

1.) LOGIN ONTO TERADATA:- we logon into the teradata fast load utility.2.) DEFINING INPUT DATA FILE:- Now we comes the DEFINE statement. Fast load must know the structure and the name of the flat file to be used as the input flat file or source file for loading the data.3.) DEFINING THE ERROR FILES:- In this script, there is no create table statement for the error tables. This error files work as error table. Fast load automatically generate both error table. In this instance, they are named err2_cust and UV_cust table. First error table (err2_cust) contains only constraints based related error and second error table (UV_cust) contains only unique violation error.4.) FAST LOAD PERFORM END LOADING:- The target table must be empty at the loading time and if the loading has completed, then we use the END LOADING statement. If we don't use the END LOADING statement then we get error i.e. Target table being loaded and that time we can't access the loading data.5.) DEFINE LOGOFF CONNECTION:- Now all the sessions has logoff. This is the last statement of this fast load script. Now the table lock has released and if no rows in the error table, then that error table automatically dropped and if there is any single row in error table then we check it, then we take appropriate action and drop that error table manually.

Step #16:- Now we execute the fast load flat file script in command prompt. This fast load script generate the output log file after the successful execution.

Step #17:- Now this fast load script has successfully executed and it has generated output log file.

2.)INSERT PHASE:- In this insert phase, we have performed the DML operation i.e. insert operation. In this insert operation, we are sending all 12 records from system generated output data file (sup_data.txt) to target table i.e. DIM_SUPPLIER table.

Step #19:- Now the data has successfully loaded into the target table with the help of fast load utility. This target table has placed in different database and this target table

is denormalize table. We can access the record from target table using the following sql query-SELECT * FROM OLAP.DIM_SUPPLIER;This sql select query help for showing the records from target table.

DIM_PRODUCT

Step #20:- we perform the load operation from multiple source tables to single target table with the help of fast load utility. All the multiple source table is normalize table which has placed in different database and after that we create the denormalize table with the help of normalize table and that denormalize table place in different database.1.) BEGIN EXPORT STATEMENT. 2.) DEFINE THE OUTPUT FILE NAME.

3.) THE SELECT PULLS DATA FROM TABLE (PRODUCTS,BRAND,CATEGORY,PACKAGING).4.) END THE JOB AND LOGOFF TERADATA.

Step #22:- Fast export script has successfully executed and it has generated output log file and system generated data file.

Step #23:- Now we are showing the data of system generated data file. This data has in unreadable form and only system understand this data. We use this data file for loading purpose in target table.

1.) LOGIN ONTO TERADATA:- we logon into the teradata fast load utility.2.) DEFINING INPUT DATA FILE:- Now we comes the DEFINE statement. Fast load must know the structure and the name of the flat file to be used as the input flat file or source file for loading the data.3.) DEFINING THE ERROR FILES:- In this script, there is no create table statement for the error tables. This error files work as error table. Fast load automatically generate both error table. In this instance, they are named err2_prod and UV_prod table. First error table (err2_prod) contains only constraints based related error and second error table (UV_prod) contains only unique violation error.

/* ++++++++++++++++++++++++++++ */ Always good to identify the /* FASTLOAD SCRIPT TO LOAD THE */ script and author in comment. /* OLTP.PRODUCTS *//* TABLE Version 1.1 *//* Created by BISP Team *//* ++++++++++++++++++++++++++++ */

4.) FAST LOAD PERFORM END LOADING:- The target table must be empty at the loading time and if the loading has completed, then we use the END LOADING statement. If we don't use the END LOADING statement then we get error i.e. Target table being loaded and that time we can't access the loading data.5.) DEFINE LOGOFF CONNECTION:- Now all the sessions has logoff. This is the last statement of this fast load script. Now the table lock has released and if no rows in the error table, then that error table automatically dropped and if there is any single row in error table then we check it, then we take appropriate action and drop that error table manually.

Step #26:- Fast load flat file script has successfully executed and it has generated output log file.

2.)INSERT PHASE:- In this insert phase, we have performed the DML operation i.e. insert operation. In this insert operation, we are sending all 6 records from system generated output data file (PROD_data.txt) to target table i.e. DIM_PRODUCT table.

Step #28:- Now we are showing the target table records. All the records has successfully loaded into the denormalize target table. We are using following sql query

for showing the result of target table- SELECT * FROM OLAP.DIM_PRODUCT;This sql query help for showing the result of table.

DIM_EMPLOYEE:-Step #29:-First we perform the load operation from multiple tables to single target table with the help of fast load utility. All the multiple table is normalize table which has placed in single database and then we create the denormalize table with the help of normalize table in other database.

Step #30:- First we use the fast export utility because this utility help for searching the data from source tables. 1.) BEGIN EXPORT STATEMENT. 2.) DEFINE THE OUTPUT FILE NAME.3.) THE SELECT PULLS DATA FROM TABLE (DEPARTMENT,EMPLOYEE).4.) END THE JOB AND LOGOFF TERADATA.

1.) LOGIN ONTO TERADATA:- we logon into the teradata fast load utility.2.) DEFINING INPUT DATA FILE:- Now we comes the DEFINE statement. Fast load must know the structure and the name of the flat file to be used as the input flat file or source file for loading the data.3.) DEFINING THE ERROR FILES:- In this script, there is no create table statement for the error tables. This error files work as error table. Fast load automatically generate both error table. In this instance, they are named err2_emp and UV_emp

/* ++++++++++++++++++++++++++++ */ Always good to identify the /* FASTLOAD SCRIPT TO LOAD THE */ script and author in comment. /* OLTP.PRODUCTS *//* TABLE Version 1.1 *//* Created by BISP Team *//* ++++++++++++++++++++++++++++ */

table. First error table (err2_emp) contains only constraints based related error and second error table (UV_emp) contains only unique violation error.4.) FAST LOAD PERFORM END LOADING:- The target table must be empty at the loading time and if the loading has completed, then we use the END LOADING statement. If we don't use the END LOADING statement then we get error i.e. Target table being loaded and that time we can't access the loading data.5.) DEFINE LOGOFF CONNECTION:- Now all the sessions has logoff. This is the last statement of this fast load script. Now the table lock has released and if no rows in the error table, then that error table automatically dropped and if there is any single row in error table then we check it, then we take appropriate action and drop that error table manually.

2.)INSERT PHASE:- In this insert phase, we have performed the DML operation i.e. insert operation. In this insert operation, we are sending all 15 records from system generated output data file (emp_data.txt) to target table i.e. DIM_EMPLOYEE table.

Step #38:- Now we are showing the target table records. All the records has successfully loaded into the denormalize target table. We are using following sql query

for showing the result of target table- SELECT * FROM OLAP.DIM_EMPLOYEE;This sql query help for showing the result of table.

DIM_STORE:-Step #39:-First we perform the load operation from multiple source tables to single target table with the help of fast load utility. All the multiple source table is normalize table which has placed in different database and after that we create the denormalize table with the help of normalize table and that denormalize table place in different database.

Step #40:- Now we use the fast export utility because this utility help for searching the data from source tables. We create the fast export script. This script help for searching the data.1.) BEGIN EXPORT STATEMENT. 2.) DEFINE THE OUTPUT FILE NAME.

3.) THE SELECT PULLS DATA FROM TABLE (STORE,STORE_REGION).4.) END THE JOB AND LOGOFF TERADATA.

1.) LOGIN ONTO TERADATA:- we logon into the teradata fast load utility.2.) DEFINING INPUT DATA FILE:- Now we comes the DEFINE statement. Fast load must know the structure and the name of the flat file to be used as the input flat file or source file for loading the data.3.) DEFINING THE ERROR FILES:- In this script, there is no create table statement for the error tables. This error files work as error table. Fast load automatically

/* ++++++++++++++++++++++++++++ */ Always good to identify the /* FASTLOAD SCRIPT TO LOAD THE */ script and author in comment. /* OLTP.STORE,STORE_REGION *//* TABLE Version 1.1 *//* Created by BISP Team *//* ++++++++++++++++++++++++++++ */

generate both error table. In this instance, they are named err2_str and UV_str table. First error table (err2_str) contains only constraints based related error and second error table (UV_str) contains only unique violation error.4.) FAST LOAD PERFORM END LOADING:- The target table must be empty at the loading time and if the loading has completed, then we use the END LOADING statement. If we don't use the END LOADING statement then we get error i.e. Target table being loaded and that time we can't access the loading data.5.) DEFINE LOGOFF CONNECTION:- Now all the sessions has logoff. This is the last statement of this fast load script. Now the table lock has released and if no rows in the error table, then that error table automatically dropped and if there is any single row in error table then we check it, then we take appropriate action and drop that error table manually.

2.)INSERT PHASE:- In this insert phase, we have performed the DML operation i.e. insert operation. In this insert operation, we are sending all 28 records from system generated output data file (str_data.txt) to target table i.e. DIM_STORE table.

3.)END LOADING PHASE:- In this END LOADING PHASE, we have completed the loading process into the target table. System has read total 28 records from output data file and now applied the insert operation in all 28 records. There is no error in error table 1 and error table 2. We can manually drop both the error table from the database, where the error table place and we are not loading any duplicate rows then total duplicate rows is zero. Step #48:- Now we are showing the target table records. All the records has successfully loaded into the denormalize target table. We are using following sql query for showing the result of target table- SELECT * FROM OLAP.DIM_STORE;This sql query help for showing the result of table.

OLTP to OLAP COnversion

Documents

OLAP and Cubes · 2010. 7. 9. · oltp vs. olap oltp olap ผู ทํางานกับข อมูลมักจะเปนระด ับ พนักงานฏ ิบัติการ

ÎHyyybrid OLTP&OLAP Database SystemsChair of Informatics III: Database Systems The mixed workload CH-BenCHmark ÎHyyybrid OLTP&OLAP Database Systems ÎReal-Time Business Intelligence

Data Blocks: Hybrid OLTP and OLAP on Compressed Storage ... fileData Blocks: Hybrid OLTP and OLAP on Compressed Storage using both Vectorization and Compilationy Harald Lang1, Tobias

Chapter 17 OLAP and Data Mining. 2 OLTP Compared With OLAP OLTPOn Line Transaction Processing – OLTP –Maintains a database that is an accurate model of

Enterprise Applications – OLTP and OLAP – Share One ... · Enterprise Applications – OLTP and OLAP – Share One Database Architecture Prof. Dr. Hasso Plattner

From OLTP to OLAP - bicortex.combicortex.com/.../post_content//2013/09/From-OLTP-to-OLAP.pdf · From OLTP to OLAP - Derivation of Sale Fee Star Schema Type of models 48! Where hierarchical

OLAP and Data Mining - cs.stonybrook.edukifer/Courses/cse532/slides/ch17.pdf · OLAP and Data Mining Chapter 17 2 OLTP Compared With OLAP • On Line Transaction Processing – OLTP

Data Blocks: Hybrid OLTP and OLAP on Compressed Storage

Compacting Transactional Data in Hybrid OLTP&OLAP Databases

OLAP and Data Mining Chapter 17. 2 OLTP Compared With OLAP OLTPOn Line Transaction Processing – OLTP –Maintains a database that is an accurate model of

OLAP vs OLTP. OLAP vs. OLTP OLTP (On Line Transaction Processing) (Procesamiento de transacción en línea) Bases de datos orientadas al procesamiento de

L-Store: A Real-time OLTP and OLAP System › papers › l-store-slides.pdfUnifying OLTP and OLAP: Velocity & Volume Dimensions Observed Trends In operational databases, there is a

Data Blocks: Hybrid OLTP and OLAP on Compressed Storage ...db.in.tum.de/downloads/publications/datablocks.pdf · Data Blocks: Hybrid OLTP and OLAP on Compressed Storage using both

BatchDB: Efﬁcient Isolated Execution of Hybrid OLTP+OLAP ...jgiceva/papers/SIGMOD_batchdb.pdf · tion of OLTP and OLAP workloads: – Explicit resource sharing depends on scheduling

L-Store: A Real-time OLTP and OLAP System · L-Store: A Real-time OLTP and OLAP System * Mohammad Sadoghi y, Souvik Bhattacherjee z, Bishwaranjan Bhattacharjee #, Mustafa Canim #

Parallel Real-Time OLAP on Multi-Core Processors and Cloud ... · OLAP vs. OLTP Source: AcceleratedAnalytics.com OLTP System OLAP System Source of data Operational data Consolidated

Olap, oltp and data mining

OLTP Compared With OLAP - hjemmesider.diku.dkhjemmesider.diku.dk/~bonnet/coursematerial/AdvancedDB06/olap-mining.pdf · 1 OLTP Compared With OLAP • On Line Transaction Processing

Seamless Integration of Archiving Functionality in OLTP/OLAP

Olap vs oltp bases datos 2