Introduction to Data Warehousing

Data Warehousing Technologies

Dr. Ilieva I. Ageenko

History of Information Processing

Most organizations began information processing on a small scale, automating one application at a time.

Systems tend to grow independently to support defined functional areas.

Each functional area tended to plan and develop systems in isolation from other areas.

Stages of Information Processing

File Transaction Processing

File Transaction Processing

The Traditional approach to file processing encouraged each functional area to develop and maintain specialized applications.

Individual applications ran on unique master files.

Problems with traditional file processing:

Data RedundancyLack of Data IntegrityProgram-Data DependencyLack of FlexibilityPoor SecurityLack of Data Sharing and Availability

Traditional File Processing

Data Redundancy and Inconsistency across all files:

File AStudent ID

NameAddressZIP Code

phone number

Registration Application

Library Application

Financial aids

Application Credit Records

Application

File BStudent IDFirst NameLast Name

Address & ZIPphone number

File CStudent ID

First & Last NameAddressZIP Code

phone number

File DSocial Security

NameAddressZIP Code

phone number

Stages of Information Processing

File Transaction ProcessingData Based Management System (DBMS)

Data Base Management Systema single source for all processing

Registration IS application

Library IS application

Financialaids application

Credit RecordsIS application

DBMS- database management system

INTEGRATED STUDENTS DATABASE

Students;name

address

Credit Records:number of credits

classes

Books:Book aBook b

Data DefinitionLanguage

Data ManipulationLanguage

CommonData Dictionary

Reasons for Extract Programs

Accessibility move data out of online processing systems

Performance perform analytical functions separate from

online processing functionsControl

shift in control of the data the end-user ends up “owing” it

Problems with naturally evolution of data extraction

Credibility of dataLow Productivity Inability to transform data into information

Stages of Information Processing

File Transaction ProcessingData Based Management System (DBMS)Extract ProcessingDecision Support Systems (DSS)

Decision Support Systems

Computer system at the management level of an organization that combines data, sophisticated analytical models, and user-friendly software to support semi-structured and unstructured decision making.

DSS often tend to be stand-alone systems, developed by end-user groups not under central IS control

Components of DSS

DSS database A collection of current or historical data from a

number of applications or groups

Model base A collection of analytical (math , statistic) models

that can easily be made accessible to the DSS user.DSS software system

The DSS component that permits easy interaction between the users of the system and the DSS database model base.

Extract Processing

Student IDName

AddressZIP Code

phone number

Registration System

Library System

Financial aidsSystem

Credit RecordsSystem

Student IDFirst NameLast Name

Address & ZIP

Student IDFirst & Last Name

AddressZIP Code

phone number

Social SecurityName

AddressZIP Code

phone number

Report

Back Office Intensive Manual Work - DSS

Ad-hocreport A

Ad-hocreport B

Ad-hocreport C

Ad-hocreport D

DSS

Dilemma- Most of the Business Analysts time is not spent in true data analysis

These logistics factors can negatively impact and slow down efficiency and effectiveness of business analysis:

Growing Volume of DataData stored in many different systems and formatsThe criticality of quick decision makingIntroduction to new products and Market dynamicsChange in organizational strategies

Stages of Information Processing

File Transaction ProcessingData Based Management System (DBMS)Extract ProcessingDecision Support Systems (DSS) Data Warehouses

DATA WAREHOUSE

Multidimensional database with reporting and query tools, that stores current and historical data extracted from various operational systems and consolidated for management reporting and analysis.

Addresses the problem of integrating key operational data from around the company in a form that is consistent , reliable, and easily available for reporting.

Data Warehouse Enterprise Architecture

Customer data

DATAWAREHOUSE

Deposits

Savings

CreditCards

Collections

Transaction Processing Systems (Legacy)

DA

TA

EX

TR

AC

TIO

N

CL

EA

NIN

G a

nd C

ON

DI T

ION

ING

TR

AN

SFO

RM

AT

ION

DATAMARTS

SAS BUSINESSOBJECTS

SQL

Marketing

Credit Card

Small Business

Stages of Information Processing

File Transaction ProcessingData Based Management System (DBMS)Extract ProcessingDecision Support Systems (DSS) Data WarehousesOLAP

Data Warehouse Architecture and OLAP

Customer data

DATAWAREHOUSE

Deposits

Savings

CreditCards

Collections

Transaction Processing Systems (Legacy)

DA

TA

EX

TR

AC

TIO

N

CL

EA

NIN

G a

nd C

ON

DI T

ION

ING

TR

AN

SFO

RM

AT

ION

DATAMARTS

OLAP

SAS BUSINESSOBJECTS

SQL

Marketing

Credit Card

Small Business

OLAP

OLAP

OLAP

OLTP vs. OLAP

OLTP database applications are developed to meet the day-to-day and operational data retrieval needs of end-users

Provides read-write capability

Data Warehouses along with OLAP tools are being developed to meet information exploration and historical trend analysis management needs

Provides read-only capability

Stages of Information Processing

File Transaction ProcessingData Based Management System (DBMS)Extract ProcessingDecision Support Systems (DSS) Data WarehousesOLAPData Mining

Data Mining

The exploration and analysis, by automatic or semiautomatic means, of large quantities of data in order to discover valid , meaningful patterns and rules to assist with business decisions.

Data Warehouse Architecture and OLAP

Customer data

DATAWAREHOUSE

Deposits

Savings

CreditCards

Collections

Transaction Processing Systems (Legacy)

DA

TA

EX

TR

AC

TIO

N

CL

EA

NIN

G a

nd C

ON

DI T

ION

ING

TR

AN

SFO

RM

AT

ION

DATAMARTS

OLAP

DATA MINING

Marketing

Credit Card

Small Business

OLAP

OLAP

OLAP

Warehousing data outside the operational systems

The primary concept of data warehousing is that the data stored for business analysis can most effectively be accessed by separating it from the data in the operational systems.

Fundamental differences between operational and informational (DW) environment: Nature of the data Development Cycle Supporting technology User community Processing characteristics