Data warehouse implementation

“ What is the Challenge ? “

– Faster processing of OLAP queries

Requirements of a Data Warehouse system

Efficient cube computation Better access methods Efficient query processing

Cube computation

COMPUTE CUBE OPERATOR Definition :

“ It computes the aggregates over all subsets of the dimensions specified in the operation “

Syntax : Compute cube cubename

Example

Consider we define the data cube for an electronic store “Best Electronics” Dimensions are :

CityItemYear

Measure :Sales_in_dollars

4

Cube Operation

• Cube definition and computation in DMQL

define cube sales[item, city, year]: sum(sales_in_dollars)

compute cube sales

• Transform it into a SQL-like language (with a new operator cube by, introduced by Gray et al.’96)

SELECT item, city, year, SUM (amount)

FROM SALES

CUBE BY item, city, year• Need compute the following Group-Bys

(date, product, customer),(date,product),(date, customer), (product, customer),(date), (product), (customer)()

(item)(city)

()

(year)

(city, item) (city, year) (item, year)

(city, item, year)

5

Efficient Data Cube Computation

• Data cube can be viewed as a lattice of cuboids – The bottom-most cuboid is the base cuboid– The top-most cuboid (apex) contains only one cell– How many cuboids in an n-dimensional cube with L levels?

• Materialization of data cube– Materialize every (cuboid) (full materialization), none (no

materialization), or some (partial materialization)– Selection of which cuboids to materialize

• Based on size, sharing, access frequency, etc.

)11(

n

i iLT

6

Iceberg Cube

• Computing only the cuboid cells whose count or other aggregates satisfying the condition like

HAVING COUNT(*) >= minsup

Motivation Only a small portion of cube cells may be “above the water’’

in a sparse cube Only calculate “interesting” cells—data above certain

threshold Avoid explosive growth of the cube

Suppose 100 dimensions, only 1 base cell. How many aggregate cells if count >= 1? What about count >= 2?

Compute cube operator

• The statement “ compute cube sales “

• It explicitly instructs the system to compute the sales aggregate cuboids for all the subsets of the set { item, city, year}

• Generates a lattice of cuboids making up a 3-D data cube ‘sales’

• Each cuboid in the lattice corresponds to a subset

Figure from Data Mining Concepts & Techniques

By Jiawei Han & Micheline Kamber

Page # 72

Compute cube operator

Advantages

– Computes all the cuboids for the cube in advance– Online analytical processing needs to access different cuboids for different queries.– Precomputation leads to fast response time

Disadvantages– Required storage space may explode if all of the cuboids in the data cube are

precomputed

• Consider the following 2 cases for n-dimensional cube

– Case 1 : Dimensions have no hierarchies

• Then the total number of cuboids computed for a n-dimensional cube = 2 n

– Case 2: Dimensions have hierarchies

• Then the total number of cuboids computed for a n-dimensional cube =

» Where Li is the number of levels associated with dimension i

“ What is chunking ?”

• MOLAP uses multidimensional array for data storage

• Chunk is obtained by partitioning the multidimensional array such that it is small enough to fit in the memory available for cube computation

So from the above 2 points we get :

“ Chunking is a method for dividing the n-dimensional array into small n-dimensional chunks “

Multiway Array Aggregation

Multiway Array Aggregation

• It is a technique used for the computation of data cube• It is used for MOLAP cube construction

Example

• Consider 3-D data array• Dimensions are A,B,C• Each dimension is partitioned into 4

equalized partitions• A : a0,a1,a2,a3

• B : b0,b1,b2,b3

• C : c0,c1,c2,c3

• 3-D array is partitioned into 64 chunks as shown in the figure

Figure from Data Mining Concepts & TechniquesBy Jiawei Han & Micheline Kamber

Page # 76

Multiway Array Aggregation (contd )

• The cuboids that make up the cube are

– Base cuboid ABC• From which all other cuboids are

generated• It is already computed and corresponds

to given 3-D array

– 2-D cuboids AB,AC,BC– 1-D cuboids A,B,C– 0-D cuboid (apex cuboid)

Figure from Data Mining Concepts & TechniquesBy Jiawei Han & Micheline KamberPage # 76

Better access methods

For efficient data accessing :• Materialized View• Index structures

• Bitmap Indexing – allows quick searching on Data Cubes, through record_ID lists.• Join Indexing – creates a joinable rows of two

relations from a relational database.

“ Materialized views contains aggregate data (cuboids) derived from a fact table in order to minimize the query response time “

There are 3 kinds of materialization(Given a base cuboid )

1. No Materialization – Precompute only the base cuboid

• “ Slow response time ”2. Full Materialization

– Precompute all of the cuboids • “ Large storage space “

3. Partial Materialization– Selectively compute a subset of the cuboids

• “ Mix of the above “

Materialized View

Bitmap Indexing• Used for quick searching in data cubes• Features

– A distinct bit vector Bv ,for each value v in the domain of the attribute– If the domain has n values then the bitmap index has n bit vectors

Example

Dimensions• Item• city

Where:

H=Home entertainment, C=Computer

P=Phone, S=Security

V=Vancouver, T=Toronto

Join Indexing• It is useful in maintaining the relationship between the foreign key and its matching primary key

Consider the sales fact table and the dimension tables for location and item

Join Indexing

Efficient query processing• Query processing proceeds as follows given materialized

views :

– Determine which operations should be performed on the available cuboids

• Transforming operations (selection, roll-up, drill down,…) specified in the query into corresponding sql and/or OLAP operations.

– Determine to which materialized cuboid(s) the relevant operations should be applied • Identifying the cuboids for answering the query

• Select the cuboid with the least cost

Consider a data cube for “Best Electronics” of the form

• “sales [time, item, location]:sum(sales_in_dollars)• Dimension hierarchies used are :

– “ day<month<quarter<year ” for time – “ item_name<brand<type” for item– “ street<city<province_or_state<country “ for location

• Query :{ brand,province_or_state} with year = 2000

• Materialized cuboids available are• Cuboid 1: { item_name,city,year}• Cuboid 2: {brand,country,year}• Cuboid 3: {brand,province_or_state,year}• Cuboid 4: {item_name,province_or_state} where year=2000

“ Which of the above four cuboids should be selected to process the query ? “

• Cuboid 2– It cannot be used

» Since finer granularity data cannot be generated from coarser granularity data» Here country is more general concept than province_or_state

• Cuboid 1,3,4• Can be used

• They have the same set or a superset of the dimensions in the query• The selection clause in the query can imply the selection in the cuboid• The abstraction levels for the item and location dimensions are at a finer level

than brand and province_or_state respectively

“How would the cost of each cuboid compare if used to process the query”• Cuboid 1 : – Will cost more

• Since both item_name and city are at a lower level than brand and province_or_state specified in the query

• Cuboid 3 : • Will cost least

• If there are not many year values associated with items in the cube but there are several item_names for each brand

• Cuboid 3 will be smaller than cuboid 4

• Cuboid 4 : • Will cost least

• If efficient indices are available

“Hence some cost based estimation is required in order to decide which set of cuboids must be selected for query processing “

21

Indexing OLAP Data: Bitmap Index• Index on a particular column• Each value in the column has a bit vector: bit-op is fast• The length of the bit vector: # of records in the base table• The i-th bit is set if the i-th row of the base table has the value for the

indexed column• not suitable for high cardinality domains

Cust Region TypeC1 Asia RetailC2 Europe DealerC3 Asia DealerC4 America RetailC5 Europe Dealer

RecID Retail Dealer1 1 02 0 13 0 14 1 05 0 1

RecIDAsia Europe America1 1 0 02 0 1 03 1 0 04 0 0 15 0 1 0

Base table Index on Region Index on Type

22

Indexing OLAP Data: Join Indices

• Join index: JI(R-id, S-id) where R (R-id, …) S (S-id, …)

• Traditional indices map the values to a list of record ids– It materializes relational join in JI file and speeds

up relational join • In data warehouses, join index relates the values of

the dimensions of a start schema to rows in the fact table.– E.g. fact table: Sales and two dimensions city and

product• A join index on city maintains for each distinct

city a list of R-IDs of the tuples recording the Sales in the city

– Join indices can span multiple dimensions

23

Efficient Processing OLAP Queries

• Determine which operations should be performed on the available cuboids

– Transform drill, roll, etc. into corresponding SQL and/or OLAP operations, e.g., dice

= selection + projection

• Determine which materialized cuboid(s) should be selected for OLAP op.

– Let the query to be processed be on {brand, province_or_state} with the condition

“year = 2004”, and there are 4 materialized cuboids available:

1) {year, item_name, city}

2) {year, brand, country}

3) {year, brand, province_or_state}

4) {item_name, province_or_state} where year = 2004

Which should be selected to process the query?

• Explore indexing structures and compressed vs. dense array structs in MOLAP

24

From data warehousing to data mining

25

Data Warehouse Usage

• Three kinds of data warehouse applications

– Information processing

• supports querying, basic statistical analysis, and reporting using crosstabs, tables, charts and graphs

– Analytical processing

• multidimensional analysis of data warehouse data

• supports basic OLAP operations, slice-dice, drilling, pivoting

– Data mining

• knowledge discovery from hidden patterns

• supports associations, constructing analytical models, performing classification and prediction, and presenting the mining results using visualization tools

26

From On-Line Analytical Processing (OLAP) to On Line Analytical Mining (OLAM)

• Why online analytical mining?– High quality of data in data warehouses

• DW contains integrated, consistent, cleaned data– Available information processing structure surrounding data

warehouses• ODBC, OLEDB, Web accessing, service facilities, reporting

and OLAP tools– OLAP-based exploratory data analysis

• Mining with drilling, dicing, pivoting, etc.– On-line selection of data mining functions

• Integration and swapping of multiple mining functions, algorithms, and tasks

27

An OLAM System Architecture

Data Warehouse

Meta Data

MDDB

OLAMEngine

OLAPEngine

User GUI API

Data Cube API

Database API

Data cleaning

Data integration

Layer3

OLAP/OLAM

Layer2

MDDB

Layer1

Data Repository

Layer4

User Interface

Filtering&Integration Filtering

Databases

Mining query Mining result

OLAP APPLICATIONS• Financial Applications• Activity-based costing (resource allocation)• Budgeting• Marketing/Sales Applications• Market Research Analysis• Sales Forecasting• Promotions Analysis• Customer Analyses• Market/Customer Segmentation• Business modeling• Simulating business behaviour• Extensive, real-time decision support system for managers

BENEFITS OF USING OLAP

• OLAP helps managers in decision-making through the multidimensional data views that it is capable of providing, thus increasing their productivity.

• OLAP applications are self-sufficient owing to the inherent flexibility provided to the organized databases.

• It enables simulation of business models and problems, through extensive usage of analysis-capabilities.

• In conjunction with data warehousing, OLAP can be used to provide reduction in the application backlog, faster information retrieval and reduction in query drag..