Information Systems for Engineers 9. OLAP and Data Cubes

Information Systems for Engineers9. OLAP and Data Cubes

Ghislain FournySpring 2020

Information Systems for EngineersGhislain Fourny

fotoreactor / 123RF Stock Photo

The road to analytics

Aurelio Scetta / 123RF Stock Photo

Another history of data management(T. Hofmann)

Age of Transactions1970s – 2000s


Age of Transactions

Age of Business Intelligence

1970s – 2000s

1995 -


Age of Transactions

Age of Business Intelligence

Age of Big Data

1970s – 2000s

1995 -

2000s -

Paradigms

OLTP

Paradigms

OLTP OLAP

vs.

OnLine Transaction Processing

ConsistentandReliableRecord-Keeping


Transactionsand resultson

small portionsof data


Lotsof transactionson

small portionsof data


NormalizedData

OnLine Analytical Processing

Data-basedDecisionSupport arturaliev / 123RF Stock Photo

OLAP is Big

Large portionsof the data

OLAP is Big

Possiblymany joins


OLAP is Big

Possiblymany joins


Few longheavy queries

OLAP Examples

Web analytics

Sales analytics

Management support

Statistical analysis (census)

Scientific databases (e.g., bio-informatics)

OLTP vs. OLAP

DetailedIndividualRecords

HistoricalSummarizedConsolidated

DataAurelio Scetta / 123RF Stock Photo

OLTP OLAP

vs.

OLTP vs. OLAP

OLTP

Lots of writes

OLAP

Lots of reads

vs.

OLTP vs. OLAP

OLTP

Small setsof records

OLAP

vs.Analysis

over big chunks

OLTP vs. OLAP

OLTP OLAP

Slow interactive

vs.

fully interactive(< 1s)

OLTP vs. OLAP

OLTP OLAP

Consistency

RedundancyRedundancyRedundancy

OLAP

Aurelio Scetta / 123RF Stock Photo

A data warehouse

... is a

A data warehouse

... is a

subject-oriented

A data warehouse

... is a

subject-orientedintegrated

A data warehouse

... is a

subject-orientedintegratedtime-variant

A data warehouse

... is a

subject-orientedintegratedtime-variantnonvolatile

A data warehouse

... is a

subject-orientedintegratedtime-variantnonvolatilecollection of data

in support of management'sdecision-making process

Subject-oriented

customers

products

sales

events

Integrated

Time-variant

Timein data warehouses is

paramount(not so in OLTP systems)

Time-variant

2019Y-1Y-2Y-3Y-4Y-5Y-6Y-7Y-8Y-9

Often past 5-10 years

Non-volatile

Milosh Kojadinovich / 123RF Stock Photo

Load.

Non-volatile


Load.Access.

Non-volatile


Load.Access.Period.

Non-volatile


Load.Access.Period.

no updates

Architecture

ERP

CRM

OLTP

Files

Architecture

ERP

CRM

OLTP

ETL

Files

Architecture

ERP

CRM

OLTP

ETL

Files

Analyze

Report

Mine

OLAP: Redundancy

OLAP: Redundancy

Materializedviews

(denormalized)

1st Normal Form (tabular) – The Key

2nd Normal Form (not joined) – The Whole Key

3rd Normal Form – Nothing But The Key

Why materialize?

Operational data sourcesare too heterogeneous

OLAP: Derived data

OLAP: Special-purpose indices

Querying OLAP

0

1

2

3

4

5

6

Category 1 Category 2 Category 3 Category 4

Slow interactive

Series 1 Series 2 Series 3

1 - 10s

Querying OLAP

vs.Continuous

monitoring/tracking0

1

2

3

4

5

6

Category 1 Category 2 Category 3 Category 4

Slow interactive

Series 1 Series 2 Series 3

1 - 10s hours

Summary of differences

OLTP OLAPSource Original (operational) Derived (consolidated)Purpose Business tasks Decision supportInterface Snapshot Multidimensional viewsWriting short and fast, by end user period refreshes, by batch jobsQueries Simple, small results Complex and aggregatingDesign Many normalized tables Few denormalized cubesPrecision ACID Sampling, confidence intervalsFreshness Serializability ReproducibilitySpeed Very fast Often slowOptimization Inter-query Intra-querySpace Small, archiving old data Large, less space efficientBackup Very important Re-ETL

Data Model

Data Cubes

Data is stored in

multidimensionalhypercubes

Data Cubes

Year

Data Cubes

Country

Data Cubes

Product

Fact

2016CH

Server

Dimensions

Where

What?

Who? etc.

Which currency?

When?

Fact table

Where?

Germany 2016 Peter 1,000$

Germany 2015 Mary 15,000$

Switzerland 2016 Mary 1,500$

Switzerland 2015 Peter 3,000$

Australia 2015 Peter 6,000$

China 2015 Mary 1,000$

Aggregation

Where?







Aggregation

Aggregation

Where?







Aggregation

2016 Peter 1,000$

2015 Mary 16,000$

2016 Mary 1,500$

2015 Peter 9,000$

Slicing

Slicers and Dicers

Dicers Slicers

Slicers and Dicers

Usually between1 and 3 dicers,

often 2

Dicers Slicers

Slicers and Dicers

Servers

World

USD

Slicers

Slicers and Dicers

2014 2015 2016

Peter 1,000,000$ 1,500,000$ 1,400,000$

Mary 2,000,000$ 2,300,000$ 2,200,000$

Servers

World

USD

Dicers

Slicers

Products: the big three

Essbase

Cognos

Analysis Services

ETLing

OLAP: Derived data

OLAP: Derived data

ETL

ETL

ExtractTransformLoad

Extract

Triggers

Incremental updates

Extract

Triggers Gateways

Incremental updates

Extract

Triggers Gateways

Incremental updates Log extraction

Transform

Derivation

Transform

Derivation Value transformation

Herr

Mister

Transform


Herr

Mister

Cleaning

Transform


Herr

Mister

CleaningFilter, split, merge, join

Load

Integrity constraints

Load

Integrity constraints Sorting

Load


Build indices

Load


Build indices Partition

Considerations

When?

Considerations

When?Granularity

Considerations

When?Granularity

Infrastructure

Implementation

Two flavors of OLAP

ROLAP

Two flavors of OLAP

ROLAP MOLAP

Fact table (ROLAP)

Dim1 Dim2 Dim3 Dim4 Dim5 Value

Fact table (ROLAP), with multiple measures

Dim1 Dim2 Dim3 Dim4 Dim5 Mes1 Mes2 Mes3

Star Schema


Snow-flake schema


NormalizeMore

Querying

Querying cubes

Tables:SQL

Querying cubes

Tables:SQL

Cubes:MDX

Sample Cube (ROLAP)

SalesDate Product Customer Quantity2017-01-01 iPhone X Peter 12017-01-01 Samsung Galaxy S9 Peter 22017-01-01 iPhone X Mary 52017-01-01 Samsung Galaxy S9 Mary 32016-01-01 iPhone X Peter 32016-01-01 Samsung Galaxy S9 Peter 22016-01-01 iPhone X Mary 62016-01-01 Samsung Galaxy S9 Mary 32015-01-01 iPhone X Peter 32015-01-01 Samsung Galaxy S9 Peter 12015-01-01 iPhone X Mary 22015-01-01 Samsung Galaxy S9 Mary 3

Satellite tables

TimeDate Year Quarter2017-01-01 2017 12016-01-01 2016 12015-01-01 2015 1

CustomerName Country PhonePeter UK 555-55-55Mary Switzerland 555-55-56

ProductName Brand PriceiPhone X Apple 1000Samsung Galaxy S9 Samsung 800

Querying with SQL

SELECT *FROM Sales s

Querying with SQL

SELECT *FROM Sales s

Sales

Date Product Customer Quantity

2017-01-01 iPhone X Peter 1

2017-01-01 Samsung Galaxy S9 Peter 2

2017-01-01 iPhone X Mary 5

2017-01-01 Samsung Galaxy S9 Mary 3









Joining with satellite tables

SELECT t.Year, s.Product, s.Customer, s.QuantityFROM Sales s, Time tWHERE s.Date = t.Date

Joining with satellite tables


Year Product Customer Quantity

2017 iPhone X Peter 1

2017 Samsung Galaxy S9 Peter 2

2017 iPhone X Mary 5

2017 Samsung Galaxy S9 Mary 3









Slicing


AND t.Year = 2017

Slicing


AND t.Year = 2017

Year Product Customer Quantity





Slicing

SELECT t.Year, s.Product, s.Customer, s.QuantityFROM Sales s, Time t, Product pWHERE s.Date = t.Date

AND s.Product = p.NameAND t.Year = 2017AND p.Brand = 'Samsung'

Slicing

SELECT t.Year, s.Product, s.Customer, s.QuantityFROM Sales s, Time t, Product pWHERE s.Date = t.Date

AND s.Product = p.NameAND t.Year = 2017AND p.Brand = 'Samsung' Year Product Customer Quantity



Aggregating

SELECT t.Year, SUM(s.Quantity)FROM Sales s, Time t, Product p

WHERE s.Date = t.DateAND s.Product = p.NameAND p.Brand = 'Samsung'

GROUP BY t.Year

Aggregating

SELECT t.Year, SUM(s.Quantity)FROM Sales s, Time t, Product p

WHERE s.Date = t.DateAND s.Product = p.NameAND p.Brand = 'Samsung'

GROUP BY t.Year

Date Quantity

2017 5

2016 5

2015 4

Aggregating (Rolling up)

SELECT SUM(s.Quantity)FROM Sales s

Aggregating (Rolling up)


Quantity

34

Aggregating (Drilling down)

SELECT t.Year, p.Brand, SUM(s.Quantity)FROM Sales s, Time t, Product p

WHERE s.Date = t.DateAND s.Product = p.Name

GROUP BY t.Year, p.Brand

Aggregating (Drilling down)



GROUP BY t.Year, p.BrandSales

Year Brand Quantity

2017 Apple 6

2017 Samsung 5

2016 Apple 9

2016 Samsung 5

2015 Apple 5

2015 Samsung 4

Materializing all aggregates

SELECT t.Year, p.Brand, SUM(s.Quantity)FROM Sales s, Time t, Product pWHERE s.Date = t.Date

AND s.Product = p.NameGROUP BY t.Year, p.Brand

SELECT t.Year, SUM(s.Quantity)FROM Sales s, Time tWHERE s.Date = t.DateGROUP BY t.Year







Roll up






Year Brand Quantity

2017 Apple 6

2017 Samsung 5

2016 Apple 9

2016 Samsung 5

2015 Apple 5

2015 Samsung 4






Year Brand Quantity

2017 Apple 6

2017 Samsung 5

2016 Apple 9

2016 Samsung 5

2015 Apple 5

2015 Samsung 4

Year Quantity

2017 11

2016 14

2015 9






Year Brand Quantity

2017 Apple 6

2017 Samsung 5

2016 Apple 9

2016 Samsung 5

2015 Apple 5

2015 Samsung 4

Year Quantity

2017 11

2016 14

2015 9

Quantity

34




SELECT t.Year, null, SUM(s.Quantity)FROM Sales s, Time tWHERE s.Date = t.DateGROUP BY t.Year

SELECT null, null, SUM(s.Quantity)FROM Sales s

Year Brand Quantity

2017 Apple 6

2017 Samsung 5

2016 Apple 9

2016 Samsung 5

2015 Apple 5

2015 Samsung 4




SELECT t.Year, null, SUM(s.Quantity)FROM Sales s, Time tWHERE s.Date = t.DateGROUP BY t.Year

SELECT null, null, SUM(s.Quantity)FROM Sales s

Year Brand Quantity

2017 Apple 6

2017 Samsung 5

2016 Apple 9

2016 Samsung 5

2015 Apple 5

2015 Samsung 4

Year Quantity

2017 NULL 11

2016 NULL 14

2015 NULL 9

Quantity

NULL NULL 34


(SELECT t.Year, p.Brand, SUM(s.Quantity)FROM Sales s, Time t, Product pWHERE s.Date = t.Date

AND s.Product = p.NameGROUP BY t.Year, p.Brand)UNION(SELECT t.Year, null, SUM(s.Quantity)FROM Sales s, Time tWHERE s.Date = t.DateGROUP BY t.Year)UNION(SELECT null, null, SUM(s.Quantity)FROM Sales s)



AND s.Product = p.NameGROUP BY t.Year, p.Brand)UNION(SELECT t.Year, null, SUM(s.Quantity)FROM Sales s, Time tWHERE s.Date = t.DateGROUP BY t.Year)UNION(SELECT null, null, SUM(s.Quantity)FROM Sales s)

Year Brand Quantity

2017 Apple 6

2017 Samsung 5

2016 Apple 9

2016 Samsung 5

2015 Apple 5

2015 Samsung 4

2017 NULL 11

2016 NULL 14

2015 NULL 9

NULL NULL 34




GROUP BY GROUPING SETS ((t.Year, p.Brand),(t.Year),()

)

Year Brand Quantity

2017 Apple 6

2017 Samsung 5

2016 Apple 9

2016 Samsung 5

2015 Apple 5

2015 Samsung 4

2017 NULL 11

2016 NULL 14

2015 NULL 9

NULL NULL 34




GROUP BY ROLLUP (t.Year, p.Brand)

Year Brand Quantity

2017 Apple 6

2017 Samsung 5

2016 Apple 9

2016 Samsung 5

2015 Apple 5

2015 Samsung 4

2017 NULL 11

2016 NULL 14

2015 NULL 9

NULL NULL 34

Cross tabulation of a roll-up

Year BrandApple Samsung

2017 5 6 112016 9 5 142015 5 4 9Total 34





SELECT p.Brand, SUM(s.Quantity)FROM Sales s, Product pWHERE s.Product = p.NameGROUP BY p.Brand




AND s.Product = p.NameGROUP BY t.Year, p.Brand)UNION(SELECT t.Year, null, SUM(s.Quantity)FROM Sales s, Time tWHERE s.Date = t.DateGROUP BY t.Year)UNION(SELECT null, p.Brand, SUM(s.Quantity)FROM Sales s, Product pWHERE s.Product = p.NameGROUP BY p.Brand)UNION(SELECT null, null, SUM(s.Quantity)FROM Sales s)



AND s.Product = p.NameGROUP BY t.Year, p.Brand)UNION(SELECT t.Year, null, SUM(s.Quantity)FROM Sales s, Time tWHERE s.Date = t.DateGROUP BY t.Year)UNION(SELECT null, p.Brand, SUM(s.Quantity)FROM Sales s, Product pWHERE s.Product = p.NameGROUP BY p.Brand)UNION(SELECT null, null, SUM(s.Quantity)FROM Sales s)

Year Brand Quantity

2017 Apple 6

2017 Samsung 5

2016 Apple 9

2016 Samsung 5

2015 Apple 5

2015 Samsung 4

2017 NULL 11

2016 NULL 14

2015 NULL 9

NULL Apple 20

NULL Samsung 14

NULL NULL 34


(SELECT t.Year, p.Brand, SUM(s.Quantity)FROM Sales s, Time t, Product p


GROUP BY GROUPING SETS ((t.Year, p.Brand),(t.Year),(p.Brand),()

)

Year Brand Quantity

2017 Apple 6

2017 Samsung 5

2016 Apple 9

2016 Samsung 5

2015 Apple 5

2015 Samsung 4

2017 NULL 11

2016 NULL 14

2015 NULL 9

NULL Apple 20

NULL Samsung 14

NULL NULL 34


(SELECT t.Year, p.Brand, SUM(s.Quantity)FROM Sales s, Time t, Product p


GROUP BY CUBE (t.Year, p.Brand)

Year Brand Quantity

2017 Apple 6

2017 Samsung 5

2016 Apple 9

2016 Samsung 5

2015 Apple 5

2015 Samsung 4

2017 NULL 11

2016 NULL 14

2015 NULL 9

NULL Apple 20

NULL Samsung 14

NULL NULL 34

Cross tabulation (grid)


2017 5 6 112016 9 5 142015 5 4 9Total 20 14 34

Cross tabulation


2017 1 2 32016 3 2 52015 3 1 4Total 7 5 12

(Slicer: Peter only)

Cross tabulation

Customer BrandApple Samsung

Peter 1 2 3Mary 5 3 8Total 6 5 11

(Slicer: 2017)

MDX stands for...

Multi-DimensionaleXpressions

Measures

Amount of licenses

Revenues

Taxes paid

...

Dimensions

Quarter

Salesperson

Product

Country

In short...

A cube is a list of

dimensionsindexing a list of

measures

Hierarchies

Dimension values are organized in hierarchies.

[Location]

[Geo] [Economy]i.e., slice and aggregateby geographic region, etc

i.e., slice and aggregateby economic partnership, etc

Members

Members correspond to levels in a hierarchy.

[Geo][Europe] [Asia] [America]

[Switzerland][ZH][BE]

[Germany]...

[China][India]...

[Canada][USA][Brazil]...

[Africa][Ocenia]

Identifying a member

[Location].[Geo].[Europe].[Switzerland].[ZH].[Zurich]

MDX statements: dicing

SELECT[Measures].Members ON COLUMNS,[Location].[Geo].Members ON ROWS

FROM [Sales]

MDX statements: slicing

SELECT[Measures].Members ON COLUMNS,[Location].[Geo].Members ON ROWS

FROM [Sales]

WHERE [Products].[Line].[Laptops].[MBP]

Syntax

XBRL Architecture

Instance (.xml)

Schema (.xsd)

Linkbase (.xml)

Discoverable Taxonomy Set

Technologies

XML

XML SchemaXML Link

XML Names

Fact

<us-gaap:AssetscontextRef="FI2012Q4"decimals="-6"id="Fact-600212FD4D06E63B4F8F6874C6E5BE74"unitRef="usd">

86174000000</us-gaap:Assets>

Dimension ValueWhat? AssetsWho? Coca ColaWhen? Dec 31, 2011Of what? USD

Context

<xbrli:context id="FI2011Q4"><xbrli:entity>

<xbrli:identifier scheme="http://www.sec.gov/CIK">0000021344

</xbrli:identifier></xbrli:entity><xbrli:period>

<xbrli:instant>2011-12-31</xbrli:instant></xbrli:period>

</xbrli:context>December 2011

1 2 3 4

5 6 7 8 9 10 11

12 13 14 15 16 17 18

19 20 21 22 23 24 25

26 27 28 29 30 31

Unit

<xbrli:unit id="usd"><xbrli:measure>iso4217:USD</xbrli:measure>

</xbrli:unit>

Concept (XML Schema)

<xs:elementid='us-gaap_Assets'name='Assets'nillable='true'substitutionGroup='xbrli:item'type='xbrli:monetaryItemType'xbrli:balance='debit'xbrli:periodType='instant' />

Graphs

DAGs

Trees

Node: locator

<locxlink:href="http://xbrl.fasb.org/us-gaap/2013/elts/us-

gaap-2013-01-31.xsd#us-gaap_Assets"xlink:label="loc_us-

gaap_Assets_102D7A4D204ED45AC0DEDA6BBC78F386"

xlink:type="locator" />

Node: resource

<link:labelid="lab_ko_NetChangeInOperatingAssetsAndLiabilitiesDisclosureAbstract_A6469A522E35CBF355816876394722EE_label_en-US"xlink:label="lab_ko_NetChangeInOperatingAssetsAndLiabilitiesDisclosureAbstract_A6469A522E35CBF355816876394722EE"xlink:role="http://www.xbrl.org/2003/role/label"xlink:type="resource"xml:lang="en-US">NET CHANGE IN OPERATING ASSETS AND LIABILITIES

DISCLOSURE [Abstract]</link:label>

Edge

<presentationArcorder="10"preferredLabel="http://www.xbrl.org/2003/role/totalLabel"xlink:arcrole="http://www.xbrl.org/2003/arcrole/parent-child"xlink:from="loc_us-

gaap_AssetsAbstract_2F55ECB2BF7C1A62009CDA6BBC757094"xlink:to="loc_us-

gaap_Assets_102D7A4D204ED45AC0DEDA6BBC78F386"xlink:type="arc" />

Summary

Architecture

ERP

CRM

OLTP

ETL

Files

Analyze

Report

Mine

Documents

Information Systems for Engineers 9. OLAP and Data Cubes