Data Warehousing 101(and a video)

Data Warehousing 101Everything

you never wantedto know aboutbig databases

but were forcedto find out anyway

Josh BerkusOpen Source Bridge 2011

contentscovering● concepts of DW● some DW

techniques● databases

not covering● hardware● analytics/reporting

tools

BIGDATA

1970

What is a“data warehouse”?

Big Data?

OLTP vs DW ● many single-row

writes● current data● queries generated

by user activity● < 1s response

times● 1000's of users

● few large batch imports

● years of data● queries generated

by large reports● queries can run for

minutes/hours● 10's of users

OLTP vs DW

big data for many

concurrent requests to

small amounts of data each

big data

for low concurrency

requests to very large amounts of data each

synonyms&

subclasses

archiving

archivingWORN data: “write once, read never”● grows indefinitely● usually a result of regulatory compliance● main concern: storage efficiency

data mining

data miningthe database where you don't know what's in there, but you want to find out● lots of data (TB to PB)● mostly “semi-structured”● data produced as a side effect of other

business processes● needs CPU-intensive processing

BI: Business IntelligenceDSS: Decision Support OLAP: Online Analytical

ProcessingAnalytics

BI/DSS/OLAP/Analytics

BI/DSS/OLAP/Analyticsdatabases which support visualization of large amounts of data● data is fairly well understood● most data can be reduced to categories,

geography, and taxonomy● primarily about indexing

What is a“dimension”?

dimensions vs. facts

FactTable

customers / accounts

categorysubcategory

sub-subcategory

dimension examples● location/region/country/quadrant● product categorization● URL● transaction type● account heirarchy● IP address● OS/version/build

dimension synonyms

● facet● taxonomy●secondary index●view

What is ETL?

Extract, Transform, Load● how you turn external raw data into useful

database data● Apache logs → web analytics DB● CSV POS files → financial reporting DB● OLTP server → 10-year data warehouse

● also called ELT when the transformation is done inside the database

Purpose of ETL/ELTgetting data into the data warehouse

● clean up garbage data● split out attributes● “normalize” dimensional data● deduplication● calculate materialized views / indexes

ETL Tools

K.E.T.T.L.E.

ETL Tools

Ad-hoc scripting

ELT Tipsthink volume

● bulk processing or parallel processing● no row-at-a-time, document-at-a-time

● insert into permanent storage should be the last step● no updates

Queues not Extract

What kind of database should I

use for DW?

5 Types

1. Standard Relational

2. MPP

3. Column Store

4. Map/Reduce

5. Enterprise Search

` `

standard relational

standard relationalthe all-purpose solution for not-that-big data● adequate for all tasks

● but not excellent at any of them● easy to use

● low resource requirements● well-supported by all software● familiar

● not suitable for really big data

MySQL

PostgreSQL

DW Database

0 5 10 15 20 25 30

0 5 10 15 20 25 30

Sweet Spots

What's MPP?

MassivelyParallelProcessing

appliance software

MPPcpu-intensive data warehousing

● data mining, some analytics● supporting complex query logic● moderately big data (1-200TB)● drawbacks: proprietary, expensive● now hybridizes

● with other types

What's acolumn store?

column store

column store

inversion of a row store:indexes become datadata becomes indexes

column stores

column storesfor aggregations and transformations of highly structured data● good for BI, analytics, some archiving● moderately big data (0.5-100TB)● bad for data mining● slow to add new data / purge data● usually support compression

What's map/reduce?

map/reduce

map/reduce

map/reduce// mapfunction(doc) { for (var i in doc.links) emit([doc.parent, i], null); }}// reducefunction(keys, values) { return null;}

map/reduce// Mapfunction (doc) { emit(doc.val, doc.val)}

// Reducefunction (keys, values, rereduce) { // This computes the standard deviation of the mapped results var stdDeviation=0.0; var count=0; var total=0.0; var sqrTotal=0.0;

if (!rereduce) { // This is the reduce phase, we are reducing over emitted values from // the map functions. for(var i in values) { total = total + values[i]; sqrTotal = sqrTotal + (values[i] * values[i]); } count = values.length; } else { // This is the rereduce phase, we are re-reducing previosuly // reduced values. for(var i in values) { count = count + values[i].count; total = total + values[i].total; sqrTotal = sqrTotal + values[i].sqrTotal; } }

var variance = (sqrTotal - ((total * total)/count)) / count; stdDeviation = Math.sqrt(variance);

// the reduce result. It contains enough information to be rereduced // with other reduce results. return {"stdDeviation":stdDeviation,"count":count, "total":total,"sqrTotal":sqrTotal};};

map/reduce vs. MPP● open source● petabytes● write routines by

hand● inefficient● generic● cheap HW / cloud● DIY tools

● proprietary● terabytes● advanced query

support● efficient● specific● needs good HW● integrated tools

What's enterprise search?

enterprise search

ElasticSearch

enterprise searchwhen you need to do DW with a huge pile of partly processed “documents”● does: light data mining, light BI/analytics

● best “full text” and keyword search● supports “approximate results”● lots of special features for web data

E.S. vs. C-Store● batch load● semi-structured

data● uncompressed● star schema● sharding● approximate results

● batch load● fully normalized

data● compressed● snowflake schema● parallel query● exact results

What's awindowing query?

regular aggregate

windowing function

TABLE events (event_id INT,event_type TEXT,start TIMESTAMPTZ,duration INTERVAL,event_desc TEXT

);

SELECT MAX(concurrent)FROM (SELECT SUM(tally) OVER (ORDER BY start)AS concurrent

FROM (SELECT start, 1::INT as tally

FROM events UNION ALL SELECT (start + duration), -1 FROM events ) AS event_vert) AS ec;

stream processing SQL● replace multiple queries with a single

query● avoid scanning large tables multiple times

● replace pages of application code● and MB of data transmission

● SQL alternative to map/reduce● (for some data mining tasks)

What's a materialized view?

query results as table● calculate once, read many time

● complex/expensive queries● frequently referenced

● not necessarily a whole query● often part of a query

● might be manually or automatically updated● depends on product

non-relational matviews● CouchDB Views

● cache results of map/reduce jobs● updated on data read

● Solr / Elastic Search “Faceted Search”● cached indexed results of complex searches● updated on data change

maintaining matviewsBEST: update matviews

at batch load time

GOOD: update matview accordingto clock/calendar

FAIR: update matview on data request

BAD for DW: update matviewsusing a trigger

matview tips● matviews should be small

● 1/10 to ¼ of RAM on each node● each matview should support several

queries● or one really really important one

● truncate + append, don't update● index matviews like crazy

● if they are not indexes themselves

What's OLAP?

cubesSite

Repeat V

isitors

Browse

r

drill-down

OLAP● OnLine Analytical Processing● Visualization technique

● all data as a multi-dimensional space● great for decision support

● CPU & RAM intensive● hard to do on really big data

● Works well with column stores

Contact● Josh Berkus: josh@pgexperts.com

● blog: blogs.ittoolbox.com/database/soup● twitter: @fuzzychef

● PostgreSQL: www.postgresql.org● pgexperts: www.pgexperts.com

This talk is copyright 2011 Josh Berkus and is licensed under the Creative Commons Attribution license. Many images were taken from google images and are copyright their original creators, whom I don't actually know. Logos are trademark their respective owners, and are used here under fair use.