I H ADOOP T D EMISE OF D ATA W AREHOUSING - Senturus · 2016-08-30 · OF D ATA W AREHOUSING? T HOUGHTS ON THE IMPACT OF H ADOOP ON BI S YSTEMS AND D ATA W AREHOUSING ... ± New wave

IS HADOOP THE DEMISE OF DATA WAREHOUSING?

THOUGHTS ON THE IMPACT OF HADOOP ON BI SYSTEMS AND DATA WAREHOUSING

Part of our BI Demystified

Series

John Peterson

CEO & Co-Founder

Senturus

Today’s Presenter

2

With thanks to:

Guy Wilnai, Sujee Maniyam and Knowledge @ Senturus

• INTRODUCTION

• THE DATA CHALLENGE

• WHAT IS HADOOP?

• ADVANTAGES & CHALLENGES

• IMPLICATIONS, PREDICTIONS & MISC. MUSINGS

• CONCLUSIONS

• Q&A

AGENDA

3 Copyright 2014 Senturus, Inc. All Rights Reserved

PRESENTATION SLIDE DECK ON WWW.SENTURUS.COM

Copyright 2014 Senturus, Inc. All Rights Reserved 4

WHO WE ARE

SENTURUS INTRODUCTION

Our Team:

Business depth combined with technical expertise. Former CFOs, CIOs, Controllers, Directors, BI Managers

SENTURUS: BUSINESS ANALYTICS CONSULTANTS


Business Intelligence Enterprise Planning Predictive Analytics

Creating Clarity from Chaos

• Former Head of BI/ Lead Architect – VISA

• Former Chief BI Architect – Jamba Juice

• Former Head of BI – Dole

• Former Chief BI Architect – Cisco

• Former Chief BI Architect – Central Garden & Pet

• Former Head of BI – Experian

• Former Head of BI – Robert Half International

• Former Head of Training (IBM Cognos, Southern California)

• Former Controller – The GAP

• Two former CFO’s

• Former Partner - PWC ($50million+ projects)

• Several former Vice Presidents of Marketing, Sales & Manufacturing/Supply Chain

• Several former COO’s

• Several former CIO’s

• Average experience = over 20 years

A FEW OF OUR TEAM MEMBERS (FORMER ROLES) Deep & Pragmatic Experience

Copyright 2014 Senturus, Inc. All Rights Reserved. 7

750+ CLIENTS, 1600+ PROJECTS, 13+ YEARS

Copyright 2014 Senturus, Inc. All Rights Reserved. 8

Outpacing our ability to harness it

THE DATA CHALLENGE

THE CHALLENGES (AND OPPORTUNITIES)

10 Copyright 2014 Senturus, Inc. All Rights Reserved.

• Data volumes & velocity increasing exponentially

• Data types proliferating

• Rapid emergence of less structured (or unstructured) data sources

• Value of Data increasing

• Traditional ETL is time-consuming and costly

• Traditional storage costs skyrocketing (not $/TB)

• Business users increasingly frustrated at not being able to get access to information

THE NET RESULT


Something is bound to

happen

A WARNING ABOUT TODAY’S FOCUS


IS ABOUT:

Hadoop as a potential platform or tool for Business Analytics & DW

IS NOT ABOUT:

Yet another “How Big Data will change the world” paradigm-shift prediction

ROLE OF HADOOP IN YOUR ENVIRONMENT

QUICK POLL

Under the Covers

WHAT IS HADOOP?

WHAT IS HADOOP?


Hadoop is a stuffed elephant

WHAT IS HADOOP REALLY?


Database Tables

• Hadoop is an open source distributed storage and processing framework

• Hadoop vs. RDBMS

System Tables

SQL Query Engine

Typical RDBMS

HDFS Files*

Hcatalog & YARN

Multiple Engines

Hadoop Stack

Storage

Metadata

Queries

*Raw data

to highly

structured

All layers combined in a

proprietary bundle

All layers separate and

independent allowing flexible access

REFERENCE ARCHITECTURE

17 Copyright 2014 Senturus, Inc. All Rights Reserved. Source: Hortonworks

REFERENCE ARCHITECTURE (DETAILED)

18 Copyright 2014 Senturus, Inc. All Rights Reserved. Source: Hortonworks

HADOOP STACK DISTRIBUTIONS


Distribution Open Source Premium

Apache Y N

Cloudera Y Y

Horton Works Y N

MapR Y (?) Y

Intel N Y

EMC Greenplum HD N Y

ADVANTAGES OF HADOOP (FOR BI)


• Dramatically lower cost

– 50x to 100x (or more)

• Can store virtually any data type

• Can support multiple analytic engines

• Massively scalable

– Both Size and Performance

– 100’s of nodes, TB of RAM, PB of storage

• Open-source leads to rapid innovation

HADOOP OFFERS COST EFFECTIVE STORAGE

“A recent survey of large financial services firms,

telecommunications carriers and retailers indicated that

storing data in an RDBMS typically runs between $30,000

and $100,000 (USD) per TB per year in total costs”

--- Cloudera white paper

- Hadoop can bring down the cost to ~$1,000 / TB

BIG DATA COST COMPARISON

Source : Neustar

BIG DATA COST COMPARISON

Source: HortonWorks

COST CASE STUDY (TELECOM)

• The carrier’s previous data processing environment was costing $59 million (USD) each year to manage 1PB of data, broken down as follows:

– $2 million (USD) per year = storage for 1PB raw archive data on network-attached storage (NAS) at $2,000 per TB per year

– $55 million (USD) per year = management and backup of 1PB processed data on EDW at $55,000 per TB per year

– $2 million (USD) per year = administration costs calculated at $1,000 per TB per year

• Calculating costs for moving data processing onto Cloudera, the carrier

reduced infrastructure costs to $5.1 million (USD) total

– $5 million (USD) per year = hardware, software and infrastructure for 1PB at $5,000 per TB per year

– $100,000 (USD) per year = administration costs calculated at $100 per TB per year

HADOOP CAN STORE ANY DATA TYPE

• Key-value pairs

• Text and binary data

• Structured

– Database records

• Semi-structured

– Sensor & Machine data

– Log files

• Un-structured

– Emails, tweets

“Set structure at query time”

Can retain

atomic level

data

ANALYTICS IN HADOOP

• ‘Batch’ or ‘offline’ analytics

– MapReduce based tools (java mapreduce, streaming, pig, hive)

– Have been there from the start, Well understood

• Fast Ad-Hoc querying

– New wave of processing, answer to MPP databases (Teradata .etc)

– Impala (Cloudera), stinger / Tez (Hortonworks), Shark on Spark (Apache)

• Streaming / Near-RealTime workloads

– Storm, Spark

– Propelled by YARN processing framework in Hadoop version 2.x

ANALYTICS IN HADOOP (CONT.)

• BI Tools integration – Rich BI tool integration

– Various levels of integration (basic, native, high-speed)

– Lots of vendors : Datameer, Pentaho, Tableau, QlikView, IBM Cognos…

• NOSQL store – Find data very quickly (milliseconds, just like a traditional database)

– Hbase

• Statistical Tools – R

• And, of course, the old favorite – SQL

– Example: InfiniDB (Calpont)

CHALLENGES OF HADOOP


• Everything is very NEW

• Playing field is changing DAILY

– The Wild West

• Tools still in v1.0 mode (at best)

• Does not eliminate the need for dimensional modeling

• Security TBD

• No “standard” (winners) declared yet

• Lots of rough edges still

• Simple things, like surrogate keys…

A DIZZYING FIELD OF PLAYERS • Alpine Data Labs, San Mateo, CA.

• Cloudera, Palo Alto, CA.

• Concurrent, San Francisco, CA.

• Continuum Analytics, Austin, TX.

• Continuuity, Palo Alto, CA.

• Couchbase, Mountain View, CA.

• Datameer, San Mateo, CA.

• DataSift, San Francisco, CA.

• DataStax, San Francisco, CA.

• DataXu, Boston, MA.

• Enigma, New York, NY.

• Factual, Los Angeles, CA.

• GoodData, San Francisco, CA.

• Gravity, New York, NY.

• Guavus, San Mateo, CA.

• Hadapt, Cambridge, MA

• Hopper, Cambridge, MA.

• Hortonworks, Palo Alto, CA.

• KarmaSphere, Cupertino, CA

• Lattice Engines, San Mateo, CA.

• MapR Technologies, San Jose, CA.

• MemSQL, New York, NY.

• Mortar Data, New York, NY.

• Mu Sigma, Northbrook, IL + India.

• Neo Technology, San Mateo, CA

• Opera Solutions, San Diego, CA + India.

• ParAccel, Campbell, CA.

• Pivotal Software, Palo Alto, CA

• Platfora:, San Mateo, CA.

• RainStor, San Francisco, CA.

• Rocket Fuel, Redwood City, CA.

• SiSense, Redwood Shores, CA and Israel.

• Skytree, Atlanta, GA.

• Splice Machine, San Francisco, CA.

• Splunk, San Francisco, CA

• Statwing, San Francisco, CA.

• SumAll, New York, NY.

• Talend, Los Altos, CA.

• WibiData, San Francisco, CA.

• Zettaset, Mountain View, CA

• Zoomdata, Reston, VA.

• 10gen, New York, NY

• 1010data, New York, NY.

29 Copyright 2014 Senturus, Inc. All Rights Reserved. Partial snapshop as of May 2014

IMPLICATIONS, PREDICTIONS & MISC. MUSINGS

TSUNAMI WARNING

IMPLICATIONS, PREDICTIONS & MUSINGS


• Hadoop as a Data Staging environment

• Hadoop as an Archive

• Hadoop as the Data Warehouse

– “Enterprise Data Hub”

• Future role of RDBMS’s ?? – For OLTP

– For Data Warehouse

• How much Transformation and where?

TYPICAL “BEST PRACTICES” BI ARCHITECTURE INTEGRATED BUSINESS PROCESS DIMENSIONAL MODELS WITH METADATA LAYER(S)


ERP Data

CRM Data

Data

Inte

grati

on

Conforming Business Process

Dimensional Models

Standard Reports

Web P

ort

al

Other Sources

Information Security

Data Warehouse

Data

Abst

ract

ion M

odel

Ad h

oc

Query

ing Planning Data

Slic

ing

&

Dic

ing

Dash

board

Auth

ori

ng

Report

Auth

ori

ng

Dashboards/ Scorecards

Sourc

e S

yste

ms

of

Reco

rd

Thre

shold

Ale

rtin

g

Self-service Reporting & Analysis

Single Version of the Truth

Threshold-based Alerts

POTENTIAL BI ARCHITECTURE USING HADOOP INTEGRATED BUSINESS PROCESS DIMENSIONAL MODELS WITH METADATA LAYER(S)


ERP Data

CRM Data

Data

Inte

grati

on

Conforming Business Process

Dimensional Models

Standard Reports

Web P

ort

al

Other Sources Information Security

Data Warehouse

Data

Abst

ract

ion M

odel

Ad h

oc

Query

ing Planning Data

Slic

ing

&

Dic

ing

Dash

board

Auth

ori

ng

Report

Auth

ori

ng

Dashboards/ Scorecards

Sourc

e S

yste

ms

of

Reco

rd

Thre

shold

Ale

rtin

g

Self-service Reporting & Analysis

Single Version of the Truth

Threshold-based Alerts

Hadoop Data Staging

IMPLICATIONS, PREDICTIONS & MUSINGS (CONT.)


• What have I got to learn?

– MapReduce = No

– Hand-coding = No

– Scoop = Maybe

– SQL = YES

• Role of Existing Tools going forward

– ETL

– BI Front-ends

• Role of DW Appliances?

– HANA

– IBM PureData System (formerly Netezza), etc.

IMPLICATIONS, PREDICTIONS & MUSINGS (CONT.)


• What is the impact on end-users seeking information?

• We still need:

– Data delivered in business user-friendly state

– Rich, relevant and conforming dimensions

– Ability to account for dimension changes over time

– Good performance (transformation and aggregation)

– Ability to integrate with existing systems

JP’S CONCLUSION #1


Wow, this stuff is a BIG game changer



It’s too early to call on the specifics



DW Architectures & Technologies

are in a huge state of flux

But…

DW Principles still apply

Resources, Upcoming Events, Q&A

NEED MORE INFO?

• Cloudera & Ralph Kimball – Best Practices for the Hadoop Data Warehouse: EDW 101 for Hadoop

Professionals – http://www.cloudera.com/content/cloudera/en/resources/library/recordedwebinar/

best-practices-for-the-hadoop-data-warehouse-video.html

– Building a Hadoop Data Warehouse: Hadoop 101 for EDW Professionals – http://www.cloudera.com/content/cloudera/en/resources/library/recordedwebinar/

building-a-hadoop-data-warehouse-video.html

• MapR & Jack Norris – How (and Why) Hadoop is Changing the Data Warehousing Paradigm

– http://tdwi.org/articles/2013/08/13/hadoop-changing-dw-paradigm.aspx

• HortonWorks – http://hortonworks.com/hadoop/

• Senturus.com – http://senturus.com/resources/

– [email protected] or [email protected]

ADDITIONAL RESOURCES


Contact us for

help on a POC

http://www.cloudera.com/content/cloudera/en/resources/library/recordedwebinar/best-practices-for-the-hadoop-data-warehouse-video.html
















http://www.cloudera.com/content/cloudera/en/resources/library/recordedwebinar/building-a-hadoop-data-warehouse-video.html












http://tdwi.org/articles/2013/08/13/hadoop-changing-dw-paradigm.aspx







http://hortonworks.com/hadoop/

http://senturus.com/resources/

mailto:[email protected]

www.senturus.com

UPCOMING EVENTS



More info….

Q & A

Helping Companies Learn From the Past, Manage the Present and Shape

the Future

www.senturus.com 888-601-6010 [email protected]

Thank You

Copyright 2014 by Senturus, Inc. This entire presentation is copyrighted and may not be reused or

distributed without the written consent of Senturus, Inc.

Documents

I H ADOOP T D EMISE OF D ATA W AREHOUSING - Senturus · 2016-08-30 · OF D ATA W AREHOUSING? T HOUGHTS ON THE IMPACT OF H ADOOP ON BI S YSTEMS AND D ATA W AREHOUSING ... ± New wave