Hadoop Summit Tokyo HDP Sandbox Workshop

HDPSandboxWorkshopRafaelCossCommunityTeamDataEvangelist@racossrafael@hortonworks.com

2 ©HortonworksInc.2011– 2016.AllRightsReserved

AgendaFutureofData

TraditionalDataArchitectures

What’sApacheHadoop?

DataAccesswithHadoop

LabIntro

3 ©HortonworksInc.2011– 2016.AllRightsReserved3 ©HortonworksInc.2011– 2016.AllRightsReserved

PaymentTracking

DueDiligence

SocialMapping

ProductDesign M&ACall

AnalysisMachineData

DefectDetecting

FactoryYields

CustomerSupport

BasketAnalysis Segments

CustomerRetention

SentimentAnalysis

OptimizeInventories

SupplyChain

Cross-Sell

VendorScorecards

AdPlacement

OPEXReduction

HistoricalRecords

MainframeOffloads

DeviceDataIngest

Rapid Reporting

DigitalProtection

Dataasa

Service

FraudPrevention

PublicData

Capture

INNOVATE

RENOVATE

EXP LORE OPT IM I Z E TRANS FORM

ACT IVEARCH IVE

ETLONBOARD

DATAENR ICHMENT

DATAD ISCOVERY

S INGLEV IEW

CyberSecurity

DisasterMitigation

InvestmentPlanning

AdPlacement

RiskModeling

ProactiveRepair

InventoryPredictions

NextProductRecs

PRED ICT IVEANALYT ICSCustomers are building Modern Data

Applications to transform their industries, renovating their IT architectures and innovating their Data in Motion and Data at Rest platforms to power actionable intelligence.

PaymentTracking

DueDiligence

SocialMapping

AnalysisMachineData

DefectDetecting

FactoryYields

CustomerSupport

CustomerRetention

SentimentAnalysis

OptimizeInventories

SupplyChain

Cross-Sell

VendorScorecards

AdPlacement

OPEXReduction

HistoricalRecords

MainframeOffloads

DeviceDataIngest

Rapid Reporting

DigitalProtection

Dataasa

Service

FraudPrevention

PublicData

Capture

INNOVATE

RENOVATE

ACT IVEARCH IVE

ETLONBOARD

DATAENR ICHMENT

DATAD ISCOVERY

S INGLEV IEW

CyberSecurity

DisasterMitigation

InvestmentPlanning

AdPlacement

RiskModeling

ProactiveRepair

NextProductRecs

PRED ICT IVEANALYT ICS

PaymentTracking

DueDiligence

SocialMapping

AnalysisMachineData

DefectDetecting

FactoryYields

CustomerSupport

CustomerRetention

SentimentAnalysis

OptimizeInventories

SupplyChain

Cross-Sell

VendorScorecards

AdPlacement

OPEXReduction

HistoricalRecords

MainframeOffloads

DeviceDataIngest

Rapid Reporting

DigitalProtection

Dataasa

Service

FraudPrevention

PublicData

Capture

INNOVATE

RENOVATE

ACT IVEARCH IVE

ETLONBOARD

DATAENR ICHMENT

DATAD ISCOVERY

S INGLEV IEW

CyberSecurity

DisasterMitigation

InvestmentPlanning

AdPlacement

RiskModeling

ProactiveRepair

NextProductRecs

Mustdotomakemoderndataapplicationspossible

PaymentTracking

DueDiligence

SocialMapping

AnalysisMachineData

DefectDetecting

FactoryYields

CustomerSupport

CustomerRetention

SentimentAnalysis

OptimizeInventories

SupplyChain

Cross-Sell

VendorScorecards

AdPlacement

OPEXReduction

HistoricalRecords

MainframeOffloads

DeviceDataIngest

Rapid Reporting

DigitalProtection

Dataasa

Service

FraudPrevention

PublicData

Capture

INNOVATE

RENOVATE

ACT IVEARCH IVE

ETLONBOARD

DATAENR ICHMENT

DATAD ISCOVERY

S INGLEV IEW

CyberSecurity

DisasterMitigation

InvestmentPlanning

AdPlacement

RiskModeling

ProactiveRepair

NextProductRecs

Powerfulmeanstooptimize currentbusiness

PaymentTracking

DueDiligence

SocialMapping

AnalysisMachineData

DefectDetecting

FactoryYields

CustomerSupport

CustomerRetention

SentimentAnalysis

OptimizeInventories

SupplyChain

Cross-Sell

VendorScorecards

AdPlacement

OPEXReduction

HistoricalRecords

MainframeOffloads

DeviceDataIngest

Rapid Reporting

DigitalProtection

Dataasa

Service

FraudPrevention

PublicData

Capture

INNOVATE

RENOVATE

ACT IVEARCH IVE

ETLONBOARD

DATAENR ICHMENT

DATAD ISCOVERY

S INGLEV IEW

CyberSecurity

DisasterMitigation

InvestmentPlanning

AdPlacement

RiskModeling

ProactiveRepair

NextProductRecs

Powerfulmeanstooptimize currentbusiness

Pathwaytotransform forstrategicadvantageandnewrevenuestreams

FutureofData

INTERNETOF

ANYTHING

TheFutureofDataisaboutactionableintelligencederivedfromaconstantlyconnectedsociety,withinteractivesecureaccesstorichdatasetscomingfromtheInternetofAnything

TheFutureofDataActionableIntelligence

D ATA I N M O T I O N

STORAG

ESTORA

GROUP2GROUP1

GROUP4GROUP3

D ATA AT R E S T

INTERNETOF

ANYTHING

ConnectedDataArchitectureAcrossyourDataPlane

ispoweringActionableIntelligence

Anyandalldatafromsensors,machines,

geolocation,clicks,files,social.

Securepoint-to-pointandbi-directionaldataflows

Collectandcuratealldata.

DataPowersHighwaySafety

ConnectedDataDrivestheConnectedCarInsurancePremiums

Warranties

RecallsPricingModels

DesignInnovation

AutonomousDriving

ConnectedCity Infotainment

Sensors

ScheduledMaintenance

PredictiveMaintenance

RouteOptimization

INSURANCECOMPANIES

GOVERNMENTAGENCIES

INFOTAINMENTPROVIDERS

SOFTWARECOMPANIES

AUTOMAKERS

NewDataParadigmOpensUpNewOpportunity

2.8zettabytesin2012

44zettabytesin2020

1 zettabyte (ZB) = 1 million petabytes (PB); Sources: IDC, IDG Enterprise, and AMR Research

Clickstream

ERP,CRM,SCM

Web&social

Geolocation

InternetofThings

Serverlogs

Files,emails

Transformeveryindustryviafullfidelityofdataandanalytics

Opportunity

T R A D I T I O N A L

LAGGARDS

LEADERS

AbilitytoConsumeData

EnterpriseBlindSpot

Whatdisruptedthedatacenter?

TraditionalDataArchitectures

SystemsofIntelligence

SystemsofEngagements

SystemsofInteractions

DataSystems

SystemsofRecord

EventsInGray

ActionableIntelligence

OperatorsDevelopers

Products

AnalyticsIn

SystemsofInsight

ModernDataApplicationsDataScopeNext Generation Analytics

Iterative & ExploratoryData is the structure

Traditional AnalyticsStructured & Repeatable

Structure built to store data

Capacity constrained down sampling of available information Whole population analytics connects the dots

Carefully cleanse all information before any analysis Analyze information as is & cleanse as needed

Next Generation AnalyticsIterative & ExploratoryData is the structure

Traditional AnalyticsStructured & Repeatable

Structure built to store data

ModernDataApplicationsapproachtoInsights

Start with hypothesisTest against selected data

Data leads the way Explore all data, identify correlations

Analyze after landing… Analyze in motion…

Next Generation AnalyticsData leads the way

Explore all data, identify correlations

Traditional AnalyticsStart with hypothesis

Test against selected data

AgileAnalyticsenhancedbyHadoop

1. Requirements2. DataSourcing3. ETL4. AppDev5. Delivery

1. Landrawdata2. DataScience/ML3. DataSourcing4. Requirements5. AppDev6. LandMoreData7. Iterate

Data as an independent business process(Silo’s of data) Data as a by product of Extended

Enterprise View

Unstructured

Visualization&Dashboards

BusinessAnalytics

DataMarts

DataMarts Archive

StatisticsOLAP

FileServer

ClickstreamLogs

Web&SocialLogs

AudioVideo

LogsLogs

Geolocation

POS CRM ERP

Filter

AppServer

MessageBus

Documents

Unstructured

BusinessAnalytics

DataMarts

DataMarts Archive

StatisticsOLAP

FileServer

ClickstreamLogs

Web&SocialLogs

AudioVideo

LogsLogs

Geolocation

POS CRM ERP

Filter

AppServer

MessageBus

Documents

Ã Tooexpensiveandslowasdatagrowthkeepsaccelerating

Ã Tooslowtogetthedatapreparedforanalytics

Ã Analyticsisonlyleveragingalimiteddataset

Ã Colddatabecomesarchivedandisnolongerusableforanalytics

Ã DataingestisrigidandslowfornewIoAT datatypes

Ã Limitedrealtimeinsights

TraditionalDataArchitectureChallengeswithBigData

Unstructured

BusinessAnalytics

DataMarts

DataMarts Archive

StatisticsOLAP

FileServer

ClickstreamLogs

Web&SocialLogs

AudioVideo

LogsLogs

Geolocation

POS CRM ERP

Filter

AppServer

MessageBus

Documents

What’sApacheHadoop?

HadoopArchitecture

DataAccessEngines

DistributedReliableStorage

DistributedComputeFrameworkResourceManagement,DataLocalityDataOperatingSystem

Batch Interactive Real-time

Governance&

IntegrationSecurity

Applications

DeployAnywhere

25 ©HortonworksInc.2011– 2016.AllRightsReserved runson

RDBMSImport/Export

DistributedStorage&ProcessingFramework

SecureNoSQL DB

SQLonHBase

NoSQL DB

WorkflowManagement

StreamingDataIngestion

ClusterSystemOperations

SecureGateway

DistributedRegistry

Search&Indexing

EvenFasterDataProcessing

DataManagement

MachineLearning

HadoopEcosystem

OpenEnterpriseHadoopCapabilities

YARN : Data Operating System

DATA ACCESS SECURITYGOVERNANCE & INTEGRATION OPERATIONS

1 ° ° ° ° ° ° ° ° °

° ° ° ° ° ° ° ° ° °

Data Lifecycle & Governance

FalconAtlas

AdministrationAuthenticationAuthorizationAuditingData Protection

RangerKnoxAtlasHDFSEncryptionData Workflow

SqoopFlumeKafkaNFSWebHDFS

Provisioning, Managing, & Monitoring

AmbariCloudbreakZookeeper

Scheduling

MapReduce

Script

Search

HBaseAccumuloPhoenix

Stream

In-memory

Others

ISV Engines

Tez Tez Slider Slider

DATA MANAGEMENT

HortonworksDataPlatform

DeploymentChoiceLinux Windows On-Premise Cloud

HDFS Hadoop Distributed File System

HORTONWORKSDATAPLATFORM

DATAMGMT

HDP2.2Dec2014

HDP2.1April2014

HDP2.0Oct2013

HDP2.2Dec2014

HDP2.1April2014

HDP2.0Oct2013

OngoingInnovationinApache

HDFSYARNMapReduceHadoopCore

WhatisApacheHadoop?

Yahoo!2006

HortonworksOct2011

Yahoo!startfocusonmultipleHadoopapps&clustersContributesHadooptoApache

HDP1.0Oct2012

ApacheHadoopv2YARN

GooglepublishesGFS&MapReduce papers2004-2005

HDP2.4March2016 2.7.1

HDP2.2Dec2014HDP2.3July2015 2.7.1

HDP 2.5Aug2016

Apache Hadoop = Storage + Compute

storage storage

HadoopDistributedFileSystem(HDFS)

CPU RAM

YetAnotherResourceNegotiator(YARN)

+ /directory/structure/in/memory.txt

Resource management + schedulingDisk, CPU, Memory

CoreNameNode

ResourceManagerYARN

Hadoop daemon

User application

DataNodeHDFS

NodeManagerYARN

Worker Node

Hadoop Distributed File System (HDFS)

Fault Tolerant Distributed Storage• Dividefilesintobigblocksanddistribute3copiesrandomly acrossthecluster• ProcessingDataLocality

• NotJuststoragebutcomputation

10110100101001001110011111100101001110100101001011001001010100110001010010111010111010111101101101010110100101010010101010101110010011010111010

Logical File

Blocks

Cluster

HDFSStorageArchitecture- Before

Before• DataNode isasinglestorage• Storageisuniform- OnlystoragetypeDisk

• Storagetypeshiddenfromthefilesystem

Alldisksassinglestorage

CloudStorage

HDFSStorageArchitecture- Now

New Architecture• DataNode isacollectionofstorages• Supportdifferenttypesofstorages

– Disk,SSDs,Memory

Block Storage Policies– DescribeshowtostoredatablocksinHDFS

Collectionoftierstorage

It Looks Like a File System

Batch Processing in HadoopMapReduceBatch Access to DataOriginal data access mechanism for Hadoop

• FrameworkMadefordevelopingdistributedapplicationstoprocessvastamountsofdatain-parallelonlargeclusters

• ProvenReliableinterfacetoHadoopwhichworksfromGBtoPB.But,batchoriented– Speedisnotit’sstrongpoint.

• EcosystemPortedtoHadoop2torunonYARN.SupportsoriginalinvestmentsinHadoopbycustomersandpartnerecosystem.

DataNode1

Mapper

Dataisshuffledacrossthenetwork

&sorted

MapPhase Shuffle/Sort ReducePhase

MapReduce JobLifecycle

SayingthatMapReduceisdeadispreposterous- Wouldlimitsustoonlynewworkloads- ALLHadoop clustersusemapreduce

- ProvenatEnterpriseScale

DataNode2

Mapper

DataNode3

Mapper

DataNode1

Reducer

DataNode2

Reducer

DataNode3

Reducer

YARN:DataOperatingSystem

Interactive Real-TimeBatch

What is MapReduce?Break a large problem into sub-solutionsMap

• Iterate over a large # of records

• Extract something of interest fromeach record

Shuffle

• Sort Intermediate results

Reduce

• Aggregate, summarize, filter or transform intermediate results

• Generate final output

MapProcess

DataData

DataData MapProcess

ReduceProcess

Read&ETL

Shuffle&Sort Aggregation

DataData

1st GenHadoop:CostEffectiveBatchatScale

HADOOP1.0BuiltforWeb-ScaleBatchApps

SingleAppBATCH

SingleAppINTERACTIVE

SingleAppBATCH

SiloscreatedfordistinctusecasesSingleApp

SingleAppONLINE

HadoopemergedasfoundationofnewdataarchitectureApacheHadoopisanopensourcedataplatformformanaginglargevolumesofhighvelocityandvarietyofdata

• BuiltbyYahoo!tobetheheartbeatofitsad&searchbusiness

• DonatedtoApacheSoftwareFoundationin2005withrapidadoptionbylargewebproperties&earlyadopterenterprises

• Incrediblydisruptivetocurrentplatformeconomics

TraditionalHadoopAdvantages

ü Managesnewdataparadigmü Handlesdataatscaleü Costeffectiveü Opensource

TraditionalHadoopHadLimitations

Batch-onlyarchitectureSinglepurposeclusters,specificdatasetsDifficulttointegratewithexistinginvestmentsNotenterprise-grade

Application

StorageHDFS

Batch ProcessingMapReduce

YARNextendsHadoopintodatacenterleaders

YARNThe Architectural Center of Hadoop

• Common data platform, many applications

• Support multi-tenant access & processing

• Batch, interactive & real-time use cases

• Supports 3rd-party ISV tools

(ex. SAS, Syncsort, Actian, etc.)

YARN Ready Applications Facilitates ongoing innovation and enterprise adoption via ecosystem of new and existing “YARN Ready” solutions

YARN : Data Operating System

BATCH, INTERACTIVE & REAL-TIME DATA ACCESS

1 ° ° ° ° ° ° ° ° °

° ° ° ° ° ° ° ° ° °

DATA MANAGEMENT

MapReduce

Script

Search

HBaseAccumuloPhoenix

Stream

In-memory

Others

ISV Engines

Tez Tez Slider Slider

WhatdoesiOS 6andWindows3.1haveincommon?

HadoopBeyondBatchwithYARN

SingleUseSysztemBatchApps

MultiUseDataPlatformBatch,Interactive,Online,Streaming,…

Ashiftfromtheoldtothenew…

HADOOP 1

MapReduce(cluster resource management

& data processing)

Data FlowPig

SQLHive

Others

API,Engine,

andSystem

YARN(Data Operating System: resource management, etc.)

Data FlowPig

SQLHive

OtherISV

Apache Yarn as a Base

System

Engine

API’s

1 ° ° ° ° °

° ° ° ° ° N

HDFS (redundant, reliable storage)

1 ° ° ° ° ° ° ° ° °

° ° ° ° ° ° ° ° ° N

BatchMapReduce

Tez Tez

MapReduce as the BaseHADOOP 2

ArchitectureEnabledbyYARNAsinglesetofdataacrosstheentireclusterwithmultipleaccessmethodsusing“zones”forprocessing

1 ° ° ° ° ° ° °

° ° ° ° ° ° ° °

° ° ° ° ° ° ° n

SQLHive

InteractiveSQLQueryforAnalytics

Script-basedETLAlgorithmexecutedinbatchtoreworkdatausedbyHiveandHBaseconsumers

• Maximize compute resources to lower TCO

• No standalone, silo’d clusters

• Simple management & operations

…all enabled by YARN

StreamProcessingStorm

Identify&actonreal-timeevents

NoSQLHbase

Accumulo

Low-latencyaccessservingupawebfrontend

HadoopWorkloadEvolution

SingleUseSystemBatchApps

MultiUseDataPlatformBatch,Interactive,Online,Streaming,…

Ashiftfromtheoldtothenew… MultiUsePlatformData&Beyond

HADOOP 1

HADOOP 2

1 ° ° ° °

° ° ° ° N

1 ° ° °

° ° ° N

MapReduce

HADOOP.Next

YARN ‘

1 ° ° ° ° ° °

° ° ° ° ° ° N

DATA ACCESS APPS

Docker

MySQLMR2 Others(ISV Engines)

Multiple(Script, SQL, NoSQL, …)

MR2 Others(ISV Engines)

Multiple(Script, SQL, NoSQL, …)

Docker

Tomcat

Docker

Gartner:WhatisHadoop?

Ã CommonApacheProjects– ALL=7(6)– Exceptfor1=3(5)– Exceptfor2=4(4)² About14CommonProjects

Ã UncommonProjects– Only1=9(1)– Only2=7 (2)– Only3=6 (3)² About22UncommonProjects

http://blogs.gartner.com/merv-adrian/2015/07/02/now-what-is-hadoop/

ODPi ODPi ODPi

HORTONWORKSDATAPLATFORM

op&YAR

HDP2.3isApacheHadoop;not“basedon”Hadoop

dbreak

Slider

Zookeepe

Falcon

Ranger

Accumulo

4.10.2

DATAMGMT DATAACCESS GOVERNANCE&INTEGRATION OPERATIONS SECURITY

HDP2.2Dec2014

HDP2.1April2014

HDP2.0Oct2013

HDP2.2Dec2014

HDP2.1April2014

HDP2.0Oct2013

0.12.0 0.12.0

0.12.1 0.13.0 0.4.0

1.4.4 1.4.4 3.3.23.4.5

0.4.00.5.0

0.14.0 0.14.0 3.4.6 0.5.0 0.4.00.9.30.5.2

4.0.04.7.2

1.2.1 0.60.0 0.98.4 4.2.0 1.6.1 0.6.0 1.5.21.4.5 4.1.02.0.0

1.4.0 1.5.1 4.0.0

1.5.1 1.4.4 3.4.5

2.7.1 1.4.6 1.0.0 0.6.0 0.5.02.1.00.8.2 3.4.61.5.25.2.1 0.80.0 0.5.01.7.04.4.0 0.10.0 0.6.10.7.01.2.10.15.0HDP2.3Oct2015 4.2.0

0.96.1

0.98.0 0.9.1

1.4.1 1.1.2

2.7.3 1.4.6 1.3.0 0.9.0 0.6.02.4.00.10.0 3.4.61.5.25.5.1 0.91.0 0.7.01.7.04.7.0 1.0.1 0.10.00.7.01.2.1+2.1***0.16.0

HDP2.5*2H2016

4.2.01.6.2+2.0** 1.1.2

2.7.1 1.4.6 1.2.0 0.6.0 0.5.02.2.10.9.0 3.4.61.5.25.2.1 0.80.0 0.5.01.7.04.4.0 0.10.0 0.6.10.7.01.2.10.15.0HDP2.4Mar2016 4.2.01.6.0 1.1.2

OngoingInnovationinApache

*HDP2.5– ShowscurrentApachebranchesbeingused.FinalcomponentversionsubjecttochangebasedonApachereleaseprocess.

**Spark1.6.2+Spark2.0– HDP2.5supportinstallationofbothSpark1.6.2andSpark2.0.Spark2.0isTechnicalPreviewwithinHDP2.5.

***Hive2.1isTechnicalPreviewwithinHDP2.5.

NextGenerationDataVendorsInvestmentfortheEnterprise

VerticalIntegration with YARN and HDFSEnsure engines can run reliably and respectfully in a YARN based cluster

Provision, Manage & Monitor

AmbariZookeeper

Scheduling

Loaddataandmanageaccordingtopolicy

Providelayeredapproachto

securitythroughAuthentication,Authorization,Accounting,andDataProtection

SECURITYGOVERNANCE

Deployandeffectivelymanagetheplatform

° ° ° ° ° ° ° ° ° ° ° ° ° ° °

Script

JavaScala

Cascading

Stream

Search

HBaseAccumulo

BATCH, INTERACTIVE & REAL-TIME DATA ACCESS

In-Memory

Others

ISV Engines

1 ° ° ° ° ° ° ° ° ° ° ° ° ° °

YARN: Data Operating System(ClusterResourceManagement)

HDFS (Hadoop Distributed File System)

Tez Slider SliderTez Tez

OPERATIONS

Horizontal Integration for Enterprise ServicesEnsure consistent enterprise services are applied across the Hadoop stack

Whatdodistributionsdo?

Ã Defineastackofcomponents• RichandlatestsetofApacheProjects(opensource&opencommunity)withoutlockin

Ã VerticalandHorizontalintegrationofcomponents• Vertical:BestSpeedandScale• Horizontal:OpenEnterpriseReady

Ã ProvisionandUpgradestack• Robust,EasyandAnywhere

Ã Acceleratetimetovalue(easyofuse)• NewFaceofHadoopwithUis fromAmbari,AmbariViews,Ranger,Falcon,Atlas

Ã PartnerEcosystem• RichandDeep

Ã Support• Industry’sbest,SmartSense andinfluencecommunity

HadoopOperations&Tools

How Do You Operate a Hadoop Cluster?

Apache™ Ambari isaplatformtoprovision,manageandmonitorHadoopclusters

Ambari Core Features and Extensibility

Install&Configure

Operate,Manage&Administer

Develop

Optimize&Tune

Developer

DataArchitect

Ambariprovidescoreservicesforoperations,developmentandextensionspointsforboth

ExtensibilityFeatures

Stacks,Blueprints&RESTAPIs

CoreFeatures

InstallWizard&Web

Web,OperatorViews,Metrics&Alerts

UserViews

ViewsFramework&RESTAPIs

ViewsFramework

ClusterAdmin

Newuserinterfaceenablesfast&easySQLdefinitionandexecution.

DataWorkerandDevOpsTooling

Ã Apluggablewaytoprovideacommonuserexperienceacrossmultipleuserpersonas.

Ambari Views

SystemAdmin/operators

DataWorkers

ApplicationDevelopers

AMBAR I

Ã Singlepointofentryforallusers.

Ã OpenCommunity

New User Views for DevOps

CapacitySchedulerViewBrowseandmanageYARNqueues

Tez ViewViewinformationrelatedtoTez jobsthatareexecutingonthecluster

NewUserViewsforDevelopment

PigViewAuthorandexecutePigScripts.

HiveViewAuthor,executeanddebugHive

queries.

FilesViewBrowseHDFSfilesystem.

ApacheZeppelin

• Web-basednotebookfordataengineers,dataanalystsanddatascientists• Bringsinteractivedataingestion,data

exploration,visualization,sharingandcollaborationfeaturestoHadoopandSpark

• Moderndatasciencestudio• ScalawithSpark• PythonwithSpark• SparkSQL• ApacheHive,andmore.

HadoopDataAccess

Access patterns enabled by YARN

YARN: Data Operating System

1 ° ° ° ° ° ° ° ° °

° ° ° ° ° ° ° ° °

Applications BatchNeeds to happen but, no timeframe limitations

InteractiveNeeds to happen at Human time

Real-Time Needs to happen at Machine Execution time.

Apache Hive: SQL in Hadoop

• Created by a team at Facebook

• Provides a standard SQL interface to data stored in Hadoop

• Quickly find value in raw data files

• Proven at petabyte scale

• Compatible with ALL major BI tools such as Tableau, Excel, MicroStrategy, Business Objects, etc…

SensorMobile

WeblogOperational

SQLQueries

Hive Architecture

User issues SQL query

Hive parses and plans query

Query converted to MapReduce and executed on Hadoop

Web UI

JDBC / ODBC

CLIHiveSQL

1HiveServer2 Hive

MR/Tez Compiler

Optimizer

Executor

MetaStore(MySQL, Postgresql,

Oracle)

MapReduce, Tez or Spark Job

Data DataData

Hadoop 3Data-local processing

Hive and the Stinger Initiative

BaseOptimizationsGeneratesimplifiedDAGsIn-memoryHashJoins

VectorQueryEngineOptimizedformodernprocessor

architectures

TezExpresstasksmoresimply

EliminatediskwritesPre-warmedContainers

ORCFileColumnStore

HighCompressionPredicate/FilterPushdowns

YARNNext-genHadoopdataprocessing

framework

QueryPlannerIntelligentCost-BasedOptimizer

PerformanceOptimizations100x+fastertimetoinsightDeeperanalyticalcapabilities

Stinger.next andSub-SecondSQL

Emergence of LLAP brings Sub-Second SQL response times within reach with Hive.

BATCH & INTERACTIVE BATCH & INTERACTIVE BATCH, INTERACTIVE & SUB-SECONDSPEED

DELIVERY

UPDATES

ENGINES

STINGERD E L I V E R E D

PROGRESSD E L I V E R E D FINALVERSION

HDP 2.1VERSION

0.13VERSION

HDP 2.3VERSION

SQL:2003+ SQL:2011 SUBSET

READ-ONLY SQL INSERT/UPDATE/DELETE

MR, TEZ MR, TEZ

F U T U R ES T I N G E R N E X T

COMPLETE ACID SUPPORT INCLUDING MERGE

COMPREHENSIVE SQL:2011 BASED ANALYTICS

MR, TEZ, LLAP

DELIVERED IN DEVELOPMENT

TieredDataStorage

Stinger.next Phase3

YARN:ContainerizedApplications

DataTypes SQL Features File Formats Latest Additions…Numeric CoreSQLFeatures Columnar ScalableCrossProduct

FLOAT/DOUBLE Date,Time andArithmeticalFunctions ORCFile PrimaryKey/Foreign KeyDECIMAL INNER,OUTER,CROSSandSEMIJoins Parquet Non-EquijoinINT/TINYINT/SMALLINT/BIGINT DerivedTableSubqueries

TextTechPreview:Proc.Extensions(PL/SQL)

BOOLEAN Correlated+ UncorrelatedSubqueries CSV FutureString UNIONALL Logfile ACIDMERGE

CHAR/VARCHAR UDFs, UDAFs,UDTFs Nested/Complex MultiSubquerySTRING CommonTableExpressions Avro Comparisontosub-selectBINARY UNIONDISTINCT JSON INTERSECT andEXCEPT

Date, Time AdvancedAnalytics XMLDATE OLAPandWindowingFunctions CustomFormatsTIMESTAMP CUBE andGroupingSets OtherFeaturesIntervalTypes NestedDataAnalytics XPath Analytics

ComplexTypes NestedDataTraversalARRAY LateralViewsMAP ACIDTransactionsSTRUCT INSERT/UPDATE/DELETEUNION

ApacheHive:JourneytoSQL:2011Analytics

LegendExisting

Future

NewwithHive2.0

Columnar Storage

ORCFile Parquet

Unstructured Data

JSON CSV

Text Avro

Custom

Weblog

SQL Engines

RowEngine VectorEngineSQ

SQL Support

SQL:2011 Optimizer HCatalog HiveServer2

Block Cache

LinuxCache

Hadoop 1

MapReduce

Hadoop 2

Tez Spark

Vector Cache

Persistent Server

Historical

Current

In-development

Legend

Apache Hive: Modern Architecture

ApacheTezisacriticalinnovationoftheStingerInitiative.

• Along with YARN, Tez not only improves Hive, but improvesallthingsbatchand interactiveforHadoop;Pig,Cascading…

• More Efficient Processing than MapReduce

• Reduceoperationsandcomplexityofbackendprocessing• AllowsforMapReduceReducewhichsavesharddiskoperations• Implementsa“service”whichisalwayson,decreasingstarttimesofjobs• AllowsCachingofDatainMemory

Cascading/ Scalding

WhyisTez Important?

°1 ° ° ° ° ° ° ° °

° ° ° ° ° ° ° ° °

° ° ° ° ° ° °

° ° ° ° ° ° N

HDFS (Hadoop Distributed File

System)

Scripting

Tez Tez

Applications

ApacheTez

Hive– MapReduce Hive– Tez

SELECT a.state, COUNT(*), AVG(c.price) FROM a

JOIN b ON (a.id = b.id)JOIN c ON (a.itemId = c.itemId)

GROUP BY a.state

SELECTa.state

JOIN(a,c)SELECTc.price

SELECTb.id

JOIN(a,b)GROUPBYa.state

COUNT(*)AVG(c.price)

SELECTa.state,c.itemId

JOIN(a,c)

JOIN(a,b)GROUPBYa.state

COUNT(*)AVG(c.price)

SELECTb.id

Tez avoidsunneededwritestoHDFS

Tez allowsReducer-onlyjobswithintheDAG

Sub-secondQueriesinHive:LLAP(LiveLongandProcess)

Ã Persistentdaemons– Savestimeonprocessstartup(eliminatescontainerallocationandJVMstartuptime)– AllcodeJITed withinaqueryortwo

Ã Datacachingwithanasync I/Oelevator– Hotdatacachedinmemory(columnaraware,soonlyhotcolumnscached)– Whenpossibleworkscheduledonnodewithdatacached,ifnotworkwillberuninothernode

Ã OperatorscanbeexecutedinsideLLAPwhenitmakessense– Large,ETLstylequeriesusuallydon’tmakesense– UsercodenotruninLLAPforsecurity

Ã Workingoninterfacetoallowotherdataenginestoreadsecurelyinparallel

Ã Betain2.0

Hive2withLLAP:ArchitectureOverview

Storage

HDFS S3+OtherHDFSCompatibleFilesystems

YARNCluster

LLAPDaemon

QueryExecutors

In-MemoryCache

LLAPDaemon

QueryExecutors

In-MemoryCache

LLAPDaemon

QueryExecutors

In-MemoryCache

LLAPDaemon

QueryExecutors

In-MemoryCache

QueryCoordinators

Coord-inator

HiveServer2(Query

Endpoint)

ODBC/JDBC SQL

Queries

HiveWithLLAPExecutionOptions

Tez Only LLAP + Tez

LLAP only

Scripting Data Pipeline & ETLApache Pig

• Dataflowengineandscriptinglanguage(PigLatin)• Allowsyoutotransform dataanddatasets

Advantages over MapReduce• Reducestimetowritejobs• Communitysupport• PiggybankhasasignificantnumberofUDF’stohelpadoption• TherearealargenumberofexistingshopsusingPIG

PigLatin

• Pigexecutesinauniquefashion:oDuringexecution,eachstatementisprocessedbythePiginterpreteroIfastatementisvalid,itgetsaddedtoalogicalplan builtbytheinterpreter

oThestepsinthelogicalplandonotactuallyexecuteuntilaDUMPorSTOREcommandisused

WhyusePig?

• Maybewewanttojointwodatasets,fromdifferentsources,onacommonvalue,andwanttofilter,andsort,andgettop5sites

Resource Management

Storage

Elegant Developer APIsDataFrames, Machine Learning, and SQL

Made for Data ScienceAll apps need to get predictive at scale and fine granularity

Democratize Machine LearningSpark is doing to ML on Hadoop what Hive did for SQL on Hadoop

CommunityBroad developer, customer and partner interest

Realize Value of Data Operating SystemA key tool in the Hadoop toolbox

ApacheSparkenthusiasm

Applications

SparkCoreEngine

ScalaJava

Pythonlibraries

MLlib(Machinelearning)

SparkSQL*

SparkStreaming*

SparkCoreEngine

Apache Spark & Apache Hadoop Perfect Together

General Purpose Data Access Engineforfast,large-scaledataprocessing

Designed for Iterative, In-Memorycomputationsandinteractivedatamining

Expressive Multi-Language APIsforJava,Scala,PythonandR

Built-in LibrariesEnabledataworkerstorapidlyiterateoverdatafor:ETL,MachineLearning,SQLandStreamprocessing

ScalaJava

PythonR

Spark Core Engine

Spark SQL

Spark StreamingMLlib GraphX

1 ° ° ° ° ° ° ° ° °

° ° ° ° ° ° ° ° ° °

Apache Projects Enable Access Patterns

Various open source projects have incubated in order to meet these access pattern needs

Today, they can all run on a single cluster on a single set of data because of YARN

All powered by a broad open community

YARN: Data Operating System

1 ° ° ° ° ° ° ° ° °

° ° ° ° ° ° ° ° °

InteractiveSolr

SparkHivePig

Real-TimeHBase

AccumuloStorm

BatchMapReduce

Applications

HandsonLabOverview

HDP2.5Sandbox

Ã ProvidesFreepreconfiguredHDP– RunsinaVirtualMachine,

DockerorAzureHortonworks.com/sandbox

Ã EasytoUse– Operations

• Ambari– DevandDevOps

• AmbariUserViews– WebNotebook

• Zeppelin

Ã Workswith60+FreetutorialHortonworks.com/tutorials

SandboxStart

Ã ImportSandboxImage

Ã Takesabout4min

Ã SplashScreen– VirtualBoxSampleSplashScreen– SavethehostIP

http://localhost:8888

DataDiscoveryLab• Elefante WineCompanyhasafleetofover100trucks.

• Thegeolocationdatacollectedfromthetruckscontainseventsgeneratedwhilethetruckdriversaredriving.

• Thecompany’sgoalwithHadoopistoMitigateRisk:o Understandcorrelationsbetweenmilesdrivenandeventso Computetheriskfactorforeachdriverbasedonmileage&events

o LabEnvo Sandbox2.5

o LabDoco URL:http://tinyurl.com/hello-hdpo LoadDatao QueryDatao ProcessData

Elefante Wine Current Challenges

The CompanyElefante Wine is a boutique wine fulfillment company with a large fleet of trucks. It delivers wine in a highly-regulated industry with stringent transportation requirements.

The SituationRecently a number of driver violations led to fines and increased insurance rates

The Challenges• Rising Operational Costs• Driver Safety• Risk Management• Logistics Optimization

Elefante WineCompanyhasalargefleetoftrucksinUSA

Atruckgeneratesmillionsofeventsforagivenroute;aneventcouldbe:

§ 'Normal'events:starting/stoppingofthevehicle

§ ‘Violation’events:speeding,excessiveaccelerationandbreaking,unsafetaildistance

Companyusesanapplicationthatmonitorstrucklocationsandviolationsfromthetruck/driverinreal-timetocalculaterisk

Route?Truck?Driver?

Analystsqueryabroadhistorytounderstandiftoday’sviolationsarepartofalargerproblemwithspecificroutes,trucks,ordrivers

Elefante Wine Risk and Driver Safety Challenges

Trucksoutfittedwithnewsensorsgeneratinglargevolumesofnewdata:

• Location

• Speed

• DriverViolations

Needtobeintegratereal-time&historicaldata

Increase safety and reduce liabilitiesAnticipate driver violations BEFORE they happen and take precautionary actionsFindpredictivecorrelationsindriverbehavioroverlargevolumesofreal-timedata

Difficult to deliver timely insights to the right people and systems to take action

Data DiscoveryUncover new findings

Predictive Analytics Identify your next best action

Better Understandingof the Past

Better Prediction of the Future

What’sourgoal?

Ã Solution:– CollectadditionaldataviasensorsintruckstobetterunderstandRiskFactors

Ã How:– Quicklystorenewsensordatainacommonrepository– Preparethedataforanalysis– Explorethedata– CalculateRisk– Generateareport

geolocation.csv

trucks.csv

TemporarytableB Geolocation

TemporarytableT Trucks

csv ORC

ORCSQL

TemporaryTablesCreatedLoadedDeleted

Geolocation

Trucks

PIGorSparkRiskCalculation

Truck_mileage

Avg_mileage

DriverMileage

RiskFactor

Events

Trucking Risk Analysis – Hadoop ELT

CalculateRisk

GettingStartedResources

ConnectedDataArchitecturewithHDCforAWS

C LO U DIdealUseCases:DataScienceandExploration(Spark,Zeppelin)

ETLandDataPreparation(Hive,Spark)

AnalyticsandReporting(Hive2w/LLAP,Zeppelin)

CloudDataProcessing

(HDCforAWS)

TechnicalPreview

hortonworks.github.io/hdp-aws

BigDataTutorials

Ã GetStarted– hortonworks.com/tutorials– ApacheHadoop&Ecosystem

• tinyurl.com/hello-hdp– ApacheSpark

• tinyurl.com/hwx-spark-intro– ApacheNiFi

• tinyurl.com/nifi-intro– UseCase

• IoT• SocialMedia

developer.hortonworks.com

HortonworksNourishestheCommunityHORTONWORKS

COMMUN I T Y CONNECT IONHORTONWORKS PARTNERWORKS

https://community.hortonworks.com

WanttocontinuethetechnicalIntroduction?

Ã HadoopSummitCrashCourses– Replays– Free

Ã hadoopsummit.org/san-jose/agenda– ApacheHadoop– ApacheSpark– ApacheNiFi– IoT &Streaming– DataScience

Thankyou!

rafael@hortonworks.com

@racoss

ProtectingtheElephantintheCastle…..Kerberos,

WireEncryption

HDFS Encryption

ApacheRangerNetworkSegmentation,

Firewalls

LDAP/AD

ApacheKnox

Hadoop Summit Tokyo HDP Sandbox Workshop

Technology

MICROSOFT AZURE€¦ · Apache code •SQL •Business analytics •Custom apps. Hadoop 2.0. TCP-DS Query 95 –200GB of Data. 570.8s 356.7s 140.3s. Hive 10 HDP 1.3 / Hive 11 HDP

Discover HDP 2.1: Interactive SQL Query in Hadoop with Apache Hive

Discover HDP 2.2: Comprehensive Hadoop Security with Apache Ranger and Apache Knox

Visual Analytics Sandbox - Information Technologyvvr3254/CMPS598/Notes/... · 2018. 1. 27. · HUE: Hadoop User Experience An open-source Web interface that supports Apache Hadoop

HDP Advanced Security: Comprehensive Security for Enterprise Hadoop

Porting your hadoop app to horton works hdp

HDP - technidrive.co.uktechnidrive.co.uk/media/uploads/hdp seris catalouge.pdf · HDP parallel shaft gear unit range are: • sizes from HDP 60 to HDP 90 with double and triple reduction

Discover HDP 2.1: Apache Solr for Hadoop Search

Discover HDP 2.1: Using Apache Ambari to Manage Hadoop Clusters

Infrastruktur & Datenarchitekturen für Big-Data-Szenarien ...€¦ · HDInsight verwendet die Hortonworks Data Platform (HDP)-Hadoop-Distribution Ein Data Lake Dienst, skalierbar

NAVAL & SPECIAL SHIP BUSINESS UNIT · 2020. 12. 15. · HDP-1500 HDP-1000 HDP-500 HDP-4000 16 HDP-5000 HDP-3000 HDP-1000 ... The comprehensive R&D activities of HMRI comprise all

Nanocarbon Hdp

Installing 9.6 BDE binaries on hadoop data nodes Snapshots captured from Cloudera sandbox

WANDISCO FUSION MICROSOFT AZURE DATA BOX · Hadoop • Amazon EMR • Cloudera CDH • Google Cloud Dataproc • Hortonworks (HDP) • ®IBM BigInsights • MapR • Microsoft Azure

Discover HDP 2.2: Apache Falcon for Hadoop Data Governance

HDP Security Overview - Cloudera · Ranger manages access control through a user interface that ensures consistent policy administration across Hadoop data access components. Security

大数据基础组件的安全解决方案 - Huodongjia.com · Spark 生态圈 Spark是 ... Apche Hadoop 2.7 Cloudera cdh 5.8.1 Hortonworks Sandbox 2.4 Hortonworks HDP 2.4 星环Transwarp

Upgrading from-hdp-21-to-hdp-24

Hortonworks Data Platform - Meetupfiles.meetup.com/13696062/HortonWorksOverview.pdf · Hortonworks Data Platform: Enterprise Hadoop The Hortonworks Data Platform (HDP) delivers all

Talend Big Data Sandbox · 2.2 Cloudera Services Status The Talend Big Data Sandbox is configured such that all necessary Hadoop services should be running once the VM is started