董西成

HADOOP YARN在异构环境下应用与实践

AGENDA

bull Background

bull Label-based scheduling

bull Framework On YARN

bull Experience

bull Q amp A

ARCHITECTURE ndash YARN

YARN distributed operating system

MapReduce Spark

(Streaming) Voidbox

machine bdquo machine

NestoPresto

machine

ARCHITECTURE ndash YARN

YARN modules

Resource Manager

Client

NodeManager

HDFS

NodeManager

HDFS

NodeManager

HDFS

YARN application1

Container (MapTask)

MR AppMaster

Container (MapTask)

Client

Application

Master

Container

YARN application2

bull Batch Jobs bull Take from a few seconds to a few days to complete

bull Eg MapReduce Spark

bull Long-running Services bull Should ldquoneverrdquo go down handle short-lived latency-sensitive requests

bull EgPrestoNesto Spark Streaming Web Services

bull YARN Data Operating System bull ResourceManager HA amp Recovery

bull NodeManager Recovery

bull Resource Scheduling(eg label-based scheduling)

bull Apache Slider amp Apache Twill

DISTRIBUTED APPLICATION TYPES

AGENDA

bull Background

bull Label-based scheduling

bull Framework On YARN

bull Experience

bull Q amp A

ARCHITECTURE ndash YARN

YARN distributed operating system

MapReduce Spark

(Streaming) Voidbox

machine bdquo machine

NestoPresto

machine

ARCHITECTURE ndash YARN

YARN modules

Resource Manager

Client

NodeManager

HDFS

NodeManager

HDFS

NodeManager

HDFS

YARN application1

Container (MapTask)

MR AppMaster

Container (MapTask)

Client

Application

Master

Container

YARN application2

bull Batch Jobs bull Take from a few seconds to a few days to complete

bull Eg MapReduce Spark

bull Long-running Services bull Should ldquoneverrdquo go down handle short-lived latency-sensitive requests

bull EgPrestoNesto Spark Streaming Web Services

bull YARN Data Operating System bull ResourceManager HA amp Recovery

bull NodeManager Recovery

bull Resource Scheduling(eg label-based scheduling)

bull Apache Slider amp Apache Twill

DISTRIBUTED APPLICATION TYPES

ARCHITECTURE ndash YARN

YARN distributed operating system

MapReduce Spark

(Streaming) Voidbox

machine bdquo machine

NestoPresto

machine

ARCHITECTURE ndash YARN

YARN modules

Resource Manager

Client

NodeManager

HDFS

NodeManager

HDFS

NodeManager

HDFS

YARN application1

Container (MapTask)

MR AppMaster

Container (MapTask)

Client

Application

Master

Container

YARN application2

bull Batch Jobs bull Take from a few seconds to a few days to complete

bull Eg MapReduce Spark

bull Long-running Services bull Should ldquoneverrdquo go down handle short-lived latency-sensitive requests

bull EgPrestoNesto Spark Streaming Web Services

bull YARN Data Operating System bull ResourceManager HA amp Recovery

bull NodeManager Recovery

bull Resource Scheduling(eg label-based scheduling)

bull Apache Slider amp Apache Twill

DISTRIBUTED APPLICATION TYPES

ARCHITECTURE ndash YARN

YARN modules

Resource Manager

Client

NodeManager

HDFS

NodeManager

HDFS

NodeManager

HDFS

YARN application1

Container (MapTask)

MR AppMaster

Container (MapTask)

Client

Application

Master

Container

YARN application2

bull Batch Jobs bull Take from a few seconds to a few days to complete

bull Eg MapReduce Spark

bull Long-running Services bull Should ldquoneverrdquo go down handle short-lived latency-sensitive requests

bull EgPrestoNesto Spark Streaming Web Services

bull YARN Data Operating System bull ResourceManager HA amp Recovery

bull NodeManager Recovery

bull Resource Scheduling(eg label-based scheduling)

bull Apache Slider amp Apache Twill

DISTRIBUTED APPLICATION TYPES

bull Batch Jobs bull Take from a few seconds to a few days to complete

bull Eg MapReduce Spark

bull Long-running Services bull Should ldquoneverrdquo go down handle short-lived latency-sensitive requests

bull EgPrestoNesto Spark Streaming Web Services

bull YARN Data Operating System bull ResourceManager HA amp Recovery

bull NodeManager Recovery

bull Resource Scheduling(eg label-based scheduling)

bull Apache Slider amp Apache Twill

DISTRIBUTED APPLICATION TYPES

bull Heterogeneous Causes bull Static factors Machines(CPU memory network OS libraries)

bull Dynamic factors Load Network Slow Disk bdquo

bull Solutions for static factors bull Distinguish machines and assign to different frameworks

bull Eg 128GB-memory machines for spark 10G-network machines for spark streamingnestopresto

bull Solutions for dynamical factors bull Solve in framework level

bull For MapReduceSpark speculative execution

bull For others

HETEROGENEOUS ISSUES

6

AGENDA

bull Background

bull Label-based scheduling

bull Framework On YARN

bull Experience

bull Q amp A

WHY LABEL-BASED SCHEDULING

bull Distinguish goodbad machines for different Apps

bull Eg Large memory machine for spark

bull Apps require special librariesservices installed

bull Spark MLlib native libraries

bull Voidbox docker engines

bull Machine-level isolation for key Apps

bull Low-latency Apps spark streamingprestonesto

LABEL-BASED SCHEDULING

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

Resource Manager (Capacity Scheduler)

node

node

node

streaming-node

mr-node

spark-nodemr-node

rack1 rack2 rack3 rack4 rack5

root

a c b

streaming-node mr-node

spark-node

spark-node node docker-node

d

docker-

node c1 c2

LEARN FROM LABEL-BASED SCHEDULING

bull Benefit

bull Staticdynamic machine-level partition

bull Elastic scalability for Apps

bull Latch to enter maintenance mode

bull Issue

bull Only Capacity Scheduler is supported

bull YARN Web Portal does not support well

bull Hurt data locality

AGENDA

bull Background

bull Label-based scheduling

bull Frameworks On YARN

bull Experience

bull Q amp A

bull MapReduce

bull Spark

bull Spark Streaming

bull Real-time processing

bull NestoPresto

bull MPP engines

bull Voidbox

bull Docker on yarn

FRAMEWORKS ON YARN

1

2

bull What is Nesto

bull A distributed query engine for parquet-like nested data

bull MPP + Parquet (On HDFS)

bull Nesto features

bull support large nested data(200+ columns 4 levels)

bull support filter groupby limit and map-side join

bull Support json expression and Programming API

bull use cases in hulu

bull Segment analysis

bull Cohort analysis

bull Funnel analysis

WHAT IS NESTO

1

3

bull Requirements

bull Run Specified N containers(servers) One container on node

bull Fault tolerance

bull Tracking web UI

bull Log rotation

bull Services (Never die)

bull Implementation

bull ApplicationMaster scheduling amp fault tolerance

bull Container NestoServer (Worker)

NESTO ON YARN (1)

1

4

Nesto Server

Nesto Server

Nesto Server

Zookeeper

NESTO ON YARN (2)

1

5

Zookeeper Resource Manager

Node Manager

Node Manager

Node Manager

Proxy

Nesto Server

AppMaster

Proxy

Nesto Server

Proxy

Nesto Server

Web server

yarn jar nesto-installnesto-010-SNAPSHOT-jar-with-dependenciesjar

comhulunestoyarnNestoYarnClient

-conf_path mntdata1homexichengdongnesto-srcyarnnesto-installconf

-framework_path hdfstmpxichengnestonesto-yarntargz

-master_memory 2048 -container_memory 2048 -container_vcores 1 -num_containers 3

bull Nesto deployment

bull Nesto is not a high-concurrency system

bull Mixed deployment with other batchOLAP systems on YARN

bull Slow taskworker

bull Slow task detection and re-dispatch

bull Worker priority scheduling based on weight

NESTO IN HETEROGENEOUS CLUSTER

1

6

WHAT IS VOIDBOX

1

7

LIGHTWEIGHT VM

RUNTIME ISOLATION

UNION FILE SYSTEM

RESOURCE MANAGEMENT

SCHEDULING

FAULT TOLERANCE

DISTRIBUTE COMPUTATION

ARBITRARY APPLICATION

FAULT TOLERANCE

VOIDBOX IS A PROGRAMMING FRAMEWORK

VOIDBOX SERVICE EXAMPLE

1

8

VOIDBOX COMPONENTS

API

RUNTIME MODE REGISTRYMGMT

VOIDBOX MASTER VOIDBOX PROXY

DAG SERVICE TASK

Core

Framework

VOIDBOX ARCHITECTURE

2

0

YARN modules

Resource Manager

Voidbox Client

NodeManager

NodeManager

NodeManager

Docker Engine

Docker Engine

Voidbox Master

State Server

Container

Voidbox Proxy

Container

Voidbox Proxy

Docker Registry In S3

HDFS

HDFS

HDFS

Voidbox Driver

Pull

Voidbox modules

Docker modules

Voidbox is a standard appliction on top of YARN just like

MapReduce

server

server

server

server

bull Batch Jobs

bull Data Dump

bull Facematch distributed application face recognition in video

bull Long-running Services

bull Nesto WebServer

bull Spark HistoryServerJobServer

bull Kinko thumbnail Generator Service

VOIDBOX IN HULU

2

1

AGENDA

bull Background

bull Label-based scheduling

bull Frameworks On YARN

bull Experience

bull Q amp A

bull Container Description

bull Host rack

bull Relax locality

bull Blacklisted Container

bull ApplicationMaster could put certain nodes into blacklist then no more containers on those nodes will be received

bull Release unused containers

bull If AM allocates K containers more containers may be received

EXPERIENCE 1 CONTAINER API

2

3

bull Whatrsquos tracking url

bull URL of web server insideoutside AM

bull Jetty or netty

bull Two tracking URL

bull Register tracking url (running)

bull Unregister tracking url (completed)

bull URL Format

bull httphostport

bull Direct URL to ajsp use ldquoajsprdquo not ldquoajsprdquo ldquoabhtmlrdquo not ldquoabhtmlrdquo

EXPERIENCE 2 TRACKING URL

2

4

AGENDA

bull Background

bull Label-based scheduling

bull Framework On YARN

bull Experience

bull Q amp A

WHY LABEL-BASED SCHEDULING

bull Distinguish goodbad machines for different Apps

bull Eg Large memory machine for spark

bull Apps require special librariesservices installed

bull Spark MLlib native libraries

bull Voidbox docker engines

bull Machine-level isolation for key Apps

bull Low-latency Apps spark streamingprestonesto

LABEL-BASED SCHEDULING

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

Resource Manager (Capacity Scheduler)

node

node

node

streaming-node

mr-node

spark-nodemr-node

rack1 rack2 rack3 rack4 rack5

root

a c b

streaming-node mr-node

spark-node

spark-node node docker-node

d

docker-

node c1 c2

LEARN FROM LABEL-BASED SCHEDULING

bull Benefit

bull Staticdynamic machine-level partition

bull Elastic scalability for Apps

bull Latch to enter maintenance mode

bull Issue

bull Only Capacity Scheduler is supported

bull YARN Web Portal does not support well

bull Hurt data locality

AGENDA

bull Background

bull Label-based scheduling

bull Frameworks On YARN

bull Experience

bull Q amp A

bull MapReduce

bull Spark

bull Spark Streaming

bull Real-time processing

bull NestoPresto

bull MPP engines

bull Voidbox

bull Docker on yarn

FRAMEWORKS ON YARN

1

2

bull What is Nesto

bull A distributed query engine for parquet-like nested data

bull MPP + Parquet (On HDFS)

bull Nesto features

bull support large nested data(200+ columns 4 levels)

bull support filter groupby limit and map-side join

bull Support json expression and Programming API

bull use cases in hulu

bull Segment analysis

bull Cohort analysis

bull Funnel analysis

WHAT IS NESTO

1

3

bull Requirements

bull Run Specified N containers(servers) One container on node

bull Fault tolerance

bull Tracking web UI

bull Log rotation

bull Services (Never die)

bull Implementation

bull ApplicationMaster scheduling amp fault tolerance

bull Container NestoServer (Worker)

NESTO ON YARN (1)

1

4

Nesto Server

Nesto Server

Nesto Server

Zookeeper

NESTO ON YARN (2)

1

5

Zookeeper Resource Manager

Node Manager

Node Manager

Node Manager

Proxy

Nesto Server

AppMaster

Proxy

Nesto Server

Proxy

Nesto Server

Web server

yarn jar nesto-installnesto-010-SNAPSHOT-jar-with-dependenciesjar

comhulunestoyarnNestoYarnClient

-conf_path mntdata1homexichengdongnesto-srcyarnnesto-installconf

-framework_path hdfstmpxichengnestonesto-yarntargz

-master_memory 2048 -container_memory 2048 -container_vcores 1 -num_containers 3

bull Nesto deployment

bull Nesto is not a high-concurrency system

bull Mixed deployment with other batchOLAP systems on YARN

bull Slow taskworker

bull Slow task detection and re-dispatch

bull Worker priority scheduling based on weight

NESTO IN HETEROGENEOUS CLUSTER

1

6

WHAT IS VOIDBOX

1

7

LIGHTWEIGHT VM

RUNTIME ISOLATION

UNION FILE SYSTEM

RESOURCE MANAGEMENT

SCHEDULING

FAULT TOLERANCE

DISTRIBUTE COMPUTATION

ARBITRARY APPLICATION

FAULT TOLERANCE

VOIDBOX IS A PROGRAMMING FRAMEWORK

VOIDBOX SERVICE EXAMPLE

1

8

VOIDBOX COMPONENTS

API

RUNTIME MODE REGISTRYMGMT

VOIDBOX MASTER VOIDBOX PROXY

DAG SERVICE TASK

Core

Framework

VOIDBOX ARCHITECTURE

2

0

YARN modules

Resource Manager

Voidbox Client

NodeManager

NodeManager

NodeManager

Docker Engine

Docker Engine

Voidbox Master

State Server

Container

Voidbox Proxy

Container

Voidbox Proxy

Docker Registry In S3

HDFS

HDFS

HDFS

Voidbox Driver

Pull

Voidbox modules

Docker modules

Voidbox is a standard appliction on top of YARN just like

MapReduce

server

server

server

server

bull Batch Jobs

bull Data Dump

bull Facematch distributed application face recognition in video

bull Long-running Services

bull Nesto WebServer

bull Spark HistoryServerJobServer

bull Kinko thumbnail Generator Service

VOIDBOX IN HULU

2

1

AGENDA

bull Background

bull Label-based scheduling

bull Frameworks On YARN

bull Experience

bull Q amp A

bull Container Description

bull Host rack

bull Relax locality

bull Blacklisted Container

bull ApplicationMaster could put certain nodes into blacklist then no more containers on those nodes will be received

bull Release unused containers

bull If AM allocates K containers more containers may be received

EXPERIENCE 1 CONTAINER API

2

3

bull Whatrsquos tracking url

bull URL of web server insideoutside AM

bull Jetty or netty

bull Two tracking URL

bull Register tracking url (running)

bull Unregister tracking url (completed)

bull URL Format

bull httphostport

bull Direct URL to ajsp use ldquoajsprdquo not ldquoajsprdquo ldquoabhtmlrdquo not ldquoabhtmlrdquo

EXPERIENCE 2 TRACKING URL

2

4

WHY LABEL-BASED SCHEDULING

bull Distinguish goodbad machines for different Apps

bull Eg Large memory machine for spark

bull Apps require special librariesservices installed

bull Spark MLlib native libraries

bull Voidbox docker engines

bull Machine-level isolation for key Apps

bull Low-latency Apps spark streamingprestonesto

LABEL-BASED SCHEDULING

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

Resource Manager (Capacity Scheduler)

node

node

node

streaming-node

mr-node

spark-nodemr-node

rack1 rack2 rack3 rack4 rack5

root

a c b

streaming-node mr-node

spark-node

spark-node node docker-node

d

docker-

node c1 c2

LEARN FROM LABEL-BASED SCHEDULING

bull Benefit

bull Staticdynamic machine-level partition

bull Elastic scalability for Apps

bull Latch to enter maintenance mode

bull Issue

bull Only Capacity Scheduler is supported

bull YARN Web Portal does not support well

bull Hurt data locality

AGENDA

bull Background

bull Label-based scheduling

bull Frameworks On YARN

bull Experience

bull Q amp A

bull MapReduce

bull Spark

bull Spark Streaming

bull Real-time processing

bull NestoPresto

bull MPP engines

bull Voidbox

bull Docker on yarn

FRAMEWORKS ON YARN

1

2

bull What is Nesto

bull A distributed query engine for parquet-like nested data

bull MPP + Parquet (On HDFS)

bull Nesto features

bull support large nested data(200+ columns 4 levels)

bull support filter groupby limit and map-side join

bull Support json expression and Programming API

bull use cases in hulu

bull Segment analysis

bull Cohort analysis

bull Funnel analysis

WHAT IS NESTO

1

3

bull Requirements

bull Run Specified N containers(servers) One container on node

bull Fault tolerance

bull Tracking web UI

bull Log rotation

bull Services (Never die)

bull Implementation

bull ApplicationMaster scheduling amp fault tolerance

bull Container NestoServer (Worker)

NESTO ON YARN (1)

1

4

Nesto Server

Nesto Server

Nesto Server

Zookeeper

NESTO ON YARN (2)

1

5

Zookeeper Resource Manager

Node Manager

Node Manager

Node Manager

Proxy

Nesto Server

AppMaster

Proxy

Nesto Server

Proxy

Nesto Server

Web server

yarn jar nesto-installnesto-010-SNAPSHOT-jar-with-dependenciesjar

comhulunestoyarnNestoYarnClient

-conf_path mntdata1homexichengdongnesto-srcyarnnesto-installconf

-framework_path hdfstmpxichengnestonesto-yarntargz

-master_memory 2048 -container_memory 2048 -container_vcores 1 -num_containers 3

bull Nesto deployment

bull Nesto is not a high-concurrency system

bull Mixed deployment with other batchOLAP systems on YARN

bull Slow taskworker

bull Slow task detection and re-dispatch

bull Worker priority scheduling based on weight

NESTO IN HETEROGENEOUS CLUSTER

1

6

WHAT IS VOIDBOX

1

7

LIGHTWEIGHT VM

RUNTIME ISOLATION

UNION FILE SYSTEM

RESOURCE MANAGEMENT

SCHEDULING

FAULT TOLERANCE

DISTRIBUTE COMPUTATION

ARBITRARY APPLICATION

FAULT TOLERANCE

VOIDBOX IS A PROGRAMMING FRAMEWORK

VOIDBOX SERVICE EXAMPLE

1

8

VOIDBOX COMPONENTS

API

RUNTIME MODE REGISTRYMGMT

VOIDBOX MASTER VOIDBOX PROXY

DAG SERVICE TASK

Core

Framework

VOIDBOX ARCHITECTURE

2

0

YARN modules

Resource Manager

Voidbox Client

NodeManager

NodeManager

NodeManager

Docker Engine

Docker Engine

Voidbox Master

State Server

Container

Voidbox Proxy

Container

Voidbox Proxy

Docker Registry In S3

HDFS

HDFS

HDFS

Voidbox Driver

Pull

Voidbox modules

Docker modules

Voidbox is a standard appliction on top of YARN just like

MapReduce

server

server

server

server

bull Batch Jobs

bull Data Dump

bull Facematch distributed application face recognition in video

bull Long-running Services

bull Nesto WebServer

bull Spark HistoryServerJobServer

bull Kinko thumbnail Generator Service

VOIDBOX IN HULU

2

1

AGENDA

bull Background

bull Label-based scheduling

bull Frameworks On YARN

bull Experience

bull Q amp A

bull Container Description

bull Host rack

bull Relax locality

bull Blacklisted Container

bull ApplicationMaster could put certain nodes into blacklist then no more containers on those nodes will be received

bull Release unused containers

bull If AM allocates K containers more containers may be received

EXPERIENCE 1 CONTAINER API

2

3

bull Whatrsquos tracking url

bull URL of web server insideoutside AM

bull Jetty or netty

bull Two tracking URL

bull Register tracking url (running)

bull Unregister tracking url (completed)

bull URL Format

bull httphostport

bull Direct URL to ajsp use ldquoajsprdquo not ldquoajsprdquo ldquoabhtmlrdquo not ldquoabhtmlrdquo

EXPERIENCE 2 TRACKING URL

2

4

LABEL-BASED SCHEDULING

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

Resource Manager (Capacity Scheduler)

node

node

node

streaming-node

mr-node

spark-nodemr-node

rack1 rack2 rack3 rack4 rack5

root

a c b

streaming-node mr-node

spark-node

spark-node node docker-node

d

docker-

node c1 c2

LEARN FROM LABEL-BASED SCHEDULING

bull Benefit

bull Staticdynamic machine-level partition

bull Elastic scalability for Apps

bull Latch to enter maintenance mode

bull Issue

bull Only Capacity Scheduler is supported

bull YARN Web Portal does not support well

bull Hurt data locality

AGENDA

bull Background

bull Label-based scheduling

bull Frameworks On YARN

bull Experience

bull Q amp A

bull MapReduce

bull Spark

bull Spark Streaming

bull Real-time processing

bull NestoPresto

bull MPP engines

bull Voidbox

bull Docker on yarn

FRAMEWORKS ON YARN

1

2

bull What is Nesto

bull A distributed query engine for parquet-like nested data

bull MPP + Parquet (On HDFS)

bull Nesto features

bull support large nested data(200+ columns 4 levels)

bull support filter groupby limit and map-side join

bull Support json expression and Programming API

bull use cases in hulu

bull Segment analysis

bull Cohort analysis

bull Funnel analysis

WHAT IS NESTO

1

3

bull Requirements

bull Run Specified N containers(servers) One container on node

bull Fault tolerance

bull Tracking web UI

bull Log rotation

bull Services (Never die)

bull Implementation

bull ApplicationMaster scheduling amp fault tolerance

bull Container NestoServer (Worker)

NESTO ON YARN (1)

1

4

Nesto Server

Nesto Server

Nesto Server

Zookeeper

NESTO ON YARN (2)

1

5

Zookeeper Resource Manager

Node Manager

Node Manager

Node Manager

Proxy

Nesto Server

AppMaster

Proxy

Nesto Server

Proxy

Nesto Server

Web server

yarn jar nesto-installnesto-010-SNAPSHOT-jar-with-dependenciesjar

comhulunestoyarnNestoYarnClient

-conf_path mntdata1homexichengdongnesto-srcyarnnesto-installconf

-framework_path hdfstmpxichengnestonesto-yarntargz

-master_memory 2048 -container_memory 2048 -container_vcores 1 -num_containers 3

bull Nesto deployment

bull Nesto is not a high-concurrency system

bull Mixed deployment with other batchOLAP systems on YARN

bull Slow taskworker

bull Slow task detection and re-dispatch

bull Worker priority scheduling based on weight

NESTO IN HETEROGENEOUS CLUSTER

1

6

WHAT IS VOIDBOX

1

7

LIGHTWEIGHT VM

RUNTIME ISOLATION

UNION FILE SYSTEM

RESOURCE MANAGEMENT

SCHEDULING

FAULT TOLERANCE

DISTRIBUTE COMPUTATION

ARBITRARY APPLICATION

FAULT TOLERANCE

VOIDBOX IS A PROGRAMMING FRAMEWORK

VOIDBOX SERVICE EXAMPLE

1

8

VOIDBOX COMPONENTS

API

RUNTIME MODE REGISTRYMGMT

VOIDBOX MASTER VOIDBOX PROXY

DAG SERVICE TASK

Core

Framework

VOIDBOX ARCHITECTURE

2

0

YARN modules

Resource Manager

Voidbox Client

NodeManager

NodeManager

NodeManager

Docker Engine

Docker Engine

Voidbox Master

State Server

Container

Voidbox Proxy

Container

Voidbox Proxy

Docker Registry In S3

HDFS

HDFS

HDFS

Voidbox Driver

Pull

Voidbox modules

Docker modules

Voidbox is a standard appliction on top of YARN just like

MapReduce

server

server

server

server

bull Batch Jobs

bull Data Dump

bull Facematch distributed application face recognition in video

bull Long-running Services

bull Nesto WebServer

bull Spark HistoryServerJobServer

bull Kinko thumbnail Generator Service

VOIDBOX IN HULU

2

1

AGENDA

bull Background

bull Label-based scheduling

bull Frameworks On YARN

bull Experience

bull Q amp A

bull Container Description

bull Host rack

bull Relax locality

bull Blacklisted Container

bull ApplicationMaster could put certain nodes into blacklist then no more containers on those nodes will be received

bull Release unused containers

bull If AM allocates K containers more containers may be received

EXPERIENCE 1 CONTAINER API

2

3

bull Whatrsquos tracking url

bull URL of web server insideoutside AM

bull Jetty or netty

bull Two tracking URL

bull Register tracking url (running)

bull Unregister tracking url (completed)

bull URL Format

bull httphostport

bull Direct URL to ajsp use ldquoajsprdquo not ldquoajsprdquo ldquoabhtmlrdquo not ldquoabhtmlrdquo

EXPERIENCE 2 TRACKING URL

2

4

LEARN FROM LABEL-BASED SCHEDULING

bull Benefit

bull Staticdynamic machine-level partition

bull Elastic scalability for Apps

bull Latch to enter maintenance mode

bull Issue

bull Only Capacity Scheduler is supported

bull YARN Web Portal does not support well

bull Hurt data locality

AGENDA

bull Background

bull Label-based scheduling

bull Frameworks On YARN

bull Experience

bull Q amp A

bull MapReduce

bull Spark

bull Spark Streaming

bull Real-time processing

bull NestoPresto

bull MPP engines

bull Voidbox

bull Docker on yarn

FRAMEWORKS ON YARN

1

2

bull What is Nesto

bull A distributed query engine for parquet-like nested data

bull MPP + Parquet (On HDFS)

bull Nesto features

bull support large nested data(200+ columns 4 levels)

bull support filter groupby limit and map-side join

bull Support json expression and Programming API

bull use cases in hulu

bull Segment analysis

bull Cohort analysis

bull Funnel analysis

WHAT IS NESTO

1

3

bull Requirements

bull Run Specified N containers(servers) One container on node

bull Fault tolerance

bull Tracking web UI

bull Log rotation

bull Services (Never die)

bull Implementation

bull ApplicationMaster scheduling amp fault tolerance

bull Container NestoServer (Worker)

NESTO ON YARN (1)

1

4

Nesto Server

Nesto Server

Nesto Server

Zookeeper

NESTO ON YARN (2)

1

5

Zookeeper Resource Manager

Node Manager

Node Manager

Node Manager

Proxy

Nesto Server

AppMaster

Proxy

Nesto Server

Proxy

Nesto Server

Web server

yarn jar nesto-installnesto-010-SNAPSHOT-jar-with-dependenciesjar

comhulunestoyarnNestoYarnClient

-conf_path mntdata1homexichengdongnesto-srcyarnnesto-installconf

-framework_path hdfstmpxichengnestonesto-yarntargz

-master_memory 2048 -container_memory 2048 -container_vcores 1 -num_containers 3

bull Nesto deployment

bull Nesto is not a high-concurrency system

bull Mixed deployment with other batchOLAP systems on YARN

bull Slow taskworker

bull Slow task detection and re-dispatch

bull Worker priority scheduling based on weight

NESTO IN HETEROGENEOUS CLUSTER

1

6

WHAT IS VOIDBOX

1

7

LIGHTWEIGHT VM

RUNTIME ISOLATION

UNION FILE SYSTEM

RESOURCE MANAGEMENT

SCHEDULING

FAULT TOLERANCE

DISTRIBUTE COMPUTATION

ARBITRARY APPLICATION

FAULT TOLERANCE

VOIDBOX IS A PROGRAMMING FRAMEWORK

VOIDBOX SERVICE EXAMPLE

1

8

VOIDBOX COMPONENTS

API

RUNTIME MODE REGISTRYMGMT

VOIDBOX MASTER VOIDBOX PROXY

DAG SERVICE TASK

Core

Framework

VOIDBOX ARCHITECTURE

2

0

YARN modules

Resource Manager

Voidbox Client

NodeManager

NodeManager

NodeManager

Docker Engine

Docker Engine

Voidbox Master

State Server

Container

Voidbox Proxy

Container

Voidbox Proxy

Docker Registry In S3

HDFS

HDFS

HDFS

Voidbox Driver

Pull

Voidbox modules

Docker modules

Voidbox is a standard appliction on top of YARN just like

MapReduce

server

server

server

server

bull Batch Jobs

bull Data Dump

bull Facematch distributed application face recognition in video

bull Long-running Services

bull Nesto WebServer

bull Spark HistoryServerJobServer

bull Kinko thumbnail Generator Service

VOIDBOX IN HULU

2

1

AGENDA

bull Background

bull Label-based scheduling

bull Frameworks On YARN

bull Experience

bull Q amp A

bull Container Description

bull Host rack

bull Relax locality

bull Blacklisted Container

bull ApplicationMaster could put certain nodes into blacklist then no more containers on those nodes will be received

bull Release unused containers

bull If AM allocates K containers more containers may be received

EXPERIENCE 1 CONTAINER API

2

3

bull Whatrsquos tracking url

bull URL of web server insideoutside AM

bull Jetty or netty

bull Two tracking URL

bull Register tracking url (running)

bull Unregister tracking url (completed)

bull URL Format

bull httphostport

bull Direct URL to ajsp use ldquoajsprdquo not ldquoajsprdquo ldquoabhtmlrdquo not ldquoabhtmlrdquo

EXPERIENCE 2 TRACKING URL

2

4

AGENDA

bull Background

bull Label-based scheduling

bull Frameworks On YARN

bull Experience

bull Q amp A

bull MapReduce

bull Spark

bull Spark Streaming

bull Real-time processing

bull NestoPresto

bull MPP engines

bull Voidbox

bull Docker on yarn

FRAMEWORKS ON YARN

1

2

bull What is Nesto

bull A distributed query engine for parquet-like nested data

bull MPP + Parquet (On HDFS)

bull Nesto features

bull support large nested data(200+ columns 4 levels)

bull support filter groupby limit and map-side join

bull Support json expression and Programming API

bull use cases in hulu

bull Segment analysis

bull Cohort analysis

bull Funnel analysis

WHAT IS NESTO

1

3

bull Requirements

bull Run Specified N containers(servers) One container on node

bull Fault tolerance

bull Tracking web UI

bull Log rotation

bull Services (Never die)

bull Implementation

bull ApplicationMaster scheduling amp fault tolerance

bull Container NestoServer (Worker)

NESTO ON YARN (1)

1

4

Nesto Server

Nesto Server

Nesto Server

Zookeeper

NESTO ON YARN (2)

1

5

Zookeeper Resource Manager

Node Manager

Node Manager

Node Manager

Proxy

Nesto Server

AppMaster

Proxy

Nesto Server

Proxy

Nesto Server

Web server

yarn jar nesto-installnesto-010-SNAPSHOT-jar-with-dependenciesjar

comhulunestoyarnNestoYarnClient

-conf_path mntdata1homexichengdongnesto-srcyarnnesto-installconf

-framework_path hdfstmpxichengnestonesto-yarntargz

-master_memory 2048 -container_memory 2048 -container_vcores 1 -num_containers 3

bull Nesto deployment

bull Nesto is not a high-concurrency system

bull Mixed deployment with other batchOLAP systems on YARN

bull Slow taskworker

bull Slow task detection and re-dispatch

bull Worker priority scheduling based on weight

NESTO IN HETEROGENEOUS CLUSTER

1

6

WHAT IS VOIDBOX

1

7

LIGHTWEIGHT VM

RUNTIME ISOLATION

UNION FILE SYSTEM

RESOURCE MANAGEMENT

SCHEDULING

FAULT TOLERANCE

DISTRIBUTE COMPUTATION

ARBITRARY APPLICATION

FAULT TOLERANCE

VOIDBOX IS A PROGRAMMING FRAMEWORK

VOIDBOX SERVICE EXAMPLE

1

8

VOIDBOX COMPONENTS

API

RUNTIME MODE REGISTRYMGMT

VOIDBOX MASTER VOIDBOX PROXY

DAG SERVICE TASK

Core

Framework

VOIDBOX ARCHITECTURE

2

0

YARN modules

Resource Manager

Voidbox Client

NodeManager

NodeManager

NodeManager

Docker Engine

Docker Engine

Voidbox Master

State Server

Container

Voidbox Proxy

Container

Voidbox Proxy

Docker Registry In S3

HDFS

HDFS

HDFS

Voidbox Driver

Pull

Voidbox modules

Docker modules

Voidbox is a standard appliction on top of YARN just like

MapReduce

server

server

server

server

bull Batch Jobs

bull Data Dump

bull Facematch distributed application face recognition in video

bull Long-running Services

bull Nesto WebServer

bull Spark HistoryServerJobServer

bull Kinko thumbnail Generator Service

VOIDBOX IN HULU

2

1

AGENDA

bull Background

bull Label-based scheduling

bull Frameworks On YARN

bull Experience

bull Q amp A

bull Container Description

bull Host rack

bull Relax locality

bull Blacklisted Container

bull ApplicationMaster could put certain nodes into blacklist then no more containers on those nodes will be received

bull Release unused containers

bull If AM allocates K containers more containers may be received

EXPERIENCE 1 CONTAINER API

2

3

bull Whatrsquos tracking url

bull URL of web server insideoutside AM

bull Jetty or netty

bull Two tracking URL

bull Register tracking url (running)

bull Unregister tracking url (completed)

bull URL Format

bull httphostport

bull Direct URL to ajsp use ldquoajsprdquo not ldquoajsprdquo ldquoabhtmlrdquo not ldquoabhtmlrdquo

EXPERIENCE 2 TRACKING URL

2

4

bull MapReduce

bull Spark

bull Spark Streaming

bull Real-time processing

bull NestoPresto

bull MPP engines

bull Voidbox

bull Docker on yarn

FRAMEWORKS ON YARN

1

2

bull What is Nesto

bull A distributed query engine for parquet-like nested data

bull MPP + Parquet (On HDFS)

bull Nesto features

bull support large nested data(200+ columns 4 levels)

bull support filter groupby limit and map-side join

bull Support json expression and Programming API

bull use cases in hulu

bull Segment analysis

bull Cohort analysis

bull Funnel analysis

WHAT IS NESTO

1

3

bull Requirements

bull Run Specified N containers(servers) One container on node

bull Fault tolerance

bull Tracking web UI

bull Log rotation

bull Services (Never die)

bull Implementation

bull ApplicationMaster scheduling amp fault tolerance

bull Container NestoServer (Worker)

NESTO ON YARN (1)

1

4

Nesto Server

Nesto Server

Nesto Server

Zookeeper

NESTO ON YARN (2)

1

5

Zookeeper Resource Manager

Node Manager

Node Manager

Node Manager

Proxy

Nesto Server

AppMaster

Proxy

Nesto Server

Proxy

Nesto Server

Web server

yarn jar nesto-installnesto-010-SNAPSHOT-jar-with-dependenciesjar

comhulunestoyarnNestoYarnClient

-conf_path mntdata1homexichengdongnesto-srcyarnnesto-installconf

-framework_path hdfstmpxichengnestonesto-yarntargz

-master_memory 2048 -container_memory 2048 -container_vcores 1 -num_containers 3

bull Nesto deployment

bull Nesto is not a high-concurrency system

bull Mixed deployment with other batchOLAP systems on YARN

bull Slow taskworker

bull Slow task detection and re-dispatch

bull Worker priority scheduling based on weight

NESTO IN HETEROGENEOUS CLUSTER

1

6

WHAT IS VOIDBOX

1

7

LIGHTWEIGHT VM

RUNTIME ISOLATION

UNION FILE SYSTEM

RESOURCE MANAGEMENT

SCHEDULING

FAULT TOLERANCE

DISTRIBUTE COMPUTATION

ARBITRARY APPLICATION

FAULT TOLERANCE

VOIDBOX IS A PROGRAMMING FRAMEWORK

VOIDBOX SERVICE EXAMPLE

1

8

VOIDBOX COMPONENTS

API

RUNTIME MODE REGISTRYMGMT

VOIDBOX MASTER VOIDBOX PROXY

DAG SERVICE TASK

Core

Framework

VOIDBOX ARCHITECTURE

2

0

YARN modules

Resource Manager

Voidbox Client

NodeManager

NodeManager

NodeManager

Docker Engine

Docker Engine

Voidbox Master

State Server

Container

Voidbox Proxy

Container

Voidbox Proxy

Docker Registry In S3

HDFS

HDFS

HDFS

Voidbox Driver

Pull

Voidbox modules

Docker modules

Voidbox is a standard appliction on top of YARN just like

MapReduce

server

server

server

server

bull Batch Jobs

bull Data Dump

bull Facematch distributed application face recognition in video

bull Long-running Services

bull Nesto WebServer

bull Spark HistoryServerJobServer

bull Kinko thumbnail Generator Service

VOIDBOX IN HULU

2

1

AGENDA

bull Background

bull Label-based scheduling

bull Frameworks On YARN

bull Experience

bull Q amp A

bull Container Description

bull Host rack

bull Relax locality

bull Blacklisted Container

bull ApplicationMaster could put certain nodes into blacklist then no more containers on those nodes will be received

bull Release unused containers

bull If AM allocates K containers more containers may be received

EXPERIENCE 1 CONTAINER API

2

3

bull Whatrsquos tracking url

bull URL of web server insideoutside AM

bull Jetty or netty

bull Two tracking URL

bull Register tracking url (running)

bull Unregister tracking url (completed)

bull URL Format

bull httphostport

bull Direct URL to ajsp use ldquoajsprdquo not ldquoajsprdquo ldquoabhtmlrdquo not ldquoabhtmlrdquo

EXPERIENCE 2 TRACKING URL

2

4

bull What is Nesto

bull A distributed query engine for parquet-like nested data

bull MPP + Parquet (On HDFS)

bull Nesto features

bull support large nested data(200+ columns 4 levels)

bull support filter groupby limit and map-side join

bull Support json expression and Programming API

bull use cases in hulu

bull Segment analysis

bull Cohort analysis

bull Funnel analysis

WHAT IS NESTO

1

3

bull Requirements

bull Run Specified N containers(servers) One container on node

bull Fault tolerance

bull Tracking web UI

bull Log rotation

bull Services (Never die)

bull Implementation

bull ApplicationMaster scheduling amp fault tolerance

bull Container NestoServer (Worker)

NESTO ON YARN (1)

1

4

Nesto Server

Nesto Server

Nesto Server

Zookeeper

NESTO ON YARN (2)

1

5

Zookeeper Resource Manager

Node Manager

Node Manager

Node Manager

Proxy

Nesto Server

AppMaster

Proxy

Nesto Server

Proxy

Nesto Server

Web server

yarn jar nesto-installnesto-010-SNAPSHOT-jar-with-dependenciesjar

comhulunestoyarnNestoYarnClient

-conf_path mntdata1homexichengdongnesto-srcyarnnesto-installconf

-framework_path hdfstmpxichengnestonesto-yarntargz

-master_memory 2048 -container_memory 2048 -container_vcores 1 -num_containers 3

bull Nesto deployment

bull Nesto is not a high-concurrency system

bull Mixed deployment with other batchOLAP systems on YARN

bull Slow taskworker

bull Slow task detection and re-dispatch

bull Worker priority scheduling based on weight

NESTO IN HETEROGENEOUS CLUSTER

1

6

WHAT IS VOIDBOX

1

7

LIGHTWEIGHT VM

RUNTIME ISOLATION

UNION FILE SYSTEM

RESOURCE MANAGEMENT

SCHEDULING

FAULT TOLERANCE

DISTRIBUTE COMPUTATION

ARBITRARY APPLICATION

FAULT TOLERANCE

VOIDBOX IS A PROGRAMMING FRAMEWORK

VOIDBOX SERVICE EXAMPLE

1

8

VOIDBOX COMPONENTS

API

RUNTIME MODE REGISTRYMGMT

VOIDBOX MASTER VOIDBOX PROXY

DAG SERVICE TASK

Core

Framework

VOIDBOX ARCHITECTURE

2

0

YARN modules

Resource Manager

Voidbox Client

NodeManager

NodeManager

NodeManager

Docker Engine

Docker Engine

Voidbox Master

State Server

Container

Voidbox Proxy

Container

Voidbox Proxy

Docker Registry In S3

HDFS

HDFS

HDFS

Voidbox Driver

Pull

Voidbox modules

Docker modules

Voidbox is a standard appliction on top of YARN just like

MapReduce

server

server

server

server

bull Batch Jobs

bull Data Dump

bull Facematch distributed application face recognition in video

bull Long-running Services

bull Nesto WebServer

bull Spark HistoryServerJobServer

bull Kinko thumbnail Generator Service

VOIDBOX IN HULU

2

1

AGENDA

bull Background

bull Label-based scheduling

bull Frameworks On YARN

bull Experience

bull Q amp A

bull Container Description

bull Host rack

bull Relax locality

bull Blacklisted Container

bull ApplicationMaster could put certain nodes into blacklist then no more containers on those nodes will be received

bull Release unused containers

bull If AM allocates K containers more containers may be received

EXPERIENCE 1 CONTAINER API

2

3

bull Whatrsquos tracking url

bull URL of web server insideoutside AM

bull Jetty or netty

bull Two tracking URL

bull Register tracking url (running)

bull Unregister tracking url (completed)

bull URL Format

bull httphostport

bull Direct URL to ajsp use ldquoajsprdquo not ldquoajsprdquo ldquoabhtmlrdquo not ldquoabhtmlrdquo

EXPERIENCE 2 TRACKING URL

2

4

bull Requirements

bull Run Specified N containers(servers) One container on node

bull Fault tolerance

bull Tracking web UI

bull Log rotation

bull Services (Never die)

bull Implementation

bull ApplicationMaster scheduling amp fault tolerance

bull Container NestoServer (Worker)

NESTO ON YARN (1)

1

4

Nesto Server

Nesto Server

Nesto Server

Zookeeper

NESTO ON YARN (2)

1

5

Zookeeper Resource Manager

Node Manager

Node Manager

Node Manager

Proxy

Nesto Server

AppMaster

Proxy

Nesto Server

Proxy

Nesto Server

Web server

yarn jar nesto-installnesto-010-SNAPSHOT-jar-with-dependenciesjar

comhulunestoyarnNestoYarnClient

-conf_path mntdata1homexichengdongnesto-srcyarnnesto-installconf

-framework_path hdfstmpxichengnestonesto-yarntargz

-master_memory 2048 -container_memory 2048 -container_vcores 1 -num_containers 3

bull Nesto deployment

bull Nesto is not a high-concurrency system

bull Mixed deployment with other batchOLAP systems on YARN

bull Slow taskworker

bull Slow task detection and re-dispatch

bull Worker priority scheduling based on weight

NESTO IN HETEROGENEOUS CLUSTER

1

6

WHAT IS VOIDBOX

1

7

LIGHTWEIGHT VM

RUNTIME ISOLATION

UNION FILE SYSTEM

RESOURCE MANAGEMENT

SCHEDULING

FAULT TOLERANCE

DISTRIBUTE COMPUTATION

ARBITRARY APPLICATION

FAULT TOLERANCE

VOIDBOX IS A PROGRAMMING FRAMEWORK

VOIDBOX SERVICE EXAMPLE

1

8

VOIDBOX COMPONENTS

API

RUNTIME MODE REGISTRYMGMT

VOIDBOX MASTER VOIDBOX PROXY

DAG SERVICE TASK

Core

Framework

VOIDBOX ARCHITECTURE

2

0

YARN modules

Resource Manager

Voidbox Client

NodeManager

NodeManager

NodeManager

Docker Engine

Docker Engine

Voidbox Master

State Server

Container

Voidbox Proxy

Container

Voidbox Proxy

Docker Registry In S3

HDFS

HDFS

HDFS

Voidbox Driver

Pull

Voidbox modules

Docker modules

Voidbox is a standard appliction on top of YARN just like

MapReduce

server

server

server

server

bull Batch Jobs

bull Data Dump

bull Facematch distributed application face recognition in video

bull Long-running Services

bull Nesto WebServer

bull Spark HistoryServerJobServer

bull Kinko thumbnail Generator Service

VOIDBOX IN HULU

2

1

AGENDA

bull Background

bull Label-based scheduling

bull Frameworks On YARN

bull Experience

bull Q amp A

bull Container Description

bull Host rack

bull Relax locality

bull Blacklisted Container

bull ApplicationMaster could put certain nodes into blacklist then no more containers on those nodes will be received

bull Release unused containers

bull If AM allocates K containers more containers may be received

EXPERIENCE 1 CONTAINER API

2

3

bull Whatrsquos tracking url

bull URL of web server insideoutside AM

bull Jetty or netty

bull Two tracking URL

bull Register tracking url (running)

bull Unregister tracking url (completed)

bull URL Format

bull httphostport

bull Direct URL to ajsp use ldquoajsprdquo not ldquoajsprdquo ldquoabhtmlrdquo not ldquoabhtmlrdquo

EXPERIENCE 2 TRACKING URL

2

4

NESTO ON YARN (2)

1

5

Zookeeper Resource Manager

Node Manager

Node Manager

Node Manager

Proxy

Nesto Server

AppMaster

Proxy

Nesto Server

Proxy

Nesto Server

Web server

yarn jar nesto-installnesto-010-SNAPSHOT-jar-with-dependenciesjar

comhulunestoyarnNestoYarnClient

-conf_path mntdata1homexichengdongnesto-srcyarnnesto-installconf

-framework_path hdfstmpxichengnestonesto-yarntargz

-master_memory 2048 -container_memory 2048 -container_vcores 1 -num_containers 3

bull Nesto deployment

bull Nesto is not a high-concurrency system

bull Mixed deployment with other batchOLAP systems on YARN

bull Slow taskworker

bull Slow task detection and re-dispatch

bull Worker priority scheduling based on weight

NESTO IN HETEROGENEOUS CLUSTER

1

6

WHAT IS VOIDBOX

1

7

LIGHTWEIGHT VM

RUNTIME ISOLATION

UNION FILE SYSTEM

RESOURCE MANAGEMENT

SCHEDULING

FAULT TOLERANCE

DISTRIBUTE COMPUTATION

ARBITRARY APPLICATION

FAULT TOLERANCE

VOIDBOX IS A PROGRAMMING FRAMEWORK

VOIDBOX SERVICE EXAMPLE

1

8

VOIDBOX COMPONENTS

API

RUNTIME MODE REGISTRYMGMT

VOIDBOX MASTER VOIDBOX PROXY

DAG SERVICE TASK

Core

Framework

VOIDBOX ARCHITECTURE

2

0

YARN modules

Resource Manager

Voidbox Client

NodeManager

NodeManager

NodeManager

Docker Engine

Docker Engine

Voidbox Master

State Server

Container

Voidbox Proxy

Container

Voidbox Proxy

Docker Registry In S3

HDFS

HDFS

HDFS

Voidbox Driver

Pull

Voidbox modules

Docker modules

Voidbox is a standard appliction on top of YARN just like

MapReduce

server

server

server

server

bull Batch Jobs

bull Data Dump

bull Facematch distributed application face recognition in video

bull Long-running Services

bull Nesto WebServer

bull Spark HistoryServerJobServer

bull Kinko thumbnail Generator Service

VOIDBOX IN HULU

2

1

AGENDA

bull Background

bull Label-based scheduling

bull Frameworks On YARN

bull Experience

bull Q amp A

bull Container Description

bull Host rack

bull Relax locality

bull Blacklisted Container

bull ApplicationMaster could put certain nodes into blacklist then no more containers on those nodes will be received

bull Release unused containers

bull If AM allocates K containers more containers may be received

EXPERIENCE 1 CONTAINER API

2

3

bull Whatrsquos tracking url

bull URL of web server insideoutside AM

bull Jetty or netty

bull Two tracking URL

bull Register tracking url (running)

bull Unregister tracking url (completed)

bull URL Format

bull httphostport

bull Direct URL to ajsp use ldquoajsprdquo not ldquoajsprdquo ldquoabhtmlrdquo not ldquoabhtmlrdquo

EXPERIENCE 2 TRACKING URL

2

4

bull Nesto deployment

bull Nesto is not a high-concurrency system

bull Mixed deployment with other batchOLAP systems on YARN

bull Slow taskworker

bull Slow task detection and re-dispatch

bull Worker priority scheduling based on weight

NESTO IN HETEROGENEOUS CLUSTER

1

6

WHAT IS VOIDBOX

1

7

LIGHTWEIGHT VM

RUNTIME ISOLATION

UNION FILE SYSTEM

RESOURCE MANAGEMENT

SCHEDULING

FAULT TOLERANCE

DISTRIBUTE COMPUTATION

ARBITRARY APPLICATION

FAULT TOLERANCE

VOIDBOX IS A PROGRAMMING FRAMEWORK

VOIDBOX SERVICE EXAMPLE

1

8

VOIDBOX COMPONENTS

API

RUNTIME MODE REGISTRYMGMT

VOIDBOX MASTER VOIDBOX PROXY

DAG SERVICE TASK

Core

Framework

VOIDBOX ARCHITECTURE

2

0

YARN modules

Resource Manager

Voidbox Client

NodeManager

NodeManager

NodeManager

Docker Engine

Docker Engine

Voidbox Master

State Server

Container

Voidbox Proxy

Container

Voidbox Proxy

Docker Registry In S3

HDFS

HDFS

HDFS

Voidbox Driver

Pull

Voidbox modules

Docker modules

Voidbox is a standard appliction on top of YARN just like

MapReduce

server

server

server

server

bull Batch Jobs

bull Data Dump

bull Facematch distributed application face recognition in video

bull Long-running Services

bull Nesto WebServer

bull Spark HistoryServerJobServer

bull Kinko thumbnail Generator Service

VOIDBOX IN HULU

2

1

AGENDA

bull Background

bull Label-based scheduling

bull Frameworks On YARN

bull Experience

bull Q amp A

bull Container Description

bull Host rack

bull Relax locality

bull Blacklisted Container

bull ApplicationMaster could put certain nodes into blacklist then no more containers on those nodes will be received

bull Release unused containers

bull If AM allocates K containers more containers may be received

EXPERIENCE 1 CONTAINER API

2

3

bull Whatrsquos tracking url

bull URL of web server insideoutside AM

bull Jetty or netty

bull Two tracking URL

bull Register tracking url (running)

bull Unregister tracking url (completed)

bull URL Format

bull httphostport

bull Direct URL to ajsp use ldquoajsprdquo not ldquoajsprdquo ldquoabhtmlrdquo not ldquoabhtmlrdquo

EXPERIENCE 2 TRACKING URL

2

4

WHAT IS VOIDBOX

1

7

LIGHTWEIGHT VM

RUNTIME ISOLATION

UNION FILE SYSTEM

RESOURCE MANAGEMENT

SCHEDULING

FAULT TOLERANCE

DISTRIBUTE COMPUTATION

ARBITRARY APPLICATION

FAULT TOLERANCE

VOIDBOX IS A PROGRAMMING FRAMEWORK

VOIDBOX SERVICE EXAMPLE

1

8

VOIDBOX COMPONENTS

API

RUNTIME MODE REGISTRYMGMT

VOIDBOX MASTER VOIDBOX PROXY

DAG SERVICE TASK

Core

Framework

VOIDBOX ARCHITECTURE

2

0

YARN modules

Resource Manager

Voidbox Client

NodeManager

NodeManager

NodeManager

Docker Engine

Docker Engine

Voidbox Master

State Server

Container

Voidbox Proxy

Container

Voidbox Proxy

Docker Registry In S3

HDFS

HDFS

HDFS

Voidbox Driver

Pull

Voidbox modules

Docker modules

Voidbox is a standard appliction on top of YARN just like

MapReduce

server

server

server

server

bull Batch Jobs

bull Data Dump

bull Facematch distributed application face recognition in video

bull Long-running Services

bull Nesto WebServer

bull Spark HistoryServerJobServer

bull Kinko thumbnail Generator Service

VOIDBOX IN HULU

2

1

AGENDA

bull Background

bull Label-based scheduling

bull Frameworks On YARN

bull Experience

bull Q amp A

bull Container Description

bull Host rack

bull Relax locality

bull Blacklisted Container

bull ApplicationMaster could put certain nodes into blacklist then no more containers on those nodes will be received

bull Release unused containers

bull If AM allocates K containers more containers may be received

EXPERIENCE 1 CONTAINER API

2

3

bull Whatrsquos tracking url

bull URL of web server insideoutside AM

bull Jetty or netty

bull Two tracking URL

bull Register tracking url (running)

bull Unregister tracking url (completed)

bull URL Format

bull httphostport

bull Direct URL to ajsp use ldquoajsprdquo not ldquoajsprdquo ldquoabhtmlrdquo not ldquoabhtmlrdquo

EXPERIENCE 2 TRACKING URL

2

4

VOIDBOX SERVICE EXAMPLE

1

8

VOIDBOX COMPONENTS

API

RUNTIME MODE REGISTRYMGMT

VOIDBOX MASTER VOIDBOX PROXY

DAG SERVICE TASK

Core

Framework

VOIDBOX ARCHITECTURE

2

0

YARN modules

Resource Manager

Voidbox Client

NodeManager

NodeManager

NodeManager

Docker Engine

Docker Engine

Voidbox Master

State Server

Container

Voidbox Proxy

Container

Voidbox Proxy

Docker Registry In S3

HDFS

HDFS

HDFS

Voidbox Driver

Pull

Voidbox modules

Docker modules

Voidbox is a standard appliction on top of YARN just like

MapReduce

server

server

server

server

bull Batch Jobs

bull Data Dump

bull Facematch distributed application face recognition in video

bull Long-running Services

bull Nesto WebServer

bull Spark HistoryServerJobServer

bull Kinko thumbnail Generator Service

VOIDBOX IN HULU

2

1

AGENDA

bull Background

bull Label-based scheduling

bull Frameworks On YARN

bull Experience

bull Q amp A

bull Container Description

bull Host rack

bull Relax locality

bull Blacklisted Container

bull ApplicationMaster could put certain nodes into blacklist then no more containers on those nodes will be received

bull Release unused containers

bull If AM allocates K containers more containers may be received

EXPERIENCE 1 CONTAINER API

2

3

bull Whatrsquos tracking url

bull URL of web server insideoutside AM

bull Jetty or netty

bull Two tracking URL

bull Register tracking url (running)

bull Unregister tracking url (completed)

bull URL Format

bull httphostport

bull Direct URL to ajsp use ldquoajsprdquo not ldquoajsprdquo ldquoabhtmlrdquo not ldquoabhtmlrdquo

EXPERIENCE 2 TRACKING URL

2

4

VOIDBOX COMPONENTS

API

RUNTIME MODE REGISTRYMGMT

VOIDBOX MASTER VOIDBOX PROXY

DAG SERVICE TASK

Core

Framework

VOIDBOX ARCHITECTURE

2

0

YARN modules

Resource Manager

Voidbox Client

NodeManager

NodeManager

NodeManager

Docker Engine

Docker Engine

Voidbox Master

State Server

Container

Voidbox Proxy

Container

Voidbox Proxy

Docker Registry In S3

HDFS

HDFS

HDFS

Voidbox Driver

Pull

Voidbox modules

Docker modules

Voidbox is a standard appliction on top of YARN just like

MapReduce

server

server

server

server

bull Batch Jobs

bull Data Dump

bull Facematch distributed application face recognition in video

bull Long-running Services

bull Nesto WebServer

bull Spark HistoryServerJobServer

bull Kinko thumbnail Generator Service

VOIDBOX IN HULU

2

1

AGENDA

bull Background

bull Label-based scheduling

bull Frameworks On YARN

bull Experience

bull Q amp A

bull Container Description

bull Host rack

bull Relax locality

bull Blacklisted Container

bull ApplicationMaster could put certain nodes into blacklist then no more containers on those nodes will be received

bull Release unused containers

bull If AM allocates K containers more containers may be received

EXPERIENCE 1 CONTAINER API

2

3

bull Whatrsquos tracking url

bull URL of web server insideoutside AM

bull Jetty or netty

bull Two tracking URL

bull Register tracking url (running)

bull Unregister tracking url (completed)

bull URL Format

bull httphostport

bull Direct URL to ajsp use ldquoajsprdquo not ldquoajsprdquo ldquoabhtmlrdquo not ldquoabhtmlrdquo

EXPERIENCE 2 TRACKING URL

2

4

VOIDBOX ARCHITECTURE

2

0

YARN modules

Resource Manager

Voidbox Client

NodeManager

NodeManager

NodeManager

Docker Engine

Docker Engine

Voidbox Master

State Server

Container

Voidbox Proxy

Container

Voidbox Proxy

Docker Registry In S3

HDFS

HDFS

HDFS

Voidbox Driver

Pull

Voidbox modules

Docker modules

Voidbox is a standard appliction on top of YARN just like

MapReduce

server

server

server

server

bull Batch Jobs

bull Data Dump

bull Facematch distributed application face recognition in video

bull Long-running Services

bull Nesto WebServer

bull Spark HistoryServerJobServer

bull Kinko thumbnail Generator Service

VOIDBOX IN HULU

2

1

AGENDA

bull Background

bull Label-based scheduling

bull Frameworks On YARN

bull Experience

bull Q amp A

bull Container Description

bull Host rack

bull Relax locality

bull Blacklisted Container

bull ApplicationMaster could put certain nodes into blacklist then no more containers on those nodes will be received

bull Release unused containers

bull If AM allocates K containers more containers may be received

EXPERIENCE 1 CONTAINER API

2

3

bull Whatrsquos tracking url

bull URL of web server insideoutside AM

bull Jetty or netty

bull Two tracking URL

bull Register tracking url (running)

bull Unregister tracking url (completed)

bull URL Format

bull httphostport

bull Direct URL to ajsp use ldquoajsprdquo not ldquoajsprdquo ldquoabhtmlrdquo not ldquoabhtmlrdquo

EXPERIENCE 2 TRACKING URL

2

4

bull Batch Jobs

bull Data Dump

bull Facematch distributed application face recognition in video

bull Long-running Services

bull Nesto WebServer

bull Spark HistoryServerJobServer

bull Kinko thumbnail Generator Service

VOIDBOX IN HULU

2

1

AGENDA

bull Background

bull Label-based scheduling

bull Frameworks On YARN

bull Experience

bull Q amp A

bull Container Description

bull Host rack

bull Relax locality

bull Blacklisted Container

bull ApplicationMaster could put certain nodes into blacklist then no more containers on those nodes will be received

bull Release unused containers

bull If AM allocates K containers more containers may be received

EXPERIENCE 1 CONTAINER API

2

3

bull Whatrsquos tracking url

bull URL of web server insideoutside AM

bull Jetty or netty

bull Two tracking URL

bull Register tracking url (running)

bull Unregister tracking url (completed)

bull URL Format

bull httphostport

bull Direct URL to ajsp use ldquoajsprdquo not ldquoajsprdquo ldquoabhtmlrdquo not ldquoabhtmlrdquo

EXPERIENCE 2 TRACKING URL

2

4

AGENDA

bull Background

bull Label-based scheduling

bull Frameworks On YARN

bull Experience

bull Q amp A

bull Container Description

bull Host rack

bull Relax locality

bull Blacklisted Container

bull ApplicationMaster could put certain nodes into blacklist then no more containers on those nodes will be received

bull Release unused containers

bull If AM allocates K containers more containers may be received

EXPERIENCE 1 CONTAINER API

2

3

bull Whatrsquos tracking url

bull URL of web server insideoutside AM

bull Jetty or netty

bull Two tracking URL

bull Register tracking url (running)

bull Unregister tracking url (completed)

bull URL Format

bull httphostport

bull Direct URL to ajsp use ldquoajsprdquo not ldquoajsprdquo ldquoabhtmlrdquo not ldquoabhtmlrdquo

EXPERIENCE 2 TRACKING URL

2

4

bull Container Description

bull Host rack

bull Relax locality

bull Blacklisted Container

bull ApplicationMaster could put certain nodes into blacklist then no more containers on those nodes will be received

bull Release unused containers

bull If AM allocates K containers more containers may be received

EXPERIENCE 1 CONTAINER API

2

3

bull Whatrsquos tracking url

bull URL of web server insideoutside AM

bull Jetty or netty

bull Two tracking URL

bull Register tracking url (running)

bull Unregister tracking url (completed)

bull URL Format

bull httphostport

bull Direct URL to ajsp use ldquoajsprdquo not ldquoajsprdquo ldquoabhtmlrdquo not ldquoabhtmlrdquo

EXPERIENCE 2 TRACKING URL

2

4

bull Whatrsquos tracking url

bull URL of web server insideoutside AM

bull Jetty or netty

bull Two tracking URL

bull Register tracking url (running)

bull Unregister tracking url (completed)

bull URL Format

bull httphostport

bull Direct URL to ajsp use ldquoajsprdquo not ldquoajsprdquo ldquoabhtmlrdquo not ldquoabhtmlrdquo

EXPERIENCE 2 TRACKING URL

2

4

bull AM or Container run command

bull $JAVA_HOMEbinjava -Dnestoservercontainerlog=ltLOG_DIRgt -Dlog4jconfiguration=log4jproperties bdquo comhuluNestoServer 1gtgtltLOG_DIRgtserverlog 2gtgtltLOG_DIRgtserverlog

bull ltLOG_DIRgt will be replaced by YARN framework

bull log4jproperties

EXPERIENCE 3 LOG ROTATION

2

5

bull Allocate more memory than JVM heap size

EXPERIENCE 4 MEMORY OVERHEAD

2

6

Java JVM

Non-Java memory

Java Server

Container Memory

Hadoop Streaming is a good example

bull Understand container environment

bull AMcontainer command is written to a shell script

bull How to check the shell script content

bull Register tracking url (running)

EXPERIENCE 5 DEBUG

2

7

Q amp A

28

bull Allocate more memory than JVM heap size

EXPERIENCE 4 MEMORY OVERHEAD

2

6

Java JVM

Non-Java memory

Java Server

Container Memory

Hadoop Streaming is a good example

bull Understand container environment

bull AMcontainer command is written to a shell script

bull How to check the shell script content

bull Register tracking url (running)

EXPERIENCE 5 DEBUG

2

7

Q amp A

28

bull Understand container environment

bull AMcontainer command is written to a shell script

bull How to check the shell script content

bull Register tracking url (running)

EXPERIENCE 5 DEBUG

2

7

Q amp A

28

Q amp A

28