HDP2 and YARN operations point

Copyright ©2015 Treasure Data. All Rights Reserved.

HDP2 and

YARN operations pointRyu Kobayashi

Treasure Data Tech Talk 11 and 12 Mar 2015


Who am I?• Ryu Kobayashi • @ryu_kobayashi • https://github.com/ryukobayashi

• Treasure Data, Inc. • Software Engineer

• Background • Hadoop, Cassandra, Machine Learning, ... • I developed Huahin(Hadoop) Framework.

http://huahinframework.org/


What is YARN?


YARN(Yet Another Resource Negotiator) Architecture


• MRv1

• JobTracker

• TaskTracker


• YARN

• ResourceManager

• NodeManager

• ApplicationMaster

• Job History Server

• YARN Timeline Server


• MRv1

• JobTracker

• TaskTracker

• YARN

• ResourceManager

• NodeManager

• ApplicationMaster

• Job History Server (We can not see the log job history If it do not install)

• YARN Timeline Server (We can not see the log YARN history If it do not install)


YARN Timeline Server • It is included container info


Note!!!


Use the Hadoop 2.4.0 and later!!!


• The versions which must not be used

• Apache Hadoop 2.2.0


• HDP 2.0(2.2.0 based)


• Currently


• CDH 5.3.2(2.5.0 based and patch)

• HDP 2.2(2.6.0 based and patch)


• Why should not use?

• Capacity Scheduler

• There is a bug

• Fair Scheduler

• There is a bug


• Any bugs?

• Each Scheduler will cause a deadlock


•In fact, there is a bug in 2.4.0 and 2.6.0…

•It is better to use the new version.

•Note: 2.7.0 and later is a different thing


Backport Patch

• I was backport some patch

• https://github.com/ryukobayashi/patches

https://github.com/ryukobayashi/patches


Backport Patch

• Included dead lock patch

• Format of the counter

• Application kill in Web UI.


Format of the counter


Format of the counter


Application kill in Web UI


Application kill in Web UI


Application kill in Web UI • Job kill in Web UI •

(default false)

• Application kill in Web UI •

(default true)

mapreduce.jobtracker.webinterface.trusted

yarn.resourcemanager.webapp.ui-actions.enabled


Backport Patch

• We want the next…

• Job task attempt kill in Web UI patch (in development)

• Currently, only command line

$ mapred job -kill-task attempt_*


Matter of resources

• total container = 4

• concurrent application = 2


Matter of resources • total container = 4


Cluster Application

App Master Container

Application



Matter of resources

• total container = 4





Cluster Application

App Master

Application

App Master

Application

App Master

Application

App Master




Cluster Application

App Master

Application

App Master

Application

App Master

Application

App Master




Cluster Application

App Master

Application

App Master

Application

App Master

Application

App Master

Livelock!




Cluster Application

App Master

Application

App Master

Application

App Master

Container

Application

App Master

Kill




• ^ squeeze the number of applications




• ^ squeeze the number of applications

• set the root maxRunningApps




• root maxRunningApps = 2

Cluster Application


Application

App Master

Application


Application

App Master

Pending


YANR Resource Managementyarn-site.xml

• yarn.nodemanager.resource.memory-mb • (yarn.nodenamager.vmem-pmem-ratio) • (yarn.scheduler.minimum-allocation-mb)

mapred-site.xml • yarn.app.mapreduce.am.resource.mb • mapreduce.map.memory.mb • mapreduce.reduce.memory.mb

fair-scheduler.xml • maxResources, minResources

etc…


e.g. Use hdp-configuration-utils.py script http://goo.gl/L2hxyq

Use Ambari http://ambari.apache.org/

See the Cloudera’s document http://goo.gl/EBreca

YANR Resource Management

http://goo.gl/L2hxyq

http://ambari.apache.org/

http://goo.gl/EBreca


Thanks!!!

Engineering

HDP2 and YARN operations point