Clustered Services with Apache Whirr: From Ops Down

Chad Metcalf@metcalfcCloudera

The Big Picture

6 Node Hadoop Cluster

# Change the cluster name herewhirr.cluster-name=hadoop

# Change the number of machines in the cluster herewhirr.instance-templates=1 hadoop-namenode+hadoop-jobtracker,5 hadoop-datanode+hadoop-tasktracker

What Does It Do?

How?

Why Use It?

Developer Way

Operator Way

Fortify with Puppet

+puppet:ntp+puppet:pig+puppet:hive+puppet:demo::users

Modules

# Gitpuppet.pig.module= git://github.com/metcalfc/puppet-pig.git

# With branchpuppet.ntp.module= git://github.com/metcalfc/puppet-ntp.gitpuppet.ntp.module.branch=puppetconf2011

# Tarballpuppet.demo.module=/tmp/demo.tgz

Manifests

# Default+puppet:pig

# Specific Class+puppet:demo::users

# Parameterized Class+puppet:ntpntp.servers=[ '0.pool.ntp.org' ]ntp.autoupdate=true

puppet apply

node default {

class {'ntp': servers => [ '0.pool.ntp.org' ], autoupdate => true, } class { 'pig': } class { 'hive': } class { 'demo::users': }}

Masterless

One Shot

Test Locally

Demo

Whats Next?

Questions?