Large-Scale Deployments With Pogo At Yahoo! Mike Schilli YAPC::NA 06/14/2012

Large-Scale Deployments With Pogo At Yahoo!

Mike SchilliYAPC::NA 06/14/2012

What’s Pogo?

• Yahoo’s deployment system– Highly scalable (thousands of targets)– Highly parallelized rollouts– In active use for several years

• Open Source version available (pre-alpha)

Pogo in a CI environment

What’s “Deployment”?

• Run one command on many hosts• Each host “knows” what it’s supposed to look

like via a central database• One command brings host into “desired” state• Fetches package updates from edge-server

repos

Run one command on many hosts

• Act on all hosts immediately

Run one command on many hosts

• Act on all hosts immediately

Example Server Farm

Pogo Features

• Sequences• Parallelizes with constraints• Halts if errors violate constraints• Runs health checks (pre/post hooks)• Command agnostic (rpm, apt-get, custom,

etc.)

Server Farm Release via Pogo











Pogo Configuration

Run a Pogo job (tags)

Pogo UI – Single Job View

Pogo UI Overview

Pogo UI – Host View

Pogo UI – Start it Up

$ perl -Ilib bin/pogo-apipogo-api-28> Listening on 0.0.0.0:7657

$ perl -Ilib bin/pogo-ui -vpogo-ui-22> Listening on port 5000

Run a Pogo job (targets)

Pogo Configuration

State of Pogo

• Pre-alpha Open Source version on Github

• OSS Pogo will be completed in the coming months

Pogo on Github

• Developed in the open:– https://github.com/ytoolshed/pogo

• Internally used with plugins to adapt to specific data sources

CI Tests with travis-ci.org

The Gory Details

Pogo – User View

Pogo Workflow

Authentication/Authorization

• Web server authentication for submitting jobs• Target authentication via passwords or

priv/pub keys• Credentials encrypted with worker pubkey• Credentials never stored persistently (or

ZooKeeper)

Pogo “Hooks”

• Scripts that run before or after Pogo commands

• Used to prepare host, perform check health, or other functions

• Installed locally on target hosts• rc.d style invocation: run in alphanumeric

order

Pogo “Hooks” (cont.)

• Two types:– pre hook: run before the specified command– post hook: run after the specified command

• If any hook script fails:– No further scripts or commands run– Host is marked as failed

Pre-hook Examples

• Take a host out of rotation• Drain database connections• Check that host is ready to be updated

Post-hook Examples

• Check whether local webserver is serving expected content

• Ping required services• Put host back into rotation

Host Failures and Concurrency

• If a host fails:– Command returns non-zero– Pre- or post-hook returns non-zero

• The failed host counts against the acceptable number/percent of unavailable hosts

• Example:– Four hosts in an app can run at a time– One host in the app fails– Only three hosts will be acted on at a time going

forward

Implementation

• AnyEvent Framework• Asynchronous, event-based• Added Object::Event for event pub/sub

HTTP Client in AnyEvent



AnyEvent and Object::Event



Queue Processor with AnyEvent and Object::Event

AnyEvent

bin/pogo-dispatcher

Testing AnyEvent Components

All Pogo Components

All-In-One Pogo: pogo-one



Questions?

Thanks!

• Pogo on Github: – http://github.com/ytoolshed/pogo

http://github.com/ytoolshed/pogo


Lessons learned along the Way

Set up Unit Tests with Travis-ci.org

Runs immediately after Github checkins

CI Tests with travis-ci.org

Configure CI with travis-ci.org

Select Github project

Authorize travis-ci.org

Travis-ci.org API Calls

Filling in Passwords


• Test logging into machines without sshkeys• Convenient for testing• Answer to sudo prompts on targets• Answer to prompts for encrypted packages


• Good password prompt:

mschilli@localhost's password:

• Not password prompt!

… user ''@'localhost' (using password: NO)

PasswordMonkey on CPAN

Plack

Plack

Plack/PSGI

• Plack used for server-independent web application– API– Dispatcher Control Port

Alternatives

• Capistrano• Rundeck• Aegir• Drush

Pogo UI – Start it Up

$ perl -Ilib bin/pogo-apipogo-api-28> Listening on 0.0.0.0:7657

$ perl -Ilib bin/pogo-ui -vpogo-ui-22> Listening on port 5000

Pogo UI

• All Cient Side-Javascript• Server just redirects to index.html– Except assets/* js/*

Thanks!

• Pogo on Github: – http://github.com/ytoolshed/pogo

• PasswordMonkey CPAN Module:– http://search.cpan.org/dist/PasswordMonkey/

• Travis CI Service: – http://travis-ci.org


http://search.cpan.org/dist/PasswordMonkey/

http://search.cpan.org/dist/PasswordMonkey/

Documents

Large-Scale Deployments With Pogo At Yahoo! Mike Schilli YAPC::NA 06/14/2012