Joey Lynchjlynch@yelp.comData Shepherd

Automating Datastore Fleets with Puppet

Yelp’s MissionConnecting people with great

local businesses.

Yelp StatsAs of Q2 2016

92M 3272%108M

jlynch StatsAs of Q2 2016

Yelp 4+ years

Datastore Whisperer

Distributed Systems Addict

Yelp Polyglot Stack

Presentation Logic Data

But Why?

Logic

Web Search Ads API Logging Svc 5 Platform

Data

MySQL Lucene Cass ... Kafka DB 5 Zookeeper

...

...

Logic

Web Search Ads API Logging Svc 5 Platform

Data

MySQL Lucene Cass ... Kafka DB 5 Zookeeper

...

...

It’s Complicated

MUST support● Different datastores● Different configurations● Cloud and datacenter ready● Ready for production● Cope with Data Gravity

Yelp’s Datastore Problem

Yelp’s Datastore Solution

Different Datastores

Cluster Modules

Wrap the vendor modules

Manage cluster wide config

Different Configurations

Configure

● Functions help● Hierarchical config

is even better

Configure - Hierarchy

Configure - Hierarchy

Configure - Hierarchy

Configure - Adding a Cluster

Configure - Adding a Cluster

Configure - Flexible

Configure - Flexible

Configure - Flexible

Functions● No hard coding!● Changes with infra● Sometimes as

easy as $::processorcount

Cloud and Datacenter Ready

Datacenter

Datacenter

Cloud

Cloud

ENC is very nice

ENC is very nice

Production Ready

Using Puppet to Manage Datastores

● About 100 clusters● Over 500 datastore nodes● Puppet runs take ~5 minutes● Run Puppet every hour, CI/CD

○ Automation runs via separate scheduler

Monitor

Monitor Clusters

Monitor Clusters

https://engineeringblog.yelp.com/2016/06/monitoring-cassandra-at-scale.html

Monitor Clusters

https://engineeringblog.yelp.com/2016/06/monitoring-cassandra-at-scale.html

Totally fine maybeBAD

cron

KISS

DatabaseClient

configure_nerveconfigure_synapse

JSON files hacheck

ZookeeperSynapse Nerve

Healthchecks

ConfigurationConfiguration

Configuration

Healthchecks

Registration

Service Discovery

Service Discovery

Service Discovery

DatabaseClient

configure_nerveconfigure_synapse

JSON files hacheck

ZookeeperSynapse Nerve

Healthchecks

ConfigurationConfiguration

Configuration

Healthchecks

Registration

Service Discovery

Data Gravity

Autoscale!

Autoscale!

● Debian package

● Cron

Backup!

● Another Debian package

● Cron

Recap

Using Puppet to Manage Datastores

Using Puppet to Manage Datastores

cronMonitorDiscoverAutomate

Puppet enables automation

@YelpEngineering

fb.com/YelpEngineers

engineeringblog.yelp.com

github.com/yelp

● Server rack, unknown authorhttp://www.clipartkid.com/server-rack-cabinet-clip-art-at-clker-com-vector-clip-art-online-IPTtWc-clipart/

● Devops venn diagram, Kharnagyhttps://commons.wikimedia.org/wiki/File:Devops.svg

● Steve Jobs, waldrynohttps://pixabay.com/en/steve-jobs-technology-illustration-1249665/

Image Citations