25
Failing at Scale Chris Tankersley @dragonmantank Pacific Northwest PHP, September 2016 Pacific Northwest PHP, September 2016 1

Failing at Scale - PNWPHP 2016

Embed Size (px)

Citation preview

Page 1: Failing at Scale - PNWPHP 2016

Pacific Northwest PHP, September 2016 1

Failing at ScaleChris Tankersley@dragonmantankPacific Northwest PHP, September 2016

Page 2: Failing at Scale - PNWPHP 2016

Pacific Northwest PHP, September 2016 2

I think something is wrong

Page 3: Failing at Scale - PNWPHP 2016

Pacific Northwest PHP, September 2016 3

Problems with Distributed Applications• Making sure that configurations are consistent• Making sure that your code gets updated everywhere• Actually finding errors

Page 4: Failing at Scale - PNWPHP 2016

Pacific Northwest PHP, September 2016 4

I know this is a bug, but where?

Page 5: Failing at Scale - PNWPHP 2016

Pacific Northwest PHP, September 2016 5

You Can’t Rely on Physical Files• Servers/Containers Disappear• How do you know where something happened?

Page 6: Failing at Scale - PNWPHP 2016

Pacific Northwest PHP, September 2016 6

syslog

Your Server

PHP Application

Monolog

syslog

Centralized syslog Server

syslog

Page 7: Failing at Scale - PNWPHP 2016

Pacific Northwest PHP, September 2016 7

Set up the Server# /etc/init.d/syslog.conf, or something similiar

# Make sure the following is setSYSLOGD="-r -m0"

Page 8: Failing at Scale - PNWPHP 2016

Pacific Northwest PHP, September 2016 8

Set up the server# /etc/syslog.conf, or something similiar

# Set up the log file locations*.emerg /var/log/emergency.log*.alert /var/log/alert.log

# Or lump them together*.emerg /var/log/application.log*.alert /var/log/application.log

Page 9: Failing at Scale - PNWPHP 2016

Pacific Northwest PHP, September 2016 9

Set up the clients# /etc/syslog.conf, or something similiar

# Send all log types of all log severities to a # single server*.* @<hostname|IP Address>

Page 10: Failing at Scale - PNWPHP 2016

Pacific Northwest PHP, September 2016 10

Logging SaaS• Same idea as a remote syslog• Usually installs some log watching software• Pretty interface for searching/looking at logs

Page 11: Failing at Scale - PNWPHP 2016

Pacific Northwest PHP, September 2016 11

Loggly

Page 12: Failing at Scale - PNWPHP 2016

Pacific Northwest PHP, September 2016 12

ELK Stack• ElasticSearch, LogStash, Kibana• Can run locally or through a SaaS service like logz.io

Page 13: Failing at Scale - PNWPHP 2016

Pacific Northwest PHP, September 2016 13

Logz.io

Page 14: Failing at Scale - PNWPHP 2016

Pacific Northwest PHP, September 2016 14

But I swear I fixed that bug!

Page 15: Failing at Scale - PNWPHP 2016

Pacific Northwest PHP, September 2016 15

Don’t Trust Humans

We Suck

Page 16: Failing at Scale - PNWPHP 2016

Pacific Northwest PHP, September 2016 16

Anything you do by hand, Automate

Page 17: Failing at Scale - PNWPHP 2016

Pacific Northwest PHP, September 2016 17

Automating the Build Process• Composer• Grunt/Gulp/npm/asset pipelines• Putting proper config files in place

Page 18: Failing at Scale - PNWPHP 2016

Pacific Northwest PHP, September 2016 18

Track Everything

Page 19: Failing at Scale - PNWPHP 2016

Pacific Northwest PHP, September 2016 19

Automating Code Pushes

Page 20: Failing at Scale - PNWPHP 2016

Pacific Northwest PHP, September 2016 20

I’m pretty sure we have PHP… 5.6 installed?

Page 21: Failing at Scale - PNWPHP 2016

Pacific Northwest PHP, September 2016 21

Remember, Don’t Trust Humans!

Page 22: Failing at Scale - PNWPHP 2016

Pacific Northwest PHP, September 2016 22

Ansible• Self contained and can be run from anywhere• Very easy to get set up and running• Lots of “playbooks”

Page 23: Failing at Scale - PNWPHP 2016

Pacific Northwest PHP, September 2016 23

Puppet• Master-Agent Setup• Constantly checks to make sure machines are compliant• You might be using it already!

Page 24: Failing at Scale - PNWPHP 2016

Pacific Northwest PHP, September 2016 24

Vagrant Cookbook

Page 25: Failing at Scale - PNWPHP 2016

Pacific Northwest PHP, September 2016 25

Thank You!• https://github.com/dragonmantank• Author of “Docker for Developers”

• https://leanpub.com/dockerfordevs

• http://ctankersley.com• [email protected]• @dragonmantank

• https://joind.in/talk/64ad5