LEVERAGING AUTOMATION TO SCALE
JIM HOPP, LOOKOUT, INC.
HOW TO KEEP UP?
• Increasing number of services
• And desire for faster cycle time
• Increasing number of machines
• Frequent deploys (and getting more frequent)
• Engineering growing much faster than Ops
(TESTING?!)
• Important that provisioning chgs follow modern dev pipeline
• rspec (chefspec) for chef
• minitest for chef’d nodes
• code reviews
• deploys from jenkins
• continually re-provision bare-metal test nodes
GET OTHERS TO DO THE WORK
• We want engineers to write the chef cookbooks and cap config files for their service
• We want them to use the production tools during the development cycle
• Originally built vagrant-based tools. Worked well but fragile and didn’t scale past 2-3 VMs
• Developed TestLab for managing Linux containers. Works great! https://github.com/zpatten/testlab
MAKE IT SIMPLE TO USE
• Built Chef LWRPs to simplify configuration
• This is all that’s necessary for a normal service:
• LWRPs for resque, firewall, SSL certs, etc.
• Libraries for finding IPs, data bags within environments, etc.
• Office hours, pairing
lookout_passenger_app 'my_service' do rvm_gemset 'ruby-1.9.3-p448@my_service'end
DEPLOYS
• chef-repo deployed to prod on merge to master by jenkins
• Deploy HEAD
• For services, moving to developer deploy
• They know their service better than Ops
• Still figuring out migrations
• Tooling for roll-back
Keep in touch with
@lookout
/mylookout
blog.lookout.com
http://bit.ly/scaling-for-mobile
Recommended
