Upload
awilmore
View
153
Download
0
Embed Size (px)
DESCRIPTION
What happens when a simple Java developer tries to do DevOps.
Citation preview
Diving InSee what happens when a Java developer
tries to do DevOps
Copper @ an Aussie Insurance Company@awilmore
Disclaimer
Me: DevOps n00b
I’m really just here to learn (but Stephen’s making me talk!)
Overview
1. How Docker was introduced to the Copper team
2. The Copper build agents - Doing it Wrong!
3. New service opportunity - Doing it Right!
4. The challenges ahead
A short detour…
What is Copper?
The Copper Team• We are our company’s rapid software delivery team
• We are cross-functional,“innovative early adopters of technology”
• Grails + AngularJS, plus many other technologies
• Provide a faster speed-to-market for customers
• Provide an alternative path to production for our business
Leo, Pete, Mike: DevOps expert consultants from Odecee
Their research project: Docker
Meanwhile…
Build Engineering Team had a problem:
Need more servers, for more Bamboo build agents!
So… In the beginning…
Docker-based Build Agents!
Solution…
Went from approx. 10 to approx. 50 build agents!
Some Copper devs were still not happy…
“Too much job queuing in Bamboo!”
So I forked the Bamboo build agents and created…
Grails-Dedicated Build Agents
• Manually built, manually managed
• 3.5 GB on disk (way too big)
• Rebuilds mean re-download-the-Internet
• Too much hands on
Docker: Doing it Wrong!
…and six months on…
Docker: Doing it Wrong!
• Containers simply do too much!
• Stateful and Stateless concerns mixed together
• Poor use of Dockerfiles and Image Layering
• No orchestration
First Problem
Containers Do Too Much
Containers Do Too Much
… which violates…
Single Responsibility Principle
Single Responsibility Principle
One-Concern-Per-Container Principle
Docker refers to it in places as the:
So, the build agents…
Grails Build Agent:
Grails
Maven
Gradle
Selenium
Firefox
NPM
NodeJS
Grunt
Ruby
Compass
Bamboo Remote Agent Service
Grails Build Agent:
Grails
Maven
Gradle
Selenium
Firefox
NPM
NodeJS
Grunt
Ruby
Compass
Bamboo Remote Agent Service
Grails Build Agent:Grails
Maven
Gradle
SeleniumFirefox
NPM
NodeJS
Grunt
Ruby
Compass
Bamboo Remote Agent Service
Grails Build Agent:Grails
Maven
Gradle
SeleniumFirefox
NPM
NodeJS
Grunt
Ruby
Compass
Bamboo Remote Agent Service
Grails Build Agent:Grails
Maven
Gradle
SeleniumFirefox
NPM
NodeJS
Grunt
Ruby
Compass
Bamboo Remote Agent Service
Grails Build Agent:Grails
Maven
Gradle
SeleniumFirefox
NPM
NodeJS
Grunt
Ruby
Compass
Bamboo Remote Agent Service
Grails Build Agent:Grails
Maven
Gradle
SeleniumFirefox
NPM
NodeJS
Grunt
Ruby
Compass
Bamboo Remote Agent Service
Grails Build Agent:Grails
Maven
Gradle
SeleniumFirefox
NPM
NodeJS
Grunt
Ruby
Compass
Bamboo Remote Agent Service
Grails Build Agent:Grails
Maven
Gradle
SeleniumFirefox
NPM
NodeJS
Grunt
Ruby
Compass
Bamboo Remote Agent Service
Grails Build Agent:Grails
Maven
Gradle
SeleniumFirefox
NPM
NodeJS
Grunt
Ruby
Compass
Bamboo Remote Agent Service
Grails Build Agent:Grails
Maven
Gradle
SeleniumFirefox
NPM
NodeJS
Grunt
Ruby
Compass
Bamboo Remote Agent Service
Grails Build Agent:Grails
Maven
Gradle
SeleniumFirefox
NPM
NodeJS
Grunt
Ruby
Compass
Bamboo Remote Agent Service
Grails Build Agent:Grails
Maven
Gradle
SeleniumFirefox
NPM
NodeJS
Grunt
Ruby
Compass
Bamboo Remote Agent Service
Grails Build Agent:Grails
Maven
Gradle
SeleniumFirefox
NPM
NodeJS
Grunt
Ruby
Compass
Bamboo Remote Agent Service
Doing Too Much
Grails Build Agent:Grails
Bamboo Remote Agent Service
Grails Build Agent:Grails
Bamboo Remote Agent Service
Java Build Agent:
Maven
Gradle
Bamboo Remote Agent Service
Selenium Build Agent:
SeleniumFirefox
Bamboo Remote Agent Service
Node Build Agent:NPM
NodeJS
Grunt
Ruby
Compass
Bamboo Remote Agent Service
Node Build Agent:NPM
NodeJS
Grunt
Ruby
Compass
Bamboo Remote Agent Service
Grails Build Agent:Grails
Bamboo Remote Agent Service
Grails Build Agent:Grails
Bamboo Remote Agent Service
Grails Build Agent:Grails
Bamboo Remote Agent Service
Grails Build Agent:Grails
Bamboo Remote Agent Service
Node Build Agent:NPM
NodeJS
Grunt
Ruby
Compass
Bamboo Remote Agent Service
Node Build Agent:NPM
NodeJS
Grunt
Ruby
Compass
Bamboo Remote Agent Service
Better
Grails Build Agent:Grails
Maven
Gradle
SeleniumFirefox
NPM
NodeJS
Grunt
Ruby
Compass
Bamboo Remote Agent Service
Grails Build Agent:Grails
Maven
Gradle
SeleniumFirefox
NPM
NodeJS
Grunt
Ruby
Compass
Bamboo Remote Agent Service
Grails Build Agent:Grails
Maven
Gradle
SeleniumFirefox
NPM
NodeJS
Grunt
Ruby
Compass
Bamboo Remote Agent Service
Grails Build Agent:Grails
Maven
Gradle
SeleniumFirefox
NPM
NodeJS
Grunt
Ruby
Compass
Bamboo Remote Agent Service
Grails Build Agent:Grails
Maven
Gradle
SeleniumFirefox
NPM
NodeJS
Grunt
Ruby
Compass
Bamboo Remote Agent Service
Grails Build Agent:Grails
Maven
Gradle
SeleniumFirefox
NPM
NodeJS
Grunt
Ruby
Compass
Bamboo Remote Agent Service
Grails Build Agent:Grails
Maven
Gradle
SeleniumFirefox
NPM
NodeJS
Grunt
Ruby
Compass
Bamboo Remote Agent Service
Grails Build Agent:Grails
Maven
Gradle
SeleniumFirefox
NPM
NodeJS
Grunt
Ruby
Compass
Bamboo Remote Agent Service
Grails Build Agent:Grails
Maven
Gradle
SeleniumFirefox
NPM
NodeJS
Grunt
Ruby
Compass
Bamboo Remote Agent Service
Grails Build Agent:Grails
Maven
Gradle
SeleniumFirefox
NPM
NodeJS
Grunt
Ruby
Compass
Bamboo Remote Agent Service
Grails Build Agent:Grails
Maven
Gradle
SeleniumFirefox
NPM
NodeJS
Grunt
Ruby
Compass
Bamboo Remote Agent Service
Grails Build Agent:Grails
Maven
Gradle
SeleniumFirefox
NPM
NodeJS
Grunt
Ruby
Compass
Bamboo Remote Agent Service
Doing Too Much
Doing It Wrong
Mixing Stateful and Stateless Concerns
Doing It Wrong
The next problem…
Mixing Stateful and Stateless Concerns
Grails Build Agent:
Grails Cache
Mixing Stateful and Stateless Concerns
Grails Build Agent:
Grails Cache
Mixing Stateful and Stateless Concerns
Grails Build Agent:
Grails Cache
Mixing Stateful and Stateless Concerns
Grails Build Agent:
Grails Cache
Grails Build Agent:
Grails Cache
Mixing Stateful and Stateless Concerns
Grails Build Agent:
Grails Cache
Grails Build Agent:
Grails Cache
Grails Build Agent:
Grails Cache
Grails Build Agent:
Grails Cache
Grails Build Agent:
Grails Cache
Grails Build Agent:
Grails Cache
File System
Data Volume Data Volume
Better
Grails Build Agent:
Grails Cache
Grails Build Agent:
Grails Cache
Better
Mixing Stateful and Stateless Concerns
Grails Build Agent:
Grails Cache
Grails Build Agent:
Grails Cache
Not so good
Not so good
The next problem…
Poor use of Dockerfiles and Image Layering
Poor use of Dockerfiles and Image Layering
https://registry.hub.docker.com/u/maxexcloo/postgresql/dockerfile/
Simple Dockerfile example:
https://registry.hub.docker.com/u/maxexcloo/postgresql/dockerfile/
Poor use of Dockerfiles and Image Layering
Simple Dockerfile example:
The Grails Build Agent Project
Dockerfile
Shell ScriptShell Script
Puppet Stuff
Shell Script
Shell Script
Puppet Stuff Shell Script
Grails Build Agent:
Grails
Maven
Gradle
Selenium
Firefox
NPM
NodeJS
Grunt
Ruby
Compass
Bamboo Remote Agent Service
The Grails Build Agent Project
Dockerfile
Shell Script
Shell Script Puppet Stuff
Shell Script
Shell Script
Puppet Stuff Shell Script Lots of moving parts (ie. code)
for one image
Dockerfile
Base OS
Image Inheritance
Java7 Image
Grails Image
Dockerfile
Dockerfile
Dockerfile
Base OS
Better
Java7 Image
Grails Image
Dockerfile
Dockerfile
Dockerfile
Shell ScriptShell Script
Puppet Stuff
Shell Script
Shell Script
Puppet Stuff Shell Script
Grails Build Agent:
Grails
Maven
Gradle
Selenium
Firefox
NPM
NodeJS
Grunt
Ruby
Compass
Bamboo Remote Agent Service
Poor use of Dockerfiles and Image Layering
Not so good
Not so good
The next problem…
No orchestration
No orchestration
[adam@laptop ~] ssh adam@build-server
adam@build-server:~ >> Last login: Tue Oct 28 22:13:27 2014 from 10.126.176.99
cd git/buildeng-puppet
adam@build-server:~/git/buildeng-puppet >> git checkout grails-agents-branch
adam@build-server:~/git/buildeng-puppet >> docker build -t=“agent-11” .Sending build context to Docker daemon Step 0 : FROM ubuntu:14.04 ---> e360b5673aef Step 1 : MAINTAINER Adam Wilmore "[email protected]" ---> Using cache ... ... ...(a long time later...)
When I need some new agents…
Too much hands on!
Too much hands on!
It would be better to have…… and more: http://www.mindmeister.com/389671722/docker-ecosystem
…you get the point.
But doing this…
[adam@laptop ~] ssh adam@build-server
adam@build-server:~ >> Last login: Tue Oct 28 22:13:27 2014 from 10.126.176.99
cd git/buildeng-puppet
adam@build-server:~/git/buildeng-puppet >> git checkout grails-agents-branch
adam@build-server:~/git/buildeng-puppet >> docker build -t=“agent-11” .Sending build context to Docker daemon Step 0 : FROM ubuntu:14.04 ---> e360b5673aef Step 1 : MAINTAINER Adam Wilmore "[email protected]" ---> Using cache ... ... ...(a long time later...)
Doing It Wrong
• Containers simply do too much!
• Stateful and Stateless concerns mixed together
• Poor use of Dockerfiles and Image Layering
• No orchestration
Doing It Wrong
Recap:
Next time, it’ll be perfect…
So how about
Doing It Right?
The Next Project
• Build new Artifactory* Service using Docker
• Find out a bit more about Docker in the process
• Take the learnings so far, build a bit, learn a bit more, repeat…
* Artifactory from JFrog is an artefact management tool, similar to Nexus
“Hang on a minute! How do you know that Docker is the right tool
for this type of service?”
Good question.
A short detour…
Why Copper Needs Docker
Why Copper Needs Docker
• Portable infrastructure - changing licensed platform
• Major infrastructure pressures within Copper - scaling, availability and limited resources
• Increasing demand for disposable infrastructure
• What the Copper team needs is…
Why Copper Needs Docker
The Copper PaaS!
And we need to start somewhere…
• Artifactory service comprised:
• Java app server (Jetty)
• Postgres backend
• Web-based file share
• Linux all the way down
• Performance and Availability are critical - can’t be worse than previous artefact manager
Project Requirements
Some Principles to Work With
• Use Puppet with Docker for “image creation”; and
• Favour Puppet over Docker for “doing work” *
• Stick to One Concern Per Container!
• Dive in and learn!
* Why? For reuse, “might move to EC2 one day”, already familiar with Puppet…?
“Hang on a minute! What about service discovery, orchestration, logging,
monitoring and backups?”
<crickets…>
So how did it turn out?
Container Structure
Image Inheritance
The Challenges Ahead
The State of Ops• How do we scale and load-balance for performance and
availability?
• How do we backup all critical data?
• How do we do upgrades?
• How do we keep operational configuration changes under source-control?
• Can we restore everything in the event of a disaster?
• What about service discovery, orchestration, monitoring, all that stuff to help with some of these questions?
The State of Dev• Have we correctly designed containers for extensibility?
• Are we using the right tools and libraries to build the containers?
• How do we version and release our code changes?
• Is our code set up for optimal reuse?
• How do we test our builds?
• How do we collaborate most effectively as developers writing infrastructure software?
One last detour…
…on the question about using tools properly…
Like Docker vs Puppet…
Is it…
(Puppet installs Docker)
Or is it…
(Docker installs Puppet)
The answer appears to be:
“It’s both, and it depends.”
The Copper PaaS
And so it appears there’s still lots of work to do
Which is great because…
Next time, it’ll be perfect!
Next time, it’ll be perfect!
Copper @ an Aussie Insurance Company@awilmore