8/9/2019 Deploying Yourself Into Happiness
1/28
Deploying your way into happiness
Vladimir Vuksan @vvuksan and Jeff Buchbinder
8/9/2019 Deploying Yourself Into Happiness
2/28
Just kidding!
8/9/2019 Deploying Yourself Into Happiness
3/28
No really, happiness is possible
8/9/2019 Deploying Yourself Into Happiness
4/28
The deployment of a ...., computer program, ... is itsassembly or transformation from a packaged form to an
operational working state.
Deployment implies moving a product from a temporary ordevelopment state to a permanent or desired state.
http://en.wikipedia.org/wiki/System_deployment
What is deployment ?
8/9/2019 Deploying Yourself Into Happiness
5/28
Sounds exciting ...
http://www.flickr.com/photos/32214265@N00/1977622599
http://www.flickr.com/photos/32214265@N00/1977622599http://www.flickr.com/photos/32214265@N00/19776225998/9/2019 Deploying Yourself Into Happiness
6/28
How it feels most of the time ...
http://www.flickr.com/photos/bobex_pics/3105613262
8/9/2019 Deploying Yourself Into Happiness
7/28
Why is deployment a problem?
Poor communication and coordination between the dev andops teams
Poorly tested/QAed software
Unaccounted for differences between QA and productionenvironments
Overwriting state, logs, configuration and other transientsoftware properties upon deployment e.g. ad-hoc changes toconf files to "make things work" are not propagated inconsecutive deployments
8/9/2019 Deploying Yourself Into Happiness
8/28
How does deployment improvement fit intoDevOps
Deployment is one of the biggest interconnection pointsbetween ops and development
Streamline deployment and 90% of the dev vs. ops issuesgo away #win
Lets ops spend more time on cooler things :-) such asmonitoring, performance etc.
Lets devs spend more time on cooler things since theydon't have to spend so much time dealing with
deployment issues
Quicker iteration between releases
Enables users by letting them self-service themselves.
8/9/2019 Deploying Yourself Into Happiness
9/28
Environment background
6 distinct applications to deploy. Java based usingTomcat/JBoss application servers
Different release schedules for each of the apps
Apps communicate with each other via web services =>coordination headaches
8/9/2019 Deploying Yourself Into Happiness
10/28
How deployment used to be
Deployment done two ways
Single BASH installer for core/base app ./installer -ver 5_40 -cvsup -servers snowhite,sleepingbeauty
Other apps installed by hand For other apps configurations "baked-in" binaries ie.
Need to do ant build_qa or ant build_prod to producedeployment binaries archives
Frequent deployment issues
Failure due to accidental commits into a branch
Installer breaks randomly
Baked-in configurations wrong (duh!)
8/9/2019 Deploying Yourself Into Happiness
11/28
How we did it
In next couple slides we'll cover
Application design for deployment
How to build releases
Deployment tool
How to deal with app specific complexities
How to automatically patch the DB
8/9/2019 Deploying Yourself Into Happiness
12/28
Application design
Application should have a sane default configuration options.
Any config option should be overrideable via an external file e.g.
include("default.conf");
if ( is_file( "$APP_HOME/conf/app.conf") ) include("$APP_HOME/conf/app.conf");
In most cases you only need to override database credentials (host,username, password).
Goal is to be able to use the same binary across multiple
environments and avoid baking configuration into binaries
8/9/2019 Deploying Yourself Into Happiness
13/28
Application design cont'd
Application should expose key internal metrics.
JMSenqueue=OK etc.
This is important because there are lots of things that can break inside theapplication which external monitoring may miss like JMS message can't be
enqueued, etc. Keep release notes actions to a minimum.
Instructions often not followed or partially followed.
Avoid them whenever possible
Automate as much as you can
8/9/2019 Deploying Yourself Into Happiness
14/28
Building releases
Developers are in charge of building and packaging releases.
QA or Ops will not know what to do if a build fails (this is Javaremember)
Each release has to be clearly labeled with the version andtagged in the repository. Archive file needs to contain the samee.g. Location Server 1.1.5 will be packaged as location-1.1.5.tar.gz.
Archives contain only WAR (Tomcat) or EAR (Jboss) files andDB patch files. Releases are to be uploaded into anappropriate file share ie. /share/releases/location.
8/9/2019 Deploying Yourself Into Happiness
15/28
Deployment tool data model
Support multiple applications
can use different app server containers ie. Tomcat/JBoss
can have configuration files that can be either key/valuepairs or templates.
Every application has a start and stop script
Has notion of unique domains/Customers
single dashboard that allow deployment to multipleenvironments e.g. QA staging (current release), QAdevelopment (next scheduled release), Dev playbox, etc.
Each of these domains has their own set of applications theycould deploy with their own domain specific configurationoptions
8/9/2019 Deploying Yourself Into Happiness
16/28
Application example
Location server
Needs Tomcat container
Tomcat 5.5.25 for versions < 1.1.0
Tomcat 6.0.24 for versions > 1.1.5 Configuration override file found in /conf/location.conf
which contains key/values e.g.
dbUser=location
$APP_HOME/start.sh -> startup script template
$APP_HOME/stop.sh -> stop script template
8/9/2019 Deploying Yourself Into Happiness
17/28
Deployment tool aka Deployer
A 500+ command line PHP script that is able to deploy, stop, startand undeploy services => intended for batch operations, massupgrades, etc.
Separate Web GUI that interfaces with the command line utility +
allows easy config changes => used largely by QA/dev Deployer mostly used for prep of the software base ie. building up
file structure, overlaying files, creating configuration files. Avoidsapp specific complexities
App specific customizations are implemented via BASH shell scripts(not a requirement)
Keeps core deployer simple
Easier to troubleshoot smaller BASH files
8/9/2019 Deploying Yourself Into Happiness
18/28
Deployer web GUI
Pick a version
Override default values using these
8/9/2019 Deploying Yourself Into Happiness
19/28
Deployer example
Commonly invoked this way
$ deployer version 1.2.5 server web10 domain joedev app base action deploy
Results in following actions
1. Unpack the proper app server container e.g. jboss-4.2.3.tar.gz to /prep_dir
2. Overlay/untar WAR/EAR files for the name version e.g. base-1.2.5.tar.gz
3. Build configuration files and scripts
4. Stop the server on the remote box ie. ssh web01 /run/ba_base/stop.sh
5. Rsync the contents of the packaged release /prep_dir -> /run/ba_base
6. Make sure Apache AJP proxy is configured to proxy traffic and executeapache reload
7. Start up the server ie. ssh web01 sudo -u ba_base /run/ba_base/stop.sh
8/9/2019 Deploying Yourself Into Happiness
20/28
DB Patching
"Automatically" patch the DB => no human/ops intervention
Every application has table called Patch (single column) with a list ofDB patches that have been applied.
Every app has dbpatches directory in the app archive which contains
a list of patches named with version and order in which they shouldbe applied e.g.
2.54.01-addUserColumn.sql
2.54.02-dropUidColumn.sql
During deployment startup script compares contents of the patch
table and a list of dbpatches and applies any missing ones. If thepatch script fails e-mail is sent to the QA or dev in charge of particulardomain
All this done using a 70 line BASH shell script => nomagic/complexity
Apparent which goes first
8/9/2019 Deploying Yourself Into Happiness
21/28
Production deployment prep
Usually a day or so before the deployment is scheduledhave a conference call between dev, QA and ops thatdiscusses deployment and any things to look out for.
Apply any configuration changes that can be done ahead
of time ie. add new config option
Upgrade the staging environment by importing the prodDB, then upgrade to the desired release and QA it.
8/9/2019 Deploying Yourself Into Happiness
22/28
Actual deployment
Paper trail -> The day before the release QA opens up aticket listing all the applications and versions that neededto be deployed.
On the morning of the deployment (that was our low time)
someone from ops, development and whole QA teamengage in deploying the app and resolving any observedissues.
8/9/2019 Deploying Yourself Into Happiness
23/28
Other pointers
8/9/2019 Deploying Yourself Into Happiness
24/28
Sizing QA environment
Create a couple
Staging QA => same version as production
Dev QA => next scheduled release
Should run on underpowered hardware (virtualized a goodchoice) since it is almost impossible to simulate productionload so this gives you valuable data e.g.
We discovered a number of major flaws when ourvirtualized machines ran out of disk space
Easier to spot higher CPU utilization on 1 core boxthan 32 core box
Want to load test ? Use a performance environment that issimilar to prod but used solely for load testing
8/9/2019 Deploying Yourself Into Happiness
25/28
Precreate environments
Create users/environments/network info ahead of time
Avoids provisioning problems, resource locks, etc.
Run different applications under their own user id => avoids
accidentally stoping wrong app Use a unique identifier. Easy way => 2/3/4 letter codes ie. aa,
ab, ac ....
Create all the necessary user ids for every environment
application combination ie. aa_msg, aa_base, aa_loc Precreate configurations, DBs/usernames (without schemas)
Need a new environment => assign the already createdenvironment to the new user/group ie. salesdemo_joe => af.dev_jane => aj. Deploy.
8/9/2019 Deploying Yourself Into Happiness
26/28
Multiple environments on same hardware w/ovirtualization
Assign customer specific port ranges then use applicationspecific offsets e.g.
Customer aa => start_port 10100, ab => start_port 10200
Messaging app offset +10 Location app offset +20
Results aa_msg can use 10110-10119 (althoughusually only 3 ports necessary)
Precreate these when when precreating environments Alternatively specify customer specific IP ranges then use
those. Private IPs are cheap.
8/9/2019 Deploying Yourself Into Happiness
27/28
Hostnames naming
Avoid theme names ie. planet/star names, cartoon charactersetc.
Pick functional names ie. db1, qadb1,web1,app1,mail1 etc.
Why ? @vvuksan Agree. You shouldn't have to have this mental
conversation at 3am: "Was the web server on Wolverine orCyclops? Shit, maybe Beast." (http://twitter.com/markimbriaco/status/19498142304 )
@markimbriaco through acquisition I got 300 machinesnamed after types of drink. In Finnish. Was hell :)(https://twitter.com/ripienaar/status/19498821560 )
8/9/2019 Deploying Yourself Into Happiness
28/28
Questions ?