© Code Crushing

All rights reserved and protected by the Law nº9.610, from 10/02/1998.

No part of this book can be neither reproduced nor transferred with-

out previous written consent by the editor, by any mean: photographic,

eletronic, mechanic, recording or any other.

Code Crushing

Books and programming

Rua Vergueiro, 3185 - 8º andar

04101-300 – Vila Mariana – São Paulo – SP – Brasil

Code Crushing

Dedication

“To my dad, who introduced me to the world of computers and is my life-longrole model."

i

Code Crushing

PrefaceJez Humble

Shortly a�er I graduated from university in ����, I got a job at a start-up inLondon. My boss, Jonny LeRoy, taught me the practice of continuous deploy-ment: Whenwewere�nishedwith a new feature, wewould do a quickmanualsmoke test on our workstation and then �p the relevant ASP scripts directlyonto the production server – not a practice I would recommend today, but itdid have the advantage of enabling us to get new ideas to our users quickly.

In ���� I joined�oughtWorks where my job was to help enterprises de-liver so�ware, and Iwas appalled to discover that lead times ofmonths or evenyears were common. Fortunately, I was lucky enough to work with a numberof smart people in our industry who were exploring how to improve theseoutcomes while also increasing quality and improving our ability to serve ourusers. �e practices we came up with also made life better for the peoplewe were working with (for example, no more deployments outside of busi-ness hours) – an important indication that we were doing something right.In ����, Dave Farley and I published “ Continuous Delivery,” in which wedescribe the principles and practices that make it possible to deliver small,incremental changes quickly, cheaply, and at low risk.

However, our book omits the nuts and bolts of how one actually getsstarted creating a deployment pipeline, put in place monitoring and infras-tructure as code, and the other important, practical steps needed to imple-ment continuous delivery. �us I am delighted that Danilo has written thebook that you have in front of you, which I think is an important and valu-able contribution to the �eld. Danilo has been deeply involved in helping or-ganizations implement the practices of continuous delivery for several yearsand has deep experience, and I am sure you will �nd his book practical and

iii

Code Crushing

informative.I wish you all the best with your journey.

iv

Code Crushing

About the book

Delivering so�ware in production is a process that has become increasinglydi�cult in IT department of various companies. Long testing cycles and adivision between development and operations teams are some of the factorsthat contribute to this problem. Even Agile teams that produce releasableso�ware at the end of each iteration are unable to get to production whenthey encounter these barriers.

DevOps is a cultural and professional movement that is trying to breakdown those barriers. Focusing on automation, collaboration, tools andknowledge sharing, DevOps is showing that developers and system engineershave much to learn from each other.

In this book, we show how to implement DevOps and Continuous Deliv-ery practices to increase the deployment frequency in your company, whilealso increasing the production system’s stability and reliability. You will learnhow to automate the build and deployment process for a web application, howto automate infrastructure con�guration, how tomonitor the production sys-tem, as well as how to evolve the architecture and migrate it to the cloud, inaddition to learning several tools that you can apply at work.

v

Code Crushing

Acknowledgements

To my father, Marcos, for always being an example to follow and for goingbeyond by trying to follow the code examples without any knowledge of thesubject. To my mother, Solange, and my sister, Carolina, for encouraging meand correcting several typos and grammar mistakes on preliminary versionsof the book.

To my partner and best friend, Jenny, for her care and support during themany hours I spent working on the book.

To my editor, Paulo Silveira, for giving me the chance, and knowing howto encourage me at the right time in order for the book to become a reality.To my reviewer and friend, Vivian Matsui, for correcting all my grammarmistakes.

To my technical reviewers: Hugo Corbucci, Daniel Cordeiro and CarlosVilella. �anks for helping me �nd better ways to explain di�cult concepts,for reviewing terminology, for questioning my technical decisions and help-ing me improve the overall contents of the book.

Tomy colleagues Prasanna Pendse, Emily Rosengren, EldonAlmeida andother members of the “Blogger’s Bloc” group at�oughtWorks, for encourag-ingme towritemore, aswell as as for providing feedback on the early chapters.

To my many other colleagues at �oughtWorks, especially Rolf Russell,Brandon Byars and Jez Humble, who heard my thoughts on the book andhelped me choose the best way to approach each subject, chapter by chapter.

Finally, to everyone who contributed directly or indirectly in writing thisbook.

�ank you so much!

vii

Code Crushing

About the author

Danilo Sato started to program as a child at a time when many people stilldid not have home computers. In ����, he entered the bachelor’s program inComputer Science at the University of São Paulo [USP], beginning his careeras a LinuxNetworkAdministrator at IME-USP for � years. While at universityhe began working as a Java / J�EE developer and had his �rst contact withAgile in an Extreme Programming (XP) class.

He started his Masters at USP soon a�er graduation, and supervised byProfessor Alfredo Goldman, he presented his dissertation in August ����about “E�ective Use of Metrics in Agile So�ware Development” [��].

During his career, Danilo has been a consultant, developer, systems ad-ministrator, analyst, systems engineer, teacher, architect and coach, becom-ing a Lead Consultant at �oughtWorks in ����, where he worked in Ruby,Python and Java projects in Brazil, USA and United Kingdom. Currently,Danilo has been helping customers adopt DevOps and Continuous Deliverypractices to reduce the time between having an idea, implementing it andrunning it in through production.

Danilo also has experience as a speaker at international conferences, pre-senting talks and workshops in: ����/����/���� XP, Agile ����/����, Ágiles����, Java Connection ����, Falando em Agile ����, Rio On Rails ����, Py-Con Brazil ����, SP RejectConf ����, Rails Summit Latin America ����,Agile Brazil ����/����/����, QCon SP ����/����, QCon Rio ����, RubyConfBrazil ����/����. He was also the founder of the Coding Dojo @ São Pauloand an organizer for Agile Brazil ����, ���� and ����.

ix

Code Crushing Contents

Contents

� Introduction �

�.� Traditional approach . . . . . . . . . . . . . . . . . . . . . . . �

�.� An alternative approach: DevOps and Continuous Delivery . �

�.� About the book . . . . . . . . . . . . . . . . . . . . . . . . . . . �

� Everything starts in production �

�.� Our example application: an online store . . . . . . . . . . . . ��

�.� Installing the production environment . . . . . . . . . . . . . ��

�.� Con�guring the production servers . . . . . . . . . . . . . . . ��

�.� Application build and deploy . . . . . . . . . . . . . . . . . . . ��

� Monitoring ��

�.� Monitoring other hosts . . . . . . . . . . . . . . . . . . . . . . ��

�.� Adding more speci�c checks . . . . . . . . . . . . . . . . . . . ��

�.� Receiving alerts . . . . . . . . . . . . . . . . . . . . . . . . . . ��

� Infrastructure as code ��

�.� Provision, con�gure or deploy? . . . . . . . . . . . . . . . . . ��

�.� Con�guration management tools . . . . . . . . . . . . . . . . ��

�.� Provisioning the database server . . . . . . . . . . . . . . . . . ��

�.� Provisioning the web server . . . . . . . . . . . . . . . . . . . ��

xi

Contents Code Crushing

� Puppet beyond the basics ���.� Classes and de�ned types . . . . . . . . . . . . . . . . . . . . . ���.� Using modules for packaging and distribution . . . . . . . . . ���.� Refactoring the web server Puppet code . . . . . . . . . . . . ����.� : Separation of concerns: infrastructure vs. application . . . . ����.� Puppet forge: reusing community modules . . . . . . . . . . ����.� Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ���

� Continuous integration ����.� Agile engineering practices . . . . . . . . . . . . . . . . . . . . ����.� Starting with the basics: version control . . . . . . . . . . . . ����.� Automating the build process . . . . . . . . . . . . . . . . . . ����.� Automated testing: reducing risk and increasing con�dence . ����.� What is continuous integration? . . . . . . . . . . . . . . . . . ����.� Provisioning a continuous integration server . . . . . . . . . ����.� Con�guring the online store build . . . . . . . . . . . . . . . . ����.� Infrastructure as code for the continuous integration server . ���

� Deployment pipeline ����.� Infrastructure a�nity: using native packages . . . . . . . . . ����.� Continuous integration for the infrastructure code . . . . . . ����.� Deployment pipeline . . . . . . . . . . . . . . . . . . . . . . . ����.� Next steps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ���

� Advanced topics ����.� Deploying in the cloud . . . . . . . . . . . . . . . . . . . . . . ����.� DevOps beyond tools . . . . . . . . . . . . . . . . . . . . . . . ����.� Advanced monitoring systems . . . . . . . . . . . . . . . . . . ����.� Complex deployment pipelines . . . . . . . . . . . . . . . . . ����.� Managing database changes . . . . . . . . . . . . . . . . . . . ����.� Deployment orchestration . . . . . . . . . . . . . . . . . . . . ����.� Managing environment con�guration . . . . . . . . . . . . . ����.� Architecture evolution . . . . . . . . . . . . . . . . . . . . . . ����.� Security . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ����.�� Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ���

xii

Code Crushing Contents

Bibliography ���Versão: ��.�.��

xiii

C������ �

Introduction

With the advancement of technology, so�ware has become an essential partof everyday life for most companies. When planning a family vacation —scheduling hotel rooms, buying airplane tickets, shopping, sending an SMS orsharing photos of a trip — people interact with a variety of so�ware systems.When these systems are down, it creates a problem not only for the companythat is losing business, but also for the users who fail to accomplish their goals.For this reason, it is important to invest in quality so�ware and stability fromthe moment that the �rst line of code is written until the moment it startsrunning.

�.� T���������� ��������So�ware development methodologies have evolved, but the process of trans-forming ideas into code still involves several activities such as requirements

�.�. Traditional approach Code Crushing

gathering, design, architecture, implementation and testing. Agile so�waredevelopment methods have emerged in the late ��s, proposing a new ap-proach to organize such activities. Rather than performing them in distinctphases - the process known as waterfall - they happen at the same time, inshort iterations. At the end of each iteration, the so�ware becomes more andmore useful, with new features and less bugs, and the team decides with thecustomer what should be the next slice that is developed.

As soon as the customer decides that the so�ware is ready to go live andthe code is released to production, the real users can start using the system. Atthis time, several other concerns become relevant: support, monitoring, se-curity, availability, performance, usability, among others. When the so�wareis in production, the priority is to keep it running stably. In cases of failure ordisaster, the team needs to be prepared to react quickly to solve the problem.

Due to the nature of these activities, many IT departments have a clearseparation of responsibilities between the development team and the opera-tions team. �e development team is responsible for creating new productsand applications, adding features or �xing bugs, while the operations teamis responsible for taking care of these products and applications in produc-tion. �e development team is encouraged to introduce changes, while theoperations team is responsible for keeping things stable.

At �rst glance, this division of responsibilities seems to make sense. Eachteam has di�erent goals and di�erent ways of working. While the develop-ment team works in iterations, the operations team needs to react instantlywhen something goes wrong. Furthermore, the tools and knowledge nec-essary to work in these teams are di�erent. �e development team evolvesthe system by introducing changes. On the other hand, the operations teamavoids changes because they bring a certain risk to the stability of the system.�is creates a con�ict of interest between these two teams.

Once the con�ict exists, the most common way to manage this relation-ship is by creating processes that de�ne the method of working as well as theresponsibilities of each team. From time to time, the development team pack-ages the so�ware that needs to go to production, writes some documentationexplaining how to con�gure the system, how to install it in production, andthen transfers the responsibility to the operations team. It is common to use

�

Code Crushing Chapter �. Introduction

ticket tracking systems tomanage the communication between the teams andde�ning service-level agreements (SLAs) to ensure that the tickets are pro-cessed and closed in a timely fashion. �is hand-o� o�en creates a bottleneckin the process of taking code from development and testing to production.It is common to call this process a deployment to production, or simply aproduction deploy.

Over time, the process tends to become more and more bureaucratic, de-creasing the frequency of deploys. With that, the number of changes intro-duced in each deploy tends to accumulate, also increasing the risk of eachdeploy and creating the vicious cycle shown in �gure �.�.

Figure �.�: Vicious cycle between development and operations

�is vicious cycle not only decreases the ability of the company to respondquickly to changes in business, but also impacts earlier stages of the develop-ment process. �e separation between development and operation teams, thehand-o� of code between them and the ceremony involved in the deploymentprocess, end up creating a problem known as “the Last Mile" [��].

�e last mile refers to the �nal stage of the development process that takesplace a�er the so�ware meet all its functional requirements but before beingdeployed into production. It involves several activities to verify whether theso�ware that will be delivered is stable or not, such as: integration testing,system testing, performance testing, security testing, user acceptance testing(UAT), usability testing, smoke testing, data migration, etc.

It is easy to ignore the last mile when the team is producing and show-casing new features every one or two weeks. However, there are few teams

�

�.�. An alternative approach: DevOps and Continuous Delivery Code Crushing

that are actually deploying to production at the end of each iteration. Fromthe business point of view, the company will only have a return of investmentwhen the so�ware is actually running in production. �e last mile problem isonly visiblewhen taking a holistic view of the process. To solve it wemust lookpast the barriers between the di�erent teams involved (the business team, thedevelopment team or the operations team).

�.� A� ����������� ��������: D��O�� ��� C��-�������D�������

Many successful internet businesses — such as Google, Amazon, Net�ix,Flickr, Facebook and GitHub— realized that technology can be used in theirfavor and that delaying a production deploy means delaying their ability tocompete and adapt to changes in the market. It is common for them to per-form dozens or even hundreds of deploys per day!

�e line of thinking that attempts to decrease the time between the cre-ation of an idea and its implementation in production is also known as “Con-tinuous Delivery" [��], and is revolutionizing the process of developing anddelivering so�ware.

When the deployment process ceases to be a ceremony and starts becom-ing commonplace, the vicious cycle of �gure �.� gets completely reversed. In-creasing deployment frequency causes the amount of change in each deployto decrease, also reducing the risk associated with that deploy. �is bene�t isnot something intuitive, but when something goes wrong it is much easier to�nd out what happened because the amount of changes that may have causedthe problem is smaller.

However, reducing the risk does not imply a complete removal of the pro-cesses between development and operation teams. �e key factor that allowsthe reversal of the cycle is process automation, as shown in �gure �.�. Au-tomating the deployment process allows it to run reliably at any time, remov-ing the risk of problems caused by human error.

�

Code Crushing Chapter �. Introduction

Figure �.�: DevOps practices help to break the vicious cycle through processautomation

Investing in automation is not a new idea; many teams already write au-tomated tests as part of their so�ware development process. Practices such astest-driven development (TDD) [�] or continuous integration [��] — whichwill be discussed in more detail in chapter � — are common and widely ac-cepted in the development community. �is focus on test automation, alongwith the creation of multidisciplinary teams, helped to break the barrier be-tween developers, testers and business analysts, creating a culture of collabo-ration between people with complimentary skills working as part of the sameteam.

Inspired by the success of Agile methods, a new movement emerged totake the same line of reasoning to the next level: the DevOpsmovement. Itsgoal is to create a culture of collaboration between development and opera-tion teams that can increase the �ow of completed work — higher frequencyof deploys — while increasing the stability and reliability of the productionenvironment.

Besides being a cultural change, theDevOpsmovement focuses a lotmoreon practical automation of the various activities necessary to tackle the lastmile and deliver quality code to production, such as: code compilation, au-tomated testing, packaging, creating environments for testing or production,infrastructure con�guration, data migration, monitoring, log andmetrics ag-gregation, auditing, security, performance, deployment, among others.

Companies that have implemented these DevOps practices successfullyno longer see the IT department as a bottleneck but as an enabler to the busi-

�

�.�. About the book Code Crushing

ness. �ey can adapt to market changes quickly and perform several deploysper day safely. Some of them even make a new developer conduct a deploy toproduction on their �rst day of work!

�is book will present, through actual examples, the main practices ofDevOps and Continuous Delivery to allow you to replicate the same successin your company. �e main objective of the book is to bring together thedevelopment and operations communities. Developers will learn about theconcerns and practices involved in operating and maintaining stable systemsin production, while system engineers and administrators will learn how tointroduce changes in a safe and incremental way by leveraging automation.

�.� A���� ��� �����e main objective of the book is to show how to apply DevOps and Contin-uous Delivery concepts and techniques in practice. For this reason, we hadto choose which technologies and tools to use. Our preference was to useopen source languages and tools, and to prioritize those that are used widelyin industry.

You will need to use the chosen tools to follow the code examples. How-ever, whenever a new tool is introduced, we will brie�y discuss other alter-natives, so that you can �nd the option that makes the most sense in yourcontext.

You do not need any speci�c prior knowledge to follow the examples. Ifyou have experience in the Java or Ruby ecosystem, that will be a bonus. Ourproduction environment will run on UNIX (Linux, to be more speci�c), so alittle experience using the command line can help but is not mandatory [��].Either way, you will be able to run all the examples on your own machine,regardless if you are running Linux, Mac or Windows.

Target audience

�is book is written for developers, system engineers, system adminis-trators, architects, project managers and anyone with technical knowledgewho has an interest in learning more about DevOps and Continuous Deliv-ery practices.

�

Code Crushing Chapter �. Introduction

Chapter structure

�e book was written to be read from beginning to end sequentially. De-pending on your experience with the topic of each chapter, you may prefer toskip a chapter or follow a di�erent order.

Chapter � presents the sample application that will be used through therest of the book and its technology stack. As the book’s focus is not on thedevelopment of the application itself, we use a nontrivial application writtenin Java built on top of common libraries in the Java ecosystem. At the end ofthe chapter, the application will be running in production.

With the production environment running, in chapter � we wear the op-erations team’s hat and con�gure a monitoring server to detect failures andsend noti�cationswhenever a problem is encountered. At the end of the chap-ter, we will be noti�ed that one of the servers crashed.

In chapter � we will rebuild the problematic server, this time using au-tomation and treating infrastructure as code. Chapter � is a continuation ofthe subject, covering more advanced topics and refactoring the code to makeit more modular, readable and extensible.

A�er wearing the operations team’s hat, we will turn our attention to theso�ware development side. Chapter � discusses the Agile engineering prac-tices that help writing quality code. You will learn about the various typesof automated tests and launch a new server dedicated to perform continuousintegration of our application code.

Chapter � introduces the concept of a deployment pipeline. We setupcontinuous integration for the infrastructure code, and implement an auto-mated process to deploy the newest version of the application in productionwith the click of a button.

In chapter �, we migrate the production environment to the cloud. Fi-nally, we discuss more advanced topics, including resources for you to re-search and learn more about the topics that were le� out of the scope of thisbook.

Code conventions

In the code examples, we will use ellipsis ... to omit the unimportantparts. When making changes to already existing code, we will repeat the

�

�.�. About the book Code Crushing

lines around the area that needs to be changed to give more context aboutthe change.

When the line is too long and does not �t on the page, we will use a back-slash \ to indicate that the next line in the book is a continuation of the pre-vious line. You can simply ignore the backslash and continue typing in thesame line.

In the command line examples, in addition to the backslash, we use agreater than signal > in the next line to indicate that it is still part of the samecommand. �is is to distinguish between the command that is being enteredand the output produced when the command executes. When you run thosecommands, you can simply type it all in one line, ignoring the \ and the >.

More resources

We have created a discussion forum on Google Groups where you cansend questions, suggestions, or feedback directly to the author. To subscribe,visit the URL:

https://groups.google.com/d/forum/devops-in-practice-bookAll code examples from the book are also available on the author’s GitHub

projets, in these URLs:https://github.com/dtsato/loja-virtual-devopshttps://github.com/dtsato/loja-virtual-devops-puppet

�