Scaling continuous delivery @ GeeCon 2014

ScalingContinuous Delivery

Tomas Riha

Tomas Riha

Architect @ VGT/WirelessCar

Passionate about creativity, change and improvementHorrible at following instructions and performing repetitive tasks

MAJOR Project Liability

mail: [email protected]: @TomasRihaSE

blog: continuous-delivery-and-more.blogspot.comslides: www.slideshare.net/TomasRiha

Three Years ago

New Telematics Delivery Platform

Multiple Stakeholders

Continuous Regression Testing

How hard can it be?

Regression Test

Unit Testing

Component Testing

System Testing

Rollback/Compatibility Testing

Failover Testing

Performance Testing

At first it was super easy!

Small team of just product owner architect and scrum master.

Huge productivity, natural test driven development.

Fast return on investment

All by the book.... we were doingContinuous Delivery!

Then we tried to scale it...

... and we failed in every possible way.

We lost our test driven developmentWe lost the individual responsibility

We more or less became a automated waterfall unable to delivery daily

We lost our ability to Continuously Improve the process

Three components of Continuous Delivery

Process & ImplementationDefinition of the Software Delivery process and its lead time optimization

Product IntegrityThe Architecture & Test Strategy to optimize product integrity and lead time.

People & OrganizationBehavioural change to the individuals and the organization needed to optimize

lead time and increase product integrity.

Process & Implementation“Dedicated test servers”

First Continuous Delivery implementation relied on standard corporate test environment with one system test server, one integration test server and one

pre prod server per production delivery

Only one commit tested at the time.

Build Release Deploy Test

DB

Test Server

Process & Implementation“Single threaded process”

Stacking jobs became a problem at 5 code committers, almost killed us at 20

Feedback time == Stack Depth * Pipe Exec Time

Variable feedback time results in less responsible developers

We were somewhat helped by component based architecture

Process & Implementation“First effort to scale”

Forced to use our internal Operations as infrastructure provider, no cloud and very rudimentary virtualization

Built a Server Pool based on Jenkins slaves and “sticky pipes”

Primitive but multi threaded pipes

Orchestration Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker Worker Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker


Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker


Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker


Process & Implementation“Crowded Environment”

Visibility was bad due to hundreds of jobs on same build system.

Teams blocking each other due to still insufficient server resources.

Traceability hard due to all server clutter.

Changes to one teams Test Environment affected the other teams.

Overall performance was horrible at peak with Jenkins totally dieing

Process & Implementation“Continuous Delivery of Continuous Delivery”

Live what you teach!

Our ability to Continuously Improve was seriously slowed by lack of Continuous Delivery

Deployment Scripts didn't have test automation.

No test environment for Continuous Delivery Process

Build pipes where manually maintained.

Jenkins servers where manually maintained.

Process & Implementation“Autonomous Environments for Autonomous Teams”

Started to do Continuous Delivery of Continuous Delivery

Automated provisioning of Continuous Delivery Environment

Created template based pipeline generator

Built Autonomous Continuous Delivery environments for each team


Worker

Worker

Worker

Worker

Worker

Worker

Worker


Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker


Worker

Worker

Worker

Worker

Worker

Worker

Worker


Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Process & Implementation“Portability is Scalability”

Portability in the Runtime Environments

Dev

Test

Load Test

UATPROD

INTTest

Complexity

Scale


If the Continuous Delivery process can run without a Build Server then it can scale to X number of build servers

By ensuring that anything above the portability line can run local or remote to the orchestration we ensure portability

Orchestration

ProvisionBuild Deploy Test Release

Test EnvironmentSCM ArtifactRepoPortability Line


All environments provisioned and deployed in the same way

Orchestration

ProvisionBuild Deploy Test Commit

Orchestration


Orchestration

Provision Deploy Test Accept

Orchestration

Provision Deploy Test

Local Dev Continuous Delivery Pipe

UAT Prod


Dev Test Load Test UATINT

Test

Provisioning Interface

Vagrant Impl Cloud Provider Impl PDF Document Impl

Topology Spec Env Spec

LegacyPROD

With the help of Cloud Technology and Vagrant we manage to create test and dev environments with the same complexity as our production

environments.


To make the Continuous Delivery Process portable we also need to move all reporting and monitoring out of the test servers and the build servers

Orchestration


Test EnvironmentSCM ArtifactRepo

Logging & Metrics Services

Build Data Repository



Log, Metrics, Build Data Repositories

WorkerWorker

Build Env


Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Worker

Team Environment Pre CloudOne Server Per Pipe Shared Database for all pipes

Team Environment in CloudOne Environment Per PipeIncluding Load Balance, App Servers & DB

Separating Data from Process Implementation improves the traceability increases dramatically.

Portability in Test Environments increases Quality and Scalability by removing bottlenecks in shared infrastructure.

WorkerWorkerWorker

Test Env

Worker

Worker

Worker

WorkerWorkerTest Env

WorkerTest EnvTest Env

Process & Implementation“Provisioning, Install and Dependency Management”

If one or more environments is provisioned manually it becomes a bottleneck

Application artifacts need dependency management to infrastructure.

Infrastructure has to be provisioned and installed with the application.

Databases have to be upgraded and migrated as part of the automated process.

An automated process doesn't pause for DBAs or SysAdmins to do manual work.

People & Organization“The Tools Team”

Initially we did development of the Continuous Delivery process as skunkworkz.

Quickly realized this was a bottleneck.

Added a CM to our team with disastrous result.

Finally created a Tools Team.

People & Organization“The Tools Team is a Bottleneck”

Tools Team a huge bottleneck, simplest issues took ages to fix.

Managed the Continuous Delivery process manually.

Only ones that understood the process.

Became responsible for every teams ability to interface with the process.

Only had time to do reactive maintenance.

Short on resources.

Became yet another IT department.

People & Organization“Build Police, please dont!”

Only Tools Team and few others understood the process.

Tried to help the few others by adding a rotational “Build Police” Role, as bad idea as it sounds.

People & Organization“The Tools Team matures”

Continuous Delivery as a Service

Tools Team responsible for the process implementation, its interfaces and the infrastructure it runs on.

The Development Teams responsible for integrating with these interfaces, configuring the pipe generator and maintaining green state of pipes.

Tools Team supports and helps if there are bugs in the mechanisms.

Tools Team consists of a core team of Developers, Testers and Operations specialists but is extended with part time resources from Development

Teams.

People & Organization“Lack of Consensus”

Disagreements on shared responsibility for the release.

Disagreements on what test automation is and how it should be used.

Disagreement on who implements tests and how.

A Continuous Delivery Engine without the consensus of the people that use it is just an overpriced CI System.

Product Integrity “Build Quality In”

SYSTEM

GUI


Internal Service

GUI

Internal Service

Internal Service

Service API


Internal Service

Internal Service

Internal Service

Each Service Tested as Black Box in isolationand with high detail level

Product Integrity“Build Quality In”

GUI

Service API Mock Impl

GUI Test are much faster and robust if they test the GUI in isolation. High detail level.


Internal Service

GUI

Internal Service

Internal Service

Service API

Verifying Use Case acceptance criterias. Low level of detail

Product Integrity“Build Quality In” Optimizes Lead Time






Assemble Release Deploy Test

Component Pipe

1000s of tests run in parallel testing

System Pipe

10s Use Case Requirements Verified


An Architecture & Test Strategy that goes hand in hand increases product integrity and optimizes the lead time and allows the Continuous Delivery

process to scale

People & Organization“Empower the Team!”

Increase Team responsibility.

Team is responsible for everything within the Continuous Delivery process

Analyzing RequirementsDefining Requirement Verification

Automating Requirement VerificationImplementing the System

Shipping the SystemSupporting the System

Create Team consensus on what its responsibility and delivery is.

People & Organization“Empower the Team!”

To help empower teams we have

Autonomous Team Environments where teams are responsible for their own Continuous Delivery increased the teams understanding of the process.

Teams that needed extra help sent one or more individuals to work part time on Tools Team.

Architecture & Test Strategy that works well with Continuous Delivery

Cross functional communities created to help increase consensus on Test Driven Development, Continuous Delivery and Architecture.

People & Organization“Test Driven Development Community”

Developers need to take more responsibility.

Code has to work all the time.

Test Driven Development requires developers to participate in specification of requirement verifications.

Test Automation is Code, Developers need to code much more tests.

Continuous Regression Testing means Developers get instant feedback and have to act on it.

People & Organization“Test Driven Development Community”

Test Profession changes.

Two main type of Test activities Test Automation and Exploratory Testing.

Test Automation is Code.

Let developers test.

Test Driven Development is Proactive.

Exploratory testing is NOT manual regression testing.

People & Organization“Continuous Delivery Community”

Agreement on Practices of Continuous Delivery

Agreement on Practices of Provisioning & Deployment

Competence Development

Infrastructure as Code

Testing Infrastructure as Code

Operations Specialists need to work with Developer tools and languages

People & Organization“Organizational Support”

Continuous Delivery is organizational change.

Scaling Continuous Delivery is as much creating a scalable agile organization as it is creating a scalable implementation.

Organization has to fully support Continuous Delivery through re-organization.

Organization has to fully support and invest in infrastructure, architecture, automation, process and test.

Organization needs to start measuring key values, such as cycle time, early.

Summary

Process & ImplementationPortability is Scalability.

Product IntegrityThe Architecture & Test Strategy go hand in hand.

People & OrganizationEmpowered Teams and Consensus.

Thats it!

Feedback & Any questions you forgot to ask?

http://continuous-delivery-and-more.blogspot.se@TomasRihaSE

or bythe Coffee stand!

And the slides are athttp://www.slideshare.net/TomasRiha

http://continuous-delivery-and-more.blogspot.se/

http://continuous-delivery-and-more.blogspot.se/