Scaling up CI to a CD Pipeline

BUILD TEST DEPLOY

➔ Social Casino Startup

➔ Web and Mobile

➔ 250K DAU

➔ ~20 Technical Staff

➔ No QA

@Plumbee_Gameshttp://www.plumbee.com

Engineering Manager at Plumbee Ltd@mserranommiguel@plumbee.com

Scaling up CI to a CD Pipeline

BUILD TEST DEPLOY

Mission + People

BUILD TEST DEPLOY

Any organisation’s pipeline is shaped by:

● The requirements to achieve its mission

● The views of the team on Software Development

Decisions

● Tools

● Test Strategy

● Frequency on deployment

● On-call strategy

BUILD TEST DEPLOY

it’s all about building an

Engineering Culture

BUILD TEST DEPLOY

Trunk vs Branching

Trunk vs Branching

● Fast ● Safe

Trunk vs Branching

● Fast

● Minimises merge pain

● Safe

● Asynchronous

Trunk vs Branching

● Fast

● Minimises merge pain

● Old Fashioned

● Safe

● Asynchronous

● Git is cool!

Trunk

“The most direct way to improve as a software developer is to be absolutely fearless when it comes to changing your code”

Jeff AtwoodStack Exchange/Discourse founder

Trunk

“Don’t be scared of changing your code. Pave your way to not to break things”

Miguel SerranoThe guy talking right now

Versioning

JAR1.0-SNAPSHOT

WAR1.4-SNAPSHOT

JAR1.4-SNAPSHOT

JAR1.0-SNAPSHOT

WAR1.4-SNAPSHOT

JAR1.4-SNAPSHOT

Versioning

Nondeterministic!

The exact sequence of builds is not reproducible

JAR1.0-SNAPSHOT

WAR1.4-SNAPSHOT

JAR1.4-SNAPSHOT

Versioning

Releases require:

● Update snapshots to static version number

● Ensure there are no snapshot dependencies

● Code freeze while releasing

● Manual updates

JAR1.0

WAR1.4

JAR1.4

JAR1.0-SNAPSHOT

WAR1.4-SNAPSHOT

JAR1.4-SNAPSHOT

Versioning

Releases require:

● Update snapshots to a static number

● Ensure there are no snapshot dependencies

● Code freeze while releasing

● Manual updates

} mvn release:prepare

mvn release:perform

JAR1.0

WAR1.4

JAR1.4

Versioning

Rules

● Every artifact is assigned a new version at build time

● Every artifact gets its dependencies versions updated to the latest version in every build

● Every artifact is ready to be deployed

JAR1.0

WAR1.4

JAR1.4

JAR1.0-SNAPSHOT

WAR1.4-SNAPSHOT

JAR1.4-SNAPSHOT

Versioning

JAR

JAR

JAR

JAR

JAR

WAR

Test

DEPLOY

JAR

JAR WAR DEPLOY

JAR

JAR

JAR

JAR

JAR

WAR

Test

DEPLOY

JAR

JAR WAR DEPLOY

Actual Pipeline ~= 154 builds

Pipeline Graph

C

E

D

A

StagingTestB

A developer makes a change in module A

Pipeline Graph

B

C

E

D

A

StagingTest

Pipeline Graph

B

C

E

D

A

StagingTest

Pipeline Graph

B

C

E

D

A

StagingTest

#1

#2

Pipeline Graph

B

C

E

D

A

StagingTest

x2

x3 x3

● 7 Modules, 12 Builds● 1 changeset triggers 3 deploys

#Builds(n) = #UniquePaths(A->n)

Build Explosion!

#Builds grow exponentially when the graph grows

...but computing power is cheap (linux)

Build Explosion!

#Builds grow exponentially when the graph grows

...but computing power is cheap (linux)

Option: ignore intermediate results

… while wasting resources

Test

#1

#2

#3the only one with changeset applied to all nodes

#Builds

B

C

D

C

A

WAR Test

● Join: Build doesn’t start until all the previous one have succeeded

● #Build = #Modules● Test runs once per changeset

Smart pipelines in Jenkins

● Build Flow Plugin○ https://github.com/jenkinsci/build-flow-plugin

● Workflow Plugin○ https://github.com/jenkinsci/workflow-plugin

Build Flow Plugin

join = parallel ({

build(“B”)

build(“C”),

})

build(“D”, param1 : join )

B

C

D

BUILD TEST STAGING PRODUCTION

EngineeringBusiness+ Engineering

Continuous Delivery

BUILD TEST DEPLOY

STAGING PRODUCTION

Continuous Delivery

Schema Update

IT Provisioning

Config updates

Roll out

Non-Technical

BUILD TEST STAGING PRODUCTION

Continuous Delivery

BUILD TEST PRODUCTION

Continuous Deployment

BUILD TEST PRODUCTION DEPLOY RELEASE

Release and Deploy are decoupled

● Changes in Production are incremental -> Less risky

● Bug Fixes are the sole responsibility of the Engineers

Engineering Business

Continuous Deployment

BUILD TEST PRODUCTIONDEPLOY RELEASE

Feature Toggles allow decoupling Release from Deployment

Also helpful for:● Canary Releases (release features to subset of users)● A/B Tests

Engineering Business

Continuous Deployment

BUILD TEST PRODUCTION

Continuous Deployment

Limitations:

➔ Legal & Regulatory➔ Distribution (App Stores!)➔ Not suitable for critical software

Q&A