Delivery Engines: Software & Spaceflight

cc: GlennPope - https://www.flickr.com/photos/89629638@N06

Delivery Engines(software & spaceflight)

Max Lincoln

JIM HIGHSMITH - VELOCITY IS KILLING AGILITY

▸ Focusing on [velocity] detracts from the quality of the customer experience delivered and investing enough in the delivery engine (technical quality).

▸ Giving the product owner/manager complete priority control … skews the balance of investing in new features versus the delivery engine.

▸ investment in the delivery engine is as critical as investing in new features.

▸ Management needs to allocate resources between features and engine work

SPACEFLIGHTACHIEVEMENTS

MANNED MISSIONS

SATELLITES & PROBES

ROVERS

INVESTMENT IN DELIVERY

PIPELINESSTAGE 1

You’re doing continuous delivery when:

Your software is releasable throughout its lifecycle;

Your team prioritizes keeping the software releasable over working on new features;

Anybody can get fast, automated feedback on the production readiness of your systems any time somebody makes a change to them;

You can perform push-button releases of any version of the software on demand.

PIPELINE BUILDING BLOCKS

Material (Version Control)

Git

Job

Git BuildBuildBuild

Stage

Git

BuildBuild

Build

LintLint

BuildBuild

Lint

Pipeline

GitBuild

Publish

Upload

Artifact Repo

Git

BuildBuild

Lint

Build

PublishUpload

Artifacts

Publish

Upload

Git BuildBuild

Artifacts

Publish

BuildBuild

Artifacts

Publish

AcceptanceDeploy Test

ExploratoryDeploy

BuildBuild Publish

AcceptanceDeploy Test

ExploratoryDeploy

ProdDeploy

GOOD PIPELINES:

- Incremental quality assurance- Fast feedback to the right people- Visibility and shared understanding of the progress of a release candidate- A consistent and auditable software delivery process

AND SCALING CHALLENGES:

X-AXIS:SCALING MORE STAGES

ProdSource

HOW SIMPLE CAN YOU GET?

Source Prod


TestBuild

Source Prod


TestBuild

Source Prod


TestBuild

feedback

Source Prod

MOAR TESTING

TestBuild QA

feedback

Source Prod

HMM…

TestBuild QA PerfSec

culture feedback governance

Source Prod

HMM…

TestBuild QA PerfSec

culture

governance

Source Prod

Test

Build

QA Perf

Sec

Source Prod

Test

Build

QA Perf

Sec

PARALLELISM

Source ProdBuild Test

Int

QA

Sec

Perf

X-AXIS: PATTERNS

SIMPLE LINEAR

SIMPLE PARALLEL

feedback

LAYERED

culture

LAYERED w/ ARTIFACT PROMOTION

governance culture

RELEASE CHANNELS

feedback


Y-AXIS:SCALING MORE PROJECTS


Z-AXIS:CONCURRENT RELEASES

Y-AXIS:SCALING MORE PROJECTS


Tips• Model & share you pipeline first

• Automate all that you can

• Always look for the “bottleneck” and make improvements

• https://github.com/thoughtworks/PipelinePatterns

https://github.com/thoughtworks/PipelinePatterns

https://github.com/thoughtworks/PipelinePatterns

ANY IMPROVEMENTS MADE ANYWHERE BESIDES THE BOTTLENECK ARE AN ILLUSIONTheory of Constraints

(Paraphrased in The Phoenix Project)

TOGGLESSTAGE 2

CTOGGLES

"Don't worry. If you flip the wrong one you die. No pressure."

Knight Capital• Costliest bug ever?

• Lost $172,222 per second

• Lost $440m in 30 minutes

ASSUMPTIONS

TOGGLE STATES

•On or off•Off by default•Off = previously released behavior

Opt-in

TOGGLE LIFECYCLE

•Similar to cards•Defined transitions

QA

TOGGLE IDS

•Same = approved together•Cannot reuse once accepted

No “rider

s”

DEV QA PROD

v2 v2 v1

Feature A ON ON ON

Feature B OFF OFF ON

Feature C ON OFF OFF

Feature D OFF ON OFF

HOW?

ENVIRONMENT MAPPING

DERIVING STATES

R1 (WIP) R2 (DONE) R3 (READY)

env criteria

CI >= WIP ON ON ON

SHOWCASE >= WIP ON ON ON

DEV1 >= WIP ON ON ON


QA RELEASE >= READY OFF OFF ON

QA Next >= DONE OFF ON ON

Perf >= DONE OFF ON ON

Security >= DONE OFF ON ON

Prod >= READY OFF OFF ON

DR >= READY OFF OFF ON


env criteria

CI >= WIP ON ON ON







Security >= READY ON ON ON

Prod >= READY OFF OFF ON



env criteria

CI >= WIP ON ON ON







Security >= READY ON ON ON

Prod >= READY OFF OFF FORCE OFF


Tips• Toggles exist to protects

developers & business from an “accidental release”

• Practice while it’s still safe to make mistakes

• Toggles avoid the need for “merging strategies”

COMMUNITIES

STAGE 3

EVEN WITH THE BEST TOOLS, DEVOPS IS JUST ANOTHER BUZZWORD IF YOU DON'T HAVE THE RIGHT CULTURE.Rouan Wilsenach

http://martinfowler.com/bliki/DevOpsCulture.html



HOW CAN WE BUILD DEVOPS-

FRIENDLYCOMMUNITIES?

SMALL, ALIGNED TEAMS

AMAZON: TWO PIZZA TEAM

SIZE & ALIGNMENT

CONWAY'S LAW: ORGANIZATIONS ARE CONSTRAINED TO PRODUCE APPLICATION DESIGNS WHICH ARE COPIES OF THEIR COMMUNICATION STRUCTURES.INVERSE CONWAY MANEUVER: EVOLVE YOUR TEAM AND ORGANIZATIONAL STRUCTURE TO PROMOTE YOUR DESIRED ARCHITECTURE.

ThoughtWorks Tech Radar

INVERSE CONWAY MANEUVER

AS PART OF BIGGER

COMMUNITIES

SPOTIFY: SCALING AGILE

TW OFFICES: DUNBAR’S NUMBER

SIZE & ALIGNMENT

IN FAIL-SAFE

ENVIRONMENTS

THAT ARE PREPARED FOR FIRES

Tips• DevOps will not thrive without the

right community / culture

• The most important thing is small, cross-functional teams

• … that collaborate with other teams

PLATFORMSSTAGE 4

WTF IS A PLATFORM?

EVERYONE HAS A DIFFERENT IDEA

ABOUT WHAT THE PLATFORM IS, BUT

OBVIOUSLY I’M ALWAYS RIGHT :PARCHITECT

?“Platform Engineering

Team”

PAAS

PLATFORMS ARE THINGS THAT WE BUILD SOFTWARE ON TOP OF: MOBILE TECHNOLOGIES LIKE ANDROID, VIRTUAL PLATFORMS LIKE THE JVM, OR GENERIC KINDS OF PLATFORMS LIKE HYBRID CLOUDS.

TW DEFINITION

• THE CAPABILITY PROVIDED TO THE CONSUMER IS TO DEPLOY ONTO THE CLOUD INFRASTRUCTURE CONSUMER-CREATED OR ACQUIRED APPLICATIONS CREATED USING PROGRAMMING LANGUAGES, LIBRARIES, SERVICES, AND TOOLS SUPPORTED BY THE PROVIDER.

• THE CONSUMER DOES NOT MANAGE OR CONTROL THE UNDERLYING CLOUD INFRASTRUCTURE INCLUDING NETWORK, SERVERS, OPERATING SYSTEMS, OR STORAGE, BUT HAS CONTROL OVER THE DEPLOYED APPLICATIONS AND POSSIBLY CONFIGURATION SETTINGS FOR THE APPLICATION-HOSTING ENVIRONMENT.

NIST

PAAS

THE CAPABILITY PROVIDED TO THE CONSUMER IS TO DEPLOY ONTO THE CLOUD INFRASTRUCTURE CONSUMER-CREATED OR ACQUIRED APPLICATIONS CREATED USING PROGRAMMING LANGUAGES, LIBRARIES, SERVICES, AND TOOLS SUPPORTED BY THE PROVIDER.THE CONSUMER DOES NOT MANAGE OR CONTROL THE UNDERLYING CLOUD INFRASTRUCTURE INCLUDING NETWORK, SERVERS, OPERATING SYSTEMS, OR STORAGE, BUT HAS CONTROL OVER THE DEPLOYED APPLICATIONS AND POSSIBLY CONFIGURATION SETTINGS FOR THE APPLICATION-HOSTING ENVIRONMENT.

NIST

PAAS

"DOCKER AS PROCESS, PAAS AS MACHINE, MICROSERVICES ARCHITECTURE AS PROGRAMMING MODEL"DEVELOPERS CAN THINK OF A CONTAINER AS A SELF-CONTAINED PROCESS AND THE PAAS AS THE COMMON DEPLOYMENT TARGET, USING THE MICROSERVICES ARCHITECTURE AS THE COMMON STYLE. DECOUPLING THE ARCHITECTURE ALLOWS THE SAME FOR TEAMS, CUTTING DOWN ON COORDINATION COST AMONG SILOS.

ThoughtWorks Tech Radar Trends

DOCKER / PLATFORM / MICROSERVICE

https://www.infoq.com/presentations/platform-manifesto

MYOB

https://www.infoq.com/presentations/platform-manifesto

Tips• Build on top of a platform so you

can focus on your business

• Kubernetes is a trendsetter for platforms of the future

• Follow the platform manifesto

TOOLSSTAGE 5

WE SHAPE OUR TOOLS AND AFTERWARDS OUR TOOLS SHAPE USMarshall McLuhan

http://www.devopsbookmarks.com/


https://www.thoughtworks.com/radar

https://www.thoughtworks.com/radar

EXAMPLES (HASHICORP)▸ Adopt

▸ Consul▸ Packer

▸ Trial▸ HashiCorp Vault▸ Terraform



Tips• If you want great software, the

people building & operating it need great tools

• Invest in custom tools

• But avoid tight coupling

Technology

Delivery Engines: Software & Spaceflight