Continuous Delivery with Containers: The Good ... - JAX London · Mechanical sympathy: Docker and Java • Watch for JVM cgroup/taskset awareness • getAvailableProcessors() may

ContinuousDeliverywithContainers:TheGood,theBad,andtheUgly

DanielBryant@danielbryantuk

Containers:Expectationsversusreality

10/10/2017 @danielbryantuk

“DevOps”

Settingthescene…

• Continuousdeliveryisalargetopic• Nobusinessfocustoday(valuestreametc)• PaaSandServerless aresuperinteresting…• ButI’massumingyou’reall-inoncontainers

• Focusingtodayontheprocessandtooling• Nolivecodingtoday• Mini-bookcontainsmoredetails(thanksnginx!)


bit.ly/2jWDSF7

TL;DR– ContainersandCD

• Containerimagebecomesthebuildpipeline‘singlebinary’

• Addingmetadatatocontainersimagesisvital,butchallenging

• Mustvalidatecontainerconstraints(NFRs)• Cultivatecontainer‘mechanicalsympathy’


@danielbryantuk

• IndependentTechnicalConsultant,CTOatSpectoLabs• Architecture,DevOps,Java,microservices,cloud,containers

• ContinuousDelivery(CI/CD)advocate

• Leadingchangethroughtechnologyandteams


ContinuousDelivery


ContinuousDelivery

• Producevaluableandrobustsoftwareinshortcycles

• Optimising forfeedbackandlearning

• Not (necessarily)ContinuousDeployment


Creationofabuildpipelineismandatoryforcontinuousdelivery



TheImpactofcontainersonCD


Containertechnology(andCD)

• OS-levelvirtualisation• cgroups,namespaces,rootfs

• Packageandexecutesoftware

• Containerimage==‘singlebinary’




Creatingapipelineforcontainers



Makeyourdevenvironmentlikeproduction

• Developlocallyorcopy/codeincontainer

• Mustbuild/testcontainerslocally• Perform(atleast)happypathtests


QuickAside:Running*entire*systemlocally


https://news.ycombinator.com/item?id=13960107https://opencredo.com/working-locally-with-microservices/https://www.datawire.io/telepresence/ |https://hoverfly.io/

Makeyourdevenvironmentlikeproduction

• Developlocallyorcopy/codeincontainer

• Mustbuild/testcontainerslocally• Perform(atleast)happypathtests

• Useidenticalbaseimagesfromproduction• Withsameconfiguration


Lessonlearned:Dockerfile contentissuper important

• OSchoice

• Configuration

• Buildartifacts

• Exposingports

• Java• JDKvsJREandOraclevsOpenJDK?

• Golang• Staticallycompiledbinaryinscratch?

• Python• Virtualenv?


Pleasetalktothesysadminpeople:Theiroperationalknowledgeisinvaluable


Differenttestandprodcontainers?

• Create“test”versionofcontainer• FullOS(e.g.Ubuntu)• Testtoolsanddata

• Easytoseeapp/configurationdrift

• Usetestsidecarcontainersinstead

• ONTESTproposalbyAlexiLedenev


http://blog.terranillius.com/post/docker_testing/

Dockermulti-stagebuilds


http://blog.alexellis.io/mutli-stage-docker-builds/https://github.com/moby/moby/pull/31257https://github.com/moby/moby/pull/32063

Javaspecificstuff…


github.com/oracle/docker-images/tree/master/OracleJava jdk.java.net/9/ea

Hotoffthepress:Modularity

• Createminimalruntimeimages

• “jlink deliversaself-containeddistributionofyourapplicationandtheJVM,readytobeshipped.”

• Benefits:• Reducedfootprint• Performance• Security



BuildingimageswithJenkins

• Myreportcoversthis

• Buildasusual…

• BuildDockerImage• CloudbeesDockerBuildandPublishPlugin

• Pushimagetoregistry


Storinginanimageregistry(DockerHub)


Metadata– Bewareof“latest”DockerTag

• Bewareofthe‘latest’Dockertag

• “Latest”simplymeans• thelastbuild/tagthatranwithoutaspecifictag/versionspecified

• Ignore“latest”tag• Versionyourtags,everytime• danielbryantuk/test:2.4.1


Lessonlearned:Metadataisvaluable

• Applicationmetadata• Version/GITSHA

• Buildmetadata• Builddate• Imagename• Vendor

• Qualitymetadata• QAcontrol,signedbinaries,ephemeralsupport• Securityprofiles(AppArmor),Securityauditedetc


Metadata- AddingLabelsatbuildtime

• DockerLabels

• Addkey/valuedatatoimage


Metadata- AddingLabelsatbuildtime

• Microscaling Systems’Makefile

• LabellingautomatedbuildsonDockerHub (h/tRossFairbanks)• Createfile‘/hooks/build’

• label-schema.org• microbadger.com


Metadata- AddingLabelsatruntime


$ docker run -d --labeluk.co.danielbryant.lbname=frontdoor nginx

• Can’docker commit’,butcreatesnewimage

• Notpossibletoupdaterunningcontainer

• DockerProposal:Updatelabels #21721

LizRice(andAqua)totherescue!


github.com/aquasecurity/manifesto

Externalregistrywithmetadatasupport



Componenttesting


Testing:JenkinsPipeline(ascode)



Testingindividualcontainers


Integrationtesting


IntroducingDockerCompose


DockerCompose&JenkinsPipeline


EphemeralKubernetesClusters

• Kubernaut (WIP)

• Managesapoolofclusters

• ”Claim”afreshcluster

• UseHelmtoinstalldependencies


TestingNFRsinthebuildpipeline

• PerformanceandLoadtesting• Gatling/jmeter• Flood.io

• Securitytesting• Findsecbugs /OWASPDependencycheck• Bdd-security(OWASPZAP)/Arachni• Gauntlt /Serverspec• DockerBenchforSecurity/CoreOSClair


DelayingNFRstothe‘LastResponsibleMoment’

• Newsflash!• Sometimesthelastresponsiblemomentisup-front!

• Containers/microservices don’tmakethiseasier• Sometimesmoredifficult…


Mechanicalsympathy:DockerandJava

• WatchforJVMcgroup/taskset awareness• getAvailableProcessors()mayincorrectlyreportthenumberofcpus inDocker(JDK-8140793)• Runtime.availableProcessors()ignoresLinuxtaskset command(JDK-6515172)• Default fork/jointhreadpoolsizes(andothers)isbasedfromhostCPUcount

• Setcontainermemoryappropriately• JVMrequirements=Heapsize(Xmx)+Metaspace +JVMoverhead• Accountfornativethreadrequirementse.g.threadstacksize(Xss)

• Entropy• Hostentropycansoonbeexhaustedbycryptooperations

10/10/2017 @danielbryantuk 46

Deployment


skillsmatter.com/skillscasts/10668-looking-forward-to-daniel-bryant-talk

docs.aws.amazon.com/elasticbeanstalk/latest/dg/using-features.deploy-existing-version.html

Observabilityiscoretocontinuousdelivery


www.infoq.com/articles/monitoring-containers-at-scale

Containersarenotasilverbullet


Movingtocontainers:Goingall-in?


OR

ShouldIbuildmyowncontainerplatform?

Probablynot(UnlessyouareGoogle,AWSorIBM)

Whateveryoudecide…pushitthroughapipelineASAP!


Usingcontainersdoesnotobviatetheneedforgoodarchitecturalpractices



https://speakerdeck.com/caseywest/containercon-north-america-cloud-anti-patterns

Summary


Insummary

• Continuousdeliveryisvitallyimportantinmodernarchitectures/ops

• Containerimagesmustbethe(single)sourceoftruthwithinpipeline• Andmetadataaddedasappropriate…

• Mechanicalsympathyisimportant(assertpropertiesinthepipeline)• Notalldevelopersareoperationallyaware

• Thetoolingisnowbecomingstable/mature• Weneedtore-applyexistingCDpracticeswithnewtechnologies/tooling


Bedtimereading


Thanksforlistening

• Anyquestions?

• Feelfreetocontactme• @danielbryantuk• [email protected]


bit.ly/2jWDSF7

Comingsoon!

Bonusslides(forextracontext)


Containerise anexisting(monolithic)app?

• For

• Weknowthemonolithwell

• Allowshomogenizationofthepipelineanddeploymentplatform

• Canbeademonstrablewinfortechandthebusiness

• Against

• Canbedifficult(100+linescripts)

• Oftennotdesignedforoperationwithincontainers,norcloudnative

• Puttinglipstickonapig?


Keylessonslearned

• Conductanarchitecturalreview• ArchitectureforDevelopers,bySimonBrown• ArchitectureInterview,bySusanFowler

• Lookfordataingress/egress• Filesystemaccess

• Supportresourceconstraints/transience• Optimise forquickstartupandshutdown• Evaluateapproachtoconcurrency• Storeconfiguration(secrets)remotely


Newdesignpatterns


bit.ly/2efe0TP

Microservices…

Containersandmicroservices arecomplementary

Testinganddeploymentchange


https://specto.io/blog/recipe-for-designing-building-testing-microservices.html




Documents

Continuous Delivery with Containers: The Good ... - JAX London · Mechanical sympathy: Docker and Java • Watch for JVM cgroup/taskset awareness • getAvailableProcessors() may