1.1

1.2

2.1

2.2

2.3

2.4

2.5

2.6

3.1

3.2

3.3

4.1

4.2

5.1

5.2

5.3

5.4

5.5

5.6

5.7

5.8

6.1

6.2

TableofContentsIntroduction

Self-Assessement

Architecture

BuildDifferentiators

DesignforPaceofChange

EvolutionarySystems

ScaleHorizontally

SmallandSimple

SmartsintheNodesnottheNetwork

Operational

CloudNative

DataStewardship

ProductionReady

Organisation

KeepPacewithTechnologicalChange

ModeltheBusinessDomain

Technology&Practices

SecurebyDesign

AutomatebyDefault

ContinuousDelivery

ConsistentEnvironments

Maintainability

PerformanceImportance

GetFeedbackEarlyandOften

DesignforTestability

Drafts

DesignforUse(DRAFT)

PerformanceImportance(DRAFT/rewrite)

1

6.3Maintainability(DRAFT/rewrite)

2

JohnLewisITSoftwareEngineeringPrinciplesThroughourcombinedexperienceofbuildingandreleasingsoftwarewehavediscoveredandcometovaluethefollowingprinciples.Theseprinciplesarenothardandfastrulesbutrathersomethingsthatweapplyandusetoguideusonadailybasis.Theseprinciplescouldalsobeusedina"discoveryphase"whenselectinganewproducttopurchase.

Thebasicanatomyofaprincipleis:

Name-ThisistheessenceoftheruleStatement-UnambiguouslycommunicatethefundamentalruleRationale-ThebusinessbenefitsofadheringtotheprincipleImplications-TherequirementsbothforthebusinessandITforcarryingouttheprinciple

TheconceptofcreatingprinciplesinthismannerhasbeenborrowedfromTOGAF'sArchitecturalPrinciples.

Printableversion

APDFversionisavailable.

Invitationtocontribute

Ifyou'reinvolvedinthesoftwareengineeringprocessatJohnLewis,thenwewantyourinput!TheprinciplesaremaintainedinaGitLabprojecttech-council/engineering-principles.

Forcorrectionoftypos,fixingbrokenlinks,oranythingsimilarlyunlikelytoraiseanydebate,pleaseforktheproject,makethechangeandraiseamergerequest.

Formoresignificantadjustmentstoanexistingprinciple,dothesame,butperhapsanticipatesomediscussiononthemergerequest.

Forsuggestionsofnewprinciples,orfundamentalchallengestoexistingones,pleasestartaconversationin#comm-victoria-engineering-group

Introduction

3

Self-assessmentHerearesomequestionsyoumightfindusefulwhenrunningaself-assessmentwithyourteamagainsttheseprinciples.

BuildDifferentiators

Doyouconsiderthiswhenchoosingthetechnology?GoogleCloudDatastoreorElasticsearchareexamplesof"buy"Doyoureinventthewheel,buildingsystemsalreadyoutthere?

DesignforPaceofChange

Iftherearelotsofdifferentcomponentsdoyoutakeintoaccountthepaceofchangeofthecomponents?Doyouover-engineeryoursolutions?Doyoureuseforthesakeofit?Isyoursystemgoingtobearoundforalongtime?

EvolutionarySystems

Ifyouchangeyoursystemisitanisolatedchangewithregardtoothersystems?Howmanypeopledoesittaketorelease?Howoftencanyourelease?CanOpsreleaseduringtheday?Howmanyrequirementsdoyouputintooneiteration?Howfutureproofisitforrequirementsthatmaynotexist?

ScaleHorizontally

Statecannotbestoredin-memorybutrathermustbepersistedtotheclientorashareddatastore.Itisnotalwayspossibletoscalehorizontallye.g.fortraditionalrelationaldatabases,increasedperformanceisoftenachievedthroughbetterhardwareorhigherspecificationsServicesshouldstrivetobeidempotent

SmallandSimple

Howlongdoyourteststake?Canyoudescribeyourcomponents?Canyoutestyourcomponentsindependently?Canyoudeploycomponentsindependently?

SmartsintheNodes,nottheNetwork

Doyouhavebusinesslogicinyourintegrationlayer?Doyouapplybusinessrulesinyourdatatransformations?Ifthereisbusinesslogicisitforyourownsystem’sconsumption?

Self-Assessement

4

CloudNative

Areyouleveragingtheservicesinthecloud?Didyouintendittobeinthecloudfromthestart?

ProductionReady

Doyouunderstandyourfailuremodes?Doyouunderstandhowmonitoringhelpsyouunderstandyourfailuremodes?Howcloseisthesystemtoitslimit?Aretherecircuitbreakers?Arethererollingdeployments,egkubernetes?Doyouhavedashboards,goldensignals,engineeringmetricsfortrafficandresilience?

KeepPacewithTechnologicalChange

Whattechnologyistheteamusing?Isituptodate?Isitwellsupported?Isitcurrent?Howoftenisitupgradedandpatched?Aretheteaminterestedinthisarea?Isthesoftwareonitswayout?

ModeltheBusinessDomain

DoesituseDDD(Domaindrivendesign)?Couldabusinesspersonunderstandthelanguageusedinyourcode?Howwelldoyouownyourdomain?Aretherechangesnotownedbyyourteam?

SecurebyDesign

Haveyouconsideredaccesscontrol?

ConsistentEnvironments

Areyouusinginfrastructureascode?Examplesareterraformorpuppet.Haveyougotanyprocessesinkeepingdatauptodate?Aretherelotsofmanualprocesses?

ContinuousDelivery

Areyoudoingit?AreyouusingaCItool?Whatmanualgatesareinplace?Canyoureleasestraighttoproduction?

Self-Assessement

5

Areyouusingtrunkbaseddevelopmentwithfeatureflagsasnecessary?Arebranchesshort-lived?

GetFeedbackEarlyandOften

Couldyoudodailyreleases?

Maintainability

Ifanewpersonjoins,cantheygetuptospeedquickly?Cancodebesafelyrefactoredduetogoodtestcoverage?

Self-Assessement

6

BuildDifferentiatorsIffunctionalityisadifferentiatorforthePartnershipthenweshouldprefertobuild,ratherthanbuyand/orcustomise.

Rationale

CustomisationofCommercialOff-The-Shelf(COTS)packages,andmisuseofSoftware-as-a-Service(SaaS)beyonditsdesigned-forusecases,canbecostlyincomparisontobuildingequivalentfunctionality.Italsohasalonger-standingimpactonthefuturepaceofchange,requiringmaintenanceanddevelopmentontopofoften-complexcustomisationsthatrequirespecialistknowledge.

Implications

ExtractwhatvaluewecanfromexistingCOTSpackagesandSaaS.UseCOTSpackagesandSaaSfortheirspecificstrengths,andcomposedifferentiatingsystemsaroundthemasappropriate.Legacysystems,wherewecandifferentiate,mayneedawrapperlayertofacilitatestrangulationandsubsequentreplacement.Preparefortoday’sdifferentiatorbecomingtomorrow’scommodity.Weshouldwatchthemarketandchangeourapproachwhenthereisvalueindoingso.

BuildDifferentiators

7

DesignforPaceofChangeForcomponentsthatchangeoften,favourdesigningforpaceofchangeoverre-use

Rationale

Forsystemsthatchangeoften,designingforre-usecaninhibitrapidchangebycreatingdependenciesandbottlenecks.Ifasystemchangeslessfrequentlyitsmaintenancecostbecomesmoreimportantanddesigningforreusecanhelpreducethis.

Codethatisdesignedforre-useisgenerallyhardertouseandmoreexpensivetocreatethancodedesignedforasinglepurpose .

Implications

Focusonre-useasitemergesthroughevolution.Bescepticalofcomponentsthatlooksimilarbutaren'tthesameDomain-specificcomponentsharingisananti-pattern;codere-usebetweenboundedcontextsisahazardtobeavoided .SystemsshouldexposecommonbusinessfunctionsthroughAPIs.Ifre-usedcodebecomesabottleneck,removethecouplingbyforkingorduplicatingthecode .Featuressuchasmonitoring,logging,servicediscovery,authentication/authorisationoftenneedtobeimplementedconsistentlyacrossallservices/applications.UseServiceTemplatestodefinewhichlibrariesshouldbeusedbyallservices .Sharedcomponentsrequirerobusttests,sufficientdocumentationandongoingmaintenance;awebsitetohostgenerateddocsmaybeuseful.

.[BuildingEvolutionaryArchitectures:SupportConstantChangebyNealFord,PatrickKua,andRebeccaParsons](http://shop.oreilly.com/product/0636920080237.do)↩

.[DomainDrivenDesignbyEricEvans](http://dddcommunity.org/book/evans_2003/)↩

1

2

1

1

1

2

DesignforPaceofChange

8

EvolutionarySystemsSystemsandarchitecturesshouldbedesignedandbuilttoenableeasy,incrementalchange.

Rationale

Atthispointintimeweknowlessthanwe’llknowinsixmonths.Oursoftwareshouldbeabletoevolveaswelearnmoreaboutthebusinessdomain,theoperationalenvironment,andthetechnologyitself.Changeisaconstantintheworldaroundus,anditsimpactisincreasinglyunpredictable.However,byengineeringoursoftwareforevolvabilityasaprimaryconcern,wegiveourselvesthebestchanceofexperimenting,learningandthrivinginnewconditions.

Implications

Whilstwecannotandshouldnotdesignforeverypossibleeventuality,theevolvabilityofoursystemscanbehugelyimproved,atlowcost,ifthelikelytriggersandimpactsofchangeareconsideredearlyenough.Uncertainties,“what-if”business&technologychangesandfailuremodesshouldbemadeexplicit,rehearsedandmitigatedwherethecosttodosoislowrelativetotherisk.UseSmallandSimpleandModeltheBusinessDomaintogrouptogetherfunctionalityinwaysthatminimisethesplash-zoneofpotentialchange.Whenmakingdesigndecisions,giveyourselfthebestchanceofadaptingtonewinformationbywaitingtillthelastresponsiblemoment,andpreferringchoicesthatareeasilyreversibleorthatkeepyouroptionsopen.Defineandregularlymeasurethearchitecturalqualitiesthataremostimportanttosuccess,andwhichmustbenurturedasthesystemevolves.Systemsandprocesseswillneedtobeexposedthroughclearlydefinedinterfacecontractssothattheunderlyingimplementationcanchangeandevolvewithoutimpactingthewidersystem.(Note:thisdoesnotmeanthatacomponentorsystem'sinternalinterfacesnecessarilyneedtobeapproachedinthesameway.)Componentsmustbeaccessiblethroughopen,non-proprietarystandards(withapreferenceforHTTP&inparticularREST).Embracingthebestpracticesofthewebmaximisestheopportunityforevolutionwhileminimisingtheriskoftechnicalredundancy.Standardsthatarederivedfromworkingsoftwareandnotjustworkinggroupsarepreferred.Battle-testedstandardsusedbyseveralorganisationstendtobemoreadaptablethanover-engineeredhotair.MostsuccessfulstandardstendtobetheonesthatareadoptedbyOpenSourcecommunities.AdoptpatternssuchasTolerantReaderandthePrincipleofRobustnesstoenablesystemstoreleasechangesindependentlywithoutnecessitatingparallelchangeversioningandmigration.Sometimesgradual/incrementalchangeisinsufficientasrequirementschange,soconsiderSacrificialArchitectureasapattern.Don'tbeafraidtothrowawayandre-write,asthisisoftenfasterandwillleadtobetterquality.IfwehavekeptthingsSmallandSimplethisshouldnotturnintoapainfulBigRewrite.Thisprincipledependsonanumberofotherstobeabletoapplyitinfull:SmallandSimple,ModeltheBusinessDomain,ContinuousDelivery,Maintainability.

EvolutionarySystems

9

ScaleHorizontallySystemcomponentsshouldbehorizontallyscalablewherepossible.

Rationale

Itischeapertoscaleservicesbyaddinginexpensiveresourceslikecommodityserversordatabasenodesratherthanbuyinglargerandlargerpiecesofhardwarewhich,atagivenpoint,areunabletoscalefurther.Byembracinghorizontalscalabilityearlyanddesigningoursystemstoworkinthismanner,weeliminatethecostofchangingatalaterdate.

Implications

Conversationalstatecannotbestoredin-memoryofaserviceinstancebutrathermustbepersistedtotheclientorashareddatastore.Providedtheymeetyourneeds,prefer“serverless”persistenceoptionsfromyourcloudprovider,sincethesewilltypicallyhandlethescaling-outofyourapplicationwithnoextraeffort.e.g.GoogleCloudDatastoreratherthanGoogleCloudSQL.Itisnotalwayspracticaltoscalehorizontally.Forexample,increasedperformancefortraditionalrelationaldatabasesisoftenachievedthroughbetterhardwareorhigherspecifications.

ScaleHorizontally

10

SmallandSimpleBuildsmallandsimplesystems.

Rationale

Monolithicapplicationscanbesuccessful,butithasbecomecommonfororganisationstofeelfrustrationswiththemastheygrowincomplexityandscopeovertime.Changecyclesaretiedtogether–achangemadetoasmallpartoftheapplicationrequirestheentiremonolithtoberebuilt,retestedanddeployed.Overtimeitisoftenhardtokeepagoodmodularstructure,makingitchallengingtokeepchangesthatoughttoaffectonlyonemodulecontainedwithinthatmodule.Scalingrequiresscalingoftheentireapplicationratherthanpartsofitthatrequiregreaterresources.Scalingrequiresupliftoftheentireapplicationratherthanonlythepartsthatrequiregreaterresources.

Buildingsoftwarethatissmallandsimplealsomakessystemseasiertoreasonabout,andeasiertotestindependently.Thingsthatareeasiertoreasonabout,withlesscode,tendtogeneratefewerbugsandreducethelikelihoodofcomplexissuesinProduction.

Implications

Aimtodecomposeyoursystemintoasuiteofsmallersystems,eachofwhichfocusondoingasmallnumberofthingswell.Buildsystemsaroundbusinesscapabilitiesthatchangeindependentlytoothers,ratherthanindividualfeaturesortechnologies.Bewaretherisksofcreatingadistributedmonolith.Servicesareindependentlydeployableandindependentlytestable.Reducetheburdenofmanagingseveralsmallersystemsthroughautomation.Indistributedsystems,itisimportanttodesignforfailure.Publishinterfaces.

SmallandSimple

11

SmartsintheNodesnottheNetworkSometimesreferredtoas“smartnodesanddumbpipes”,meaningthatsystemsaimtobeasdecoupledandcohesiveaspossible,andnotcentrallychoreographedinmiddleware.

Rationale

Composinglargersystemsaroundsimplerandsmallersystemsalsomeansneedingtoowntheircommunicationwithothersystemsaswell.Toallowsystemstoevolveovertimeandkeepupwiththebusiness’sdesiredpaceofchange,havingthemowntheircommunicationallowsthemtochangequicklyratherthannegotiatingachangetothepipesthatconnectthem.

Implications

Preferopenprotocols,suchasHTTP.Servicesactlikeunix-filters,acceptingrequests,applyinglogicandreturningaresponse.Implementintegrationadapterstoisolatethetechnicalitiesofintegration,andanti-corruptionlayerstoisolatethetranslationofbusinessdata,betweenboundedcontexts.Whendataneedstobeexposedorexportedoutsidetheserviceboundary,wrapthiscommunicationinaservicelayerratherthanallowingconsumerstoaccessthedatadirectly.Thishelpspreventbusinesslogicleakingoutofthesystem.

SmartsintheNodesnottheNetwork

12

CloudNativeBuildsystemsthataresuitableforrunninginthepubliccloud,usingcloud-nativetechnologiessuitedtoourproviderofchoice.

Rationale

Useofcloud-nativeserviceshelpsusfocusonbuildingdifferentiatorsandsupportingarapidpaceofchange,andreducesoperationalcomplexity.Thisisparticularlyimportantwherewerunalargersetofdistributed,smallersystemswhichneedtooperateindependently.

Implications

Prefer“Platform-as-a-Service”over“Infrastructure-as-a-Service”.Wherepossible,usetheJLDigitalPlatform,whichhasbeenbuiltwiththisprincipleinmind,andprovidesacuratedsetoftoolsandservicestoachievethis.Preferopentoolingforconfigurationanddeploymentthatworkswellwithourchoiceofcloud.Leveragetheserviceswhereourcloudproviderisstrong.Donotovercommittobeingcloud-agnosticwherethiscreatesalargeamountofengineeringeffortormissestheopportunitytomakeuseofhigh-qualitycloudfeatures.Buildsoftwaretotakeadvantageofcloudfeatures,andbetolerantofcloudfailurescenarios.Forexample,considervariablenetworkconditionsandminimisingoflocalstatetosupportautomaticscaling.Teamsshouldcontinuouslyreviewthecloudservicestheyconsumeforsuitabilityandcost-effectiveness.Compute,storageandnetworkresourcesareabstractedfromapplications,isolatingthemfromunderlyinginfrastructuredependenciesandthereforeimprovingportability.Cloud-NativetechnologiesarefocusedaroundAPIsratherthanlow-levelinfrastructureinterfaces.

CloudNative

13

DataStewardshipActivelyprotectandcareforthedatastoredandaccessedbyasystem,atleastasmuchasthefunctionalbehaviourofthatsystem.

Rationale

Informationisafundamentalassettothebusiness,criticaltoitssuccess.Itmaybere-usedoradaptedtoaddsignificantvalueinnewbusinesscontexts,wellbeyonditsoriginalintent,outlivingthesystemthatfirstcreatedit.Inadequatecarefordataconcernscanresultinmisuse,insecurityandunintentionalcouplingbetweensystems,generatingwidespreadcomplexityandflakiness,aswellassignificantfinancialandreputationalrisktothebusiness.Theresultingcostsandconstraintscanbeextremelypainfulandexpensivetoresolve.

Implications

Eachboundedcontextmustmanagethedataituses,actingasaresponsibleowner,evenifitisnotnecessarilythe“master”ofthatdata.Teamsmustunderstandandaddressanydataprivacyconcernsrelatedtotheirdata,includingpolicyandlegalcompliance.Forexample:classification&retentionpolicies,GDPR,PCI-DSS.Whereothersneedtoconsumeanapplication’sdata,itshouldexposeitviaaninterfacewithanexplicitcontract.Consumersofdataexposedbyanapplicationshouldbeabletoeasilyfindmetadataaboutitsmeaningandqualitye.g.definitions,accuracy,freshness,time-to-liveaswellasoperationalservicelevelse.g.RTO,RPO,supporthours.Itshouldbeclearwhotheyshouldcontacttoaskquestionsaboutit.Wherethereisapotentialchoiceofdatasources,thereisariskofaccidentalmisuse,complexityandflakiness.Consumersshouldfirstmodelthebusinessdomaintoidentifytheboundedcontexttheywantinformationfrom,thenaimtogetascloseaspossibletothecommonlyagreedsourceoftruthforthatcontext.Thiswillusuallybetheoriginortrusted“goldenrecord”thatparticularbusinessdomainreliesonoperationally.Consumersandprovidersmaybothneedtobalanceavailabilityandaccessibilityofthedatawithothernon-functionalconsiderationssuchasperformance,qualityandfreshness.Thesefactorsaretypicallyintensionandneedtobetradedoff.Datacanbetransformedortranslatedasitisshared.Applicationsshouldrepresenttheircopyofdatausingschemasdesignedforthecontextoftheirbusinessdomainandusecases.Whenmakingsignificantchangesthatcouldbebreakingforconsumers,includingchangesinmeaning,makeuseofversionedinterfaces.

Relatedreading

Dataintegrityattheorigin|ThoughtworksTechnologyRadarAPIsasaproduct|ThoughtworksTechnologyRadarHowtoMoveBeyondaMonolithicDataLaketoaDistributedDataMeshDataconsiderationsformicroservices|AzureArchitectureCenter

DataStewardship

14

ProductionReadySystemsareengineeringforuseinProductionfromthestart–theyarescalable,observableandtolerantoffailure.

Rationale

Buildingmore,smallersystemsandutilisingcloudenvironmentsmeansunderstandingandadaptingtochange,andhandlingfailure,ismoreimportantthanever.Ideally,asystemisself-healingorproducesonlyrelevant,actionablealerts.

Implications

ReleaseItwillbeusedasthestartingpointforwhatdefinesProductionReady.Thesystemshouldhavesufficientmonitoring,loggingandalertingtobeabletounderstandandreportonitshealth,andthehealthofitsdependencies,inaconsistentway.AnyoneinJLPshouldbeabletoaccessourtelemetrytools.Systemsshouldgracefullydegradeondisruption–forexample,byusingcircuitbreakers,bulkheads,cachingorpartialresponses.TeamsandbusinessownerswillneedtoworktogethertoidentifyrelevantbusinessandtechnicalKPIsandSLOsfortheirproduct.Teamsshouldunderstandandimprovetheresilienceoftheirservicethroughfailuremode&effectsanalysis,andbyexperimentingwithintentionalfaults.DashboardsneedtobeimplementedforthemostimportantSLOs.Notallriskstoproductionreadinessareanalysableinadvancesoexploratorytesting,productionobservability,faultinjection,post-incidentreviews,andoperationalmonitoringshouldbeusedtoexposenewinformationaboutsoftwarebehaviour.Applicationsshouldbebuiltwithadiversityofstakeholdersinmind.Operabilityandsupportabilityareimportantinmostcontexts,butsee[https://en.wikipedia.org/wiki/List_of_system_quality_attributes]foralistofothersoftware'ilities'thatmayneedparticularconsiderationinyourcontext.

ProductionReady

15

KeepPacewithTechnologicalChangeUsethelatest,mostappropriatetoolsandtechnologiesthatsolvethebusinessproblem.

Rationale

Technologyadvancesatanever-increasingrate.Pragmaticallyadoptingthelatesttoolsandtechnologieswillallowustoprovideinnovativenewproductsandservicestoourcustomersandbusinessandcompetemoreeffectivelywithourcompetitors.

Implications

TheprincipleofEvolutionarySystemsisessentialinorderforustoupdateandupgradesystemswithoutimpactingthebroaderecosystem.Thebestpeopletochoosetoolsandtechnologiesarethosethatareclosesttotheproblembeingsolved.Guidelineshelpusadoptnewtechnologiesconsistently.TheArchitecturefunctionandtheVictoriaEngineeringGroup,supportedbyourengineeringcommunities,helpaligntechnologiesandidentifysynergieswhereappropriate.Regularpatching,ifpossible,willhelpfuturesoftwareupgradesandmaintainabilityandkeepsystemsmoresecure.Technologyselectionconflictswillneedtoberesolvedthroughacollaborativeframeworkwhichisopentoadvancingtechnicalexcellence.WehavecreatedaTechnologyRadarthatidentifiesandmonitorsthetechnologiesthatwebelievewillberelevanttoourbusiness.Newtoolsandtechnologiesshouldbefilteredthroughthisprocess.Thereshouldbeacommunityofinterestaroundsoftwareengineeringtosharegoodpracticeandtechnologynews.

KeepPacewithTechnologicalChange

16

ModeltheBusinessDomainTerms,conceptsandcapabilitiesofthebusinessshouldbereflectedinthewaywewrite,structureanddeployourcodeandsystems.

Rationale

Usingacommonlanguagethroughouttheteamtodescribebusinessconceptsandprocessesallowsforacommonunderstanding.Misunderstandingsthatleadtodefectsareeasiertoidentifybecausethelanguagethatisusedallowsnon-technicalstakeholderstounderstandandcorrectit.

Structuringcomponentsystemstoreflectcoherentbusinesscapabilitiesandinfluencetheorganisationalstructurewillenableproduct/businessowners,teamsandsystemstobemorecloselyalignedtothecustomervaluetheyprovide,resultinginmoreresponsivedeliveryandoperations.

Implications

WewillneedtotakeadvantageofConway'sLaw(wheresoftwarereflectstheorganisationthatbuiltit)tovalidateandinfluenceboththesystemswewriteandhowwestructuretheorganisation.Wewillneedtoapplydomainmodellingtechniquestoidentifyboundedcontexts,usingacontextmaptoalignsystemboundariesandinterfaces.Wherepackageshavetheirownmodelandterminology,wemayneedtocreateabstractionlayerstomaptoourbusinessmodel.

ModeltheBusinessDomain

17

SecurebyDesignSystemswillbedesignedandmaintainedwiththeassumptionthattheywillbeattackedandpossiblycompromisedbyanybody.

Rationale

Thecostofhandlingasecuritybreach issignificantlygreaterthanthecostofhardeningthesysteminthefirstplace.Ifabreachdoesoccur,wecanminimisethescaleandcostbydetectingitassoonaspossible,andhavingawell-plannedresponseenablesafasttimetoclosethebreachandassessitsimpact.

Byfocussingonsecurityfromthebeginningwemitigatenotonlythefinancialimpactbutalsothereputationalimpactthatwouldbeincurredintheeventofabreach.

Implications

TeamsshouldbeawareofdataprivacyimplicationsincludingpolicyandlegalcomplianceTeamsunderstandthesecurityrisksandmodelthreatvectors.Datawillneedtobeappropriatelyclassifiedandsecured,bothintransitandatrest.Peopleatallpointsoftheengineeringprocessneedstrongawarenessofsecuritytechniquesandprinciples,andshouldapplythesethroughoutthelifecycleofthesoftwareTeamsapplytheprincipleofleastprivilege.WewillusetheOWASPguidelinesforsoftwaredevelopmentasastartingpointtohelppeopleunderstandpotentialattackvectorsandmitigationtechniques.AutomatedvulnerabilityanalysistoolscanhelpidentifyvulnerabilitiesbutarenotareplacementforaSecurebyDesignapproach.Allnetworkcommunicationshouldbedoneoverasecuretransportlayer.Thesystemshouldbeabletodetectandallowretrospectiveanalysisofattacksthroughanappropriateaudittrail.Teamsshouldhaveaclearstrategyinplacetorespondtoattacksanddatabreaches.

.A2015surveyfoundtheaveragesecuritybreachcostsalargeorganisation£1.46–£3.14million(source).↩

1

1

SecurebyDesign

18

AutomatebyDefaultTasksthatcanbeautomatedshouldbeautomatedbydefault.Thechoicenottoautomateshouldalwaysbeaconsciousdecision.

Rationale

Automationofwell-definedtasks,suchasdeploymentandtesting,improvesspeed,consistencyandreliabilitycomparedtomanualexecution.Theautomationcodecanalsobevaluableasaformofdocumentationforthecorrectprocessordesiredoutcome.

Implications

Automationrequiresanupfrontandongoinginvestmentthatusuallypaysoffquickly,butmustbebalancedwithdeliveringbusinessvalue.WhenevaluatingCOTSpackagesorSaaSsolutions,theeasewithwhichtheycanbeautomatedshouldbeasignificantfactorintheirselection.Automationmaybenefitfromuseofspecialisedtools,frameworksorlanguages,inwhichcaseteamswillneedtoacquireordeveloptheappropriateknowledge&skills.Softwareandprocessesthataredifficulttoautomatewillneedtobechangedorreplacediftheyrequiretoomucheffortorarenotamenabletobeingautomated.Automatedtasks/processesthatarecomplicatedbutinfrequentlyexecutedcanbecomeasourceofrisk,totheextentthatpeopledecidenottoperformthetaskmanuallybecausetheydon’thaveconfidenceintheautomatedscripts.DesignforTestabilitymustbeappliedsothattheautomationcanberegularlyproventowork,sothatitcanberelieduponwhenneeded.DisasterRecoveryprocessesareagoodexampleofthis.Automationscriptsmustbetreatedinthesamewayasothersoftwareandconformtotheseengineeringprinciples.

AutomatebyDefault

19

ContinuousDeliveryContinuousDeliveryisasoftwaredevelopmentdisciplinewhereyoubuildsoftwareinsuchawaythatthesoftwarecanbereleasedtoproductionatanytime.Systemswillbestructuredtoenablefast,automatedfeedbackonqualityandcorrectness.

Rationale

ContinuousIntegrationusuallyreferstointegrating,building,andtestingcodewithinthedevelopmentenvironment.ContinuousDeliverybuildsonthis,dealingwiththefinalstagesrequiredforproductiondeployment.Thisisdonetoreducedeploymentrisk,showbelievableprogress,andreducethetimetogetfeedbackfromrealusers.

Failuresidentifiedverysoonafterthecauseareeasiesttofix;detailsofthechangewillstillbefreshintheengineer'smind.Slow-runningtestswilltendtobeskippedorrunlessfrequentlyduringdevelopment.

Implications

Wewillkeepsoftwaredeployablethroughoutitslifecycleandprioritisekeepingthesoftwaredeployableoverworkingonnewfeaturesthroughconstantinvestment.Fastfeedbackisrequiredbothonanengineer'smachineforaspecificcomponentforproductionreadiness,andincontextwithothercomponents/applicationsaspartofacontinuousdeliverypipeline.Deliveryapproachshouldbeinformedbyriskandriskappetiteinordertofocusmitigationapproaches.Riskcanbemitigatedthroughacombinationofautomatedchecks,deploymentandreleaseprocessesandexploratorytesting.Wecanperformpush-buttondeploymentsofanyversionofthesoftwaretoanyenvironmentondemandWeusetrunkbaseddevelopment,mixinginbranchingbyabstractionand/orfeatureflagswhereappropriate.WeuseasingletrunkbasedCIbuildasthebeginningofourpipeline.Applicationsandsystemsshouldbebrokendownintosmallercomponents,eachofwhichcanbebuiltandtestedquickly.Applicationsshouldbecapableofbeingbuiltviaasinglecommandwherepossible.

ContinuousDelivery

20

ConsistentEnvironmentsEnvironmentsshouldhavehomogeneousapplicationconfiguration,software,operatingsystem,infrastructureanddata(whereappropriate).

Rationale

Lesstimewillbetakentoinvestigateissuescausedbyenvironmentaldiscrepancies,thiswillallowforfasterdeliveryofbusinessrequirementsandgivehigherconfidenceinwhatwearedeliveringiscorrect.

Implications

Configurationwillneedtobeenshrinedinversion-controlledcode.Softwarepackagesthatarenotamenabletobeingconsistentlyconfiguredwillbelessdesirable.Softwarethatisfree,orprovidescheaperlicensesfordevelopmentandtesting,willbepreferable.Wewillneedtoinvestheavilyinmakingdataavailableandconsistentacrossenvironments.Creating(anddestroying)newenvironments,includingappropriatedatasets,willneedtobeassimpleandasfastaspossibleandwillrelyheavilyontheprincipleofAutomatebyDefault.Particularlywherean“environment”isrequiredtoincludealargesetof(distributed)applicationsinordertofacilitateend-to-endtesting,thecreationandmaintenanceofanenvironmentcanbeveryexpensive.Reducing/limitingthenumberofenvironmentsmayactasanenablingconstraint–thefewerenvironmentswehave,theeasierandcheaperitwillbetokeepthemconsistent.Wewillrequireconsistentconfigurationofinfrastructure(includingnetworks,firewallsandservers).

ConsistentEnvironments

21

MaintainabilityEachcodebasemustbeunderstandableandeasytochangebynewdeveloperswithminimalexperienceoftheapplication.

Rationale

Systemsbecomeunmaintainablethroughsize,complexityandlackofup-to-datedocumentation.Bykeepingourcomponentssmallandwell-testedwereducecomplexityandincreasemaintainability.Bycombiningthesesmall,well-understoodcomponentsweareabletocreatelarge,complexyetmaintainablesystems

Implications

Softwaremustbecreatedwithaclearsetoftestswhichhelpthereaderunderstandtheapplication,andcanberunfromthecommandlinewithasinglecommand.Appropriatelevelsofdocumentationwillneedtobecreatedandupdated,withapreferencefordetaileddescriptionsoffunctionalbehaviourtobeenshrinedincodeastests.Differentaudiencesmayrequiredifferentartefactsforeffectivemaintenance,changeplanningandriskmanagement.Thismayinclude:thecode,thecommithistory,JIRAtickets,cross-functionalstories,architecturaldescriptions,runbooks,orformaldocumentation.Staledocumentationisworsethannodocumentation,soinvestmentwillberequiredtoensureitisuptodate.Maintainedreferencedocumentationandtransientdeliverydocumentationshouldbekeptseparate,soanyonecanquicklyunderstandwhattheycantrust.Knowledgesharingcanbesplitintotwoparts,the“WHATs”andthe“WHYs”.TheWHATsshouldbeevidentfromthecode,andfocusedreferencedocumentation,buttheWHYsshouldbecarefullydocumentedasdecisionsaremade.Decisionrecords,codecommentsand/orappropriatelynamedtestscanallbeusefulinthiscase.HandoveroftheWHYsshouldbedonetoalllevels,notjusttodevelopers.AREADMEshouldbecreatedandmaintained,describingtheapplicationanddocumentingthecommandrequiredtobuild,testandruntheapplication.Eachcodebaseshouldhaveapublishedsetofcodestyleguidelines,preferablyimportableintoyourIDEortexteditor.Ataminimum,encoding,linebreaksandtabbingstandardsshouldbedefined(takealookateditorconfig).Developersshouldrefactorcodeandtestswhenneeded,reassuredbythepresenceoftherapidfeedbacktheywillreceiveiftheybreaksomething.

Maintainability

22

PerformanceImportanceSystemwillbedesignedanddeliveredwithaconstantfocusonthetrendinperformancecharacteristics.

Rationale

Withoutacontinuedfocusontheperformanceimpactofchangestosystemsweriskimpacttobothuserexperiencesandmoreseriously,outagesleadingtoimpactofrevenue.

Implications

Everysystemshouldincorporateanalysisoftrendinthechangeofperformanceaftereverycommit.Everyoneneedsawarenessofhowsystemshavebeenengineeredmeetit'sperformancecriteria,asavoidintroducingbreakingchanges.Someperformancepatterns:

CachingShardingScalinghorizontallyEventualconsistency

PerformanceImportance

23

GetFeedbackEarlyandOftenGainfeedbackbyfrequentandearlyreleasesoffunctionality,ratherthanBigBangreleases.

Rationale

Earlyandfrequentreleasesprovidefeedbacksooner,makecodelevelintegrationeasier,allowcoursecorrectionstobemadeearlierandprovidebusinessvaluemorequickly.Problemsareeasiertoidentifyandresolveinsmallerchangesets.

Implications

Requirestheautomationoftasksinthedeliverypipeline.Encouragessplittingbigprojectintoasetofsmallerfeatures.Requiresswitchestodisableincompletepartsoffunctionalityortomakearollback.Requiresregularassessmentoffeatureflagsandremovaloftheobsoleteones.Nolong-termdevelopmentbranches(andmerges)needed.Requiresanautomated,version-controlledconfigurationmanagementstrategy.Thisprincipleappliestoallenvironmentsinthepathtolive,notjusttheproductionenvironment.Earlyvisibilityofplatformcostswillhelpmakeinformeddecisionsaboutfuturesolutions.

GetFeedbackEarlyandOften

24

DesignforTestabilityToacceleratetheachievementofshippablequality,solutionsshouldbearchitectedanddesignedforincreasedtestability.

Rationale

Testabilityisthecharacteristicofbeingeasyoramenabletotestingandchecking.Solutionsthataredesignedwithtestabilityconcernsinmind,facilitatefasterfeedbackandfasterdelivery.

Implications

Teamsshouldensuretheyaskthemselves"howarewegoingtotestthis?"DeliveryTeamsandthosearchitectingsolutionsshouldconsiderthedifferentfacetsoftestability,including:

controllability:thedegreetowhichitispossibleeasilytocontrolthestateofasoftwarecomponentandplaceitinastateneededforatestorcheckobservability:thedegreetowhichitispossibletoobservetheinternalstateofacomponentorsystemfromexternalfeaturesortoobtaintheoutputsofaprocessisolateability:thedegreetowhichthecomponentcanbeexecutedinisolationinordertoobserveitsbehaviour(duringcheckingandtesting)separationofconcerns:thedegreetowhichthecomponenthasasingle,welldefinedresponsibilityunderstandability:thedegreetowhichthecomponentundertestisdocumentedorself-explainingautomatability:thedegreetowhichitispossibletoautomatetestingofthecomponentundertestheterogeneity:thedegreetowhichtheuseofdiversetechnologiesrequirestousediversetestmethodsandtoolsinparallel

FurtherReading

TestabilityblogpostbyMichaelBolton(https://www.developsense.com/blog/2009/07/testability/)TheTeamGuidetoSoftwareTestabilitybyAshWinterandRobMeaney(https://leanpub.com/softwaretestability)

DesignforTestability

25

DesignforUse(DRAFT)Thisisadraftrewrite/replacementforDesignforPaceofChange.

Designforknownusecasesoverspeculativereuse.

Rationale

AlthoughYAGNIiscommonlyusedwhenthinkingaboutfeaturesandnon-functionaloptimisations,itcanconflictwithourinstinctsaboutwhat“good”lookslike–andwhatgoodengineersdo–whenappliedtoreuseandelegance.Webelieveitdeservesextraattentioninthatcontext.Designingforreusefromtheoutset,intheabsenceofwell-understoodusecases,requiresustomakeassumptionsaboutwhatwillbevaluableatsomepointinthefuture.Theseassumptionsusuallyturnouttobewrong.Aswellaswastingeffortandcreatingacostofdelay,thiscaninhibitchangebycreatingunnecessarycomplexity,dependenciesandbottlenecks.Codethatisdesignedforreuseisgenerallyhardertobuild,maintainandusethancodedesignedforasinglepurpose ,soweshouldfirstestablishthatthereisconcretevalueinreuseoutweighingthecosts,andacceptduplicationuntilthen.

Implications

Identifyreuseasitemergesthroughevolutionormodellingthebusinessdomain;preferduplicationoverprematureabstractionuntilyouhaveevidenceitwillnotconstraintherequiredpaceofchange.Onceappropriateabstractionshavebeenestablished,systemsshouldexposecommonbusinessfunctionalitythroughAPIs.Bescepticalofcomponentsthatlooksimilarbutaren'tthesameonceyouconsiderbusinessusecasesandwhatmighttriggerthemtochangeanddiverge.Thisprinciplecanonlybeeffectiveifreusecanbeenabledinfutureatasimilarcosttotoday.Wemustthereforefocusonbuildingeasy-to-changeEvolutionarySystems.Sharingdomain-specificcodelibrariesbetweenboundedcontextsisananti-patternandahazardtobeavoided .Ifreusedcodebecomesabottleneck,removethecouplingbyforkingorduplicatingthecode.Technicalandoperationalfeaturessuchasmonitoring,logging,servicediscovery,authentication&authorisationoftenneedtobeimplementedconsistentlyacrossallapplications.UseServiceTemplatestodefinewhichlibrariesshouldbeusedbyallservices .

.[BuildingEvolutionaryArchitectures:SupportConstantChangebyNealFord,PatrickKua,andRebeccaParsons](http://shop.oreilly.com/product/0636920080237.do)↩

.[DomainDrivenDesignbyEricEvans](http://dddcommunity.org/book/evans_2003/)↩

1

2

1

1

2

DesignforUse(DRAFT)

26

PerformanceImportance(DRAFT/rewrite)Systemsaredesignedanddeliveredwithaconstantfocusonperformancecharacteristics.

Rationale

Performanceisaboutretainingusers.Sitesshouldbefastandresponsivetouserinput.Slowsiteshaveanegativeimpactonrevenueandconversionrates.Withoutacontinuedfocusontheimpactofchangestosystemsperformanceweriskimpactingourusersexperience,oursystemsabilitytoscaleandoursystemsstability.Thiscanleadtoimpactingourrevenue,conversionratesandtosystemsoutages.

Implications

Settingaperformancebudgetperpageorserviceasearlyaspossibleandincorporatedinthebuildprocesshelpsfocussingonperformancethroughouttheproject,thesegoalshelptobetterbalanceperformancewithoutharmingfunctionalityoruserexperience.Applicationsshouldincorporateananalysisofimpacttoperformance(client-sideand/orback-end)aftereverycommitbyaddingshortcomponentperformanceteststotheCIpipelinewithleveragingmocks/stubsfordependencies(withsitespeed.ioand/orgatlingtools,forinstance).Teamsneedawarenessofhowtheirsystemshavebeenengineeredandmeettheirperformancecriteria,andavoidaccidentallyintroducingchangesthatleadtoperformanceandstabilitydegradationandimpactourusers’experience.Volume/loadtestingistobeconsideredforback-endperformancetestingandclient-sideperformancetestingforapplicationswithafront-end.Longerperformancetestscanberuninamoread-hocwaytoaddressspecificstability/reliability/availabilityrisks.Highervolumestresscomponentperformancetestsshouldbeconsideredtoaddressspecificscalabilityrisks.Aswellasimplementingcontinuouscomponentperformancetesting,teamsshouldconsiderautomatedintegration/end-to-endperformancetestingagainstanentiretechstackiftheirapplicationisperformancecriticalandafterassessingtheriskofimpactingupstreamordownstreamdependenciesperformance.Performanceproductionobservabilityistobeusedaswelltounderstandbetterthebehaviourofdistributed/complexsystemsbylookingatthedatatheygenerate.Thiscanbeusedforinstancewithdeploymentstrategiesthatenableasaferolloutwithobservabilityoverachangewithusingfeaturetogglestodarklaunchnewfeaturestocustomers.Forexample,ifyourmonitoringdashboardsshowanincreaseinlatencyafteryouenableaparticularfeature,youcanturnitoffagainwithouthavingtore-deployanentirereleasecandidate,youcanalsoslowlyrampuptrafficwithmonitoring(Goldensignalsorcustoms)toassesstheperformance.Productionperformanceassessmentarealsousefulactivitiestomitigatetheriskofenvironmental,usageoroperationaldifferencesbetweentestandproduction.Forinstance,byusingmonitoringtoassessperformanceinproductionorbyvalidatinginproductiontheperformanceworkloadprofilesusedinthetestenvironmentorbyrunningproductionperformancetesting.

PerformanceImportance(DRAFT/rewrite)

27

Maintainability(DRAFT/rewrite)Eachcodebasemustbeunderstandableandeasytochangebynewdeveloperswithminimalexperienceoftheapplication.

Rationale

Almostallsoftwarecontinuestorequirechangesthroughitslife,whethertofixbugs,addnewfeaturesoraddresssecurityvulnerabilities.Soonerorlater,mostsoftwareendsupbeingmaintainedbypeopleotherthanthosewhocreatedit.Inordertobeabletochangethesoftware,though,adeveloperneedstobeabletounderstandit—oratleastthepartofitthattheyneedtochange.Ourexperienceisthatsystemstypicallyoutlasttheirexpectedlifetime,andit’snotunusualforsystemsconsignedto“maintenancemode”tobeopenedupforchangewhenalmosteveryonewholastworkedonithasleftthebusiness.

Systemsthatareconsideredtoodifficult,tooexpensiveortooriskytochangewilleitherbecomealimitingfactoronbusinesschangeoreventuallyleadtoaBigRewrite,whichcancostmillionsandtakeseveralyears.Instead,weshouldensurethatapplications’codebases,aswellasthesystemsthatresultfromtheinteractionsbetweenthem,areeasilymaintainableandkeptthatway,sothattheyareabletochangewiththebusinessrequirements.Eventheprocessofreplacingordecommissioningasystemwillbenefitgreatlyfromitscurrentworkingsbeingunderstandable.

Implications

Softwaremustbecreatedwithaclearsetoftestswhichhelpthereaderunderstandtheapplication,andcanberunfromthecommandlinewithasinglecommand.Appropriatelevelsofdocumentationwillneedtobecreatedandupdated,withapreferencefordetaileddescriptionsoffunctionalbehaviourtobeenshrinedincodeastests.Differentaudiencesmayrequiredifferentartefactsforeffectivemaintenance,changeplanningandriskmanagement.Thismayinclude:thecode,thecommithistory,JIRAtickets,cross-functionalstories,architecturaldescriptions,runbooks,orformaldocumentation.Staledocumentationisworsethannodocumentation,soinvestmentwillberequiredtoensureitisuptodate.Maintainedreferencedocumentationandtransientdeliverydocumentationshouldbekeptseparate,soanyonecanquicklyunderstandwhattheycantrust.Knowledgesharingcanbesplitintotwoparts,the“WHATs”andthe“WHYs”.TheWHATsshouldbeevidentfromthecode,andfocusedreferencedocumentation,buttheWHYsshouldbecarefullydocumentedasdecisionsaremade.Decisionrecords,codecommentsand/orappropriatelynamedtestscanallbeusefulinthiscase.HandoveroftheWHYsshouldbedonetoalllevels,notjusttodevelopers.AREADMEshouldbecreatedandmaintained,describingtheapplicationanddocumentingthecommandrequiredtobuild,testandruntheapplication.Eachcodebaseshouldhaveapublishedsetofcodestyleguidelines,preferablyimportableintoyourIDEortexteditor.Ataminimum,encoding,linebreaksandtabbingstandardsshouldbedefined,ideallyusinganon-IDE-specific(orwidely-supported)toolsuchaseditorconfig.Developersshouldrefactorcodeandtestswhenneeded,reassuredbythepresenceoftherapidfeedbacktheywillreceiveiftheybreaksomething.

RelatedReading

Ontheproblemswiththe“BigRewrite”ThingsYouShouldNeverDo,PartI|JoelonSoftwareWhydowefallintotherewritetrap?|JustinFuller

Maintainability(DRAFT/rewrite)

28

Maintainability(DRAFT/rewrite)

29