Upload
techwell
View
56
Download
2
Embed Size (px)
Citation preview
W16 Agile Testing 10/5/16 15:00
Agile Testing for Embedded and IoT Software Development
Presented by:
Thomas Stiehm
Coveros, Inc.
Brought to you by:
350 Corporate Way, Suite 400, Orange Park, FL 32073 888-‐-‐-‐268-‐-‐-‐8770 ·∙·∙ 904-‐-‐-‐278-‐-‐-‐0524 -‐ [email protected] -‐ http://www.starwest.techwell.com/
Thomas Stiehm Thomas Stiehm has been developing applications and managing software development teams for eighteen years. As CTO of Coveros, he is responsible for the oversight of all technical projects and integrating new technologies and application security practices into software development projects. Most recently, Thomas has been focusing on how to incorporate DevOps best practices into distributed agile development projects using cloud-‐based solutions and how to achieve a balance between team productivity and cost while mitigating project risks. Previously, as a managing architect involved in agile development at Digital Focus, Thomas found that agile is the only development methodology that makes the business reality of constant change central to the development process.
9/21/16
1
©COPYRIGHT2016COVEROS,INC.ALLRIGHTSRESERVED. 1
Agility.Security.Delivered.
Agile Tes)ng for Embedded So3ware Development
©COPYRIGHT2016COVEROS,INC.ALLRIGHTSRESERVED. 2
About Coveros
• Coverosbuildssecurity-criIcalapplicaIonsusingagilemethods.
• CoverosServices• AgiletransformaIons• AgiledevelopmentandtesIng• DevOpsandconInuousintegraIon• ApplicaIonsecurityanalysis
• Agile&Securitytraining
• GovernmentqualificaIons• DCAAapprovedratesandaccounIng• TSfacilityclearance
AreasofExper8se
9/21/16
2
©COPYRIGHT2016COVEROS,INC.ALLRIGHTSRESERVED. 3
What is embedded so3ware
• EmbeddedsoTwareissoTwarethatcontrolsadevice• AdeviceisacombinaIonofsoTware,firmwareandhardwarethatmakeupasystem• Thesystemhashardwarecomponentsthatareeithersensorsormanipulators• ThesoTwareandfirmwarecontrolthehardwarecomponents,usingthesensorstodiscoverandthemanipulatorstocarryoutacIons• Examplesinclude:• Medicaldevices• Selfdrivingcars• Fitnesswearables• HVACsystems
©COPYRIGHT2016COVEROS,INC.ALLRIGHTSRESERVED. 4
What is agile tes)ng
• AgiletesIngisthethesetofpracIcesusedtobuildqualityinandmakesurethatsoTwarecreatedbytheteamisofhighquality• Thewholeteamisresponsibleforquality• TestautomaIonisintegratedintotheConInuousIntegraIonandConInuousDeliverypipeline• TesIngofnewfuncIonalityisconductedduringthesamesprintoriteraIoninwhichthefuncIonalityisdeveloped• UnittesIngisusedaspartofthequalityassuranceprocess• AcceptancetesIngisconductedusingacceptancecriteriacreateddirectlyfromtherequirements
9/21/16
3
©COPYRIGHT2016COVEROS,INC.ALLRIGHTSRESERVED. 5
User Acceptance Tes)ng (UAT)
• UATistheprocessofge]ngstakeholderstowritedownwhatresultstheywanttoseefromthesoTwareinordertodetermineitworksastherequirementsintended• Acceptancecriteriaarewri^enbeforedevelopmentofthesoTwarebegins• AcceptancecriteriaareaformofrequirementsthatareusedtolettheteamknowwhatkeyaspectsoftherequirementsneedtobemetinorderforthestakeholderstofeelthattheycanacceptthesoTwareincrement• TesIngofacceptancecriteriahappensattheendofanincrementandgivestheteamfeedbackonhowwelltheyhavecreatedsoTwarethatthestakeholdersexpected.
©COPYRIGHT2016COVEROS,INC.ALLRIGHTSRESERVED. 6
Behavior Driven Development (BDD)
• BDDisatestautomaIonprocessfortesIngsoTwarebasedontherequirements• Thetestsarewri^enbeforethecodeiswri^enoratleastindependentlyofthecode• ForsometeamsBDDisusedtoautomateUATtests• ManyBDDtoolsexistsuchasCucumberandBehave• MosttoolsusetheGherkinlanguage,whichusesthefollowingformat:• Givenini8alcondi8on• Whensomeac8onortrigger• Thenresul8ngac8on
9/21/16
4
©COPYRIGHT2016COVEROS,INC.ALLRIGHTSRESERVED. 7
Test Driven Development (TDD)
• TDD,alsocalledTestFirstDevelopment,istheprocessofusingunitteststohelpdesignthecodeforanapplicaIon• Thedeveloperwillwriteatestofwhattheyintendthecodetodoandthenwritethecodetomakethetestpass• Red,Green,Refactorcycle• ThishasanumberofposiIveresults:• Makesthecodemoretestable• Makesthecodemoremodularorselfcontained• Makesthecodemoremaintainable• Givesthecodearegressiontestsuite• Makestheintentofthecodeeasiertounderstand
©COPYRIGHT2016COVEROS,INC.ALLRIGHTSRESERVED. 8
Con)nuous Integra)on (CI)
• CIistheprocessofintegraIngandtesIngcodeuponcheckin• InCI,integraIonistheprocessofintegraIngadeveloper’scodewiththecodealreadyintherepository• ThegoalofCIistomakesurethecodecanbuildandpassaniniIalsetofregressiontestsbeforeothertesIngandqualityassuranceisappliedtoaspecificbuild• CIusesabuildserversuchasJenkinstodoacleanbuildofthecodeandrunasuiteofunittests• Ifthebuildbreaks,thepeoplethatcheckedinsincethelastsuccessfulbuildareinformedandtheyareexpectedtogetthebuildbacktoaworkingstate
9/21/16
5
©COPYRIGHT2016COVEROS,INC.ALLRIGHTSRESERVED. 9
Con)nuous Delivery (CD)
• CDistheprocessoftakingabuildthroughabuildpipelinethatconsistsofaCIprocessfollowedbyseveralqualitygatesthatverifythecodethroughvariousqualitychecksincluding:• AutomatedfuncIonaltests• StaIccodeanalysis• Automatednon-funcIonaltestslikeperformanceandsecurity• Automateddeploymentintoprogressivelyhigherenvironments• Itcouldincludeondemanddeploymenttoamanualtestenvironment• OndemanddeploymenttoaUATenvironment• Ondemanddeploymenttoademoenvironment
• ConInuousDeploymentisautomaIcallyprogressingtoproducIonincludingautomaIcallydeployingintoproducIon
©COPYRIGHT2016COVEROS,INC.ALLRIGHTSRESERVED. 10
Value Stream Mapping
• ValuestreammappingistheprocessofdetermininghowaproductiscreatedinyourenvironmentfromincepIontodeploymentanduseinproducIon• BycreaIngavaluestreammapyoucanfigureouthowmanystepsyouhaveinyourdeliveryprocessandhowlongeachsteptakes• OnceyouhaveamapofyourprocessyoucanstarttoopImizeandautomatethedifferentstepstostreamlineyourprocess• ThegoalistoreducetheImeittakestomoverequirementsthroughyourprocessandgetnewfeaturesintoproducIon
9/21/16
6
©COPYRIGHT2016COVEROS,INC.ALLRIGHTSRESERVED. 11
Build Pipeline
• Thebuildpipelineisthepartofyourvaluestreamthattakescheckedincodeandmakesitintohighquality,testedsoTwarethatiseitherinproducIonorreadytobedistributedtocustomers• CI/CDareasubsetofyourbuildpipeline• AutomaIon,includingtestautomaIon,istheonlywaytomakeabuildpipelineworkinafastandefficientmanner• ManyofthequalitygatesinabuildpipelinearetesIngfocusedandrequirestrongtesIngacumeninordertohelpdeliverhighqualitysoTware
©COPYRIGHT2016COVEROS,INC.ALLRIGHTSRESERVED. 12
Version Control
• VersioncontrolistheprocessofmanagingthecodeoverImebyallowingaprojecttokeeptrackofthestateofthecodeasitprogresses• Allcode,includingtes6ngcodeandtestautoma6oncodeneedstobecheckedintoversioncontrol• ApplicaIoncodeandtesIngcodeshouldbeversionedtogethersincethetestcodeactsontheapplicaIonandchangesoverImetokeepinsyncwithchangestotheapplicaIoncode• Versioncontroltoolsinclude:• Subversion• Git• Others
9/21/16
7
©COPYRIGHT2016COVEROS,INC.ALLRIGHTSRESERVED. 13
Configura)on Management
• ConfiguraIonManagementarethetoolsandprocessusedtodeterminehowsoTwareissetupinanygivenenvironmentandhowtoaccountforallofthedifferentfilesneededtomakethesoTwarework• Thisincludesbinaries,configuraIonfilesandenvironmentse]ngs• MostbuildpipelineshaveacomponentthatmanagesconfiguraIonelementsforeachdeploymentsothattheteamcanknowexactlywhatisdeployedandtestedforeveryenvironment
©COPYRIGHT2016COVEROS,INC.ALLRIGHTSRESERVED. 14
Tes)ng So3ware vs. Hardware vs. System
• WithembeddedsystemsyouaretesIngthreethings:• SoTware–Thecodethatmanagesthesystem• Hardware–Themechanicaland/orelectricalpartsofthesystem• System–ThecombinaIonofhardwareandsoTware
• SomeImesfirmwareistestedlikesoTware,someImesitistestedlikehardware(dependingonhowfirmitis)• IsolaIngwhatyouaretesIngcanhelpyoucreatetestautomaIonthatcoversamuchoftheproductaspossible• SomeamountofmanualtesIngmightbeunavoidable
9/21/16
8
©COPYRIGHT2016COVEROS,INC.ALLRIGHTSRESERVED. 15
So3ware Tes)ng
• DuringsoTwaretesIngyouaretesIngonlythesoTwarepartofyoursystem• Thismeansthatallinput(sensordata)shouldbesuppliedbyaknownsource,ifpossiblebyrecordingthedataandplayingitback,orsimulaIngit• Outputcanbeignoredorsimulated• Recordandplayback,orsimulaIoncanbelessexpensiveandeasiertosetup• YoushouldbefocusedontesIngthebusinesslogicofthesoTware,i.e.thecalculaIonsanddeterminaIonsthatsoTwaremakes• YoucanalsofocusontesIngtheuserinterfaceorinteracIveelementsofthesystem
©COPYRIGHT2016COVEROS,INC.ALLRIGHTSRESERVED. 16
Hardware Tes)ng
• HardwaretesIngcanbeagileaswell• HardwarecanhaveadifferentcadencethansoTware• AgilehardwaretesIngisaboutautomaIngasmuchofthetesIngprocessaspossible• SoTwaretools,bothhomegrownandthirdparty,canbeusedtoautomatehardwaretesIng• UsingthesystemsoTwarethecontrolsthathardwaretotestthehardwareisacceptablebutkeepthefocusonthehardware
9/21/16
9
©COPYRIGHT2016COVEROS,INC.ALLRIGHTSRESERVED. 17
System Tes)ng
• SystemtesIngisthecombinaIonofthesoTwareandhardwareworkingtogether• Dependingonthenatureofyourembeddedsystem,thiscanbehardtoautomatedorautomatecompletely• Dependingonthenatureofyourembeddedsystemtheremaybeagooddealoftoolingthatcanhelpyoutestthesystemortheremaybenothingatall• TesIngthesystemiscriIcalbecauseyoudon’tknowhowthesoTwareandhardwarewillinteractunIlyouactuallyhavetheminteract
©COPYRIGHT2016COVEROS,INC.ALLRIGHTSRESERVED. 18
Test Automa)on vs. Manual Tes)ng
• AgiletesIngfocusesontestautomaIonandinsometypesofapplicaIonsitispossibletoautomatenearlyallofthetesIng• WiththefocusontestautomatonitiseasytothinkthatmanuallytesInghasnoplaceinagilebutthatisn’ttrue• ManualtesIngisoTenaprecursortofuncIonaltestautomaIon• ManualtesIngcanbecheaperormoreeffecIveforsomefeaturesofanembeddedsystem• ExploratorytesIngcanexercisethesysteminawaythattestautomaIonwillnotandfinddeeperissues
9/21/16
10
©COPYRIGHT2016COVEROS,INC.ALLRIGHTSRESERVED. 19
How much can you automate?
• Lookatthisfromanumberofpointsofview:• Whatcanyouafford• WhatdoyouhavetheskillsorabiliIestoautomate• Whatispossible• Whatisthemostbeneficial
• FindingabalancebetweentestautomaIonandmanualtesIngwillbeachallengethatallprojectsface• TheanswerwillchangeoverImeasyoulearnandasyourcodeimproves
©COPYRIGHT2016COVEROS,INC.ALLRIGHTSRESERVED. 20
Coding Standards (for tests too)
• Usingcodingstandardsmeans:• Makingthecodingstandards• Enforcingcodingstandardsusingcodereviews
• Itdoesn’tma^erwhatspecificcodingstandardsyouusesomuchasyouhavethemandtheteamagreestothem• Ifpossible,useanexisIngcodingstandardforyourtechnology• Makecodingstandardscompliancepartofyourcodereview• Be^eryet,useacodeforma^ertoapplythestandardduringcodecommit• Applycodingstandardstoyourtestsaswell
9/21/16
11
©COPYRIGHT2016COVEROS,INC.ALLRIGHTSRESERVED. 21
Code Inspec)ons
• CodeinspecIonsincludeanumberofthings:• Codereviews• StaIccodeanalysis• Securecodereviews• Formatandstylereviews
• Automatewhatyoucan,formatandstyleareeasytoautomate• StaIccodeanalysiscanbeagoodinputintocodereviews• Codereviewtoolscanhelp,thingslikeCrucibleandReviewBoard• Asecurecodereviewsisdifferentthanacodereviewandisadifferentskillset,youshouldmasterboth
©COPYRIGHT2016COVEROS,INC.ALLRIGHTSRESERVED. 22
Unit Tes)ng
• Unittestsaredeveloperteststhatareusedtomakesurethatthecodethattheywritebehavesasthedeveloperintended• Unittestsarewri^eninthelanguageandtechnologyplanormoftheapplicaIoncodeandoTenuseaframeworklikeUnity(seeh^p://www.throwtheswitch.org/unity/)• Unittestsmightberunonthedevplanormorthetargetplanorm,thisoTendependsonhowthetoolingworks• Samplecode:voidtest_FuncIonWhichReturnsLocalVariable_ShouldReturnTheCurrentCounterValueAgain(void)
{ TEST_ASSERT_EQUAL(0,FindFuncIon_WhichIsBroken(78)); TEST_ASSERT_EQUAL_HEX(0x5a5a,FuncIonWhichReturnsLocalVariable()); }
9/21/16
12
©COPYRIGHT2016COVEROS,INC.ALLRIGHTSRESERVED. 23
BDD Sample Code
Scenario:SubtracttwonumbersGivenIhaveacalculatoronWhenIenter”25"intothecalculatorAndIenter”10"intothecalculatorAndIpresssubtractThentheresultshouldbe"15"onthescreenScenario:AddtwonumbersGivenIhaveacalculatoronWhenIenter”25"intothecalculatorAndIenter”45"intothecalculatorAndIpressaddThentheresultshouldbe”70"onthescreen
©COPYRIGHT2016COVEROS,INC.ALLRIGHTSRESERVED. 24
Mock Objects
• Mockobjectsarereplacementsforthedependenciesofthecodeundertest• TheyareusedtomakeiteasiertotestthecodebyrespondingthesamewayalltheIme• ThereareanumberofgoodmockobjectframeworksincludingCMock(seeh^p://www.throwtheswitch.org/cmock/)
9/21/16
13
©COPYRIGHT2016COVEROS,INC.ALLRIGHTSRESERVED. 25
Test Fixtures
• TestfixturesarethecodethatcreatesafixedstatethatisusedtobaselinetestssothatthetestenvironmentisthesameeachImethetestsarerun• TestframeworksdothisbyexposingmethodsorfuncIonsthatallowfortheconfiguraIonofdependentobjectsorforthecreaIonofmockobjects• Foradatarelatedtestatestfixturemightloadaspecificsetofdataintothedatabase• MostunittesIngframeworksandmanytestautomaIontoolshavetestfixturesbuildintothem
©COPYRIGHT2016COVEROS,INC.ALLRIGHTSRESERVED. 26
Working with Simulators
• AsimulatorissoTwareorhardwareusedfortesIngthathasthesameinterfaceastheapplicaIonorsystemitisreplacing• SimulatorsprovideaconsistentinterfacefortesIng,muchliketestfixturesormockobjects• Simulatorsaremorefullfeaturedandcanreacttoinput,inmorevariedwaysthatothertesIngtools• Theideaisthatasimulatorwillreactmoreliketherealcomponentintherealworld• SimulatorscanbeexpensivetobuildandmaintainbuttheremaybenootheropIon
9/21/16
14
©COPYRIGHT2016COVEROS,INC.ALLRIGHTSRESERVED. 27
Tes)ng services for embedded so3ware
• ManyInternetofThings(IoT)devicesuseservices• ServicesareInternetaccessiblebitsoffuncIonalitythatdothingslike:• Collectdata• Calculateresults• Coordinateresources
• TesIngtheseservicescanbeeasiertoautomatethenthedevicesthemselves• Scalecanbeanissueforservices• Networkavailabilityorreliabilitycanbeanissueforthedevices• PerformancetesIngisoTenimportantfortesIngservices
©COPYRIGHT2016COVEROS,INC.ALLRIGHTSRESERVED. 28
What to take away from the presenta)on
• TestautomaIonisthekeytoagiletesIng• DotestautomaIonplanningwhileyoudotestplanning• CreatetestsuitesthattestyoursoTwareseparatelyfromyourhardwareandanothersuitethatteststhesystemasawhole
9/21/16
15
©COPYRIGHT2016COVEROS,INC.ALLRIGHTSRESERVED. 29
Ques)ons?
• TomSIehm• [email protected]• @thomassIehm