Pre-Con Lab: Test Automation Across Nine Open Source Tools—Taurus

World®’16

TestAutomationacross9OpenSourceTools:TaurusAndreyPokhilko- ChiefScientist,BlazeMeter- CATechnologies

DO5X52L

DEVOPS

2 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD

©2016CA.Allrightsreserved.Alltrademarksreferencedhereinbelongtotheirrespectivecompanies.

Thecontentprovidedinthis CAWorld2016presentationisintendedforinformationalpurposesonlyanddoesnotformanytypeofwarranty. The informationprovidedbyaCApartnerand/orCAcustomerhasnotbeenreviewedforaccuracybyCA.

ForInformationalPurposesOnlyTermsofthisPresentation


Abstract

Inthenotsodistantpast,loadtestingtoolshadoneclearpurpose:togenerateload.Allthetoolscreatedatthattimeansweredthisrequirementperfectly.Therationalehasevolvedaswediscovered,fine-tuned,andextendedaspectsoftheprocess.Now,thetestpreparationstageissocomplexandcrucialthatwespendmoretimepreparingteststhanrunningtheload.Wewanttoseelivetestresultsalongsidethehardwaremonitoringdata.Finally,thepost-testphaserequirescheckingSLAconditions,sendingautomatednotifications,andstoringresultsforfutureanalytics.The'goodold'toolswerenotdesignedwellforsuchabroadusecase.Butitisn'tjustthescopeofworkthathaschanged.Eventhehabitsofthepeoplebuildingtheserviceshavealsochanged.Theyprimarilyusetwotypesoftools:command-linetoolsandWeb2.0services.ThinkhowoftenyouencounterLinux,Git andAnsible,Wget andcURL.Orhowmanyofthesetabs- GmailandAWS,GitHubandSplunk,SalesforceandHubspot,NewRelic andJenkins- youcurrentlyhaveopen?The"cherryontop"isthe"continuous"aspectofmoderndevelopmentprocesses.Teamswanttoautomateallpossiblepartsoftheflow,and,insuchcases,thetestingtoolshouldnaturallyfitintothe"pipelinechain"oftheCImachine.We'releftwithtwopossiblesolutions:modernizingoldtoolsorcreatingnewtoolsfromscratch.Neithersolutionisperfect,bothrequirespendingalotofresourcesonredoingsomethingthathasalreadybeendone.Inthissession,AndreyPokhilkowillexploreathirdapproach- anopensourcehybridtoolcalled"Taurus".Thistoolmeetsthebroadusecasesmentionedabove,whileenablingyoutoutilize'triedandtested'toolsforgeneratingload.Tosuittheuser'shabits,Taurusisasimple-to-learncommand-linetoolthatalsoconnectstoWeb2.0services.Thissessionwillhaveahand-onsectionthatwillprovideyouwithanopportunitytocreate,configureandexecuteSeleniumscriptsforperformancetestsintheCloudusingTaurus.

AndreyPokhilkoCATechnologiesChiefScientist,BlazeMeter


http://gettaurus.org


Agenda

WHYLEARNTAURUS?

TAURUSOVERVIEW

9TOOLSSUPPORTED

Q&A

1

2

3

4


WhyLearnTaurus?

§ DevOps-styleusageofloadtestingtools(CLI+SaaS)

§ Freedomtochoosetoolthatfitsuse-case

§ Enable“shiftleft”forloadtesting

§ EnableautomationandCIforloadtesting


Taurusis:

§ Text-basedconfigurationfileformat

§ Executedbythecommand-linetool

§ Thatwrapsexistingloadgeneratingtools

§ AndconnectstoSaaSforreportingandscale


YAMLSyntaxIntro

§ Whitespacematters <=mainsourceofmistakes

§ Dictionaries,listsandscalars

§ Noneedtoquotestringsunlessspecialcharsused

§ YAML-awareeditorhelps(Notepad++)

§ IfYAMLdoesnotworkforyou,youcanuseJSONwithTaurus


TaurusConfig Structure

§ Scenarios - collectionofscripts

§ Execution - whichscenariosexecuteandhow

§ Reporting - howtoanalyzedata

§ Services - extraactionsondifferentstages

§ Modules - changesettingsforallabove

§ Generalsettings- sometop-leveloptions


GoodDefaults

§ Youcanomitalmosteveryoption

§ Almosteverythinghasmeaningfuldefaults


WritingFirstConfig

run> bzt first.yml

---execution:- scenario:

requests:- http://blazedemo.com/


Understandingwhathappensinside

1. Loadconfigs:basedefaults,personalpreferences,config itself

2. GenerateJMXfromYAMLscenario

3. ModifyJMXtointegratewithTaurus

4. RunJMeterasparallelprocess

5. Analyzeresultsandoverwatch JMeter

6. Reportstatsintocommandlineattheend


ApplyingSomePressure

---execution:- scenario: blazedemoconcurrency: 10ramp-up: 1mhold-for: 2m

scenarios:blazedemo:requests:- http://blazedemo.com/


LearningQuickOverrides

run> bzt first.yml ⇒

-o execution.0.concurrency=20

-o modules.console.disable=true


UsingOnlineReports

run> bzt first.yml -report


ImportantFilesandDirectories

§ Artifactslocation,configurable,showntwicearun:– merged.yml,effective.yml,logs

... ... ...04:03:26 INFO: Artifacts dir: /tmp/bzt-artifacts/1107-035917... ... ...


ScalingWithCloudProvider

run> bzt first.yml –cloudYouwillneedtheAPIkeyin~/.bzt-rc forthis

---execution:- scenario: blazedemoconcurrency: 50hold-for: 5mlocations:us-west-1: 2eu-west-1: 3


ScenarioBuilding:DSLinYAML

§ Scenariobuildingisalotspecificforeachtool

§ JMetersupportisthemostrich:– extractors,– assertions,– logic,– config elements– JSR223– threadactions


ScenarioBuilding:DSLinYAML

---scenarios:from-script:requests:- url: http://blazedemo.comassert:- Welcome


ScenarioBuilding:SourceFiles

---scenarios:from-script:script: my-predefined.jmxmodifications:disable:- Constant Timer


9ToolsSupported

§ Somearecross-platform:JMeter,Gatling,Selenium,Grinder,Locust,ab

§ SomeareLinux/Unix-only:Siege,Pbench,Tsung

§ Seleniumisveryspecial:Java,Python,JavaScript,Rubysupported

§ YoucanwriteyourownandplugitintoTaurus


SwitchingTools

Available:JMeter,Gatling,Selenium,Grinder,Locust,ab,Siege,Pbench,Tsung

---execution:- scenario: blazedemohold-for: 1mexecutor: gatling


Taurus• OpenSourceinitiativesponsoredbyCABlazeMeter• Automation-friendlyframeworkforContinuousTesting• DefinescenariosinTaurusYAMLorexecuteexistingtests• Widevarietyoftestexecutors,includingJMeter,Selenium,Gatling,…• Abilitytogenerateloadlocallyorinthecloud• ConsistentreportingthroughTaurusUIorBlazeMeter• EasyintegrationwithContinuousDeliverysystems

Summary

scenarios:home-page:requests:- label: Homeurl: /method: GETthink-time: 500msassert:- contains:- Welcomesubject: bodyregexp: false


FreePlayTime!


Questions?


RecommendedSessions

SESSION# TITLE DATE/TIME

DO5T03P FinancialIndustryPanel– Theater5 11/16/2016at04:30pm

DO5X39SGM'sRemovingTestingConstraintswithRespecttoTestModelingandTDM– LagoonL

11/16/2016at04:30pm

DO5X55S WalmartBrasil—SeasonalPeakTrafficReadiness 11/16/2016at04:30pm


MustSeeDemos

ModernizeAppDeliveryIntegratedCDTheater5- DOV501P

DeliverTestDataFasterTestDataManagerTheater5- DOV511P

DeliverBetterAppsServiceVirtualizationTheater5- DOV507P

OrchestrateYourReleaseReleaseAutomationTheater5- DOV513P


Thankyou.

Stayconnectedatcommunities.ca.com

@CAWORLD#CAWORLD ©2016CA.AllRIGHTSRESERVED.29 @CAWORLD#CAWORLD

DevOps– ContinuousDelivery

FormoreinformationonDevOps– ContinuousDelivery,pleasevisit:http://cainc.to/PiTFpu

Technology

Pre-Con Lab: Test Automation Across Nine Open Source Tools—Taurus