Jenkins World

#JenkinsWorld

Jenkins at Splunk> - Bill Houston

Splunking> Jenkins - Ali Raza

Jenkins World

#JenkinsWorld

•  1minuteintroduc3ontoSplunk

•  DevelopmentprocessandbuildpipelinesatSplunkusingJenkins

•  HowReleaseEng.usesSplunktoimprovedeveloperproduc3vityandexperience

•  QualityEngineeringatSplunkusingJenkins

•  HowQTIusesSplunktoimproveQEproduc3vityandexperience

•  MakingsenseofalltheJenkins,development,andtes3ngdatausingSplunk

Agenda

Jenkins World

#JenkinsWorld

•  WhatisSplunk–  SplunkEnterprisemakesitsimpletocollect,analyze,andactuponuntappedvalueofthebigdatagenerated

byyourtechnologyinfrastructure,securitysystems,andbusinessapplica3ons(likeJenkins).Splunkgivesyoutheinsightstodriveopera3onalperformanceandbusinessresults

–  CommercialEnterpriseSoOware

–  Freedownload(500MB/day)

–  BillandIdon’tworkinsalessoyoucanrelaxJ

SplunkIntroduc3on

Jenkins World

#JenkinsWorld

•  Mul3pleJenkinsclustersatSplunkperformingavarietyofdifferentfunc3ons–  EngineeringDevCIsystemtotesteachpullrequestandcommittoBitBucket–  QECIsystemforautomatedtes3ngofmul3plebuildsadayacrossmul3plebranches–  SplunkClouddeployment–  Applica3ontes3nginShanghai–  Othermisc.housekeepingandtes3ngtasks

•  OurtalkwillfocusontheuseofJenkinsandSplunkinourDevandQECIsystems

•  WewilldoalivedemousingournewSplunkpluginforJenkinsalongwithaJenkinsAppforSplunk.BothofthesearegearedtowardtheoverallJenkinscommunity.

JenkinsatSplunk

Jenkins World

#JenkinsWorld

DevelopmentPipeline

JenkinsatSplunk

Jenkins World

#JenkinsWorld

Tes3ngofEveryCommittoCertainBranches

Testsrunforalimitednumberofbranches

Jenkins World

#JenkinsWorld

DevelopercommitsachangetoBitbucket

Tes3ngofEveryCommittoCertainBranches

Jenkins World

#JenkinsWorld

AnHTTPRequestPostreceivehooksendsatrigger

Tes3ngofEveryCommittoCertainBranches

Jenkins World

#JenkinsWorld

Whyhaveatriggerforwarder/queuesystem?

•  Providesalayerofsecurity

•  Asingleswitchisusedtoinhibittriggerforwarding

•  Mul3pleTriggerscanbeforwarded

•  Verifiesthatthetriggerwasproperlyreceived

•  ImplementedwithJenkins

Tes3ngofEveryCommittoCertainBranches

Jenkins World

#JenkinsWorld

Forwardersendsthetriggeron

Tes3ngofEveryCommittoCertainBranches

Jenkins World

#JenkinsWorld

Jenkinsmasterstartstherequiredtests

Tes3ngofEveryCommittoCertainBranches

Jenkins World

#JenkinsWorld

ResultsarepostedtoBitbucket,thedeveloperandSplunk

Tes3ngofEveryCommittoCertainBranches

Jenkins World

#JenkinsWorld

Tes3ngofEveryCommittoCertainBranchesAbouttheHipchatNo3fica3ons•  Send3melytestupdateswithout“Spamming”the

team•  Senttotargetedindividuals•  SentwithGroovyscriptusingtheHipchatRESTapijson_message="{\"message\":\"Triggerfortes3ngofPullRequest:${pr_ID}hasbeenreceived\",\"no3fy\":true}”cmd=["bash”,"-c","curl-k-v-XPOST-H'Authoriza3on:Bearer${hipChat_token}'-H'Content-Type:applica3on/json'-d'${json_message}'hnps://hipchat.splunk.com/v2/user/${emailAddress}/message"]try{HipChat_Process=newProcessBuilder(cmd)process=cmd.execute()process.waitFor()println"curlGETexitvalue:${process.exitValue()}"}catch(e){println"failedtosendtheHipchatmessage"}

Jenkins World

#JenkinsWorld

Basicallysimilartocommittests

Tes3ngofEveryPullRequest

Jenkins World

#JenkinsWorld

Atestisautoma3callytriggeredwhenapullrequestisopened

Tes3ngofEveryPullRequest

Jenkins World

#JenkinsWorld

FurthertestsaremanuallytriggeredwithabunonintheUI

Tes3ngofEveryPullRequest

Jenkins World

#JenkinsWorld

Tes3ngofEveryPullRequest

Jenkins World

#JenkinsWorld

TestprogressandresultsarepostedtotheBitbucketPRUIinacomment.Teststartno3fica3onandresultsarepostedtothedeveloperviaHipchatandtoSplunkviatheSplunkplugin.TestresultsarepostedtothepullrequestreviewersviaHipchat.

Tes3ngofEveryPullRequest

Jenkins World

#JenkinsWorld

Real-TimeNo3fica3onsinBitbucketComments

Goal:providemeaningfulinputtoreviewers,don’t“spam”thepullrequestUI.Acommentiscreatedforeachtriggerthenupdatedasthetestprogresses.

Jenkins World

#JenkinsWorld

Real-TimeNo3fica3onsinBitbucketComments

Jenkins World

#JenkinsWorld

Real-TimeNo3fica3onsinHipChat

Hipchatno3fica3onshavelinksbacktotheBitbucketpullrequestUIwheretheresultsarepostedincommentsAlinkinthepullrequestUIsendsDevelopersdirectlytothetestresultspageintheJenkinsUIMostdeveloperinterac3oncantakeplaceinBitbucketorinSplunk

Jenkins World

#JenkinsWorld

•  Theprojectsareasgenericaspossible.

•  AllscriptsandpropertyfilesarestoredinBitbucket.

•  TheJenkinsmasterpullsafreshcopyofthefilesonacommit.

•  Eachslavenodepullsafreshcopyofthefilesatthestartofajob

•  AJenkinsStagingsystemisusedfornewdevelopmentandbugfixtes3ng

Keeping the System Maintainable/Extensible

Jenkins World

#JenkinsWorld

•  Somenodesarerestartedregularly

•  Ascriptanalyzestheresultsattheendofeachtest

•  Testrunsthatfailduetoaninfrastructureissueareretriggeredonadifferentnode.

•  TheJenkinshomedirectoryisstoredonanNFSmount

•  TheSplunkpluginmonitorstheJenkinsmasterandallnodes

•  AlertsaresetupinSplunktowarnofpendingandac3vefailures

Keeping the System Reliable

Jenkins World

#JenkinsWorld

•  MaintainaBitbucketreferencerepositoryoneachnode

•  Deferlowerprioritytests/tasksduringhighloadperiods

•  DataforeachjobissenttoSplunk

•  SplunkingourJenkinssystemshasallowedustobemoreefficient,andresponsive

•  Diagnosingbuildandtestfailuresattheendofeachtest

Be Responsive to Developers Needs

Jenkins World

#JenkinsWorld

•  Commonissuessuchassourcecodemergeconflicts,compileorlinkerrorsareiden3fied

•  Eachfailureofabuildortestiscomparedwiththeresultsofprevioustests

Be Responsive to Developers Needs

Jenkins World

#JenkinsWorld

QualityEngineeringPipeline

JenkinsatSplunk

Jenkins World

#JenkinsWorld

QualityEngineeringatSplunk

•  EnterpriseSoOwareTes3ngisComplex:

•  Distributednatureofapplica3ons•  Variouscomplexdeploymenttopologies•  Destruc3vetes3ng–>preventssharingoftestenvironments

•  Variousplauorms

Jenkins World

#JenkinsWorld

QualityEngineeringatSplunk

Ini3alJenkinsrolloutstrategyforQA

•  Let’sempowereveryonetodowhatevertheywantinJenkins•  AslongwemonitorJenkins,everythingshouldbefine

Jenkins World

#JenkinsWorld

Jenkins–Doasyouplease

Jenkins World

#JenkinsWorld

Jenkins–Doasyoupleaseresults

•  JenkinsInfrastructurebecametheWildWildWest•  ConstantJenkinsinfrastructureandscalabilityissues•  Nocontrolandinsightintowhoisdoingwhatandwhy•  Lackofownership•  Ridiculousamountofdatatoprocess

•  Jenkinsconsolelogs•  TestResults•  Ar3facts(crashlogs,testlogs,etc)

Jenkins World

#JenkinsWorld

Jenkins–Myteam’sdailyjob

Jenkins World

#JenkinsWorld

Whattodo?

•  EngineeringDilemma:•  Can’tevenfigureoutwhattofixandwheretostart•  Notsureiftouchinganythingwillbreaksomethingelse

•  Let’sskipallthestageofgriefandgostraightto‘Acceptance&Hope’•  Currentsetupanddesignisnotdesignedtoscale•  Throwoutthecurrentsetupandstartfresh•  Learnfromourmistakes

Jenkins World

#JenkinsWorld

NewTeamsetup

•  QAToolsandInfrastructureTeam(QTI)•  LaserfocusedonQAproduc3vityandneeds•  InfrastructureandDeploymentservices•  TestArchitectureandTes3ngTools•  Jenkins•  Analy3csusingSplunk

•  QAEngineering•  Focusedontes3ngourproduct-Splunk•  Workssidebysidewithdeveloper,productmanagers,etc.

Jenkins World

#JenkinsWorld

JenkinsResurrec3on

•  NewJenkinsarchitectureGoals•  LimitedJenkinsprivilegesformostengineers.Onlybuildaccess•  ProvideatestframeworkabstracterlayerinJenkinsusingBuildFlow•  PerformancetuneandscalethehelloutofJenkins

•  LeverageSplunkfor

•  DiscoveringissuesinJenkinsandTes3nginfrastructure•  Real-Timealerts•  TestAnaly3cs•  Logeverything•  KPIsformanagement

Jenkins World

#JenkinsWorld

Jenkins–BuildsGalore

Jenkins World

#JenkinsWorld

JenkinsResurrec3on–Workflow

Jenkins World

#JenkinsWorld

JenkinsData-SomeUseCases

•  Jenkinsopera3onaldataforadministrators•  Monitormul3pleJenkinsclusters•  Jenkinsmastersandslaveshealth(JVM,logs,logins,etc.)•  SecurityandAudi3ngofjobsandpluginconfigura3ons•  Buildqueue3mes,longrunningbuilds,andtrends•  Real-Timeconsolelogs•  KPIsformanagement

•  JenkinsBuildsdatafornormalusers

•  Buildandjobsconsolelogs,ar3facts,results•  Deeptestsanaly3cs,trends,andtriagingcapabili3es•  Preservethisdataforaverylong3me

Jenkins World

#JenkinsWorld

WhatcanSplunk>doforyou?

•  Fixingengineeringissuesistheeasypart.Findingoutwhattofixisdifficult.

•  Splunkwon’tfixyourJenkinsinfrastructureBUTitcanbeinstrumentalinexposingissuesinyoursetup.

•  Opera3onalIntelligenceintoyourJenkinsorotherCIapplica3on

infrastructureiscrucialtoincreasingdeveloper/QAproduc3vityandkeepyourorganiza3onmovingatafasterpace

Jenkins World

#JenkinsWorld

Splunk+Jenkins

•  JenkinspluginforSplunk•  Onceconfigured,itwillsendalotofinforma3onaboutyourJenkins

environmenttoSplunk•  NoneedtoinstallanySplunkForwarder•  Noneedtomodifyanyjobs.Alljobandbuildac3vityisintercepted,

captured,andsenttoSplunkinreal-3me.•  Highlyconfigurable.Sendasmuchoraslinledataasyouneed

•  SplunkAppforJenkins

•  StandardAppinSplunk.Installin2minutes•  Prebuildcollec3onofdashboardsforprovidingdeepvisibilityintoyour

Jenkinsenvironment•  SpecificallydesignedforReleaseEngineers,JenkinsAdministrators,and

QAEngineerusecases

Jenkins World

#JenkinsWorld

LiveDemo

Jenkins World

#JenkinsWorld

JenkinsPlugin+JenkinsAppRelease

hnps://www.surveymonkey.com/r/splunk-jenkins

Jenkins World

#JenkinsWorld

Contributors

DanielleJenkins KashyapJotwani

JingShao TedXiao AlanYu

Jenkins World#JenkinsWorld

Jenkins World