Copyright©2016SplunkInc.

CenturyLinkThomWoldahlLeadEngineer

2

AboutCenturyLink•  Foundedin1930•  LeadingproviderofInternet,TV,Voice•  AglobalcommunicaIons,hosIng,cloudandITservicescompany

•  250,000-route-mileUSfibernetwork•  Morethan55globaldatacenters

•  HeadquarteredinMonroe,Louisiana•  S&P500,Fortune500

3

AboutMe•  Leadengineer•  BasedinPhoenix,AZcoveringtheMountainWestarea

(Colorado,NewMexico,Arizona)•  FirstIPTVengineerinthefieldforCenturyLinkonPRISMproduct•  Previouslyworkedinnetworkplanningandengineering

•  AsaRegionalOrganizaIon,wesupportourlocalfieldoperaIonsteam•  Install,maintenance,repairvideoandhigh-speedInternet

productsaswellasPOTS(plainoldtelephone)

•  We’remeasuredoninstallaIonandtroubleshooIngsuccessrates•  Keepcostsdown/customershappy

4

SplunkatCenturyLink•  SplunkposiIonedtocollectmachinedatafromenIreCenturyLink

network

•  Datasources:•  SyslogfromIP/MPLSbackbone(routers/switches),servers,

appliances,gateways,videodeliverysubsystems,SNMP,callcentercalllogs

•  Databaselogs,applicaIon(transacIonal)logs•  AcIve,passiveandtransacIonalmeasurementsthroughoutthe

network:datapulledintoSplunksodwareusingnetworkprobesandscriptedinputintodatabase

•  Indexingupto600GBdataperday

•  Splunkusers:750regular(2000+unique)

•  TeamsusingSplunk:NOC,Development,Architecture,Planning,OperaIons,Engineering,ServiceProvisioning,eCommerceteam,OrderProcessing(IVR),MarkeIng,Product,IT,etc.

•  SplunkApps:DeploymentMonitor,WindowsApp,SplunkonSplunk,SplunkDBConnect,ourowncustomizedSplunkApps

5

BeforeSplunk:inaccessibledata,toomanycustomervisits

•  MulIplevisitsofon-sitecustomersupport•  “chroniccustomers”(3+troublecallsinameasuredperiod)•  Iwouldonlybecalledinastechsorsupervisorswouldbecomefrustrated

withineffecIverepairsorunabletoidenIfyaspecificissue

•  Enduredlaborious,manualprocesses•  Ionlyhadaccesstotestdata,ifIwas“onsite”duringtroubleshooIng•  Technicianswerechallengedtocapturethedata,moveittoaUSBdrive,

andsendittomeinanemailforreview•  Timeconsuming,clunkyprocess(andatonof“windshieldIme”drivingto

customershomes)

•  WeknewthedatacouldtelluscriIcalinformaIonifwecouldviewthetestresults

“Techniciansweredeveloping

rela9onshipswithcustomersbecauseitwasnotuncommonforthemtobeat

theirhousesmul9ple9mesforrepair.

6

BeforeSplunk:• Istartedwith“postitnotes”withtestreadingswrikendownonthem• Begantouseaspreadsheetwithdataenteredbyhand• CreatedaMacrothatwouldimporttestresultsintothespreadsheetthattechswouldemail

•  MassiveamountofdataneededautomaIon•  Original“AutomaIon”waspulngthetestdataintoaphpbasedwebsite

•  Wehadthedatashow“redorGreen”forpassorfail…….Buttherewasnowaytoreportthetestfailures

Toobtaindata:•  TechniciansmanuallyuploadresultsfromthetestdevicetoaUSB• Plugintoalaptop•  surftoawebaddress• uploadthedatafiles•  IdenIfytheJobInformaIon•  Technicianwhoperformedthework

7

ChoosingSplunk•  Withthepreviouswebsite,Iwasnotabletomakereportsorchangeswithout

askingtheoriginaldeveloperto“pleasedoitforme”

•  Hemadechoicesofwhatdatatoparseandwhichakributetodisplay

•  Ofthe127testakributes,only27wereabletoview

•  Therewasnotawayformeto“work”withresultseasily………..

•  IlearnedaboutSplunkataCenturyLinkDSLconferenceinDenver•  AcolleaguemadeakeyintroducIon,whichledtoSplunk

•  WewereabletoputaSplunkconnecIonontheexisIngdatabasetocapturenewtestresultsaswellasingestALLtheprevious300K+testuploadsintoSplunk.

•  Iimmediatelybeganworkingwiththedata,allthewhiledreamingofwhatcouldbeaccomplishedasfaras“themeaning”ofthetestresults.

•  Itwaslikegoingfroma“closet”(whichhungdatainarelaIveorientaIon)

•  Toa“sockDrawer”whereIcoulddigaroundandlookforthemissingsock

“WithSplunk,wewereableto

figureoutwhatdrivestrouble,addressthe

standardsandadjusttosupport“performance”

8

SplunkBenefits

•  Webeganlookingatakributesofcustomercalls&dispatchIckets&andcomparingtouploadedtestdata

•  Webuiltdashboardsforallofthe127differentakributestoshowthemedianresultsandtheupperandlowerthresholds.

•  Wecouldsearchonanyakributeoranydistancewewanted

•  ThenwecouldlookupcustomercallinrecordsandtroubleIcketswhich“were”inseparatedatasilos

“Splunkintegratesintoeverypor9onofwhatwedo.

We’reabletobringsiloeddatabasesanddifferentinforma9onalltogether.”

9

SplunkBenefits•  SplunkwasthesoluIonformanagingourdataandprovidinganopportunitytomakeinformeddecisions

• SplunkhasstreamlinedandautomatedouroperaIonstestdatacollecIonandanalysis• Wecanconnect“siloed”databasesthatcontainrelatedtypesofinformaIon

•  Forinstance,thetechniciancannowsavethetestdataonhishandheldtestunitandsubmitdirecttoourSplunkdashboards.

•  Thedashboards“know”who,where,andwhenparIculartaskisworked,noneedforthemtofillintheinfotosubmitdata.

• Splunkintegratesintoeverydisciplineofwhatwedo•  WiththeconnecteddatabasesanddifferenttypesofinformaIon-wegetapictureofwhatishappeningwitha

customerservice.Wecanbegintovalidate“performance”notjusttestresults•  AllthedatacomestogetherinSplunkasoneview-asingleviewtoseemulIpledatasets•  A“singlelogon”notmul9pledisconnectedwebpages

•  WithSplunk,ourdatabasescancommunicatewitheachothertogiveusabekerview•  Splunkgivesususefulvisuals–graphicalrepresentaIons–notjustlogsandnumbers

SplunkBenefits

PerformanceTracking

Log/DataManagement

ServiceManagement/Efficiency

Visibility/ReporIng/Dashboards

12

SplunkforLog/DataManagement•  ASplunkforwardernowsendsdatadirectlyfromthemeter

toourindexers•  Eliminatesmanualinputbythetechnician•  Weknowwhodidwhatandwhere

•  WecombinealllogdataintoSplunk•  Machinedatasources•  Databases•  ModeminformaIon•  Dispatchreports•  Customercallindata

•  Ourprocessesaremuchmorestreamlined•  Independentdatabasescommunicatewitheachother•  Whichgivesmuchimprovedview,whichinturnprovidesbekercustomer

experience

“Techniciansnolongerhavetoput

inwhatjobsthey’reworkingon.Wealready

know.”

13

Users/Partners•  NaIonalOperaIonsteams•  Analysts•  DocumenIngEvaluaIng/UpdaIng/MaintainingProcessandStandards

•  RegionalFieldOperaIonsteams,•  Engineers•  Analysts•  CreatecompliancereporIngthatgoestoLocalOperaIonsteams

•  LocalOperaIonsTeams•  VP/GMs,Directors,AreaManagers,PlantSupervisors•  Technicians

•  Growingbythethousands—  Weexpect5,000+usersbytheendoftheyear

14

SplunkforVisibility/ReporIng/Dashboards

“Splunkhasbecomeour

periscope.Someofthedashboardswehavearequiteincredible,totellyouthetruth.”

•  Daily,weeklyandmonthlyreports•  AutomatedemailsfromSplunkto“management”•  Compliancereports,whouploadedwhattestdataforaservice•  InventorycontrolandtrackingofSettopBoxesandGateway/Modems,whatwas

installedandwhere•  Wasthe“process”followedduringcustomer“turnup”•  TechniciansarenowaccountabletothetestsperformedforserviceinstallaIon

•  ProcessRepeatabilityandmonitoring•  Makesuretechniciansarefollowingtherightsteps•  WeTestwiththesamemethods•  Runthesametestscriptsanduploadresults(basedoncompanystandardsto

supportperformance)•  ThetestequipmentismonitoredforFirmwareRevision•  WecantracethetestequipmenttoUserandmaintaintheInventoryofCapitol

equipment•  TheImetocompleteatask

•  Othergroupsareaskingforourhelpto“unlock”theirinformaIon

15

•  WemakereporIngavailableonalocal,regionalandnaIonallevelusingthesamedatasourcefromwithinSplunk.• WereacttothesameunfilteredinformaIon

16

SideBenefit:WeareabletoviewRevisionofFirmwareonatechnicianstestset,aswellasthelocaIonandcapabilityoftestequipment

17

18

SplunkforPerformanceTracking•  WeuseSplunktotrackkeyperformancemetrics:•  Settopboxperformance•  Modemperformance•  Copperperformance(notjust“tesIng”anylonger)

•  Wetrackproblemsolvingvariablesaswell•  Technicians,modems,copper,oracustomerissue(what’sontheLAN)•  WithSplunk,wecannarrowitdown&pinpointtheissue

•  SeeingintoallthemovingpartsofcomplexsituaIonsislikethe6sidesofaRubik’scube–wecanseetheactualsourcesofproblemsandhowtheyarerelatedtoeachother

19

SplunkasanMobileDeviceInterface

20

40Mcircuitthatthecustomerisexperiencing~7.5Mdownload

• Anexampleof“mashingdata”frommulIplesourcestocreateanopportunityforevaluaIon

21

Missingbandwidth

Bounceindicates“Bridgetap”

Fulluseofbandwidth

”looksgood”

poorperformingloop Anacceptabletest

22

SplunkUniquetool:The“PromiseChart”

Bluelinerepresenttestdataas“Losspertone”

ColoredlinesrepresentMAXlosspertoneforaService

Ifthebluelinedoesnottouchthecoloredbar,theloopcanNOTsupportthatprovisionedspeed

23

24

“goodsyncrates”

Uncorrectederrorsarepresent

25

• OriginalDispatchDataSetallowsfor45daysofhistory,wenowkeeptheinformaIonindexedinSplunkfor18months

• Wechangedthe“clunky”interface

• WeaddedMappingtothedispatch

26

SideBenefit:Weareabletoreview“technicianperformance”Didtheyperformatest,wasittherequiredtests,anddidthetestsmeetthestandard

27

28

SplunkforServiceManagement/Efficiency

•  Wecannowlookatdifferentakributesfrommanysourceswithinourcompany,tofigureoutwhatdrivesacustomercall-in/poorexperience

•  Wenarroweddown175testitemstothehandfulthatdrivetroublecalls•  Wehaveadjustedourteststandardsto“whatdrivesapoor

customerexperience”

•  Splunkhelpsusprovewhereweneedtomakeadjustmentstoavoidtroubles

•  TechniciansusethisinformaIontoknowwhattolookfortofixthecustomersproblem

“WithSplunk,Iknowwehavesaved

expensedollarsusing‘costavoidance’byadjus9ngourtes9ng

standards.”

29

WeseethelengthofImeonanyjoborcrew.Wetrackthelengthof

Imethatjobstakeasthedaymoveson…………

Wereportthepercentageofuploads

Wecandrilldowntothedetailofthejobanduploads

30

SplunkWordstotheWise….•  Youhavetohaveacoreunderstandingofwhatthedatarepresentstomakeituseful

•  The“users”havetobeapartoftheimplementaIonandputinefforttomakethedatameaningful•  Reportanddashboarddevelopment•  IdenIfyindependenthowever“Related”datasourcesfor

ingesIonintoSplunk.

31

SplunkSuccesses•  We’veimprovedmanagementsawarenessoftechnicianefficiencythroughautomated

reporIng.

•  WeknowSplunkhasreducedcost.•  PrevioustoourSplunkanalysisofourtestdata,wewoulddispatch“twotruckrolls”foraVideo

customerservedoveracopperloop.•  Wenowonlydispatchonceforaninstall•  Whereacustomermayhave“failedstandard”andwasdeniedservice,wenowhavestandards

thataremeaningfultoperformanceandwecanevaluatequickly

•  WenowcanvalidateacustomersservicebecausewehaveinstallaIonhistorythroughuploadedtestresults.

•  WeareeffecIvely“mapping”our“lastmile”performanceusingSplunk•  Installsandrepairs•  GatewayandSTBautomatedreporIng•  DSLAMperformance

32

SplunkingAhead….•  Googlemapping•  We’regoingtogetmoregraphicalintermsof

layingoverinstallsandtroubleImes•  Thiswillbekerhighlightwherewehave

performanceissues

•  IdenIfyingwheretoinvestforahigherreturnoninvestment•  Maintenanceversusinstallingnewtechnology

33

Lookingahead……….WhatifwecombinedourtestdatatodispatchinformaIon?AndthenhaveSplunkmapit?• WethencouldidenIfyissuesandfigurewheretospendourImeinmaintenance.• WecouldidenItywheretoplacenewtechnologies• WeCouldfigureoutwhyacertainareatakeslongertoinstallorwhywehaveincreasedrepairacIvity• WecouldbegintoPredict…………………..Toan9cipateresultsbasedonFactualinformaIon.

ThankYou