Copyright©2016Splunk Inc.

AlexJamesPrincipalProductManager,Splunk

SearchOptimizationKarthikSabhanatarajanSeniorSoftwareEngineer,Splunk

&

SessionOutline

WhyOptimizeSPL?Whatdoesoptimizationinvolve?What’snewin6.5?DemoWhatelsecanyouexpect?

2

Demo1– WhyoptimizeSPL?

A B C D L EA B C D L EA B C D LA B C D LA B C D

A B C D L EA B C D L EA B C D LA B C D

Taleoftwosearches

4

searchSourceType lookupL evalE searchA&L&E

Diskrawdata&index

searchSourceType&A lookupL searchL evalE searchE

• 10,000,000indexhits• 1,000,000eventscreated(i.e.extractions)• 1,000,000lookups• 1,000,000evals• 1,000,000filters• Produces100,000matchingevents

• 7,000,000fewerindexhits• 700,000 fewereventscreated• 700,000fewerlookups• 800,000fewerevals• Net500,000lessfilters• ProducesIDENTICAL100,000matchingevents

TOTALWORK

SAVINGS

WhatdoesoptimizationInvolve?

5

OptimizationPrinciplesDoaslittleworkaspossible– Retrieveonlytherequireddata– Moveaslittledataaspossible– Parallelizeasmuchworkaspossible– Setappropriatetimewindows

ImplicationsbasedonSplunkArchitecture– Filterasmuchaspossibleintheinitialsearch– Join/Lookuponlyonrequireddata– Evalontheminimumnumberofeventspossible– Delaycommandsthatbringdatatothesearchheadasmuchaspossible.

6

Retrievingonlytherequireddata• Trytofilterassoonaspossible- Inthefirstsearchifpossible

– search ERROR | search x=y– search ERROR x=y – search ERROR

| eval MB = bytes / (1024 * 1024)| lookup usertogroup uid as user OUTPUT group as group| search group = “admin” status=404

– search ERROR status=404 | eval MB = bytes / 1024 | lookup usertogroup uid as user OUTPUT group as group| search group = “admin”

• Sometimesitisnotpossible:– search field=value | eval KB=bytes/1024 | where field2=field3

• ButstilldothefilteringASAP:– searchfield1=value|wherefield2=field3|evalKB=bytes/1024

7

Don’tdounnecessarywork• Neverdothis:

search ERROR | eval MB = bytes / (1024 x 1024) | search status=404

• Dothis:search ERROR status=404 | eval MB = bytes / (1024 x 1024)

• Don’tdothis:search ERROR | stats sum(bytes) as sum by clientip | search sum >1048576 AND clientip="10.0.0.0/8"

• Dothis:search ERROR clientip="10.0.0.0/8" | stats sum(bytes) by clientip | search sum > 1048576

• Sameprincipleappliesforanythingthatinvolvessignificantcost:– i.e.Stats,Dedup,Sort,Join,Lookups,Evals

• Principle:Reduce/Augment/Reduce/Augment/etc.

8

Streamingvs Non-Streamingcommands• Somecommandsarestream-able

ê essentiallyoneeventin– one(orno)eventoutê where,search,eval, lookup etc.

• Somecommandsrequireallthedatatoproduceresults:ê stats,sort,dedup,top, append etc.

• ‘non-streaming’commandsrequiredatafromallindexerstofinish

9

NameAlex

Ben

Clint

David

Evan

NameBen

Clint

NameAlex

David

Ben

Evan

Clint

NameAlex

David

NameClint

NameBen

Evan

Non-streamingcommands

10

sortName … where …

search lookup where presorteval

search lookup where presorteval

search lookup where presortxeval

search lookup where sortNameeval … where …

Parallelizeforaslongaspossible• Non-streamingcommandsforcedatatothesearchhead

– append,stats (etal),dedup,sort

• Problems:ê Lotsofdatamovementcostsê Lossofparallelism

• Mitigations:– Pushanyworkyoucantotheleftofthenon-streamingcommand… | sort -bytes | where x > 20… | where x > 20 | sort -bytes … | append [search sourcetype=a “WARNING” | eval KB=b/1024 ] | search KB > 5… | search KB > 5 | append [search sourcetype=a “WARNING” | eval KB=b/1024 | search KB > 5]

11

NewinSplunk6.5

Howsearchworksin6.4

13

searchsourcetype=access-*(status=401orstatus=403)|lookup usertogroupuserOUTPUTgroup|where src_category=“email_server”

search lookup

1)Spliton‘|’andcreateprocessorpipeline

2)Distributebetweenindexandsearchheads,passargumentsandexecute

search lookup whereIndexer1

search lookup whereIndexer2

combine

Searchhead

where

Howsearchworksin6.5

14

searchsourcetype=access-*(status=401orstatus=403)|lookup usertogroupuserOUTPUTgroup|where src_category=“email_server”

search lookup

1)ParseintoAST

search lookupIndexer1

search lookupIndexer2combine

Searchhead

searchsourcetype=access-*(status=401orstatus=403)src_category=“email_server”|lookup usertogroupuserOUTPUTgroup

2)OptimizeAST

3)ConstructPipelinefromAST

JSONAST

OptimizedJSONAST

4)Distributebetweenindexandsearchheads,passargumentsandexecute

• Turniton/offgloballyinLimits.conf

[search_optimization]enabled = false

• Overrideglobalsettingforaspecificsearchusingnoop| datamodel Authentication Successful_Authentication search| where Authentication.user = “fred” | noop search_optimization=true

Optimizerisonbydefault

Demo2

Real-TimeSPLOptimizationinSplunk6.5

17

Automaticallyoptimizesqueryperformance

Filterresultsasearlyaspossible lookup onlyonrequireddata

eval ontheminimumnumberofeventspossible

ProcessasmuchaspossibleinparallelonIndexers

Automaticallyappliesbestpracticetechniquestooptimizeexecutionspeedofanyquery

Whatoptimizationsaredone?

18

Pushingpredicatestotheleft(ordown)– For*any*streamingcommandsthatdon’tmodifyafield:

– |rangemap field=scoreF=0-64D=65-69C=70-79B=80-89A=90-100|searchuser=“A*”– |searchuser=“A*”|rangemap field=scoreF=0-64D=65-69C=70-79B=80-89A=90-100

– SpecialHandlingforsomecommands:ê Rename– |renamesrc asip |whereip=“192.1.2.13”– |wheresrc=“192.1.2.13”|renamesrc asip

ê Eval– |evalsrc=if(isnull(src)ORsrc=“”,“unknown”,src |wheresrc =“192.1.2.13”– |wheresrc =“192.1.2.13”|evalsrc=if(isnull(src)ORsrc=“”,“unknown”,src

ê Byclausefilters– |statscountbyip |wherecidrmatch(“192.1.2.1/28”,ip)– |wherecidrmatch(“192.1.2.1”,ip)|statscountbyip

Search/Wheremerging– searchERROR|search404|wheresourcetype=“windows”– searchERROR404sourcetype=“windows”

Whatoptimizationsarecominglater?

19

PredicateSplitting– |evalx=a+b|wherex=10andy=10– |wherey=10|evalx=a+b|wherex=10

PredicateNormalization– searchERROR|where10=y– searchERRORy=10

Collapsingconsecutivecommands– |renamebasz,aasx|renamexasy– |renamebasz,aasy– |evalx=a+b|evaly=c+d– |evalx=a+b,y=c+d

ConvertingEvalFunctionsintoSearchfiltersifpossible– searchERROR|wherecidmatch(“13.4.3.1/31”,ip)– searchERRORip=“13.4.3.1/31”

ProjectionElimination– searchERROR|evalx=a*b|inputlookupusersuidOUTPUTusername|statscountbyb– searchERROR|statscountbyb

Re-usingprevioussearchresults

Whatdoesthismeanforyou?

20

FasterSearchesUpgradeto6.5Scanfor‘inefficientsearches’– Especiallyinscheduledworkloads...

UsetheJobInspectortoseeoptimizationinactionOptimizefurthermanuallyifneeded

Q&A

21

Disclaimer

23

Duringthecourseofthispresentation,wemaymakeforwardlookingstatementsregardingfutureeventsortheexpectedperformanceofthecompany.Wecautionyouthatsuchstatementsreflectourcurrentexpectationsandestimatesbasedonfactorscurrentlyknowntousandthatactualeventsorresultscoulddiffermaterially.Forimportantfactorsthatmaycauseactualresultstodifferfromthose

containedinourforward-lookingstatements,pleasereviewourfilingswiththeSEC.Theforward-lookingstatementsmadeinthethispresentationarebeingmadeasofthetimeanddateofitslivepresentation.Ifreviewedafteritslivepresentation,thispresentationmaynotcontaincurrentoraccurateinformation.Wedonotassumeanyobligationtoupdateanyforwardlookingstatementswemaymake.Inaddition,anyinformationaboutourroadmapoutlinesourgeneralproductdirectionandissubjecttochangeatanytimewithoutnotice.Itisforinformationalpurposesonlyandshallnot,beincorporatedintoanycontractorothercommitment.Splunkundertakesnoobligationeithertodevelopthefeaturesor

functionalitydescribedortoincludeanysuchfeatureorfunctionalityinafuturerelease.

MigratingSlidesforMac1. Forbestresults,simplypasteyourslidesintothis

template.

2. ApplyslidelayoutsusingtheLayout buttonundertheFormattab.

3. IfLayoutstilldoesnotreflectthedesiredMasterLayout,chooseResetLayouttoDefaultsettings.

4. Deleteunwantedtemplateslides(anyslidesafterLastSlide).

5. ChooseSaveAstosavethefilewithoutoverwritingthetemplate.

24

MigratingSlidesforPC1. Forbestresults,simplypasteyourslidesintothistemplate.

– Pastingafterabulletslideisrecommended

2. Reviewallslidesandmakeformattingadjustmentsasneeded– OntheHome ribbon,clickLayout andselectthecorrectslidelayout– ClickReset toresetallslideelementstothedefaultsizeandposition– Checkforhiddentext,suchaswhitetextonawhitebackground

3. Deleteunnecessarytemplateslides4. SaveAstosavethefilewithoutoverwritingthetemplate

25

SlideMasters• Whenimportingslidesfromanotherpresentation,the

SlideMastersassociatedwiththoseslidesmayalsoimporttothistemplate.Thisisa‘feature’ofPPTandcannotbeturnedoff.

• TodeleteunwantedSlideMasters:– makesureallslidesinthepresentationhavethenew

templateSlideMasterLayoutsassigned(first16SlideMastersshownunderLayout)

– GotoView/MastertodeleteanyunwantedSlideMasters

• ThelastSlideMasterinthistemplateiscalledLastSlide.AnySlideMastersafterthisslidewerelikelyimportedfromanotherpresentationandcanbedeleted(ifnolongerusedbyanyslides.)

26

ImportantTips• Thistemplateusesareducedslidesize.Youmayhavetomanuallydecreasethesizeofsomeitemssuchasstrokes andfonts.

• Iffontsappearbiggerthandesired,remembertoassignaLayout toyourslideandResettoDefaultSettings.

• Ifpagenumbersdonotappearorarethewrongformatting,remembertoassignaLayout toyourslideandResettoDefaultSettings.

• Thecolorsinyourgraphicswillautomaticallybeshiftedtothenewpalette.Pleaseadjustasneeded.

27

Agenda

AgendaItemAgendaItemAgendaItem

28

2012GoalsandObjectivesExample

GoalItemGoalItemGoalItem

29

SampleTitle,66pt.Calibri

30

SampleTitle,66pt.Calibri

31

Subhead

TitleOnlySlide,60pt.Calibri

32

TitleOnlySlide,54pt.Calibri

33

Samplewithscreenshot

34

Screenshothere

SampleTwo-columnFormatSubhead

35

Sampletwo-columnformat

• Sampletwo-columnformat,sentence– Secondbullet

Sampletwo-columnformat

SplunkObjectStyleandColor

36

Hardware ProductBusiness/Corporate

HighlightOnlyGenericVirtualization

Generic

Thesearesuggestedusesforcolorsonly.

AssignDefaultObjectStyle

37

ApplyingSplunkObjectStyle

38

ToapplytheSplunk objectstyletoanyshape:1. Selecttheshapewiththedesiredstyle2. ClickonFormatPainter(paintbrush)toolintoolbar3. Applystyletoanynewshape

Logos

39

CorporateLogo ProductLogo

Logos

40

Logos

41

Splunk Icons

42

search barchart lock cloud opencloud checkmark envelope

storage- 3storageiPhoneiPadandroid

server indexer forwarder searchhead desktop laptop

datacenter

Splunk server

firewall

Splunk IconsCont’d

43

application virtualmachine virtualserver network wwworglobal tools

logfile RFID router loadbalancer script shoppingcart

user users gears/settings gear messaging tag/ticket

document

alert

gps tower

Splunk Icons

44

Checkmark InfoAlert StopiPhoneiPadAndroid

Twitter Facebook LinkedIn RSS YouTube ShoppingcartGPSTower

Healthcare Hospital Officebuilding VoIPPhone Support POSCardReader RFID

Splunk Icons

45

SecurityIcons

46

FirewallAttacker,Generic

Attacker,Insider

Attacker,Nation/State

Botnet Key

Malware MalwareDocument

MalwarePackaged

SecurityBadge

SecurityServer

Shield VirusFootsteps

TheInternetofThingsIcons

47

POSCardReader

RFIDElectricCar

EMVReaderInternetofThings Meter Factory

SignatureCapture

Arrows

48

TableExample

49

ColumnTitle ColumnTitle ColumnTitle ColumnTitle

Text Text Text Text

Text Text Text Text

Text Text Text Text

Text Text Text Text

Text Text Text Text

TableExample

50

ColumnTitle ColumnTitle ColumnTitle ColumnTitle

Text Text Text Text

Text Text Text Text

Text Text Text Text

Text Text Text Text

Text Text Text Text

SampleCustomerSuccess

51

Customerlogohere

CustomernameCustomercompany

“SplunkmakesitcheaperandeasierforHughestoanalyzenetworktrafficforenterprisecustomersaswellasmanagebandwidthforconsumerandsmallbusinesscustomers.”

BulletplaceholderBulletplaceholderBulletplaceholder

Screenshotorgraphichere

TimelineChart

52

Q1 Q2 Q3 Q4

Milestone Event

ChartExample

53

PlannedActual

Number

Number

Number

Number

Number

FY09 FY10FY08PreviousYear

N%growthoverFYxx

QuoteBox

54

“Apessimistseesthedifficultyineveryopportunity;anoptimistseestheopportunityineverydifficulty.”

-WinstonChurchill

QuoteBox

55

QuoteBox

56

WhatNow?

57

Relatedbreakoutsessionsandactivities…

THANKYOU