Copyright©2016SplunkInc.

NadineMillerTechnicalSupportEngineer,Splunkaka'vraptor'onIRCandSlack

KVStore:HammerTime

Disclaimer

2

Duringthecourseofthispresentation,wemaymakeforwardlookingstatementsregardingfutureeventsortheexpectedperformanceofthecompany.Wecautionyouthatsuchstatementsreflectourcurrent

expectationsandestimatesbasedonfactorscurrentlyknowntousandthatactualeventsorresultscoulddiffermaterially.Forimportantfactorsthatmaycauseactualresultstodifferfromthosecontainedinourforward-lookingstatements,pleasereviewourfilingswiththeSEC.Theforward-lookingstatementsmadeinthethispresentationarebeingmadeasofthetimeanddateofitslivepresentation.Ifreviewedafter

itslivepresentation,thispresentationmaynotcontaincurrentoraccurateinformation.Wedonotassumeanyobligationtoupdateanyforwardlookingstatementswemaymake.Inaddition,any

informationaboutourroadmapoutlinesourgeneralproductdirectionandissubjecttochangeatanytimewithoutnotice.Itisforinformationalpurposesonlyandshallnot,beincorporatedintoanycontractorothercommitment.Splunkundertakesnoobligationeithertodevelopthefeaturesorfunctionality

describedortoincludeanysuchfeatureorfunctionalityinafuturerelease.

WhoamI?

SeniorTechnicalSupportEngineer

SearchHeadCluster&IndexClusterSME

KeeperofSupportKVStoreTrouble-ShootingDocs

SplunkTrustMember

SeniorUNIXSystemsAdministratorinaPreviousLife

3

Whatarewetalkingabout?

FocusedonKVStorewithinthecontextofSHC

Discussionofbackup/restoreimportanttoStandalone

WheretodisableKVstore

Merging!

Nodiscussionofdevelopment/customuse

4

Whatisthisbeast?

KVStoreis:

adatabase

Mongodb

Storesuser-createddatainSplunkthatcanbelinkedtoeventsviasearches

Anothermethodforlookups

Scheduler

5

WhyisKVStoreImportant?

ES,ITSI,otherpremiumapps

user-createddatastoredinKVStore

SHC

keepstrackofcompletedjobstopreventduplicatejobsifSHCmembergoesoffline

Stand-aloneSH

trackingofcompletedjobsincaseofdowntime

6

KVStore101

Mongodb-projecthasgooddocs,donothesitatetorefertothem

Logs:$SPLUNK_HOME/var/log/splunk/mongodb.log

$SPLUNK_HOME/var/log/introspection/kvstore.log

KVStorewhollyunnecessaryonIDX&HWFbydefault

unlessyouexplicitlyuseKVStore

oraSHisenabledontheinstance(maynotbenecessary,though,dependsonuse)

Note:setting"replication=true"createsCSVsthatreplicatetoIDX,notmongodbreplicationtoIDX

7

KVStore101

SearchestogetKVStorestatus

SHCstatuscommands

DMC/MCinfo

Collectioncounts,size,etc.

RESTcurl -ku admin https://<host>:<mPort>/services/kvstore/status

OthersinRESTendpointdocs

8

#1Failure

NoBackups!

Folksdon'trealizethisdataisnotinanindex

Ifyoudon'ttakeregularbackups,easytolosealldataintheKVstore

Protip:RAIDisnotabackup

10

BackupMethods

OSlevel

SHC-firstmakesureallSHsareinsync6.5orlater:splunkshowkvstore-status

before6.5:curl -s -k https://localhost:8089/services/server/info | grep kvStoreStatusCheckDMC/MCforcollectioncounts

TakeSHofflineTakeafilesystemsnapshotorcopyentireKVstoredirectory

BringSHbackon-line

11

BackupMethods

ITSI

Usebackup/restoreinWebUI

Usekvstore_to_json.py(repurpose?)

Starcher'sbackupscript

https://github.com/georgestarcher/Splunk-backupkvstore

Outputlookup

largekvstorecouldbeaproblem-highmemoryconsumption

multi-kvcouldalsobeaproblem-flattens

12

RestoringZeWholeEnchilada

Protip:TESTinadevenvironmentfirst!

OSLevelstandaloneSH

OfflineSplunk

Copykvstoredirectorybackincorrectlocation

RestartSplunk

13

RestoringinSHC

SHCversion6.3orlater:

ShutdownSHCmemberssplunk clean kvstore --cluster

Copykvstoredirectoryintoplaceononemember

VerifyKVstoreisgood

RestartotherSHCmembers

SHCpriorto6.3,complicated

Re-bootstrapclusterfromscratchaftertemporarilysettingreplication_factor=1

Advisesupportcase

14

SHC:WhenEverythingisaNail...

KVStoreComplications

KVstoreisaseparateclusterwithintheSHC

KVstorecaptainprobablydifferentfromSHCcaptain

MonitorKVstore:

Staleness

Collectioncounts:

Outofsync=problem

MonitoryourSHs,ifoneisofflineforanysignificanttime,likelytolosesync

16

UsetheMonitoringConsole,Luke

Especiallyin6.5andlater

17

KVStoreBasicStatus

18

MC->Search->KVStore:Deployment

Screenshothere

KVInstanceCollectionMetrics

19

MC->Search->KVStore:Instance

KVStoreReplicationLatency

20

MC->Search->KVStore:Deployment

OpLog

Mongodbusesacircularoperationslog,aka"oplog"

IfaKVstoremembercannotkeepupwiththetransactionsinthislog,eventuallyitwilllosesyncExamples:

SHCmemberoff-lineforaperiodoftimeexceedingoplogsizeLargeKVstorechangesareperformedinanon-transactionalway(e.g.createtemptable=>replaceexistingtable)

SHCissobusythatoplogheadisoverwrittenfasterthantheKVstorecanreplicateacrossallmembers

21

KVStoreBasicStatus

22

OotBOplogsize=1GB

Butwhy?

Mongodbbydefaultsetsepilogsizeto50GBonlargepartitions

1GBgenerallyfineforanormalSHCthat'sjustkeepingtrackofschedulerjobs

1GBevenworksfineforawhilewithpremiumapps—untilitdoesn't

23

Howbigshould"oplogSize"be?

"Itdepends"

Lookatsizeof"kvstore"directoryondiskorinMC

Hastherebeenarecentresync(compactonlyoccursonresync)?

Latency?

AnyodditiesinWindowsize?

Doinganythingusual(temptables/replace)?

Generallysomethingbetween10-20gbshouldworkunlessyou'redoingsomethingunusual

24

KVOpLogWindow

25

MC->Search->KVStore:Deployment

OtherOpLogImpacts

HowbigistheSHC?

SlowdiskI/O

Networklatencyw/inSHC

IfyouarehavingproblemsreplicatingconfigchangesinsideSHC,canimpactkvstore

MayneedtuninginSHC(hitmeinSlack/IRC)

26

MergingKVStores

ThankstoSplunk,itispossibletomergeKVStoresunlikerawMongodb

Caveats:

Mustbedoneonacollectionbycollectionbasis

Havetounderstandkeyfieldsincollection

Multi-kvflattened

Example:| inputlookup incident_review_lookup | table _key,time,rule_id,owner,urgency,status,comment,user,rule_name|outputlookup new_data.csv

27

GeneralMethod

BackuptheKVStoreyouwanttomergeinto"good"kvstore

Performtheprevioussearchontheincomplete"bad"kvstoretogetaCSVfile

Copybothtoatestinstance

Restore"good"kvstoreintotestinstance

PerformwhateversurgeryisnecessaryontheCSVfiletoremoveunwantedrecords

“Merge”usinganothersearch:|inputlookup new_data.csv |outputlookup append=true incident_review_lookup

28

OtherSuggestions

DuaneandGeorge's.conf2016talk:”ShopSmartattheKVStore:BestValueTricksfromtheSplunkKVStoreandRESTAPI”

https://conf.splunk.com/files/2016/slides/shop-smart-at-the-kv-store-best-value-tricks-from-the-splunk-kv-store-and-rest-api.pdf

George'sTA-TA-SyncKVStore

https://splunkbase.splunk.com/app/3519/GeminiKVStoreToolsforSplunk

Looksinteresting,haven'ttestedhttps://splunkbase.splunk.com/app/3536/

29

Errata

OnastandaloneootbSH,KVstoreisonlyusedforusers’searchhistory;noscheduledjobstatusistrackedusingKVstore

30

Addendum#1

Youcandoa“roundrobinresync”toincreaseopLogSize

Slowandtediousifyouhavealargecluster

31

Addendum#2

FindoutifKVStoreisinuseonyourIDXorHWF:

curl -k -u admin https://127.0.0.1:8089/servicesNS/nobody/search/storage/collections/config|egrep -i "\<title\>"

Shouldonlyreturn:

<title>collections-conf</title> <title>SavedSearchHistory</title>

Caveats:OnlyadminscanlogintoIDX/HWF;noappsareinstalled;youusedacleaninstanceasanIDX

32

Questions?

THANKYOU