ManySolutions,OneGoal.

SplunkandWindowsEventLog:BestPractices,Reductionand

EnhancementDavidShpritzAplura,LLC

BaltimoreAreaSplunkUserGroupJune,2017

ManySolutions,OneGoal.

Agenda

• GettingWindowsEventsintoSplunk:PatternsandPractices• TURNDOWNTHEVOLUME:Licensereductiontips• Makingthemmoreuseful:Improvingknowledgeobjects

ManySolutions,OneGoal.

GroundRules

• Fidelitylevels• Howcompletearetheevents?

• WindowsEventinterpretation• Thesearebinaryrecords• AgentscanreadthemdirectlyorasktheWindowsAPI• Thismeansthatyouaren’treallygettingtheeventlog,justarepresentationofit

ManySolutions,OneGoal.

GettingWindowsEventsintoSplunk

ManySolutions,OneGoal.

DifferentWaystoSkinaCat

• BesttoWorst• UniversalForwarder• WindowsEventForwarding• WMI• EVTXImport• ThirdPartySyslogAgent(Snare,forexample)

ManySolutions,OneGoal.

UniversalForwarder

• ThebestwaytogetWindowsevents(ofcoursewe’rebiased)• Pros• Highfidelity• CanbecontrolledbyDeploymentServer• CanfilterWindowsevents• Canrunscripts(batch,exe,PS)• Canalsogetadmon (greatforassetsandidentities)

• Cons• “Anotheragent!?!?”• Securityconcerns

ManySolutions,OneGoal.

WindowsEventForwarding

• NativetoWindows(2008R2andup)• Pros• NativetoWindows,noagent• CanbeconfiguredwithGPO

• Cons• Almosthighfedlity• Slower• CustomertestingshowsitconsumesmoreresourcesthanaUF

ManySolutions,OneGoal.

WMI

• UsedbyaSplunksystemtocollectWindowsEventsfromaremotesystem• Pros• Remote,noagent

• Cons• Slow• Alotofoverhead• Limitedcollectionavailability(mayneedmultiplesystemstopullallofyourWindowshosts)• Lowfidelity• Dealingwithpermissions

ManySolutions,OneGoal.

EVTXImport

• Canbeusedtoexporteventlogsfromasystemandthenimporttherawfilesonanothersystem• Oftenseenin”air-gapped”environments• Pros• Nonetworkconnectionneededfromtheclientsystemstothetargetindexers

• Cons• Lowfidelity(rememberthat“interpretation”thingearlier?)• Movingandremovingthefilesisamanualprocess• Opentoeventduplication

ManySolutions,OneGoal.

ThirdPartySyslogAgent(Snare)

• It’sathing,theseagentsexist• Pros• Canworkwithyourexistingsysloginfrastructure

• Cons• Superlowfidelity• Unreliable(syslogneverdies)• Remoteconfiguration?

ManySolutions,OneGoal.

TURNDOWNTHEVOLUME:Licensereductiontips

ManySolutions,OneGoal.

• Splunkestimatesbetween200-300mbperday,persystem• Ofcourse,thatcanvarywildly• Lotsofrepeatedeventswithlittletonovalue(lookingatyou4662)• Dowereallyneedallofthese?• Doweneedeverypartofallofthese?

Thesethingsarechatty

ManySolutions,OneGoal.

Stratergery

• Pickyoursystemscarefully• Pickyourinputscarefullyonthosesystems• WhitelistandBlacklistcarefully• Resolvingobjects• Baseline?• Current_only?Start_from?• XmlWinEventLog• Filteringandcleaningup

ManySolutions,OneGoal.

Whichsystems?

• JustActiveDirectoryservers?• Endpoints?• Servers?• Sorry,thisisonacasebycasebasis

ManySolutions,OneGoal.

Pickingyourinputs(notyournose)

• SetabaselineforwhichlogsALLofyoursystemsshouldbesending• Forothereventlogs,useanindividualappforturningonthatinput(DS-Input-wineventlog_application)• Doyouneedadmon fromallofyoursystems?Probablynot,justonafewADsystems• Makesureyouaren’tusinglegacyinputs(WMIvsPerfmon)• LookoutforWindowsFirewallEvents(maybeStreaminstead?)

ManySolutions,OneGoal.

WhitelistingandBlacklisting

• Canhaveabigimpactonyourlicenseusage• Investingthetimein“whichevents”canpayoffbig• Carefulwithawhitelist-onlyapproach• Notethatthereisalimittothenumberoflists• Performedattheforwarder,sodoesnotusenetworktraffic

ManySolutions,OneGoal.

Someniceblacklistoptionstostartwith

• https://gist.github.com/automine/a3915d5238e2967c8d44b0ebcfb66147

ManySolutions,OneGoal.

ADObjectResolution

• ResolvesthingslikeSIDsandGIUDs• YoucantellSplunkwhichDCstousetoresolvethese• Canaddsomeoverhead(CPUandMemory),butusuallylowimpact• Recommendationistoresolvethem(lookattheevt_*)optionsininputs.conf forWindowsEventLogs

ManySolutions,OneGoal.

BaseliningAD

• WillcollectyourwholeADschema• CantakeupalotofmemoryonADcontrollers• ButbaseliningisusefulforAssetsandIdentitiesinES• Sobecarefulwhichsystemsyoubaselineon

ManySolutions,OneGoal.

Current_only vs.start_from

• Current_only tellsSplunktoonlygrabthelatestevents(liketail–f,ifWindowshadsuchathing)• Usefultomakesureyoudon’tgetallofthehistoricaldata• Maywanttosetthatto“true”oninitialdeployment• Thensetto“false”,restart,anditshouldpickupfromthecheckpoint• Start_from shouldbe“oldest”• Settingitto“newest”canbeusedtogrababacklogofevents• I’veneverseenthisinthewild

ManySolutions,OneGoal.

XmlWinEventLog

• Shouldreducelicenseusage(claimsareupto70%)• ItwillalwaysbeinEnglish(pro?Con?)• Hardertoread,Imean,it’sXML• QualityofCIMcompliancehasbeenvariedinthepast• Itdoesn’t”looklikeWindowsevents”andsomeauditorsarenotbright• Whatifyoucouldgetthesamelogsavingsandthereadability

ManySolutions,OneGoal.

Filteringandcleaningup

• Don’tuse“suppress_text”• It’stempting,buttheregoesthebabywiththebathwater• Maybejustcleanupthetextyoudon’tneed

ManySolutions,OneGoal.

Filteringandcleaningup

• IPv6supportineventlogsresultsinalotof“::”and“ffff”andothergarbage• Let’scleanupalot(thankstoalotofpeopleforthis)

• https://gist.github.com/automine/5c8ef5b50e1df38249dfba01a70f2875

ManySolutions,OneGoal.

MakingThemMoreUseful

ManySolutions,OneGoal.

Sorry,Iranoutoftime

• GotES?TakealookatRyanFaircloth’sSecKit work• https://splunkbase.splunk.com/app/3059/• https://bitbucket.org/SPLServices/seckit_sa_idm_windows

• AlternativeTAs• ShouldhelpwithKOoverhead• https://github.com/my2ndhead/TA-microsoft-windows (candoXMLevents)• https://bitbucket.org/SPLServices/seckit_ta_microsoft_windows (forusewithSecKit)