• Home

  • Documents

  • Splunk and Windows Event Log: Best Practices, Reduction ... · •Current_onlytells Splunk to only...
25
Many Solutions, One Goal. Splunk and Windows Event Log: Best Practices, Reduction and Enhancement David Shpritz Aplura, LLC Baltimore Area Splunk User Group June, 2017

Splunk and Windows Event Log: Best Practices, Reduction ... · •Current_onlytells Splunk to only grab the latest events (like tail –f, if Windows had such a thing) •Useful to

  • Upload
    others

  • View
    7

  • Download
    0

Embed Size (px)

Citation preview

Page 1: Splunk and Windows Event Log: Best Practices, Reduction ... · •Current_onlytells Splunk to only grab the latest events (like tail –f, if Windows had such a thing) •Useful to

ManySolutions,OneGoal.

SplunkandWindowsEventLog:BestPractices,Reductionand

EnhancementDavidShpritzAplura,LLC

BaltimoreAreaSplunkUserGroupJune,2017

Page 2: Splunk and Windows Event Log: Best Practices, Reduction ... · •Current_onlytells Splunk to only grab the latest events (like tail –f, if Windows had such a thing) •Useful to

ManySolutions,OneGoal.

Agenda

• GettingWindowsEventsintoSplunk:PatternsandPractices• TURNDOWNTHEVOLUME:Licensereductiontips• Makingthemmoreuseful:Improvingknowledgeobjects

Page 3: Splunk and Windows Event Log: Best Practices, Reduction ... · •Current_onlytells Splunk to only grab the latest events (like tail –f, if Windows had such a thing) •Useful to

ManySolutions,OneGoal.

GroundRules

• Fidelitylevels• Howcompletearetheevents?

• WindowsEventinterpretation• Thesearebinaryrecords• AgentscanreadthemdirectlyorasktheWindowsAPI• Thismeansthatyouaren’treallygettingtheeventlog,justarepresentationofit

Page 4: Splunk and Windows Event Log: Best Practices, Reduction ... · •Current_onlytells Splunk to only grab the latest events (like tail –f, if Windows had such a thing) •Useful to

ManySolutions,OneGoal.

GettingWindowsEventsintoSplunk

Page 5: Splunk and Windows Event Log: Best Practices, Reduction ... · •Current_onlytells Splunk to only grab the latest events (like tail –f, if Windows had such a thing) •Useful to

ManySolutions,OneGoal.

DifferentWaystoSkinaCat

• BesttoWorst• UniversalForwarder• WindowsEventForwarding• WMI• EVTXImport• ThirdPartySyslogAgent(Snare,forexample)

Page 6: Splunk and Windows Event Log: Best Practices, Reduction ... · •Current_onlytells Splunk to only grab the latest events (like tail –f, if Windows had such a thing) •Useful to

ManySolutions,OneGoal.

UniversalForwarder

• ThebestwaytogetWindowsevents(ofcoursewe’rebiased)• Pros• Highfidelity• CanbecontrolledbyDeploymentServer• CanfilterWindowsevents• Canrunscripts(batch,exe,PS)• Canalsogetadmon (greatforassetsandidentities)

• Cons• “Anotheragent!?!?”• Securityconcerns

Page 7: Splunk and Windows Event Log: Best Practices, Reduction ... · •Current_onlytells Splunk to only grab the latest events (like tail –f, if Windows had such a thing) •Useful to

ManySolutions,OneGoal.

WindowsEventForwarding

• NativetoWindows(2008R2andup)• Pros• NativetoWindows,noagent• CanbeconfiguredwithGPO

• Cons• Almosthighfedlity• Slower• CustomertestingshowsitconsumesmoreresourcesthanaUF

Page 8: Splunk and Windows Event Log: Best Practices, Reduction ... · •Current_onlytells Splunk to only grab the latest events (like tail –f, if Windows had such a thing) •Useful to

ManySolutions,OneGoal.

WMI

• UsedbyaSplunksystemtocollectWindowsEventsfromaremotesystem• Pros• Remote,noagent

• Cons• Slow• Alotofoverhead• Limitedcollectionavailability(mayneedmultiplesystemstopullallofyourWindowshosts)• Lowfidelity• Dealingwithpermissions

Page 9: Splunk and Windows Event Log: Best Practices, Reduction ... · •Current_onlytells Splunk to only grab the latest events (like tail –f, if Windows had such a thing) •Useful to

ManySolutions,OneGoal.

EVTXImport

• Canbeusedtoexporteventlogsfromasystemandthenimporttherawfilesonanothersystem• Oftenseenin”air-gapped”environments• Pros• Nonetworkconnectionneededfromtheclientsystemstothetargetindexers

• Cons• Lowfidelity(rememberthat“interpretation”thingearlier?)• Movingandremovingthefilesisamanualprocess• Opentoeventduplication

Page 10: Splunk and Windows Event Log: Best Practices, Reduction ... · •Current_onlytells Splunk to only grab the latest events (like tail –f, if Windows had such a thing) •Useful to

ManySolutions,OneGoal.

ThirdPartySyslogAgent(Snare)

• It’sathing,theseagentsexist• Pros• Canworkwithyourexistingsysloginfrastructure

• Cons• Superlowfidelity• Unreliable(syslogneverdies)• Remoteconfiguration?

Page 11: Splunk and Windows Event Log: Best Practices, Reduction ... · •Current_onlytells Splunk to only grab the latest events (like tail –f, if Windows had such a thing) •Useful to

ManySolutions,OneGoal.

TURNDOWNTHEVOLUME:Licensereductiontips

Page 12: Splunk and Windows Event Log: Best Practices, Reduction ... · •Current_onlytells Splunk to only grab the latest events (like tail –f, if Windows had such a thing) •Useful to

ManySolutions,OneGoal.

• Splunkestimatesbetween200-300mbperday,persystem• Ofcourse,thatcanvarywildly• Lotsofrepeatedeventswithlittletonovalue(lookingatyou4662)• Dowereallyneedallofthese?• Doweneedeverypartofallofthese?

Thesethingsarechatty

Page 13: Splunk and Windows Event Log: Best Practices, Reduction ... · •Current_onlytells Splunk to only grab the latest events (like tail –f, if Windows had such a thing) •Useful to

ManySolutions,OneGoal.

Stratergery

• Pickyoursystemscarefully• Pickyourinputscarefullyonthosesystems• WhitelistandBlacklistcarefully• Resolvingobjects• Baseline?• Current_only?Start_from?• XmlWinEventLog• Filteringandcleaningup

Page 14: Splunk and Windows Event Log: Best Practices, Reduction ... · •Current_onlytells Splunk to only grab the latest events (like tail –f, if Windows had such a thing) •Useful to

ManySolutions,OneGoal.

Whichsystems?

• JustActiveDirectoryservers?• Endpoints?• Servers?• Sorry,thisisonacasebycasebasis

Page 15: Splunk and Windows Event Log: Best Practices, Reduction ... · •Current_onlytells Splunk to only grab the latest events (like tail –f, if Windows had such a thing) •Useful to

ManySolutions,OneGoal.

Pickingyourinputs(notyournose)

• SetabaselineforwhichlogsALLofyoursystemsshouldbesending• Forothereventlogs,useanindividualappforturningonthatinput(DS-Input-wineventlog_application)• Doyouneedadmon fromallofyoursystems?Probablynot,justonafewADsystems• Makesureyouaren’tusinglegacyinputs(WMIvsPerfmon)• LookoutforWindowsFirewallEvents(maybeStreaminstead?)

Page 16: Splunk and Windows Event Log: Best Practices, Reduction ... · •Current_onlytells Splunk to only grab the latest events (like tail –f, if Windows had such a thing) •Useful to

ManySolutions,OneGoal.

WhitelistingandBlacklisting

• Canhaveabigimpactonyourlicenseusage• Investingthetimein“whichevents”canpayoffbig• Carefulwithawhitelist-onlyapproach• Notethatthereisalimittothenumberoflists• Performedattheforwarder,sodoesnotusenetworktraffic

Page 17: Splunk and Windows Event Log: Best Practices, Reduction ... · •Current_onlytells Splunk to only grab the latest events (like tail –f, if Windows had such a thing) •Useful to

ManySolutions,OneGoal.

Someniceblacklistoptionstostartwith

• https://gist.github.com/automine/a3915d5238e2967c8d44b0ebcfb66147

Page 18: Splunk and Windows Event Log: Best Practices, Reduction ... · •Current_onlytells Splunk to only grab the latest events (like tail –f, if Windows had such a thing) •Useful to

ManySolutions,OneGoal.

ADObjectResolution

• ResolvesthingslikeSIDsandGIUDs• YoucantellSplunkwhichDCstousetoresolvethese• Canaddsomeoverhead(CPUandMemory),butusuallylowimpact• Recommendationistoresolvethem(lookattheevt_*)optionsininputs.conf forWindowsEventLogs

Page 19: Splunk and Windows Event Log: Best Practices, Reduction ... · •Current_onlytells Splunk to only grab the latest events (like tail –f, if Windows had such a thing) •Useful to

ManySolutions,OneGoal.

BaseliningAD

• WillcollectyourwholeADschema• CantakeupalotofmemoryonADcontrollers• ButbaseliningisusefulforAssetsandIdentitiesinES• Sobecarefulwhichsystemsyoubaselineon

Page 20: Splunk and Windows Event Log: Best Practices, Reduction ... · •Current_onlytells Splunk to only grab the latest events (like tail –f, if Windows had such a thing) •Useful to

ManySolutions,OneGoal.

Current_only vs.start_from

• Current_only tellsSplunktoonlygrabthelatestevents(liketail–f,ifWindowshadsuchathing)• Usefultomakesureyoudon’tgetallofthehistoricaldata• Maywanttosetthatto“true”oninitialdeployment• Thensetto“false”,restart,anditshouldpickupfromthecheckpoint• Start_from shouldbe“oldest”• Settingitto“newest”canbeusedtogrababacklogofevents• I’veneverseenthisinthewild

Page 21: Splunk and Windows Event Log: Best Practices, Reduction ... · •Current_onlytells Splunk to only grab the latest events (like tail –f, if Windows had such a thing) •Useful to

ManySolutions,OneGoal.

XmlWinEventLog

• Shouldreducelicenseusage(claimsareupto70%)• ItwillalwaysbeinEnglish(pro?Con?)• Hardertoread,Imean,it’sXML• QualityofCIMcompliancehasbeenvariedinthepast• Itdoesn’t”looklikeWindowsevents”andsomeauditorsarenotbright• Whatifyoucouldgetthesamelogsavingsandthereadability

Page 22: Splunk and Windows Event Log: Best Practices, Reduction ... · •Current_onlytells Splunk to only grab the latest events (like tail –f, if Windows had such a thing) •Useful to

ManySolutions,OneGoal.

Filteringandcleaningup

• Don’tuse“suppress_text”• It’stempting,buttheregoesthebabywiththebathwater• Maybejustcleanupthetextyoudon’tneed

Page 23: Splunk and Windows Event Log: Best Practices, Reduction ... · •Current_onlytells Splunk to only grab the latest events (like tail –f, if Windows had such a thing) •Useful to

ManySolutions,OneGoal.

Filteringandcleaningup

• IPv6supportineventlogsresultsinalotof“::”and“ffff”andothergarbage• Let’scleanupalot(thankstoalotofpeopleforthis)

• https://gist.github.com/automine/5c8ef5b50e1df38249dfba01a70f2875

Page 24: Splunk and Windows Event Log: Best Practices, Reduction ... · •Current_onlytells Splunk to only grab the latest events (like tail –f, if Windows had such a thing) •Useful to

ManySolutions,OneGoal.

MakingThemMoreUseful

Page 25: Splunk and Windows Event Log: Best Practices, Reduction ... · •Current_onlytells Splunk to only grab the latest events (like tail –f, if Windows had such a thing) •Useful to

ManySolutions,OneGoal.

Sorry,Iranoutoftime

• GotES?TakealookatRyanFaircloth’sSecKit work• https://splunkbase.splunk.com/app/3059/• https://bitbucket.org/SPLServices/seckit_sa_idm_windows

• AlternativeTAs• ShouldhelpwithKOoverhead• https://github.com/my2ndhead/TA-microsoft-windows (candoXMLevents)• https://bitbucket.org/SPLServices/seckit_ta_microsoft_windows (forusewithSecKit)


Splunk Developer & Admin Certification Training...This Splunk course also includes various topics of Splunk, such as installation and configuration, Splunk Syslog, Syslog Server, log

Splunk Developer & Admin Certification Training...This Splunk course also includes various topics of Splunk, such as installation and configuration, Splunk Syslog, Syslog Server, log

Documents
Netfilter Iptables for Splunk Documentation - Read … · Netfilter Iptables for Splunk Documentation, Release 0 Splunk Answers Splunk has a strong community of active users and

Netfilter Iptables for Splunk Documentation - Read … · Netfilter Iptables for Splunk Documentation, Release 0 Splunk Answers Splunk has a strong community of active users and

Documents
Rivium Splunk Windows · o Splunk Enterprise Security * o uberAgent* o Splunk App for Web Analytics Common Splunk Apps & Add-ons 15 SplunkingWIndows o Splunk Add-on for Microsoft

Rivium Splunk Windows · o Splunk Enterprise Security * o uberAgent* o Splunk App for Web Analytics Common Splunk Apps & Add-ons 15 SplunkingWIndows o Splunk Add-on for Microsoft

Documents
Splunk user group - automating Splunk with Ansible

Splunk user group - automating Splunk with Ansible

Technology
Introduction to Splunk · Splunk “From bits to information” ... AWS Cloudtrail. Amazon S3 Azure Windows Azure Cloud amazon webservices Weblogic. WebSphere. Tomcat Application

Introduction to Splunk · Splunk “From bits to information” ... AWS Cloudtrail. Amazon S3 Azure Windows Azure Cloud amazon webservices Weblogic. WebSphere. Tomcat Application

Documents
Program Overview - Splunk...mission-critical services. None. Splunk Enterprise System Administration Splunk Enterprise Data Administration Splunk Cloud Administration Implementing

Program Overview - Splunk...mission-critical services. None. Splunk Enterprise System Administration Splunk Enterprise Data Administration Splunk Cloud Administration Implementing

Documents
Splunk 教學 - Splunk Documentationdocs.splunk.com/images/d/db/Tutorial_zh_TW_502.pdf · 簡介 歡迎使用 Splunk 教學！ Splunk 是什麼？ Splunk 是一個可從構成您

Splunk 教學 - Splunk Documentationdocs.splunk.com/images/d/db/Tutorial_zh_TW_502.pdf · 簡介 歡迎使用 Splunk 教學！ Splunk 是什麼？ Splunk 是一個可從構成您

Documents
データ分析プラットフォーム「Splunk」eww.gp-club.panasonic.co.jp/data/splunk/splunk.pdfTitle データ分析プラットフォーム「Splunk」 Author パナソニック

データ分析プラットフォーム「Splunk」eww.gp-club.panasonic.co.jp/data/splunk/splunk.pdfTitle データ分析プラットフォーム「Splunk」 Author パナソニック

Documents
FireEye Splunk InterFireEye + Splunk: Intermediate Guidemediate Guide

FireEye Splunk InterFireEye + Splunk: Intermediate Guidemediate Guide

Documents
Admission Control using Splunk Enterprise Deployment Guide€¦ · Configuring Splunk Enterprise ... For more information on Splunk configuration, see Splunk documentation. Troubleshooting

Admission Control using Splunk Enterprise Deployment Guide€¦ · Configuring Splunk Enterprise ... For more information on Splunk configuration, see Splunk documentation. Troubleshooting

Documents
Draw Toolbar plus the standard screen grab facility ...jbelisle/cs1300/09Tutorials... · Draw Toolbar plus the standard screen grab facility available in Windows. 1 Pages Exercises

Draw Toolbar plus the standard screen grab facility ...jbelisle/cs1300/09Tutorials... · Draw Toolbar plus the standard screen grab facility available in Windows. 1 Pages Exercises

Documents
Splunk For IT Operational Intelligenceuploads.integrity360.com/1425903243-Splunk-for-IT-Operations.pdf · • Splunk Apps –Splunk App for VMware –Splunk Apps for Citrix & Hyper

Splunk For IT Operational Intelligenceuploads.integrity360.com/1425903243-Splunk-for-IT-Operations.pdf · • Splunk Apps –Splunk App for VMware –Splunk Apps for Citrix & Hyper

Documents
Splunk のご紹介 - hitachi.co.jp · Splunkの歴史. 5 . 2006 . Splunk 1 . Splunk 2 . 2007 . Splunk 3 . 2009 . Splunk 4 . 2010 . Splunk 4.1 . 2011 Splunk 4.2 . App for Enterprise

Splunk のご紹介 - hitachi.co.jp · Splunkの歴史. 5 . 2006 . Splunk 1 . Splunk 2 . 2007 . Splunk 3 . 2009 . Splunk 4 . 2010 . Splunk 4.1 . 2011 Splunk 4.2 . App for Enterprise

Documents
Splunk Spark Integration - GitHub Pageslitaotao.github.io/files/4. splunk_spark.pdf · Splunk Spark Integration ... • Splunk"products:""! Splunk"Enterprise"! ... splunk_spark Created

Splunk Spark Integration - GitHub Pageslitaotao.github.io/files/4. splunk_spark.pdf · Splunk Spark Integration ... • Splunk"products:""! Splunk"Enterprise"! ... splunk_spark Created

Documents
Splunk conf2014 - Splunk for Data Science

Splunk conf2014 - Splunk for Data Science

Technology
Splunk App: Windows Security Operation Center

Splunk App: Windows Security Operation Center

Documents
Industrial Data Internet of Things splunk (REST API, SDKs) splunk . TESCO splunk 500 Y—Jt1-3— 41328 . SHANON 1/5 splunk . htt . (500MB/B) s lunk splunk . Created Date:

Industrial Data Internet of Things splunk (REST API, SDKs) splunk . TESCO splunk 500 Y—Jt1-3— 41328 . SHANON 1/5 splunk . htt . (500MB/B) s lunk splunk . Created Date:

Documents
Splunk App: Windows Security Operation Center · 41 toolsmith: Windows Security Operation Center | Russ McRee Process Tracking is also a great view to monitor on critical servers

Splunk App: Windows Security Operation Center · 41 toolsmith: Windows Security Operation Center | Russ McRee Process Tracking is also a great view to monitor on critical servers

Documents
Splunk Cloud の製品概要 › ... › product-briefs › splunk-cloud.pdfSplunk App で機能を強化 Splunk Cloud は、Splunk Enterprise Security や Splunk IT Service Intelligence

Splunk Cloud の製品概要 › ... › product-briefs › splunk-cloud.pdfSplunk App で機能を強化 Splunk Cloud は、Splunk Enterprise Security や Splunk IT Service Intelligence

Documents
Splunk conf2014 - Splunk Monitoring - New Native Tools for Monitoring your Splunk Deployment

Splunk conf2014 - Splunk Monitoring - New Native Tools for Monitoring your Splunk Deployment

Technology
DATE; September l, 1991 LOCATION AND ACCESS; SPECIFIC ...€¦ · 1-33 1A 2A 3A 4A 5A 6A 7A B H Grab Grab Grab Grab Grab Grab Grab Grab Grab Grab Grab Grab See attached 'Government

DATE; September l, 1991 LOCATION AND ACCESS; SPECIFIC ...€¦ · 1-33 1A 2A 3A 4A 5A 6A 7A B H Grab Grab Grab Grab Grab Grab Grab Grab Grab Grab Grab Grab See attached 'Government

Documents
Referent / Redner Benjamin Tiggemann · 2015-07-08 · Splunk for Exchange Splunk for Enterprise Security* Splunk for Vmware Splunk for PCI Compliance* Splunk for Windows Splunk for

Referent / Redner Benjamin Tiggemann · 2015-07-08 · Splunk for Exchange Splunk for Enterprise Security* Splunk for Vmware Splunk for PCI Compliance* Splunk for Windows Splunk for

Documents
Splunk Conf2010: Corporate Express presents Splunk with SAP

Splunk Conf2010: Corporate Express presents Splunk with SAP

Technology
データ分析プラットフォーム「Splunk」 - Panasonicdl.it-sol.jpn.panasonic.com/data/splunk/splunk.pdfTitle データ分析プラットフォーム「Splunk」 Author パナソニック

データ分析プラットフォーム「Splunk」 - Panasonicdl.it-sol.jpn.panasonic.com/data/splunk/splunk.pdfTitle データ分析プラットフォーム「Splunk」 Author パナソニック

Documents
Agency Chargeback Models to Enable Enterprise Splunk ......Workshops: Get Splunk Hands -on Experience Attend a Splunk Workshop Upcoming Schedule December 1: Introduction to Splunk

Agency Chargeback Models to Enable Enterprise Splunk ......Workshops: Get Splunk Hands -on Experience Attend a Splunk Workshop Upcoming Schedule December 1: Introduction to Splunk

Documents
SecKit IDM Windows Documentation•SecKit IDM Common >=3.0.0 •Splunk TA Windows >=5.0.1 3.2Installation Installation of the apps is intended to be minimally impactful to a Splunk

SecKit IDM Windows Documentation•SecKit IDM Common >=3.0.0 •Splunk TA Windows >=5.0.1 3.2Installation Installation of the apps is intended to be minimally impactful to a Splunk

Documents
Time*ACer*Time*–** Comparing*Time*Ranges*in*Splunk* · AboutMe*! Splunk*Senior*Instructor*since*2009*! Frequentcontributor*to*Splunk*Answers*! Love*Splunk*search*language*puzzles*

Time*ACer*Time*–** Comparing*Time*Ranges*in*Splunk* · AboutMe*! Splunk*Senior*Instructor*since*2009*! Frequentcontributor*to*Splunk*Answers*! Love*Splunk*search*language*puzzles*

Documents
SplunkingYour Mobile Apps - .conf19 | Splunk · Introducing Splunk for Mobile Intelligence (Splunk MINT) Deploying Splunk MINT – Intro to SDKs – Getting Started Using Splunk MINT

SplunkingYour Mobile Apps - .conf19 | Splunk · Introducing Splunk for Mobile Intelligence (Splunk MINT) Deploying Splunk MINT – Intro to SDKs – Getting Started Using Splunk MINT

Documents
ReversingLabs Explainable Threat Intelligence Enriches ... Data Sheets/RL-Splunk...sent to Splunk. • In Splunk, the TitaniumScale report is correlated with other available Splunk

ReversingLabs Explainable Threat Intelligence Enriches ... Data Sheets/RL-Splunk...sent to Splunk. • In Splunk, the TitaniumScale report is correlated with other available Splunk

Documents
Splunk Enterprise 6.5.0 Splunk Enterprise 보안

Splunk Enterprise 6.5.0 Splunk Enterprise 보안

Documents