Upload
doantruc
View
249
Download
0
Embed Size (px)
Citation preview
Copyright © 2014 Splunk Inc.
David Veuve SE, Splunk
Security Ninjutsu Using Splunk for CorrelaEon, Anomaly DetecEon and Response AutomaEon
Who Am I?
2
! David Veuve – Sales Engineer for Major Accounts in Northern California
! [email protected] ! Former Splunk Customer (For 3 years, 3.x through 4.3) ! Security Guy ! Primary author of Splunk Search Usage app ! Primary area of Splunk ExperEse: Search Language ! Stands on the shoulders of giants
Disclaimer
3
During the course of this presentaEon, we may make forward looking statements regarding future events or the expected performance of the company. We cauEon you that such statements reflect our current expectaEons and
esEmates based on factors currently known to us and that actual events or results could differ materially. For important factors that may cause actual results to differ from those contained in our forward-‐looking statements,
please review our filings with the SEC. The forward-‐looking statements made in the this presentaEon are being made as of the Eme and date of its live presentaEon. If reviewed a^er its live presentaEon, this presentaEon may not contain current or accurate informaEon. We do not assume any obligaEon to update any forward looking statements we may make. In addiEon, any informaEon about our roadmap outlines our general product direcEon and is subject to change at any Eme without noEce. It is for informaEonal purposes only and shall not, be incorporated into any contract or other commitment. Splunk undertakes no obligaEon either to develop the features or funcEonality described or to
include any such feature or funcEonality in a future release.
Agenda • Visibility – Analysis – AcEon in Four Scenarios 1. Threat List IntegraEon leads to Firewall Blocks 2. Anomaly DetecEon leads to Opening a Ticket 3. Behavioral Profiling leads to Manager ConfirmaEon 4. Visual CorrelaEon of Security Indicators
4
Being Covered 1. Tools and Searches and Demos 2. All of these examples and concepts come from actual customer requirements and actual customer deployments. No smoke and mirrors.
3. Github with data gens and accoutrement at end of presentaEon
5
Who Are You? 1. Security Engineer / SOC Analyst / Threat Analyst / Someone Technical Who Cares about Security
2. Splunk skill level is basic-‐advanced 3. No Enterprise Security required (though it can make things easier at scale)
6
Visibility – Analysis – AcEon • Framework for evaluaEng data and responding Splunk • Applies to all exisEng frameworks, as it’s the Splunk side of the loop. • For example, Let’s look at the lateral movement secEon of the kill chain. (Not familiar with the kill chain? It’s a great way to understand the phases of an agack. Check the URL below.)
• Visibility: What data will let you detect Lateral Movement? • Analysis: What will you do to that data to come to a decision? • Ac2on: What will you do in response to that decision?
– Can we automate all of this? • Kill Chain: hgp://www.lockheedmarEn.com/content/dam/lockheed/data/corporate/documents/LM-‐White-‐Paper-‐Intel-‐Driven-‐Defense.pdf
7
Scenario One
C&C DetecEon and Blocking
Command and Control DetecEon and Blocking • New threat list intel (or any other source of detecEng agackers) has become available, and we are trying to block any outbound Command and Control.
• The formal firewall policy can’t be pushed except every Wed night and Sunday night – not fast enough.
• Goal: Take in the firewall logs, leverage our available intelligence to detect C&C behavior, and then block the desEnaEons, all in near realEme.
• Visibility: Firewall Logs, Threat Intel Sources • Analysis: IntersecEon (lookup) of the two • Ac2on: Apply dynamic firewall blocks
9
What / Where is Threat Intelligence
10
! A feed of known bad IPs/DNS Names/MD5s/URLs/etc from a vendor or non-‐profit that specializes in discovering Indicators of Compromise.
! Great sources of Open Source Threat Intel include: – Emerging Threats: hgp://rules.emergingthreats.net/ – I-‐Blocklist: hgps://www.iblocklist.com/lists.php – MalwareDomains: hgp://www.malwaredomains.com/ – Zeus Tracker: hgps://zeustracker.abuse.ch/
! Many great commercial enEEes too (generally beger ranking / quality): – Norse (Splunk Partner), iSight Partners, Verizon iDefense, Commercial
Versions of most of the above, and many many more
Visibility Palo Alto Networks Firewall Log
Sep 15 19:02:06 1,2014/09/15 19:02:06,0004C104559,TRAFFIC,end,1,2014/09/15 19:02:05,10.2.2.14,206.16.215.101,206.16.216.158,214.34.245.101,Internet Traffic,,, salesforce-‐base,vsys1,Trust,Untrust,ethernet1/8,ethernet1/2,MyLogForwarding,2014/09/15 19:02:05,24238,1,61845,443,57339,443,0x400000,tcp,allow,1275,761,514,14,2014/09/15 19:01:31,5,any,0,358477769,0x0, 10.0.0.0-‐10.255.255.255, United States,0,8,6
11
ConnecEon End Date
Src and Dest IPs Firewall Rule
ApplicaEon To/From Zone Dest Port
Threat Intel Lookup: bad_ip,threat_intel_source 115.29.46.99/32,zeus_c2s 61.155.30.0/24,cymru_hgp
Analysis
• First, we want to pull out all firewall traffic coming from inside our network, going outside our network.
• Then, we want to cross-‐reference that data with our Threat Intel list. This is accomplished in the Splunk world via a lookup.
• Finally, we want to pull just the logs that have Threat Intel
12
index=pan_logs sourcetype=pan_traffic src=“10.*” dest!=“10.*” | lookup ThreatIntel dest | search ThreatList=*
Name of our lookup, and the key field
Name of our lookup, and the key field
Data held in Lookup Table
Analysis -‐ Challenges
13
! Performance – you get lots of traffic, maybe you have lots of threat intel entries. – SoluEon: Enterprise Security is built to solve this problem at scale. – Alternate SoluEon: data models help substanEally with the first half. You
can fragment the lookups if you get to very high numbers. ! MulEple Threat Lists – DeprioriEze Open source threat list vs Premium threat list – SoluEon: Enterprise Security has this fixed as well with deduping and
prioriEzing – Alternate SoluEon: | inputlookup Premium| append [|inputlookup
OpenSource] | munge | outputlookup MyList
Analysis – Value Adds
14
! Strength of AutomaEon in Splunk is high fidelity alerts. ! This was a simple example, but you could also make it more impressive by tracking whether the IP is in the US:
! AlternaEvely, you could look to see whether that parEcular host had a recent malware event:
| join host [| `tstats` count from datamodel=Malware by Malware_Agacks.dest | stats count by Malware_Agacks.dest | rename Malware_Agacks.dest as host]
AcEon • PANBlock! (Or other Network Response, see below) • Challenges:
– Many organizaEons fear automaEc response due to potenEal for downEme ê SoluEon: Start with high confidence alerts and limited list of assets, verify success.
ê Alternate SoluEon: Don’t go automaEc response. This works through the UI too.
– You don’t run Palo Alto Networks ê SoluEon: While PAN/Splunk have made this work out of the box, this has been implemented many Emes with a number of products, Incl but not limited to: – Cisco Border Router: Expect Script to block – Check Point: R80 Rest Interface (Talk to me if you want to do this, I want in)
15
AcEon – Example Customer Workflow
16
Demo – Palo Alto Logs
17
Demo – Threat Lookup
18
Demo – Threat Lookup – Table View
19
Demo – Add panblock
20
Where to Learn More About PAN Blocking
21
! Have a Palo Alto device and like this parEcular feature? Visit – Docs: hgps://live.paloaltonetworks.com/docs/DOC-‐6593 – App Page: hgp://apps.splunk.com/app/491/
! Or beger yet, go see those talks: – AutomaEc Malware DetecEon, Analysis and MiEgaEon in Splunk
Jose Hernandez, SoluEons Security Architect, Splunk You just missed it! Get the PDF and watch the video later
– MiEgaEng Cybersecurity Risk with Palo Alto Networks and Splunk Marc Benoit, Sr. Director, Palo Alto Networks Breakout Session: 10/09/2014, 2:15-‐3:15
Scenario Two
Anomaly DetecEon EssenEals
Anomaly DetecEon EssenEals • File audiEng is a common pracEce, and it can be accomplished quickly and easily in Splunk.
• It becomes harder at scale, but data model acceleraEon helps. • UlEmately, by conquering anomaly detecEon, you can more effecEvely find the difficult to detect in your systems.
• Visibility: Carbon Black Logs • Analysis: System DistribuEon, accelerated via Data Models • Ac2on: Security Incident CreaEon
23
What is Standard DeviaEon?
24
! A measure of the variance for a series of numbers. ! One file is opened on 100, 123, 79, and 145 hosts per day – average of 111.75 and a standard deviaEon of 28.53.
! Another file is opened on 100, 342, 3 and 2 hosts per day – average of 111.75, but a stdev of 160.23.
Visibility – Log Examples
25
{"acEon": "write", "Emestamp": 1410911994, "path": "c:\\Program Files\\Splunk\\bin\\splunk-‐perfmon.exe", "type": "filemod", "process_guid": 36661217281}
How To Accelerate
26
• AcceleraEon facilitates beger and broader analysis. • Splunk has a few ways of acceleraEng content: • Report AcceleraEon • Data Model AcceleraEon • TSCollect • Summary Indexing • Pre-‐processing of logs
• Check out Gerald Kanapathy’s Session on Friday: Title: Splunk Search AcceleraEon Technologies Speaker: Gerald Kanapathy, Sr. Director Product Management, Splunk When: 10/09/2014, 10:30 AM – 11:30 AM
Analysis – Create Data Model
27
Create a data model and accelerate
Analysis – Create Pivot Search
28
• Create a baseline pivot search and Open in Search. • In this case, split dc(host) by path • Add a filter for criEcal paths
Analysis – Create AddiEonal StaEsEcs
29
Add addiEonal stats command on top of accelerated Pivot search.
Analysis – Only Show Suspect Entries
30
AcEon – Create a New Incident
31
! Will work with essenEally any EckeEng system, maybe via a scripted alert. – Every TickeEng System Accepts Emails too!
! Known to work with: – Remedy: hgp://wiki.splunk.com/Community:Use_Splunk_alerts_with_scripts_to_create_a_Ecket_in_your_EckeEng_system – ServiceNow: hgp://answers.splunk.com/answers/47086/service-‐now-‐Ecket-‐generaEon-‐via-‐splunk-‐alerts.html – PagerDuty: hgp://www.pagerduty.com/docs/guides/splunk-‐integraEon-‐guide/ – ArcSight: hgps://apps.splunk.com/app/1847/ – Q1 – NetCool – Anything AccepEng Email – Anything Scriptable: hgp://docs.splunk.com/DocumentaEon/Splunk/6.1.3/alert/ConfiguringScriptedAlerts
Demo – ModificaEons of Exec Files in System32
32
Scenario Three
Behavioral Anomaly DetecEon
Behavioral Anomaly DetecEon • DetecEng known bad is great, but leaves you vulnerable. • Augment with syntheEc checks of sensiEve systems. • StaEsEcs can consume all your Eme
– Generally easiest to leverage so^ approval (e.g., emails to managers) with standard deviaEon.
– AddiEonally, use hard enforcement for large deviaEon (e.g., FW isolaEon)
• In this scenario, we are a hospital tracking paEent chart opens. • Visibility: CharEng System Logs • Analysis: Frequency Analysis by User, Role, etc. • Ac2on: Email the employees’ manager to invesEgate
34
What is Standard DeviaEon?
35
! A measure of the variance for a series of numbers. In this case, let’s say chart opens.
! Over a few days, Jane opens 100, 123, 79, and 145 charts per day with an average of 111.75 and a standard deviaEon of 28.53.
! Over the same period, Jack opens 100, 342, 3 and 2 charts per day, also with an average of 111.75, but a stdev of 160.23.
! When Jack and Jane both open 500 records some day, that will be 13.6 standard deviaEons (z=13.6) for Jane but only 2.42 for Jack.
! Z score = number of standard devia2ons away from average
Visibility – Log Examples <audit_list><audit_version>1</audit_version> <event_dt_tm>2014-‐09-‐06 23:59:59.52</event_dt_tm> <outcome_ind>0</outcome_ind> <user_name>AHARVEY</user_name> <prsnl_id>117499</prsnl_id> <prsnl_name>Angel Harvey</prsnl_name> <role>DBA</role> <role_cd>24209801</role_cd><enterprise_site>HNAM</enterprise_site><audit_source>Test/Domain</audit_source><audit_source_type>600005</audit_source_type><network_acc_type>1</network_acc_type><network_acc_id>MTYVQ-‐ACTX03</network_acc_id><applicaEon>HNA: Powerchart</applicaEon><task>RUN PowerView Preferences</task><request>cps_ens_ppa</request><appl_ctx>346793285</appl_ctx><perform_cnt>69</perform_cnt><event_list><event_name>Maintain Person</
event_name> <event_type>Chart Access Log</event_type> […….]</audit_list>
36
Analysis • Core Metric: Chart Opens Per Day, Per Employee • Dimensions to Compare:
– Over Eme for the same user, others with same Etle – Others with the same Etle in the same city or with the same years of experience
• Why MulEple Dimensions? 1. Comparing mulEple metrics reduces false posiEves. 2. Provides more context. 3. If I open 25 Emes as many charts, but so does every other nurse in my facility
because we’re under inspecEon, that should be evident.
• What about performance? – Good point! Data Models turn this into a 30 seconds per 5M events search on my
laptop. Tscollect is manual but turns it into a quarter second search.
37
Analysis – Basic
38
index=cerner | eval EmployeeID=spath(_raw, "audit_list.prsnl_id") | eval EmployeeName = […] | eval RecordNum= […]
| bucket _Eme span=1d | stats dc(RecordNum) as NumRecords by EmployeeName, EmployeeID, _Eme | stats first(NumRecords) avg(NumRecords) stdev(NumRecords) by EmployeeName, EmployeeID | where ‘first(NumRecords)’ > ‘avg(NumRecords)’ + ‘stdev(NumRecords)’ * 6
! Basic Data Set ! Field Munging ! Pull the number of stats per
employee, per day ! Pull the average, standard
deviaEon, and most recent daily number per employee
! Find instances where the most recent number is more than 6 standard deviaEons away from the average
Demo
39
40 minutes later…
How To Accelerate
40
• AcceleraEon facilitates beger and broader analysis. • Splunk has a few ways of acceleraEng content: • Report AcceleraEon • Data Model AcceleraEon • TSCollect • Summary Indexing • Pre-‐processing of logs
• Check out Gerald Kanapathy’s Session on Friday: Title: Splunk Search AcceleraEon Technologies Speaker: Gerald Kanapathy, Sr. Director Product Management, Splunk When: 10/09/2014, 10:30 AM – 11:30 AM
Analysis – AcceleraEon
41
index=cerner | eval Role=spath(_raw, "audit_list.role") | eval RoleID = […] | eval EmployeeID= […] | eval EmployeeName = […] | eval PaEentNum= […]
| bucket _Eme span=1d | stats dc(PaEentNum) as NumRecords by EmployeeName, EmployeeID, Role, RoleID _Eme
| lookup HR_IS.csv EmployeeID
| tscollect retain_events=t Cerner
! Basic Data Set ! Field Munging
! Stats split by as many dimensions as required, but not more.
! Lookup occurs a^er stats
! Store the results in a local tsidx (could also do this with datamodels)
Analysis – Find StaEsEcal Outliers Pt 1
42
| tstats local=t first(NumCharts) as Recent_NumCharts avg(NumCharts) as Avg_NumCharts stdev(NumCharts) as Stdev_NumCharts from Cerner groupby EmployeeName, EmployeeID, Username, Role, RoleID, City, YearsAtCompany
| join type=outer RoleID [| tstats local=t avg(NumCharts) as Role_Avg_NumCharts stdev(NumCharts) as Role_Stdev_NumCharts from Cerner groupby Role, RoleID ]
! How many charts is typical (and what is the standard deviaEon) for this person. Also, how many did they open yesterday?
! How many chart opens is standard for people in this role?
Analysis – Find StaEsEcal Outliers Pt 2
43
[… conEnued from previous slide …] | eval Personal_Z = abs(Recent_NumCharts-‐Avg_NumCharts)/Stdev_NumCharts | eval Role_Z = abs(Recent_NumCharts-‐Role_Avg_NumCharts)/Role_Stdev_NumCharts | eval Z_Min = min(Role_Z, Personal_Z) | where Z_Min > 6
! How unusual is this acEvity, for this person or versus others in this role? – Z score = how many StDev
away from average. – Consider other metrics, such as
years at the company, facility. – Goal is to capture normal
across dimensions, to idenEfy trends across organizaEon (e.g., a facility audit).
AcEon • Email the Manager • This opEon is mostly just forma�ng. Join to the HR / LDAP database and uElize sendemail +
map. • Could also escalate big violaEons to the SOC or GRC. | lookup LDAPSearch sAMAccountManager as username OUTPUT manager | lookup LDAPSearch dn as manager OUTPUT mail as ManagerEmail “
44
| map maxsearches=100 search=“ | stats count | eval ManagerEmail=$ManagerEmail$ | eval EmployeeName=$EmployeeName$ | eval ZAvg = $Z_Avg$ | sendemail to=ManagerEmail sendresults=f subject=EmployeeName . \“ excess Chart Opens\” message=EmployeeName . \“ has opened more charts than normal (\“ . ZAvg . \“ stdev). _._Please Follow Up.\”
Demo
45
Scenario Four
Visual Event CorrelaEon
Visual Event CorrelaEon • A^er conquering the essenEals of ge�ng some alert data, it’s important to be able to understand an agacker’s acEon plans. – Progress through kill chain – Movement toward criEcal assets – Et Cetera
• Easiest with Enterprise Security, but possible without
47
Visibility – Log Examples • Anything. This should encompass all of your log sources, correlaEon rules, alerts, and etc.
• Ideally include operaEonal data here too (e.g., website response Eme change)
48
Analysis • Examples thus far have centered around automated analysis, but Splunk is also a great tool for data visualizaEon and analysis.
• CapabiliEes here are virtually endless, but here are a few examples.
49
AcEon • Need more informaEon? Enterprise Security has many built in work flow acEons to go pull more data.
• Go pull more informaEon from your Endpoint Threat DetecEon and Response app: – Tanium: hgp://apps.splunk.com/app/1862/ – Tripwire / nCircle ip360: Ask your SE – Bit9 / Carbon Black: hgps://www.bit9.com/soluEons/splunk/ – Many Others also exist
• File a Ecket with your EckeEng – Remedy: hgp://answers.splunk.com/answers/122019
• Open a new Notable Event in ES 50
Demo – Separate Product Lines (ES)
51
Demo – Kill Chain Swimlanes (ES)
52
Demo – Visualizing By Priority
53
• While not as slick as the ES version, you can get much of the same value by leveraging mulEple reports on one dashboard, or with stacked column charts.
Security is a Team Sport
140+ security apps Splunk App for Enterprise Security
Splunk Security Intelligence Pla�orm
55
Palo Alto Networks
NetFlow Logic
FireEye
Blue Coat Proxy SG
OSSEC Cisco Security Suite
AcEve Directory
F5 Security
Juniper Sourcefire
Talk to your neighbor We’re all in this together.
56
Go Play With Data
57
Github with DataGens and searches: www.davidveuve.com/go/conf-‐security
Shameless Plug
Splunk Search Usage Splunk Search Usage and AdopEon Tracking, with security reports.
•
hgp://www.davidveuve.com/go/ssu 59
THANK YOU