Embed Size (px)
Citation preview
Copyright © 2014 Splunk Inc.
Peter Lam Security Analyst
Closed Loop DetecAon, MiAgaAon, and ConAnuous Compliance
Bio…
! Who am I? – Peter Lam – A Happy “Splunker” – InformaAon Security Staff in a naAonal bank – 20+ Years of Professional IT Experience
ê Experience in: ê System/PlaVorm Engineering, ApplicaAon Development, Security, Incident Response
2
Disclaimer
! All content in this presentaAon reflect my views exclusively and not that of my employer
3
What is The Problem?
The “What” Title: Closed-‐loop fraud detecAon and miAgaAon (automated blocking) using Splunk Abstract: Fraud has negaAve impact to any company with an online presence and Splunk can help. This session will describe this financial insAtuAon implemented an auditable methodology to defend against online fraud using a closed-‐loop detecAon, miAgaAon, validaAon, and reporAng process. The approach leverages Splunk as the "brain” and log entries from devices, transacAon monitoring and customer informaAon as the "blood cells” carrying oxygen to the brain to determine the threat and to drive acAon to block fraudulent transacAons. This session will describe the necessary data to feed into the Splunk, the consideraAons and processes to implement automated blocking and the reporAng metrics on how long it took to detect, respond and miAgate threats. All of these can be done with Splunk saved queries and some simple scripAng.
5
Challenges
! Many, many challenges faced by the financial industry – Industry regulatory – Industry operaAonal efficiency – Cyber criminal acAviAes – Market fluctuaAon risk
6
OFAC As An Example
! Important requirement – OFAC is only one of the many – Office of Foreign Assets Control
! Purpose: – Enforce sancAons per government requests
ê Foreign governments/countries, individuals, businesses, criminal groups
7
Regulatory Requirement
Operational Efficiency
Criminal Activity
OFAC
Why is OFAC Important
! Apply to ALL, not just banks ! PenalAes are substanAal
– YTD PenalAes -‐> $1.21B (as of 9/3/2014) – Average penalty per offense -‐> $70MM (as of 9/3/2014)
ê hlp://www.treasury.gov/resource-‐center/ sancAons/CivPen/Pages/civpen-‐index2.aspx
! Historical enforcements – CiA (9/2014), BB&T (8/2014),
BNP (6/2014), AIG (5/2014), Royal Bank of Scotland (12/2013)
8
Regulatory Requirement
Operational Efficiency
Criminal Activity
OFAC
Regulatory Requirement
Operational Efficiency
Criminal Activity
OFAC
OFAC Penalty
BNP got fined for ~$1B
9
Boss Wants
! “Right Thing” – more than check the box – Add value to the banking products
ê Security posture as a business alribute ! IT Security Group’s Goal
– Compliance – ConAnuous alestaAon and verificaAon – Cost effecAve
! Decision: – No sancAoned enAty is allowed
10
Goals… OFAC (II)
11
Criminal Ac9vity
Opera9onal Efficiency
Regula9ons
OFAC To opera9onalize • Minimize total cycle Ame • Create repeatable pracAce for other risk
To report & assure • Report as suspicious acAvity • Provide evidence of effecAve control
To detect & prevent • Detect acAvity from sancAoned countries / enAAes
• Deny access
High Level “Want” ! Automated 24x7 detect and prevent response with 8x5 manual follow-‐up (detect & prevent, operaAonalize)
! No/Low false posiAve (detect) ! Minimize total detect → act → report cycle Ame (operaAonalize)
! Incident report (internal & external) (report) ! Control proof (assure) ! Repeatable pracAce for other types of incidents (operaAonalize)
! Target maturity level >= managed → CMM4 (assure)
12
Requirements ! Business Support
– Fail to comply to costly – high stakes ê Brand damage ê Red flag for regulators ê Profit eliminaAon due to hevy penalty
– When stakes are high, block first, ask quesAon later/second ê Purely business decision ê Risk, benefit, loss comparison
! Data sources (logs) to complete the picture into Splunk ! IdenAfy enforcement points – where to control
– Firewall – Proxy – AddiAonal enforcement – depends on the threat, it can talk to other control points
ê Switch port can be used to stop a virus outbreak type of threat
13
Methodology
Define What, how, when?
Detect Saved searches
Protect AcAon scripts
Assure Assurance for management and proof for regulator
Report Deep-‐dive and regulatory report
Value
14
Steps to Success
15
01 Logs from various sources. e.g FireWall (FW), Operating System (OS)
02 External threat intelligence, e.g. Threat actor IP, Malicious account number
03 Frequency & threshold (Security Requirement)
04 Internal Observation, E.g. Newly identified vulnerabilities, detection
Saved Search
>
Define
! The “What” – External Intelligence
ê Finger print ê Indicators ê Threat actors
– Internal intelligence ê “Secret Sauce”
! The “When” – Frequency/occurrence
16
16
16
Define
01 02 03 04
Detect
! Collects banking web applicaAon logs Into Splunk – Match criteria
ê SensiAve URLs – Ex – wire transfer, TX type , and amount
ê Countries – Ex – blacklisted country
ê Account numbers – Ex – account numbers known to be bad from other InsAtuAons
– Historical acAviAes ê Cross check with previous history ê New account – higher fraud rate
17
Saved Search Example Logic of the find – blacklisted country, touching high risk transacAon URLs,
MaxMind is a vendor that offers geoip
! Using MaxMind as an example – Index=APACHE_LOG access_request=“SENSITIVE_TRANSACTION” |
lookup geoip client as IP | where client_country=“EVIL COUNTRY”| table IP | collect index=SUMMARY ê Save Search as “BLOCK_OFAC”
! MaxMind app located at – hlps://apps.splunk.com/app/291/
18
Protect
! Detect -‐> script in saved search, Splunk iniAates script configured on the the saved search page
! This script can be anything, but in our case, it will be a dynamic firewall block commands against the evil – Firewall (FW) must support dynamic control (check with vendor)
! hlp://docs.splunk.com/DocumentaAon/Splunk/6.0/Alert/Configuringscriptedalerts
19
Protect–Process
20
> firewall
script
Detected “Bad”
Saved search result usually in the form of IPs, or URL, etc.
Script parses the detected result, which Is located at ARG#8, and executes down-‐stream “enforcement” instrucAons (e.g. block commands at Firewall).
Protect Example A sample Python setup #!/usr/bin/env python
f = gzip.open(sys.argv[8], "rb")
l=f.readline()
For l in f:
x = l.strip('\n')
y = x.replace('\"','')
EVIL_IP = y.strip(',')
subprocess.Popen(['ssh','-‐i',’SSH_KEY',’SSH_USER',”FW_COMMAND”,EVIL_IP])
f.close()
21
Stripes out forma�ng characters
Expands the file located at sys.argv[8], reads the content 1
line at a Ame
Actual FW command using
SSH
Overall Picture (Define, Detect & Protect)
22
ApplicaAon
“Run A Script” opAon on saved search page
>
Saved Search
Evil
Firewall
Alert
Proxy
Users Switch port
Report
! OFAC FW Block -‐> Fraud invesAgator confirms ! If False PosiAve, Unblock IP
! Management report – IP, account name involved, Ame of incident – Report as suspicious acAvity for deep-‐dive follow up
ê Aver confirmaAon, report to government
! FW Block noAficaAon saved search ê index=CUSTOMER_LOGON earliest=-‐24h| lookup geoip client as IP | table _Ame, CUSTOMER_NAME, client_country, IP | join IP [search index=SUMMARY earliest=-‐24h search_name=“BLOCK_OFAC” | table ip ]
23
Report (cont.)
24
! InvesAgator can easily take a deep dive into the alert
Time Customer Name IP Country
9/11/2014 07:23:00
ABC Company 1.2.3.4 Evil Country
9/11/2014 10:45:00
XYZ Company 9.8.7.6 Evil Country
Report-‐II
! Confirm “Evil” is indeed blocked ê index=SUMMARY earliest=-‐5min search_name=“BLOCK_OFAC” | table _Ame, IP, search_name | join type=lev IP [search index=FW earliest=-‐5min acAon=“ADD RULE”| dedup IP | table IP, FW_MSG] | where isNull(FW_MSG)
! If search returns row count > 0, block failed – “Distress” call to support staff
25
Time IP Search_name FW_MSG
9/11/2014 06:38:00 9.8.7.6 BLOCK_OFAC “Added Block AcAon”
9/11/2014 07:23:00 1.2.3.4 BLOCK_OFAC <EMPTY>
Problem, block did not take place!
Report-‐III
! Find “False PosiAve” ê index=SUMMARY earliest=-‐1d search_name=“BLOCK_OFAC” | table _Ame, IP, search_name | join IP [search index=FW earliest=-‐1d acAon=“CANCELLED RULE”| dedup IP | table IP, FW_MSG]
! If search returns row count > 0, block has been revoked – False posiAve rate for threat intelligence
26
Assure
! Demonstrate to management and auditors that control ALWAYS works as intended
! Confirm: – Saved search always fire – Saved search always finish – Saved search always complete with SLA
! Status =“Skipped” indicates search not executed, ie, “Defect Rate”
27
Assure Sample-‐I
! Confirm search execuAon and completes ! Index=_internal savedsearch_name=“BLOCK_OFAC” NOT status=“conAnued”| where scheduled_Ame >= relaAve_Ame(now(), ‘-‐1d@d’) AND scheduled_Ame <= relaAve_Ame(now(), ‘0d@d’)| stats count(status) by status
! For example, if saved search fires every hour, then search needs to fire for 24 Ames in a day
ê Percentage of missed, complete search determines defect rate
28
Assure Sample-‐II
! Confirm search performance ! Index=_internal savedsearch_name=“BLOCK_OFAC” | eval Z_LAG=_Ame-‐scheduled_Ame| stats earliest(_Ame) as Z_START_TIME, latest(_Ame) as Z_END_TIME by scheduled_Ame, savedsearch_name | eval Z_LAG_TO_START=Z_START_TIME -‐ scheduled_Ame | eval Z_TIME_TO_COMPLETE=Z_END_TIME-‐scheduled_Ame| stats avg(Z_LAG_TO_START) as START_LAG, avg(Z_TIME_TO_COMPLETE) as PROCESS_TIME by savedsearch_name
29
Savedsearch_name Avg(START_LAG) Avg(PROCESS_TIME)
BLOCK_OFAC 123 456
Report/Assure–Data Flow
30
App Log Index • Execute saved search • Detect bad FW Log Index
• Records adding “bad” IP into block list
• Records cancelling of block request Summary Index
• Records “bad” IP and TimeStamp
_internal Index • Records when saved search was executed
• Use to calculate defect rate of control
Summary + FW Index • Use to confirm “bad” IP is indeed added to be blocked within SLA • Calculate “Ame-‐to-‐remediate” • Use to calculate false posiAve rate based on “cancel” events
OperaAonalize and QuanAfy ! Monthly metrics of quality (KQI)
– Performance ê Time-‐to-‐Detect -‐> Bad acAvity found ê Time-‐to-‐Remediate -‐> Block acAon ê Defect Rate -‐> Missed search ê False PosiAve Rate -‐> Bad intelligence
! Time-‐to-‐Detect – (index=apache earliest=-‐1d [search index=SUMMARY earliest=-‐1d savedsearch_name=“BLOCK_OFAC” | dedup
IP | table IP]) | stats min(_Ame) as FIRST_APPEARANCE by IP | join IP [search index=SUMMARY earliest=-‐1d savedsearch_name=“BLOCK_OFAC” |eval DETECT_TIME=info_search_Ame | dedup IP | table IP, DETECT_TIME, FIRST_APPEARANCE] | eval TIME_TO_DETECT = DETECT_TIME-‐FIRST_APPEARANCE
! Time-‐to-‐Remediate – index=SUMMARY savedsearch_name=“BLOCK_OFAC” | eval DETECT_TIME=_Ame| table DETECT_TIME, IP,
search_name | join IP [search index=FW *Added rule* |eval BLOCK_TIME=_Ame | dedup IP | table IP, BLOCK_TIME, DETECT_TIME] | eval TIME_TO_REMEDIATE=BLOCK_TIME-‐DETECT_TIME| table search_name, IP, TIME_TO_REMEDIATE | sort -‐TIME_TO_REMEDIATE
31
Value
OperaAonalize and QuanAfy – II
! Defect Rate – Same as previous secAon, but run search over month
! False PosiAve Rate – index=SUMMARY earliest=-‐1mon search_name=“BLOCK_OFAC” | table
_Ame, IP, search_name | join IP [search index=FW earliest=-‐1mon acAon=“CANCELLED RULE”| dedup IP | table IP, FW_MSG]
32
Value
Summary
! Business goal achieved – Our case – OFAC MiAgaAon
ê 24x7 Auto miAgaAon -‐> ProtecAon and compliance ê ConAnuous alestaAon and verificaAon -‐> Demonstrable control ê Cost effecAve -‐> SystemaAc control
– ConAnuous improvement based on data-‐driven decision -‐> defect rate/false posiAve kept within limit
! Methodology allows repeatable and auditable pracAce – JusAficaAon for auto-‐miAgaAon
! Uses built-‐in Splunk capabiliAes
33
Value
34
Security office hours: 11:00 AM – 2:00 PM @Room 103 Everyday Geek out, share ideas with Enterprise Security developers
Red Team / Blue Team -‐ Challenge your skills and learn new tricks Mon-‐Wed: 3:00 PM – 6:00 PM @Splunk Community Lounge Thurs: 11:00 AM – 2:00 PM
Learn, share and hack
Birds of a feather-‐ Collaborate and brainstorm with security ninjas Thurs: 12:00 PM – 1:00 PM @Meal Room
THANK YOU
