Copyright © 2014 Splunk Inc.

George Starcher Security Engineer, Peak Hosting

From Tool to Team Member: Controlling Systems with Splunk Alert Scripts

About Me

2

• George Starcher, Information Security Engineer • CISSP, Splunk Certified Knowledge Manager and Splunk

Certified Administrator • Splunk IRC Channel • Looking to kick off a Nashville, TN - Splunk User Group • www.georgestarcher.com • www.peakhosting.com

Agenda

Splunk from Tool to a Team Member How it Works Getting into the Code

Alert Script to Intrusion Prevention System Control Alert Script to X-ARF Abuse Reporting

3

“Using Alert Scripts to take action on our behalf, we can transform Splunk from a tool to a team member.”

Splunk from Tool to Team MemberManual Abuse Scanning Process

Reviewed SSH, RDP, VNC etc daily Consumed 30-45 minutes per day Permanent blacklist entries

Moved to automated process Scheduled Splunk Searches driven by any log source Web Services (REST) calls to the IPS Greatly reduced time and static blacklist maintenance

4

Splunk from Tool to Team Member

5

Outlook Web Access - Phishers

Started Feb 10, 2014

• Blocked for any access from Nigeria every 5 minutes

Expanded Multi Country Feb 15, 2014

• Blocked for combination from certain countries & a lookup table of hosted providers

Feb 17, 2014

• Noticed unexpected Exchange OWA from Nigeria

Splunk from Tool to Team Member

6

Outlook Web Access - Phishers

Single User by src_ip_country:

Hosted Lookup users

by src_ip:

How it Works

How it Works

8

http://blogs.splunk.com/2011/03/15/storing-encrypted-credentials/

http://www.georgestarcher.com/splunk-alert-scripts-automating-control/

$splunk_home/etc/auth/splunk.secret

How it Works

9

Setup a service account to own the Alert Searches: svc-alert Create a role just for the alert account That role must have ‘admin_all_objects’ The role must have access to all indexes that might have the data for the scheduled search alert. Consider restricting access to the rest API port 8089 to only Splunk servers and any other dedicated systems.

Alert Service Account

How it Works

10

Intrusion Prevention Appliance

How it Works

11

SPLUNK_ARG

How it Works

12

Alert Script in Action

13

Avoids manual reporting Ensures timely action Consistent Reporting Format Accurate Evidence Data Splunky the Intern works around the clock and doesn't need coffee

X-ARF Abuse Reporting

Alert Script in Action

14

X-ARF Abuse Reporting

Alert Script in Action

15

X-ARF Abuse Reporting

Crawling into the Code

@SplunkDev Team - THANKS!!

17

@gblock - Glenn Block

@damiendallimore -Damien Dallimore

David Noble - Twitter App

Where can you get the code?

18

Github Repository https://github.com/georgestarcher/Splunk-Alert General Intrusion Prevention System Example Code Google Spreadsheet Upload Code X-ARF Abuse Reporting Code

The Google Spreadsheet Example http://www.georgestarcher.com/splunk-alert-scripts-automating-control/

Arguments Sent to Alert Scripts

19

http://docs.splunk.com/Documentation/Splunk/6.1.3/Alert/Configuringscriptedalerts

SPLUNK_ARG_0 Script name SPLUNK_ARG_1 Number of events returned SPLUNK_ARG_2 Search terms SPLUNK_ARG_3 Fully qualified query string SPLUNK_ARG_4 Name of report SPLUNK_ARG_5 Trigger reason (for example, "The number of events was greater than 1") SPLUNK_ARG_6 Browser URL to view the report SPLUNK_ARG_8 File in which the results for this search are stored (contains raw results)

The Code Modules - IPS

20

credentialsFromSplunk.py A Python class to fetch the saved service account

targetlist.py The Python class for data to be handled

ips.py The Python class for an IPS rest API interface

alert_script.py The main Python alert script

credentialsFromSplunk.py

21

a re-usable Python class to fetch stored user credentials from Splunk provide the app where credentialed is stored: splunkapp provide the purpose name used when saving the credentials: realm provide the username to be retrieved: username call the getPassword method

credentialsFromSplunk.py

22

# Define the source in Splunk for the stored credential splunkapp = "myadmin" realm = 'ips' username = 'splunk'

# Get the stored credential from Splunk try: ipsCredential.getPassword(sessionKey) except Exception, e: logError("Splunk Credential Error: %s" % str(e)) exitAlertScript(_SYS_EXIT_FAILED_SPLUNK_AUTH)

# Define the ips connection ipsCredential = credential(splunkapp,realm,username)

from alert_script.py

targetlist.py

23

a simple Python class for a single column list of source IPs populated by the alert search returning only source IPs takes argument of path to the search results to load the list

# Obtain the path to the alert events compressed file and load the search results to the list

alertEventsFile = os.environ['SPLUNK_ARG_8']

try: alertTargetList = targetlist(alertEventsFile) except Exception, e: logError("Target File Error: %s" % str(e)) exitAlertScript(_SYS_EXIT_FAILED_TARGET_FILE)

from alert_script.py

ips.py

24

an example Python class to interface with our Intrusion Detection System Rest API setup and retrieve the credential from splunk: ipsCredential provide the IPS quarantine policy name: policy_name provide IP address of the IPS management Interface: ips_ip activate the IPS rest connection object loop through the alertTargetList having the IPS quarantine each IP

Make your Own REST API wrapper class to control other systems.

ips.py

25

# Active the ips connection object try: ssh_ips = ips(ips_ip,ipsCredential.username, ipsCredential.password,policy_name) except Exception, e: logError("IPS Error: %s" % str(e)) exitAlertScript(_SYS_EXIT_FAILED_IPS)

from alert_script.py

alert_script.py

26

the main script called by Splunk for our alert search imports all our classes parses the sessionKey connects to our IPS pulls in the search result list of IP addresses loops through the IP list and tells the IPS to quarantine them

alert_script.py

27

The Hash Bang: #!/opt/splunk/bin/python

# Quarantine each source ip in the alert results table

for address in alertTargetList.targetlist: try: ssh_ips.addQuarantine(address) except Exception, e: logError("IPS Quarantine Error: %s" % str(e)) exitAlertScript(_SYS_EXIT_FAILED_IPS)

# Obtain the Splunk authentication session key … # Adjust the returned sessionKey text based on Splunk version …

Extended Abuse Reporting - X-ARF

28

Much more complex code Search results driving results is a table of data not a simple IP list Pulls email settings from Splunk Builds the email body using the Python Mako template (mail merge to search results) Improved alert script action logging sending into index=_internal Attaches Alert Event Search results from Splunk REST API Calls

http://www.x-arf.org/

BONUS

The Search for Alerting

29

tag=authentication action=failure app=sshd NOT src_ip=10.0.0.0/8 | stats count values(user) AS Users first(_time) AS EndTime last(_time) AS StartTime by src_ip, host, app | where count>4 | lookup abuseLookup ip AS src_ip| lookup dnsLookup ip AS src_ip OUTPUT hostname AS src_hostname | eval src_hostname=if(isNull(src_hostname),"unknown",src_hostname) | lookup dnsLookup hostname AS host OUTPUT ip AS dest_ip | eval dest_host=host | iplocation src_ip | eval StartTime=strftime(StartTime,"%Y-%m-%d %T %z") | eval EndTime=strftime(EndTime,"%Y-%m-%d %T %z") | eval City=if(isNull(City),"unknown city",City) | eval Region=if(isNull(Region),"unknown region",Region) | eval Country=if(isNull(Country),"unknown country",Country)| eval DistinctUserNames=mvcount(Users) | table src_ip, src_hostname, City, Region, Country, abusecontact, StartTime, EndTime, dest_ip, dest_host, app, count, DistinctUserNames, Users | search dest_ip=* NOT abusecontact=spam* NOT abusecontact="abuse@erratasec.com" NOT src_hostname=*.shodan.io

The Code Modules - X-ARF

30

abuselist.py The data to be handled, a different version than for IPS

emailSplunkXARF.py A python class to for the mail settings and sending reports

xarf-abuse.tmpl Abuse report Email mako template

alert_to_xarf.py The main alert script

abuselist.py

31

method getEvidence holds the evidence search executed against the Splunk REST API this method also manipulates the earliest/latest timestamp coming from the search results automatically to go into the detail evidence search

emailSplunkXARF.pymethod getMailSettings is Splunk REST API call to fetch the settings from your Splunk server

Evidence Search

32

search_query = 'search tag=authentication action=failure app=sshd src_ip='+self.source+' earliest="'+earliestTimestamp+'" latest="'+latestTimestamp+'" | head '+numEvents+' | eval eventTime=strftime(_time,"%Y-%m-%d %T %z") | eval eventLog=substr(_raw,timeendpos+1) | eval eventDetail=eventTime+" "+eventLog | table eventDetail'

alert_to_xarf.py

33

All the X-ARF values are at the top of the script method getSplunkVersion gets the running Splunk version from the REST API to help auto adjust the sessionKey method getSplunkUser gets the username the Alert executed under from Splunk needed for the evidence search fetch logging writes with proper timestamp GMT to $SPLUNK_HOME/var/log/splunk/…

You could use this to make your own highly customized alert email based on search results

Thank You!

34

Other resources Splunk IRC ( EFNet #splunk ) Splunk Answers ( http://answers.splunk.com ) Splunk community wiki ( http://wiki.splunk.com ) http://www.georgestarcher.com/ http://blog.splunk.com/ http://www.meetup.com/Splunk/Nashville-TN/

Other “must-see” .conf 2014 presentations Avoid the SSLippery Slope of Default SSL - Duane Waddle and George Starcher In Depth With Deployment Server - Dave Shpritz, Aplura Using Lesser Known Commands in Splunk Search Processing Language (SPL) - Kyle Smith, The Hershey Company Masters of IRC - panel talk on the Splunk Community Stage

THANK YOU