Sydney Python Presentation (October 2010) - Splunk

Splunk and Python

Sydney Python October 2010

Kelvin Nicholson

What is Splunk?

“Splunk is the world’s leading software used to monitor, report and analyze live streaming IT data as well as terabytes of historical data – located on-premises or in the cloud.” -Splunk.com

“Splunk is like google for log files.” -Kelvin

Installing Splunk (on Ubuntu)

$ sudo dpkg -i splunk-4.1.5-85165-linux-2.6-intel.deb$ sudo splunk enable boot-start$ sudo /etc/init.d/splunk start

Splunk Welcome Screen

Configuring Splunk●Configure Splunk to allow syslog traffic●Configure devices to send syslog to Splunk

○ Linux (syslog-ng) destination loghost { udp("192.168.83.11" port (514)); }; log { source(s_all); destination(splunk); };

●Cisco IOS no logging console no logging monitor logging 192.168.83.11

● OSSEC <syslog_output> <server>192.168.83.11</server> <port>8514</port> </syslog_output>

Splunk Search Screen

Why I Like Splunk (Abridged)

●Dashboards of Search terms■ Security alerts “login failed for”■ STP network issues (“LEARNING AND FORWARDING”■ Duplex mismatches■ Wildcard searches, e.g. “-server2k3-”

●My “WTF” filter (easy filter building)●Beautiful trending (“cold start” AND “switch01”)

Splunk Simple Filtering

Extending Splunk with Python

●REST API. (Search only)●Custom search command. (iplocation)●Configuring scripted alerts. (tweet X alert)●Directly to backend using Splunk's built-in

modules. (Full module access)

Accessing Splunk Datastore>>> import splunk.auth, splunk.search>>> key = splunk.auth.getSessionKey('admin','changeme')>>> my_job = splunk.search.dispatch('search sypy', namespace='search')>>> event_list = []>>> for event in my_job.events:... event_list.append(event.fields)... >>> print event_list

kelvinn@splunk:/opt/splunk/bin$ ./splunk cmd python

[{'_si': splunk,main, 'index': main, 'sourcetype': syslog, 'source': udp:514, '_kv': 1, 'splunk_server': splunk, '_time': 2010-10-06T19:40:37+1100, 'host': 192.168.83.5, '_sourcetype': syslog, '_raw': Oct 6 19:40:37 192.168.83.5 Oct 6 19:40:38 mini kelvinn: hello SyPy, hope you are doing well., '_serial': 0, '_cd': 0:275}, {'_si': splunk,main, 'index': main, 'sourcetype': syslog, 'source': udp:514, '_kv': 1, 'splunk_server': splunk, '_time': 2010-10-06T19:39:33+1100, 'host': 192.168.83.5, '_sourcetype': syslog, '_raw': Oct 6 19:39:33 192.168.83.5 Oct 6 19:39:34 mini kelvinn: sypy, '_serial': 1, '_cd': 0:251}]

>>> event_list[0]['_raw']Oct 6 19:40:37 192.168.83.5 Oct 6 19:40:38 mini kelvinn: hello SyPy, hope you are doing well.

Splunk Architecture

CherryPy built-in, sweet. What can we do with that?

Built-in CherryPy Funkelvinn@splunk:/opt$ cat splunktest.py import cherrypyimport splunk.auth, splunk.search

def get_splunk_data():key = splunk.auth.getSessionKey('admin','changeme') # replace with your credentialsmy_job = splunk.search.dispatch('search sypy', namespace='search', earliest_time='-24h')

event_list = []for event in my_job.events:event_list.append(event.raw)return event_listclass HelloWorld:def index(self):splunk_list = get_splunk_data()return str(splunk_list)index.exposed = True

cherrypy.config.update({'server.socket_host': '0.0.0.0','server.socket_port': 9999,})cherrypy.quickstart(HelloWorld())kelvinn@splunk:/opt$ /opt/splunk/bin/splunk cmd python /opt/splunktest.pyP.S. I'm not a CherryPy expert, but it looks pretty fun.

View CherryPy Page

Resources + ThanksSplunk introduction:

http://www.splunk.com/base/Documentation/4.1.5/Installation/Splunksarchitectureandwhatgetsinstalled

Splunk REST Search (with Python httplib example):

http://www.splunk.com/base/Documentation/4.1.5/Developer/RESTCreateSearch

Custom search command (iplocation):

http://www.splunk.com/base/Documentation/latest/SearchReference/Customsearchiplocation

How to write custom alerts:

http://www.splunk.com/base/Documentation/4.1.5/Admin/Configurescriptedalerts

Using Splunk's built-in Python modules:

http://answers.splunk.com/questions/14/can-i-use-splunks-built-in-python-sdk-in-my-own-scripts

Some information about Splunk's Python SDK:

http://www.splunk.com/base/Documentation/4.1.5/Developer/PySDK

Thanks.