Upload
splunk
View
571
Download
13
Tags:
Embed Size (px)
Citation preview
Copyright © 2014 Splunk Inc.
Andrew Duca Sr. Professional Services Consultant, Splunk
Data On-‐Boarding
Disclaimer
2
During the course of this presentaGon, we may make forward-‐looking statements regarding future events or the expected performance of the company. We cauGon you that such statements reflect our current expectaGons and
esGmates based on factors currently known to us and that actual events or results could differ materially. For important factors that may cause actual results to differ from those contained in our forward-‐looking statements,
please review our filings with the SEC. The forward-‐looking statements made in the this presentaGon are being made as of the Gme and date of its live presentaGon. If reviewed aQer its live presentaGon, this presentaGon may not contain current or accurate informaGon. We do not assume any obligaGon to update any forward-‐looking statements we may make. In addiGon, any informaGon about our roadmap outlines our general product direcGon and is subject to change at any Gme without noGce. It is for informaGonal purposes only, and shall not be incorporated into any contract or other commitment. Splunk undertakes no obligaGon either to develop the features or funcGonality described or to
include any such feature or funcGonality in a future release.
About Me
! Senior Professional Services Consultant based in Boston, MA ! 14+ Years of world-‐wide Professional Services ConsulGng with the last two at Splunk
! Involved in 20+ deployments from 1GB to 5TB
3
Agenda
! Data ! Splunk Components ! Index Data ! Proper Parsing ! Challenging Data ! Advanced Inputs
4
Are You in The Right Room?
5
! You have used Splunk at least once, or at least read about it ! You are interested in Splunk best pracGces ! You like to use Splunk’s default parsing rules ! You just took over a Splunk deployment and you’re not sure what to do
! This is not an educaGon class; it’s best pracGce
Data
6
! Machine data is more than just logs -‐ it's configuraGon data, data from APIs and message queues, change events, the output of diagnosGc commands and more
! Log types: ApplicaGon, Web Access and Proxy, Call Detail Records (CDR), Clickstream, Message Queues, Packet, Database audit and tables, File audit, Syslog, WMI, PerfMon
! Manual: Gecng Data Inhdp://docs.splunk.com/DocumentaGon/Splunk/latest/Data/WhatSplunkcanmonitor
Splunk is the engine for machine data
Splunk Apps
7
! Look to Splunk Apps first and uGlize Technical Add-‐On (TA) ! Applies the Common InformaGon Model (CIM)
! CIM details the standard fields, event type tags, and host tags that Splunk uses when it processes most IT data
! Example TAs: Windows Unix Exchange AcGve Directory VMware Vcenter WebSphere
Splunk Distributed Components
8
Search Head
Deployment Server
Indexer
Forwarder
Test Environment
9
! Every Splunk deployment should have a test environment
! It can be a laptop, virtual machine or spare server
! Should have the same version of Splunk running in producGon
! Accessible to other Splunk developers and administrators
One Shot
10
! Easiest way to get data into your test environment ! Components of the oneshot:
./splunk add oneshot user_conf.txt –index indexname –sourcetype sourcetype name
! Where to find more informaGon:hdp://docs.splunk.com/DocumentaGon/Splunk/latest/Data/MonitorfilesanddirectoriesusingtheCLI
Data -‐ Broken
11
Props
12
! Always set these six parameters
# USER CONFERENCE
[user_conf_2014]
TIME_PREFIX = ^
TIME_FORMAT = %Y-‐%m-‐%d %H:%M:%S
MAX_TIMESTAMP_LOOKAHEAD = 19
SHOULD_LINEMERGE = False
LINE_BREAKER = ([\n\r]+)\d{4}-‐\d{2}-‐\d{2}\s\d{2}:\d{2}:\d{2}
TRUNCATE = 10000
Props
13
! Defaults to empty
# USER CONFERENCE
[user_conf_2014]
TIME_PREFIX = ^
TIME_FORMAT = %Y-‐%m-‐%d %H:%M:%S
MAX_TIMESTAMP_LOOKAHEAD = 19
SHOULD_LINEMERGE = False
LINE_BREAKER = ([\n\r]+)\d{4}-‐\d{2}-‐\d{2}\s\d{2}:\d{2}:\d{2}
TRUNCATE = 10000
Props
14
! strpGme Style format
# USER CONFERENCE
[user_conf_2014]
TIME_PREFIX = ^
TIME_FORMAT = %Y-‐%m-‐%d %H:%M:%S
MAX_TIMESTAMP_LOOKAHEAD = 19
SHOULD_LINEMERGE = False
LINE_BREAKER = ([\n\r]+)\d{4}-‐\d{2}-‐\d{2}\s\d{2}:\d{2}:\d{2}
TRUNCATE = 10000
Props
15
! By default MAX_TIMESTAMP_LOOKAHEAD = 150 characters
# USER CONFERENCE
[user_conf_2014]
TIME_PREFIX = ^
TIME_FORMAT = %Y-‐%m-‐%d %H:%M:%S
MAX_TIMESTAMP_LOOKAHEAD = 19
SHOULD_LINEMERGE = False
LINE_BREAKER = ([\n\r]+)\d{4}-‐\d{2}-‐\d{2}\s\d{2}:\d{2}:\d{2}
TRUNCATE = 10000
Props
16
! By default set to True
# USER CONFERENCE
[user_conf_2014]
TIME_PREFIX = ^
TIME_FORMAT = %Y-‐%m-‐%d %H:%M:%S
MAX_TIMESTAMP_LOOKAHEAD = 19
SHOULD_LINEMERGE = False
LINE_BREAKER = ([\n\r]+)\d{4}-‐\d{2}-‐\d{2}\s\d{2}:\d{2}:\d{2}
TRUNCATE = 10000
Props
17
! By default set to ([\r\n]+); change to posiGve lookahead
# USER CONFERENCE
[user_conf_2014]
TIME_PREFIX = ^
TIME_FORMAT = %Y-‐%m-‐%d %H:%M:%S
MAX_TIMESTAMP_LOOKAHEAD = 19
SHOULD_LINEMERGE = False
LINE_BREAKER = ([\n\r]+)\d{4}-‐\d{2}-‐\d{2}\s\d{2}:\d{2}:\d{2}
TRUNCATE = 10000
Props
18
! By default set to 10000 bytes; set to 0 to never truncate
# USER CONFERENCE
[user_conf_2014]
TIME_PREFIX = ^
TIME_FORMAT = %Y-‐%m-‐%d %H:%M:%S
MAX_TIMESTAMP_LOOKAHEAD = 19
SHOULD_LINEMERGE = False
LINE_BREAKER = ([\n\r]+)\d{4}-‐\d{2}-‐\d{2}\s\d{2}:\d{2}:\d{2}
TRUNCATE = 10000
Data -‐ Fixed
19
6.2 Splunk Web Data On-‐Boarding
Why to Use Splunk Web to On-‐board?
21
Quick and easy way to… ! Easily visualize the data into events rather then lines of text ! Quickly get the data properly broken into events ! Accurately get the Gme stamp extracted
All in a wicked cool GUI Once everything is good you take your PROPS secngs and deploy
Splunk Web Data On-‐Boarding
22
! Locate the source file on the Splunk Server’s file system
Splunk Web Data On-‐Boarding
23
! Validate event breaking and Gmestamp recogniGon
Splunk Web Data On-‐Boarding
24
! Resolve event breaking
Splunk Web Data On-‐Boarding
25
! Set Gmestamp format even if Splunk figures it out automaGcally
Splunk Web Data On-‐Boarding
26
! Copy the props.conf secngs and deploy in a custom app
Challenging Data
Limit Indexed Data
28
! Anonymize data: [source::.../accounts.log]
SEDCMD-‐accounts = s/ssn=\d{5}(\d{4})/ssn=xxxxx\1/g s/cc=(\d{4}-‐){3}(\d{4})/cc=xxxx-‐xxxx-‐xxxx-‐\2/g
! Rewrite raw data: [source::.../sql.log]
SEDCMD-‐sqllog = s/(.*?)Command:EXECUTE[.\d\D\w\W]*/\1/g
! Discard events: props
[source::/var/log/user_conf.txt]
TRANSFORMS-‐null= setnull
transforms [setnull] REGEX = (?i)DEBUG DEST_KEY = queue FORMAT = nullQueue
Limit Indexed Data
29
! Anonymize data: [source::.../accounts.log]
SEDCMD-‐accounts = s/ssn=\d{5}(\d{4})/ssn=xxxxx\1/g s/cc=(\d{4}-‐){3}(\d{4})/cc=xxxx-‐xxxx-‐xxxx-‐\2/g
! Rewrite raw data: [source::.../sql.log]
SEDCMD-‐sqllog = s/(.*?)Command:EXECUTE[.\d\D\w\W]*/\1/g
! Discard events: props
[source::/var/log/user_conf.txt]
TRANSFORMS-‐null= setnull
transforms [setnull] REGEX = (?i)DEBUG DEST_KEY = queue FORMAT = nullQueue
Limit Indexed Data
30
! Anonymize data: [source::.../accounts.log]
SEDCMD-‐accounts = s/ssn=\d{5}(\d{4})/ssn=xxxxx\1/g s/cc=(\d{4}-‐){3}(\d{4})/cc=xxxx-‐xxxx-‐xxxx-‐\2/g
! Rewrite raw data: [source::.../sql.log]
SEDCMD-‐sqllog = s/(.*?)Command:EXECUTE[.\d\D\w\W]*/\1/g
! Discard events: props
[source::/var/log/user_conf.txt]
TRANSFORMS-‐null= setnull
transforms [setnull] REGEX = (?i)DEBUG DEST_KEY = queue FORMAT = nullQueue
Limit Indexed Data
31
6.X or later Windows forwarders
! Whitelist events or blacklist specific events ! Inputs.conf ConfiguraGon
Index ExtracGons
32
! Provides reliable and consistent indexing of data with headers. ! Address issue on forwarder:
INDEX_EXTRACTIONS = {CSV | W3C | TSV | PSV | JSON} ! Supports custom header parsing and easy mode for common formats. ! Extract IIS fields using Props.conf on Windows forwarder:
[iis] INDEX_EXTRACTIONS = w3c
MulGple Timestamps
33
datePme.xml <datetime>
<define name=”two_tz" extract="day, litmonth, year, hour, minute, second, zone">
<text><![CDATA[^(\d+)-‐(\w+)-‐(\d+),(\d+):(\d+):(\d+),(?:[^,]*,){2}([\w\-‐]*)]]></text> </define>
<timePatterns>
<use name=”two_tz">
</timePatterns>
<datePatterns>
<use name=”two_tz">
</datePatterns>
</datetime>
props.conf # USER CONF
[user_conf]
DATETIME_CONFIG = /etc/apps/splk_ps_user_conf_props/local/datetime.xml
* Do not set TIME_FORMAT
12-‐Sep-‐2014,09:01:00,12-‐Sep-‐2014,09:02:00,-‐4 INFO Gtle="User Conference" msg="Splunk hosted user conference in Las Vegas." 12-‐Sep-‐2014,19:01:00,12-‐Sep-‐2014,19:02:00,-‐5 DEBUG Gtle="User Conference" msg="Gecng Data In, Correctly is a solid session."
Database Connect
Database Connect
35
! Allows for indexing data from database sources directly ! Allows for adding meta data to events from database sources using lookups
Caveats ! Java required on Splunk server ! Search head pooling requires custom configuraGon to share the DB connecGon passwords. Not meant for data input sources
Database Connect Best PracGces
36
! Normalize Gmestamps naGvely inside the SQL Query ! Filter results down in SQL Query to reduce garbage in Splunk index ! Repeated DBLookups should be converted to staGc lookup ! Search head pooling requires encrypted password replicaGon
Modular and Scripted Inputs
Modular and Scripted Inputs
38
Benefits
! Almost any program that can output text can be used to index ! Modular inputs allow for configuraGon files and configuraGon secngs inside Splunk Differences
! Scripted inputs require configuraGon to be done in the script ! Modular inputs can be configured via deployed .conf files and accessed via REST API ! Scripted inputs need are specific to the OS deployed on where modular inputs can
support mulGple Examples
vmstat, iostat, Checkpoint Opsec, Twider, Stream, Amazon S3 Online storage and more…
Scripted Inputs Example
39
! Shell script saved in /opt/splunk/bin/scripts/ OR in a specific app ! Allows you to execute any program on Splunk forwarder and index
STDOUT data. ! UGlizing key value pairs makes for easier searching.
Sample output from custom script /Applica3ons/Splunk/bin/scripts/FantasyFootball.sh
Scripted Inputs Example
40
Shell script calls local system binary programs and can provide configuraGon opGons.
Use Inputs.conf to define INDEX, SOURCETYPE, and INTERVAL for the scripted input
ProducGon Deployment
ProducGon Environment
42
! Complexity managing configuraGons across tens, hundreds, or thousands of forwarders
! Not all indexers and search heads receive the same configuraGons
! Should think about version control for deployment apps, e.g., GitHub
SHP
Deployment Server Terminology
43
! Deployment Server -‐ A Splunk instance that acts as a centralized configuraGon manager, grouping together and collecGvely managing any number of Splunk instances. Any Splunk instance can act as a deployment server, even one that is indexing data locally. Splunk instances that are remotely configured by deployment servers are called deployment clients.
! Deployment Client -‐ A Splunk instance that is remotely configured by a deployment server.
! Server Class -‐ Represents a configuraGon of Splunk deployment clients. Server classes enable the management of a group of deployment clients as a single unit. A server class can be used to group deployment clients together by applicaGon, OS, data type to be indexed, or any other feature of your Splunk deployment.
Deployment App
44
! A deployment app (configuraGon bundle) is a set of deployment content (including configuraGon files) deployed as a unit to clients of a server class
! Located in $SPLUNK_HOME/etc/deployment-‐apps and pushed to deployment client’s $SPLUNK_HOME/etc/apps folder
! DO NOT store configuraGons in $SPLUNK_HOME/etc/system/local ! Use deployment apps regardless of your deployment tool
Deployment App -‐ Naming ConvenGon
45
org
acme
acme
splk
splk
group
finance
markeGng
all
ps
applicaGon
apache
iis
indexer
user_conf
configuraGon
inputs
props
Base
inputs
Deployment App -‐ Naming ConvenGon
46
org
acme
acme
splk
splk
group
finance
markeGng
all
ps
applicaGon
apache
iis
indexer
user_conf
configuraGon
inputs
props
base
inputs
Deployment App -‐ Naming ConvenGon
47
org
acme
acme
splk
splk
group
finance
markeGng
all
ps
applicaGon
apache
iis
indexer
user_conf
configuraGon
inputs
props
base
inputs
Deployment App -‐ Naming ConvenGon
48
org
acme
acme
splk
splk
group
finance
markeGng
all
ps
applicaGon
apache
iis
indexer
user_conf
configuraGon
inputs
props
base
inputs
Deployment App -‐ Naming ConvenGon
49
org
acme
acme
splk
splk
group
finance
markeGng
all
ps
applicaGon
apache
iis
indexer
user_conf
configuraGon
inputs
props
base
inputs
Deployment App -‐ Naming ConvenGon
50
org
acme
acme
splk
splk
group
finance
markeGng
all
ps
applicaGon
apache
iis
indexer
user_conf
configuraGon
inputs
props
base
inputs
splk_ps_user_conf_inputs
Deployment Apps
51
! SplunkForwarder
! SplunkLightForwarder
! Splunk_for_AcGveDirectory
! Splunk_for_Exchange
! splk_all_deploymentclient
! splk_all_forwarder_outputs
! splk_all_indexer_base
! splk_all_search_base
! splk_ps_user_conf_inputs
! splk_ps_user_conf_props
! splk_ps_user_conf_web
! splunk_app_was
• user-‐prefs
mba13:apps $ ls -‐la
CollecGng Syslog
52
! Send device, e.g., routers, firewalls to a syslog collector
! Write files to this directory structure: /sourcetype/host/log.txt
! Monitor the sourcetype level cisco_asa
my.firewall.name # CISCO ASA [monitor:///data/cisco_asa/…/] sourcetype = cisco_asa host_segment = 3 index = firewall
Summary
53
! Test in a non-‐producGon environment ! Always use key props parameters:
– TIME_PREFIX – TIME_FORMAT – MAX_TIMESTAMP_LOOKAHEAD – SHOULD_LINEMERGE – LINE_BREAKER – TRUNCATE
! Deploy apps to /etc/apps; not /etc/system/local ! Clear predictable naming convenGon ! When you’re stuck, use Answers and Re-‐Use apps from Apps.Splunk.com
Resources
54
! Get educated: hdp://www.splunk.com/view/educaGon/SP-‐CAAAAH9 ! Download Splunk applicaGons: hdp://apps.splunk.com/ ! Hire Splunk Professional Services: hdp://www.splunk.com/view/professional-‐services/SP-‐CAAABH9
! Watch some videos: hdp://www.splunk.com/videos
THANK YOU