View
21
Download
0
Category
Preview:
Citation preview
Instrumentation with Splunk
11/17/2016 2www.datavail.com
About Datavail
13+ Years delivering data services
300+ customers with average client retention of 7 years
Managed services, projects, and staffing
11/17/2016 3www.datavail.com
The Problem…
Oracle Applications have allot of moving pieces…
11/17/2016 4www.datavail.com
The Problem…
Oracle Applications can have allot of moving pieces…
In allot of different locations…
11/17/2016 5www.datavail.com
The Problem…
Root causes are quite often buried and difficult to find.
11/17/2016 6www.datavail.com
The Problem…
Root causes are quite often buried and difficult to find.
What if there was a neat way to federate the telemetry from all your key pieces?
11/17/2016 7www.datavail.com
The Problem…
Root causes are quite often buried and difficult to find.
What if there was a neat way to federate the telemetry from all your key pieces? Including the contents of your critical logs?
11/17/2016 8www.datavail.com
A Solution…
Web Based GUI Search Engine
Filtered Reports on Key Perf Indicators
Raw Log File Repository
11/17/2016 9www.datavail.com
A Solution…
11/17/2016 10www.datavail.com
What is Splunk?
Splunk is an enterprise application.
Splunk was made to monitor.
Splunk is agnostic.
Splunk is a framework!
11/17/2016 11www.datavail.com
What Splunk isn’t
Not a collection of purchased lock-downed modules (Solar Winds etc.)
It’s not an application that lives on a database.
It’s not an out of the box solution.
11/17/2016 12www.datavail.com
Splunk’s Pieces Forwarder
• Installed on the server
• Light weight
• Has a watch list
• Basic filtering
• Basic classification
• Looks for file changes
• Looks for new files
• Sends it all to the Indexer
11/17/2016 13www.datavail.com
Splunk’s Pieces
Indexer
• Transforms the raw data into events.
• An event is like a line in your log file.
• Can be set up in HA configs.
11/17/2016 14www.datavail.com
Splunk’s Pieces
Search Head
• The GUI interface to your data.
• Provides API and CLI support.
• Runs on *NIX, OSX and Windows.
Splunking Your DataWhat can I do with Splunk?
11/17/2016 16www.datavail.com
Splunk’s Features
Interactive Searching• Has robust SPL to search with.
• Combination of grep, regex, and custom functions like eval, average,
• Has grouping and de-dupe functions
• Searches can be saved as carts or reports.
11/17/2016 17www.datavail.com
Splunk’s FeaturesData is organized as Events.
Events assign metadata to the raw data.
11/17/2016 18www.datavail.com
Splunk’s FeaturesMetadata is automatically aggregated and displayed as dynamic ‘tool tips’.
11/17/2016 19www.datavail.com
Splunk’s Features
REGEX Field Extractions
Splunk extracts your KPIs using REGEX quickly parsing through your collected data to identify the information you are most interested in.
(?i) ORA-(?P<OracleAlertError>.+)
11/17/2016 20www.datavail.com
Splunk’s Features
Automated Field Extractions
Splunk can also automatically generate regex extractions for many common patterns of data.
11/17/2016 21www.datavail.com
Splunk’s Features
Powerful Data Classification Tools
Data Models provide easy access to extracted data that can be used to easily create pivot charts and tables.
11/17/2016 22www.datavail.com
Splunk’s Features
Data Models
Select FieldsSelect Pivots
11/17/2016 23www.datavail.com
Splunk’s Features
Interactive Charts
11/17/2016 24www.datavail.com
Splunk’s Features
Interactive Charts
Customer Use Case
11/17/2016 26www.datavail.com
Scripted Inputs For GC Monitoring
Use Cases• Access data that is not available as an ordinary file.
• Access data that cannot be sent using TCP or UDP.
• Stream data from command-line tools, such as SQLPlus.
• Reformat complex data so you can more easily parse the data into events and fields.
• Attach a timestamp to transient data such as iostat.
Methods of Implementation• Shell Scripts
• Batch Files
• Python/Perl Scripts
• Command Line Output
• Anything that writes to STDOUT
Methods of Capture• Direct from STDOUT
• Write a file to be indexed
11/17/2016 27www.datavail.com
Forwarder Recap
The forwarder is the beginning of classification.
Source, source types, hosts, and filenames are collected and sent to the indexer.
Forwarders are self contained. They do not have to be ‘installed’. They can run as any OS user.
Forwarders run on Windows, *NIX, and OSX
[monitor:///.../opmn]disabled = falsesourcetype = OPMNLogsindex=euebpignoreOlderThan = 7d
[monitor:///.../Apache/access_log*]disabled = falsesourcetype = ApacheAccessLogsindex=euebpignoreOlderThan = 7d
[script://$SPLUNK_HOME/etc/apps/<appName>/bin/rcat.sh] disabled = false host = rmanhostindex = main interval = 30 #frequency to run the script, in seconds sourcetype = RMAN
11/17/2016 28www.datavail.com
Garbage Collection Monitor Use Case
Based On jstat
jstat –gcutil <ospid>
Use perl to parse for pid and format into this.
11-03-2016 07:05 OSPid=28733 Proc=forms-c4ws_server1 S0=0.00 S1=8.99 E=56.21 O=66.80 P=85.68 YGC=83
YGCT=13.092 FGC=21 FGCT=80.713 GCT=93.80511-03-2016 07:05
11/17/2016 29www.datavail.com
Implementing the Script
Create the perl script
Create the shell script
Move scripts into place
Add script stanza to the input
11/17/2016 30www.datavail.com
Activate the Script
Inputs.conf Stanza
[script://$SPLUNK_HOME/etc/apps/<appName>/bin/jstat.sh]
disabled = false
host = rmanhost
index = main interval = 30 #frequency to run the script, in seconds
sourcetype = RMAN
11/17/2016 31www.datavail.com
Graph The Results
How to Get StartedIf I Don’t Have Splunk?
11/17/2016 33www.datavail.com
Splunk Installation Reqs
Splunk can be downloaded for free. You will be limited to 500mb of ingested data a day. This is actually can be sufficient for one environment if you are judicious about what you log.
Splunk for *NIX can be downloaded in tar ball format. This can be unpacked in any directory and does not have to be installed by or run by root.
Splunk stores everything in its self contained path, so you just have to delete the directory to remove it.
You can install Splunk directly on the system you wish to monitor (not always a good idea)
Splunk uses ports 8000 and higher for the browser and the forwarders so again, no root user is needed.
11/17/2016 34www.datavail.com
Existing Splunk?
If your company already uses Splunk you can ask your admin for the following.
• Your own index (The index is actually the directory structure and files Splunk uses to store the data).
• Additional forwarders, or…
• If your server already has a forwarder on it, you just need to get your log locations added to inputs.conf and sent to your index.
• Your own Application. An application is just a collection of settings like searches, chart and report descriptions and etc. This way you won’t interfere with the network and security guys.
Amazon Echo Giveaway
Recommended