Splunk for JMX

Splunk for JMX

Damien DallimoreDeveloper Evangelist

Copyright © 2013, Splunk Inc.

Splunk for JMX

2

• Connect to any local or remote JVM's JMX server, Hotspot/JRockit/IBM J9/OpenJDK/Azul Zing

• Query any MBean running on that server

• Extract any MBean attributes (simple, composite or tabular)

• Invoke MBean operations

• Write attributes and operation results out in a default key/value format, or plugin your own custom format, for SPLUNK indexing and searching

• Transport events over STD OUT(default), TCP, Syslog, Splunk REST endpoint or direct to file.

• Declare clusters of JVM's for larger scale JVM deployments

• Runs on *Nix and Windows

• Out of the box dashboards for common JVM MBeans

• Freely available from SplunkBase, all source code is on GitHub


Connectivity Options

3

• Remote JMX interface– rmi (JSR160 Standard Implementation and MX4J's JSR160 Implementation)– iiop (JSR160 Standard Implementation and MX4J's JSR160 Implementation)

• Direct Process attachment – Connect directly to a locally running JVM process

• MX4J HTTP connectors (requires MX4J in the target JVM also)– soap , soap+sssl– hessian, hessian+ssl– burlap, burlap+ssl


Setup and Configuration

4

• The main goal of the app was to make it as simple and intuitive as possible to connect to your JVMs and start Splunking JMX data

• Enable your target JVM’s remote JMX interface , test connectivity with JConsole• Install Splunk for JMX• Extract Splunk for JMX tarball to SPLUNK_HOME/etc/apps• Restart Splunk• At the setup screen, choose a scripted input for your platform (Nix / Windows)

• Setup your JMX configuration file• The default config.xml file is pre configured for common JVM MBeans• Browse your JVM (using JConsole) for other MBeans that you wish to poll and configure these• You can have as many config files as you require, and you might set these up to fire off at different

scheduled frequencies


Configuration Examples - Simple

5


Configuration Examples - Wildcards

6

MBean Object name format “domain:key=value,key2=value2”

Around 25KBytes per dump on Hotspot JVMs

Don’t know the domain or properties name or have many Mbeans with the same attributes ?

* and ? wildcards are supported in the Mbean name


Configuration Examples - Clusters

7

• Define clusters of JVM’s that share the same MBean definitions

• Note , in these examples, for brevity I am using “dumpAllAttributes” , but in production you’d want to pick and choose specific MBean attributes you are interested in, and perhaps split definitions over multiple files run at varying frequencies


Configuration Examples - Operations

8

• Invoke JMX operations that return a value or simply perform some action on the target JVM

• Operation definitions can take parameters

Use Case 1 : your developers might code a JMX operation that returns a CSV or JSON formatted snapshot of some metrics for Splunking

Use Case 2 : dynamically trigger HPROF dumps.The “com.sun.management:type=HotSpotDiagnostic” Mbean exposes a “dumpHeap” operation


Configuration Examples - Connecting

9

• IP Address with credentials• Hostname• Static Process ID• Process ID lookup from file• Process ID lookup from command output• Raw JMX Service URL• MX4J HTTP Connector


Custom Formatters/Transports

10

• The Splunk for JMX configuration is user extensible• You can code and configure your own Formatters and Transports

Formatters• Takes the raw MBean polled output and formats it for Splunking• A Java implementation of the "com.dtdsoftware.splunk.formatter.Formatter" interface• If the optional formatter declaration is omitted, then the default formatter will be used

Transports • Takes the formatted output and transports it to a destination• A Java implementation of the "com.dtdsoftware.splunk.transport.Transport" interface• If the optional transport declaration is omitted, then the default transport(STD out) will be used


Formatter Examples

11


Transport Examples

12


Deployment Architectures 1

13

• Simplest scenario• Monolithic Splunk installation• Splunk for JMX polling 1 or more remote/local JVMs

via the remote JMX interface• There is support for many target JVM’s in the

configuration schema but to really scale out, you need a more advanced Splunk architecture


Deployment Architectures 2

14

• Run Splunk UF locally with target JVM.Can connect use remote JMX interface or direct process attachment.

• Each tier scales out horizontally.• Can overcome firewall issues that are

sometimes inherent with Java RMI• Deploy Splunk for JMX components and

configurations with Splunk Deployment Server, Puppet or Chef.

Demo

Copyright © 2013, Splunk Inc.16

Contact me

Email : [email protected] : @damiendallimoreSkype : damien.dallimoreGithub : damiendallimoreSplunkbase : damiendSlideshare : http://www.slideshare.net/damiendallimore Blogs : http://blogs.splunk.com/devWeb : http://dev.splunk.com

http://blogs.splunk.com/dev

http://dev.splunk.com/

http://dev.splunk.com/

Technology

Splunk for JMX