Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/

GAINING APPLICATION LIFECYCLE

INTELLIGENCE WITH SPLUNK

By Damien Dallimore , Dev Evangelist @ Splunk

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/

Who am I ?

2

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/3

From Middle Earth

Make things

JVM background

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/4

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/5

apps.splunk.com

github.com/damiendallimore

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/

Agenda

6

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/7

Overview of Splunk and build a simple app

How Splunk can help in the Application Development Lifecycle

Various ways to get data into Splunk and demos

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/

Data Data Everywhere

8

VOLUME

VARIETY

VERACITY

VELOCITY

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/

How can Splunk help ?

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/

Spelunking

10

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/

Platform for machine data

11

Splunk storage Other Big Data stores

DeveloperPlatform

Data collection

and indexing

Report and

analyze

Custom dashboards

Monitor and alert

Ad hoc search

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/

DeveloperPlatform

Report and

analyze

Custom dashboards

Monitor and alert

Ad hoc search

Platform for machine data

12

Splunk storage Other Big Data stores

Data

collection

and indexing

Any amount, any location, any source.Schema at read time, not write time

Data in any format

No RDBMS

Very Extensible

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/

What Does Machine Data Look Like?

13

Sources

Twitter

Care IVR

Middleware Error

Order Processing

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/

Machine Data Contains Critical Insights

14

Customer ID Order ID

Customer’s Tweet

Time Waiting On Hold

Twitter ID

Product ID

Company’s Twitter ID

Sources

Twitter

Care IVR

Middleware Error

Order Processing

Customer IDOrder ID

Customer ID

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/

Machine Data Contains Critical Insights

15

Order ID

Customer’s Tweet

Time Waiting On Hold

Product ID

Company’s Twitter ID

Sources

Twitter

Care IVR

Middleware Error

Order Processing

Order ID

Customer ID

Twitter ID

Customer ID

Customer ID

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/16

How are we best

going wrangle this

data ?

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/

Release the Developers

17

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/

Very Extensible Platform for Developers

18

REST API

Build Splunk Apps Extend and Integrate Splunk

Simple XML

JavaScript

Django

Web

FrameworkJava

JavaScript

Python

Ruby

C#

PHP

Data Models

Search

Extensibility

Modular Inputs

SDKs

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/

Lets build something simple

19

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/

Simple Swarm App (ex Foursquare)

20

Get my actual checkin data in via REST

Search over this data

Visualize

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/

Application Lifecycle Data

21

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/

BuildUnit Testing

Code

Check-inIntegration

Testing Deploy

Staging

22

Application Development Challenges

Lack of visibility across the product

development lifecycle

Pressure to increase velocity and

agility with DevOps

Limited insights into behavior and

performance from application logs

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/

Quickly trace and identify errors anywhere

in the codebase with real-time search

and monitoring

Instrument your app logs to gain application intelligence

Break down dev tool silos with real-time

insights from machine data

GAIN END-TO-END VISIBILITY

ACROSS THE DEV TOOL CHAIN

FIND AND FIX

ISSUES FASTER

PUSH BETTER

CODE USING

ANALYTICS

Splunk for Application Lifecycle Intelligence

23

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/

Real-time dashboards show error

rate in production and impact of

pushing new builds

Developers can search and

visualize web logs, Java logs—

without production access

Alerts notify developers as soon as

a problem arises

24

Find and Fix Issues Faster

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/

Gain end-to-end visibility to make

informed decisions

Analytics insights without the need

for additional analytics tools

Ask questions while exploring and

collecting data

void submitPurchase(purchaseId){

log.info("action=submitPurchaseSt

art, purchaseId=%d", purchaseId)

//these calls throw an exception on error

submitToCreditCard(...)

generateInvoice(...)

generateFullfillmentOrder(...)

log.info("action=submitPurchaseCo

mpleted, purchaseId=%d", purchaseId)

}

25

Push Better Code Using Analytics

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/

End-To-End Visibility Across The Dev Tool Chain

26

CI / Build

Servers

Project and

Issue Tracking

Code

Repository

QA / Testing

ToolsDeployment

Servers

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/

App Development Lifecycle Demo

27

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/

Getting your data into Splunk

28

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/29

Log DataLog Files

Splunk Logging Appenders

CodingSplunk Java SDK

Splunk Spring Integration Adaptors

JMX

MessagingJMS

AMQP w/Rabbit

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/

Log Data

30

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/

Standard Log Files

31

Oct 21, 2013 4:42:15 PM org.apache.catalina.startup.Catalina load

INFO: Initialization processed in 1153 ms

Oct 21, 2013 4:42:15 PM org.apache.catalina.core.StandardService startInternal

INFO: Starting service Catalina

Application logs that are part of the product

Developer logs for any code that was deployed

Written to local disk or network storage

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/

Structured and Unstructured Data

32

Ideally events are in a best practice semantic format

key=value format , JSON

You can perform index time and search time extractions in Splunk

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/

Logging best practices

33

Clearly timestamp every event , human readable at beginning of line

Log in text , binary needs decoding

Categorize – Use INFO, WARN, ERROR, DEBUG, Event type etc...

Log unique identifiers

Log anything that can add value when aggregated, charted or further

2012-08-07 15:54:06:644+1200 name="Failed Login" event_id="someID" app="myapp" user="jane" somefieldname="foobar"

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/

SplunkJavaLogging

34

Sometimes you can’t write to file

Appenders for Java Util Logging , Log4J , Logback

Simply add a logging appender to your logging configuration

file

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/

LogBack Appender Example

35

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/

Code

36

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/

Better Exception Logging

37

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/

Easier to work with in Splunk

38

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/

Coding

39

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/

Splunk SDK for Java

40

Use the SDK from any JVM Language , Java / Groovy / Scala etc….

Send log events via REST , UDP or TCP directly to Splunk from your

code

Search over data in Splunk

SDK available from dev.splunk.com

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/41

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/

Spring Integration Adaptors

42

Inbound Adapter

Used to execute Splunk searches and get data out

Outbound Adapter

Write data to Splunk via REST, TCP , UDP

Write to a named index, submit a REST request, write to a data input bound

to a server TCP port

Get the code on Github

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/

Inbound Adaptor

43

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/

Outbound Adaptor

44

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/

JMX

45

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/

What is JMX

46

JMX = Java Management Extensions

Monitor JVM via MBean attributes , operations and

notifications

JVM MBeans

Vendor MBeans

Custom Coded MBeans

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/

Getting this data into Splunk

47

Runs on all supported Splunk platforms

Works with all main JVM variants

100% Free and Open Source

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/

Simple to Configure

48

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/

Many Connectivity Options

49

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/

Messaging

50

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/

JMS

51

Not a messaging protocol , but a programming interface to many

different underlying message providers

WebsphereMQ , Tibco EMS , ActiveMQ , HornetQ , SonicMQ etc…

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/

AMQP

52

Built with Rabbit Java Client Library

AMQP 0.9.1, 0.9, 0.8

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/

But wait , there’s more…..

53

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/54

Poll data from any REST API

Pull data directly off the wire

Capture output from executing any commands

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/

Closer look at capturing command output

55

Let’s see what the host Operating System can tell us : top

External programs that provide additional JVM insights : jstat

Index this data in Splunk and correlate

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/

top

56

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/57

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/

jstat

58

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/59

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/60

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/

Splunk options galore

61

Splunk> Enterprise : Free to download and use. Index 500 MB/day.

Splunk> Cloud : Premium, cloud hosted. Full Enterprise stack.100% uptime.

Splunk> AMI : BYOL versions for Amazon AWS Cloud.

Splunk> Sandbox : Spin up a cloud instance in minutes.Load in data.

Hunk> : Splunk for data in Hadoop HDFS , MongoDB

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/

More Info

62

Splunk Docs , Downloads , Vids : http://www.splunk.com

Download Splunk Apps : http://apps.splunk.com

Ask : http://answers.splunk.com

Watch the App Dev / Devops Video : http://www.splunk.com/goto/appdev

Splunk Developer Platform : http://dev.splunk.com

Splunk on Github : https://github.com/splunk

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/

Thankyou.

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/

Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a

Creative Commons Attribution-NonCommercial l icense: http://creativecommons.org/licenses/by-nc/3.0/

Questions ?@damiendallimore

ddallimore@splunk.com