If you can't read please download the document
Upload
damien-dallimore
View
5.242
Download
0
Embed Size (px)
Citation preview
PowerPoint Presentation
Using the Splunk Java SDKPresented by Damien DallimoreDeveloper Evangelist at Splunk
Copyright 2012 Splunk Inc.
About meDeveloper Evangelist at Splunk since July 2012http://dev.splunk.com http://splunk-base.splunk.comSlides available for my Splunking the JVM session Splunk Community MemberSplunk4JMXSplunkJavaLoggingSplunkBase AnswersSplunk Architect and Administrator Coder, hacker, architect of Enterprise Java solutions around the globe in many different industries(aviation, core banking, card payments etc)Yes, I do have an accent , so please restrain all your sheep, Lord of the Rings and Kim Dotcom heckles until beer oclock
2
AgendaOverview of the Splunk PlatformREST API & SDKsJava SDK overviewCode, Code, Code !Thinking outside the SquareAlternate JVM LanguagesMaking it easier for developers to log to SplunkSplunk is not just for ProductionQuestions (feel free to yell out at any time also)
Accelerate development & testing with proactive monitoring
Integrate data from Splunk into your existing IT environment for operational visibility
Quickly deliver real-time business insights from Big Data outside of IT
4REST APIs
Splunkd
Custom/Existing Applications
SDKsSearch, chart and graphSave and schedule searches as alertsExport search resultsManage inputs and indexesAdd & remove users and roles
Splunk & Developers
SplunkUI(Splunk Apps)
Machine Data
4
REST API & SDKs
What you can do with the SDKs & APIIntegrate with third-party reporting tools and portals Log directly to Splunk Integrate Splunk search results into your applicationExtract data for archiving, compliance Build a custom UI of your choice 6
Splunk REST APIExposes an API method for every feature in the productWhatever you can do in the UI you can do through the API.Run searchesManage Splunk configurationsAPI is RESTfulEndpoints are served by splunkdRequests are GET, POST, and DELETE HTTP methodsResponses are Atom XML Feeds JSON coming in 5.0Versioning coming in 5.0Search results can be output in CSV/JSON/XML
7
Language SDKsThe SDKs make it easier for you to use the raw REST API , abstracting away much of the lower level plumbing, so you can instead just focus on developer productivityHandling HTTP accessAuthenticatingManaging namespacesSimplifying access to REST endpointsBuilding the correct URL for an endpointDisplaying simplified output for searchesInput of data to a Splunk indexPython, Java, Javascript in beta - SupportedPHP available now!Still study the core REST API though , if youre anything like me you like to know what is going on under the hood
8
Java SDK overview
Java SDK Design PrinciplesProvide comprehensive coverage of the REST APIHave a 1:1 mapping of endpoint to class in the SDKProvide implementation that felt intuitive to a Java developer Lowest common denominator for build ANT (Any maven people out there?)Project support for Eclipse and IntelliJ to ease getting started10
Get the Java SDK setupOpen sourced under the Apache v2.0 licenseClone from Github : git clone https://github.com/splunk/splunk-sdk-java.gitCurrent release status is betaProject level support for Eclipse and Intellij IDEsI use Eclipse with the eGit pluginPre-requisitesJRE 6+Ant (builds, javadoc generation)Splunk installedRun the unit tests and examplesSetup a .splunkrc file in your users home directoryRun an Ant buildRun examples with the command line wrappersRun the Junit tests from Ant or within your IDE
11
Key Java SDK ConceptsNamespacesowner : splunk usernameapp : app contextsharing : user | app | global | systemDefaults to current user and default appService classInstantiate an object to connect and loginEntry point for REST API callsClient/Server stateNeed to maintain state explicitlyupdate() : to push changes to splunkdrefresh() : to get changes from splunkd
12
Java SDK Class Model13ServiceResourceResourceCollectionEntityEntityCollectionApplicationIndexHTTPServiceInputInputCollectionSavedSearchCollectionCollections use a common mechanism to create and remove entitiesEntities use a common mechanism to retrieve and update property values, and access entity metadataService is a wrapper that facilitates access to all Splunk REST endpoints
public String codeTime(){ return Lets Rock n Roll;}
Connecting / Authenticating15
Simple Entity Retrieval16
Logging Events via HTTP REST17
Uses receivers/simple endpointUses receivers/stream endpoint
Logging Events via Raw TCP18
If you dont already have a TCP port listening, simply create one via the REST API
SetupLog to SplunkTeardown
Searching OverviewSearch querya set of commands and functions you use to retrieve events from an index or a real-time stream , "search * | head 10".Saved searcha search query that has been saved to be used again and can be set up to run on a regular scheduleSearch joban instance of a completed or still-running search operation.Using a search ID you can access the results of the search when they become available. Job results are saved for a period of time on the server and can be retrieved Search ModesNormal : asynchronous , poll job for status and resultsBlocking : synchronous , a job handle is returned when search is completedOneshot : synchronous , no job handle is returned, results are streamedExport : synchronous, not a search per say, doesnt create a job, results are streamed oldest to newest
Heaps more juicy examples here : http://dev.splunk.com/view/SP-CAAAEHQ
19
For those searches that stream the results (oneshot and export), the search results are not saved. If the stream is interrupted for any reason, the results are not recoverable without running the search again.19
Blocking Searches20
No Job is createdA Job is created
20
Non-Blocking Search21
There is code in the develop branch (which we should probably push into main before .conf) that obviates the need for job.refresh()
isDone() and isReady() refresh behind your back.21
Non-Blocking Search (with Paging)22
maxresultrows in Splunk config default 50KNot recommended to change thisIf result set > 50K , then page through results
In order to get all events, you have to use the export endpoint. But the export endpoint has different behavior than a normal job. An export cannot be "restarted" when getting events if the network hiccups. A search job can just do another getResults() with the appropriate offset this is because the export endpoint doesn't save the results like a search job does. But a search job has a limited number of events it will store on the server which can be affected by status_buckets but there is no way to guarantee the upper limit. With the default status_buckets we can get to 500K events. Itay and I experimented with hundreds of stratus_buckets but were only to get up to about 1M events, out of 13M available events.22
Realtime Search23
Saved Search24
Processing CSV/JSON/XML results25
Client/Server State26
Namespaces27
Thinking outside the square
Alternate JVM LanguagesScalaGroovyClojureJavascript(Rhino)JRubyPHP(Quercus)CeylonKotlinJython
29We dont need SDKs for these languages , we can just use the Java SDK !
Scala SDK30
Groovy SDK31
SplunkJavaLoggingA logging framework to allow developers to as seamlessly as possible integrate Splunk best practice logging semantics into their code.Custom handler/appender implementations(REST and Raw TCP) for the 3 most prevalent Java logging frameworks in play. Splunk events directly from your code.LogBackLog4jjava.util.loggingImplementation of the SPLUNK CIM(Common Information Model)32
Developers just log as they are used to332012-08-07 15:54:06:644+1200 name="Failed Login" event_id="someID" app="myapp" user="jane" somefieldname="foobar"
BetterA-HA
Logging Framework takes care of the Splunk transport , REST or Raw TCP34
Typical Java Stacktraces in logs35
SplunkJavaLogging is your friend36
Java Stacktraces in Splunk37
Use Splunk in dev/test => better quality delivered to prod 38Its not good enough to assert that your software is production ready because the load test doesnt make it fall overSplunk the apps machine data throughout the lifecycle of the testVia a Splunk SDK, enrich your test harness pass/fail assertions with outputs of Splunk searchesCatch malignant code tumors that may have flown under the radar that your harness alone couldnt possibly know about.Grinder + Java SDK + Assertions
Contact DetailsAlways more than happy to be contacted for questions, feedback, collaborations, ideas that will change the world etc
Email : [email protected]: damiendGithub: damiendallimoreTwitter : @damiendallimoreBlog : http://blogs.splunk.com/devSplunk Dev Platform Team : [email protected]
39
LinksGists for all code examples : https://gist.github.com/damiendallimoreJava SDK Homepage : http://dev.splunk.com/view/java-sdk/SP-CAAAECNJava SDK Github repository : https://github.com/splunk/splunk-sdk-javaSplunkJavaLogging : https://github.com/damiendallimore/SplunkJavaLoggingSplunk Best Practice Logging : http://dev.splunk.com/view/logging-best-practices/SP-CAAADP6Splunk REST API : http://docs.splunk.com/Documentation/Splunk/latest/RESTAPI/RESTcontents
40
//Thanks for coming !System.exit(5150);