Upload
conner-swann
View
303
Download
1
Embed Size (px)
Citation preview
YOU’VE GOT JUNK IN YOUR SPLUNK
Conner SwannNAU Information Technology Services
YOU’VE GOT JUNK IN YOUR SPLUNK - THE PROBLEM
WHAT IS THE PROBLEM?
▸ Most enterprise data is machine-generated
▸ Machine data is often-times not human readable
▸ Numerous disparate data sources and formats
▸ Different implementations and architectures
▸ Virtualized Applications
▸ 3rd Party Off-Site Solutions (“The Cloud”)
▸ On-Site Hardware
YOU’VE GOT JUNK IN YOUR SPLUNK - THE PROBLEM
SERIOUSLY? THIS IS A PROBLEM?
▸ Dan the developer is asked to help figure out why his code is crashing on Sundays at Midnight
▸ Sally the SysAdmin has no idea why users from one office location can’t log in to their computers
▸ Ivan the InfoSec Analyst has no idea a hacker in Bulgaria is sending spam from his servers
▸ Billy the Business Analyst needs to figure out what localities are using his company’s applications
▸ Molly the Marketing Executive needs to analyze her affiliate marketing campaigns to see if improvements can be made
YOU’VE GOT JUNK IN YOUR SPLUNK - THE PROBLEM
YES, IT’S A PROBLEM.
▸ Machine Data is the most rapidly growing and complex segment of “Big Data”
▸ It’s generated 24/7/365 by nearly every device in existence and will continue to be generated forever
▸ Contains categorical record of every activity and behavior
▸ Value from this data is largely untapped — extremely difficult to process and analyze in a timely manner by traditional means
YOU’VE GOT JUNK IN YOUR SPLUNK - THE DATA
SOMETHING’S GOT TO GIVE - UNDERSTANDING IMPORTANT DATA
▸ Business Application Data
▸ Relational Data, highly structured, inflexible schema
▸ Financial Records, multidimensional data, computationally intense at times
▸ Rare reports, never realtime
YOU’VE GOT JUNK IN YOUR SPLUNK - THE DATA
SOMETHING’S GOT TO GIVE - UNDERSTANDING IMPORTANT DATA
▸ Human Generated Data
▸ Created as a result of Human-Human interaction
▸ Email, IM, Voice, Text, Video
▸ Stored in central corporate data centers, on mobile devices and on individual PCs
YOU’VE GOT JUNK IN YOUR SPLUNK - THE DATA
SOMETHING’S GOT TO GIVE - UNDERSTANDING IMPORTANT DATA
▸ Machine Data
▸ Time series, diverse, unstructured, no predefined all-encompassing schema
▸ Encapsulates Human Generated Data
▸ Generated by all IT systems
▸ Absolutely ridiculous volume of data
YOU’VE GOT JUNK IN YOUR SPLUNK - MACHINE DATA
WHAT DOES “MACHINE DATA” LOOK LIKE?
2015-10-17 13:08:51-0700 [SSHService ssh-userauth on HoneyPotTransport,2323,93.158.203.167] login attempt [root/12345] succeeded
64.242.88.10 - - [07/Mar/2004:16:05:49 -0800] "GET /twiki/bin/edit/Main/Double_bounce_sender?topicparent=Main.ConfigurationVariables HTTP/1.1" 401 12846
{"created_at":"Mon Sep 28 19:39:04 +0000 2015”,”user”:”yourbuddyconner", "id":648582717068587000,"id_str":"648582717068587009","text":"The amount of local news stations treating the Facebook outage as news is too damn high. #FacebookDown #TwitterIsUp #Facebook”,"entities":{"hashtags":[{"text":"FacebookDown","indices":[89,102]},{"text":"TwitterIsUp","indices":[103,115]},{"text":"Facebook","indices":[116,125]}],"symbols":[],"user_mentions":[],"urls":[]}}
message_id=53088 timestamp="2015-02-03 20:30:06" date_read="2015-02-03 20:29:20" is_from_me=1 is_read=1 handle=+9999999999 service=iMessage message="I mean, I can, those pancakes were so good"
Honeypot Logs:
Webserver Logs:
Tweets:
Text Messages:
SERVICE NAME
USERNAME PASSWORD STATUS MESSAGEIP ADDRESS
HTTP METHOD
TIMESTAMP TWITTER HANDLE
HASHTAGS
PHONE NUMBER
MESSAGE
TIMESTAMP
YOU’VE GOT JUNK IN YOUR SPLUNK - THE SPLUNK
ENTER SPLUNK
YOU’VE GOT JUNK IN YOUR SPLUNK - THE SPLUNK
WHAT THE HECK IS SPLUNK?
▸ Splunk consumes text and provides insights about the data contained within
▸ Splunk stores your historical data and allows you to look at how the baselines have changed over time
▸ Splunk helps identify anomalies which might affect business decisions
▸ Splunk allows people who know their data to share it with people who don’t
YOU’VE GOT JUNK IN YOUR SPLUNK - THE SPLUNK
WHAT THE HECK IS SPLUNK?
REACTIVE
PROACTIVE
SEARCH AND INVESTIGATE
PROACTIVE MONITORING AND
ALERTING
OPERATIONAL VISIBILITY
REAL-TIME BUSINESS INSIGHTS
YOU’VE GOT JUNK IN YOUR SPLUNK - THE FUN
NOW FOR THE FUN PART!
YOU’VE GOT JUNK IN YOUR SPLUNK - THE FUN
CASE STUDIES AND EXAMPLES
▸ 7/11 - Uses Splunk to gain a business foothold in Indonesia, predicting shopping trends based on weather, among other things
▸ Information Security - Northern Arizona University uses splunk to trace intrusion attempts across our network
▸ Conner Swann (That’s Me) - Used splunk to glean metadata from text messages
YOU’VE GOT JUNK IN YOUR SPLUNK - THE CASE STUDY (7/11)
7/11 - THE CLIMATE
▸ Expanding to a new market (2009)
▸ Had to offer an attractive alternative to existing businesses
▸ Offer local foods, became a place local teens would hang out
▸ Caused competitors to adapt to new climate, occupying new niches
YOU’VE GOT JUNK IN YOUR SPLUNK - THE CASE STUDY (7/11)
7/11 - THE PROBLEM
▸ In order to retain their new customers, the company had to offer the best fast food as well as any daily necessities customers might need
▸ Necessitates a technological solution for providing behavioral insights on consumers
▸ Original data analytics solution was rigid, involved several rounds of manual analysis
▸ Analysis took 3-6 business days to complete
▸ Promotional campaigns took ~3 months to prepare
YOU’VE GOT JUNK IN YOUR SPLUNK - THE CASE STUDY (7/11)
7/11 - THE SOLUTION
▸ 7/11 now uses Splunk for their POS analysis
▸ Assets are dynamically organized, delivering comprehensive overview of POS data from multiple perspectives
▸ System also leverages data from external systems (i.e weather, telecom)
▸ Data is processed in minutes instead of days
YOU’VE GOT JUNK IN YOUR SPLUNK - THE CASE STUDY (7/11)
7/11 - THE RESULT
▸ Promotion planning time slashed by 80% - 2 weeks
▸ All people involved have access to the same data and visualizations with little training
▸ Promotions are evaluates for effectiveness as they occur
▸ ROI is apparent
YOU’VE GOT JUNK IN YOUR SPLUNK - THE CASE STUDY (NAU INFOSEC)
NAU INFORMATION SECURITY - EXAMPLE USE CASE
▸ Information Security is best when efforts are proactive
▸ Identify unwanted activity or actors and see if that data shows up anywhere else
▸ Honeypots on the network are used to collect data about intruders
▸ That data can be used to identify other anomalous behavior
YOU’VE GOT JUNK IN YOUR SPLUNK - THE CASE STUDY (NAU INFOSEC)
HOW IT WORKS Northern Arizona University
Hacker
IP Address: 68.55.90.112
Login Attempt From:68.55.90.112
HoneyPot
LouieSuccessful Login From:68.55.90.112
SplunkAnomalous Events Detected:
68.55.90.112 Sources:
- Honeypot- Peoplesoft
YOU’VE GOT JUNK IN YOUR SPLUNK - THE CASE STUDY (NAU INFOSEC)
THE IMPACT
▸ All event detection is done in real-time
▸ Incident response occurs as the event happen
▸ Remediation is simpler than in the past
▸ Easy to share impacts with non-technical people
YOU’VE GOT JUNK IN YOUR SPLUNK - THE CASE STUDY (ME!)
TEXT MESSAGES
YOU’VE GOT JUNK IN YOUR SPLUNK - THE CASE STUDY (ME!)
TEXT MESSAGES - THE WHY
▸ Personal analytics is HUGE
▸ Look for trends in communication
▸ Shows how much inferential data can be gleaned from behavior
YOU’VE GOT JUNK IN YOUR SPLUNK - THE CASE STUDY (ME!)
TEXT MESSAGES - THE HOW
▸ Extracted messages from iPhone backup’s SQLite database
YOU’VE GOT JUNK IN YOUR SPLUNK - THE CASE STUDY (ME!)
TEXT MESSAGES - THE RESULTS
▸ Average sentiment of outgoing texts over time
▸ index=text_messages is_from_me=1 | sentiment twitter message | timechart avg(sentiment) as sentiment span=1mon
▸ Conclusion: Sentiment fluctuates over time
YOU’VE GOT JUNK IN YOUR SPLUNK - THE CASE STUDY (ME!)
TEXT MESSAGES - THE RESULTS
▸ Average sentiment of outgoing texts with baseline over time
▸ index=text_messages is_from_me=1 | sentiment twitter message |eval diff=sentiment-0.788400| eval count=count| timechart avg(diff) as sentiment, count span=14d
▸ Conclusion: Sentiment might correlate with life events and text message frequency
YOU’VE GOT JUNK IN YOUR SPLUNK - THE CASE STUDY (ME!)
TEXT MESSAGES - THE RESULTS
▸ Comparing incoming sentiment with outgoing sentiment
▸ index=text_messages is_from_me=0 | sentiment twitter message | eval diff=sentiment-0.788400 | timechart avg(diff) as sentiment_from span=1mon | appendcols [search index=text_messages is_from_me=1 | sentiment twitter message | eval diff2=sentiment-0.788400 | timechart avg(diff2) as sentiment_me span=1mon]
▸ Conclusion: Outgoing sentiment is at times closely coupled with incoming sentiment
YOU’VE GOT JUNK IN YOUR SPLUNK - CONCLUSION
PUT SOME JUNK IN YOUR SPLUNK!
▸ Splunk is free to play with
▸ (Developer Licenses are easy to come by)
▸ http://www.splunk.com/
▸ Provide value to the shareholders!