Upload
firstmark
View
325
Download
6
Tags:
Embed Size (px)
DESCRIPTION
Sumo Logic CEO Vance Loiselle presented at September 2014's edition of Data Driven NYC. Sumo Logic turns machine data into smart decisions.
Citation preview
Sumo Logic Confidential
400+ Industry-Leading Customers
3
Leading InvestorsExperienced Team15 of the Global 500
500% Bookings Growth
200% Customer Growth
98% Renewal Rate
Applications Mobile Internet of ThingsNetwork and Server
10TB/Day 200+ TB/Day 10TB/Day 40TB/Hour
Machine Data Is Everywhere
Sumo Logic Confidential6
Issues with machine data analysis
Sumo Logic Confidential5
2013-10-29 19:11:42,010 -0700 ERROR [hostId=somehost-2] [module=STREAM] [localUserName=fdsfa] [logger=streasdf_pipasdfine.csharp.ogsfdtors.AbstsdfgourceOperator$$anon$1] [thread=RawOutputProcessor-Session-9CCBF82C187-1] [auth=User:[email protected]:00000000qw53rA:000safdadasf000:false:fdfaulmUer] [sessionId=C6D689147BD] [remote_ip=12.212.42.3] [web_session=18ipejcn...] [module=strater] unexpected exception caught while processing element PLUS com.somecomp.util.csharp.caching.DiskStoreDeletedException: Disk store at temp/cached-output/cache-1384532089902--789623452346869513425600 has already been deleted at com.caching.IndexedDiskStore.append(IndexedDiskStore.csharp:42) at com.somecomp.adh_pipe.glue.ElementStore.addElement(ElementStore.csharp:75) at com.somecomp.adh_pipe.glue.CachedOutputWire.addElement(CachedOutputWire.csharp:164) at com.somecomp.adh_pipe.glue.CachedOutputWire.send(CachedOutputWire.csharp:129) at com.somecomp.adh_pipe.core.Producer.output(Producer.csharp:117) at com.somecomp.adh_pipe.csharp.operators.LookupOperator.processPlus(LookupOperator.csharp:239) at com.somecomp.adh_pipe.csharp.operators.LookupOperator.process(LookupOperator.csharp:261) at com.somecomp.adh_pipe.glue.DefaultDataFlowWire.send(DefaultDataFlowWire.csharp:30) at com.somecomp.adh_pipe.core.Producer.output(Producer.csharp:117) at com.somecomp.adh_pipe.csharp.operators.KeyValueOperator.processPLUS(KeyValueOperator.csharp:210) at com.somecomp.adh_pipe.csharp.operators.KeyValueOperator.process(KeyValueOperator.csharp:220) at com.somecomp.adh_pipe.glue.DefaultDataFlowWire.send(DefaultDataFlowWire.csharp:30) at com.somecomp.adh_pipe.core.Producer.output(Producer.csharp:117) at com.somecomp.adh_pipe.csharp.operators.AbstractSourceOperator.protected$output(AbstractSourceOperator.csharp:50) at $processInternal(AbstractSourceOperator.csharp:50) at com.somecomp.adh_pipe.csharp.operators.AbstractSourceOperator$$anon$1.com$somecomp$adh_pipecom.somecomp.adh_pipe.csharp.operators.AbstractSourceOperator$$anon$1$$anonfun$processOnOutputThread$1.apply(AbstractSourceOperator.csharp:38) at com.somecomp.util.csharp.choose(FeatureFlag.csharp:20) at com.somecomp.adh_pipe.csharp.operators.AbstractSourceOperator$$anon$1.processOnOutputThread(AbstractSourceOperator.csharp:38) at com.somecomp.adh_pipe.csharp.operators.SingleThreadedOutputStream$$anon$1$$anonfun$$init$$1.apply(SingleThreadedOutputStream.csharp:31) at $runAndLogException(ExecutionContextRunnableWrapper.csharp:32) at com.somecomp.util.csharp.context.ExecutionContextRunnableWrapper$$anonfun$run$1.apply$mcV$sp(ExecutionContextRunnableWrapper.csharp:24) at com.somecomp.util.csharp.context.ExecutionContextRunnableWrapper$$anonfun$run$1.apply(ExecutionContextRunnableWrapper.csharp:24) at $1.apply(SingleThreadedOutputStream.csharp:31) at $runAndLogException(ExecutionContextRunnableWrapper.csharp:32) at com.somecomp.util.csharp.context.ExecutionContextRunnableWrapper$$anonfun$run$1.apply(ExecutionContextRunnableWrapper.csharp:24) at com.somecomp.util.csharp.context.RichExecutionContextThreadLocal$.doInExecutionContext(RichExecutionContextThreadLocal.csharp:19) at com.somecomp.util.concurrent.BlockingThreadPoolExecutor$1.run(BlockingThreadPoolExecutor.csharp:53) at csharp.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.csharp:1145) at csharp.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.csharp:615) at csharp.lang.Thread.run(Thread.csharp:724)
Huge Volume (V #1)– 1TB = 2 Billion Events
High Variability (V #2)– Largely unstructured and schema free
Tremendous Velocity (V #3)– Most valuable right when it is generated
2013-10-29 18:14:05,164 -0700 WARN [hostId=receiver2] [module=receiver] [localUserName=cqmerger] [logger=adh_pipe.operators.Select] [auth=User:[email protected]:00000000000170E9:DefaultSumoSystemUser] [module=cqmerger] Error while processing element Type: PLUS - Tuple:foster web marketing::::0000000000036FE0::::::::::::5.54599925E8 com.somecomp.adh_pipe.glue.Warning: cannot process null $class.safeInvoke(Function.csharp:27) at com.somecomp.adh_pipe.csharp.evaluators.Method$2.safeInvoke(MethodResolver.csharp:165)
169.107.162.237 - - [Wed Oct 30 01:50:38 UTC 2013] "GET www.somecomp.com/form/submit/includes/follow/follow_us.php HTTP/1.1" 503 1566 "http://www.google.com" "Mozilla/5.0 (Linux; U; Android 2.3.4; en-us; SCH-R720 Build/GINGERBREAD) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1"
2013-10-29 18:21:01,464 -0700 INFO [hostId=search3] [module=STREAM] [thread=MTP-Session-AC84E072A559FA03-1] [sessionId=A1B3909A5] [explainPlan] exiting search, customerId=000000000000, queryId=AC84E072A559FA03, exitCode=0, query='DefaultPerCustomerLagTracker "current lag” message='Finished successfully', executionStartTime='2013-10-29 18:21:00,579', executionEndTime='2013-10-29 18:21:01,464', executionDuration=885, millisToFirstCount=-1, firstBucketEndTime='N/A', firstBucketDelta=-1, costToStream=4
[**] [1:254:4] DNS SPOOF query response with TTL of 1 min. and no authority [**] [Classification: Potentially Bad Traffic] [Priority: 2] 08/23-18:26:59.915786 172.68.10.13:63 -> 12.10.20.49:39291 UDP TTL:64 TOS:0x0 ID:10725 IpLen:20 DgmLen:97
What machine data can tell you
Sumo Logic Confidential6
1999-09-07 13:44:11 192.168.1.179 1001 ORDER/ OrderConfirmation &5&UPS+-+US+2nd+Day+Air&10806&$893.48&Supersonic+Stereo+System&/ Electronics/ Music&1&150.0000&70.0000&10338&HealthRider+Home+Pro+-+Chrome&/ Sports/ Equipment&1&543.4800&434.7800&10800-1&French+Language+Courses&/ Language+Courses&1&200.0000&125.0000 - -
2010-02-03 01:49:09.077 -0800 wafbox1 WF ALERT SQL_INJECTION_IN_PARAM 192.168.128.7 39661 192.168.132.21180 webapp1:deny_ban_dir GLOBAL LOG NONE "[type=""sql-injection-medium"" pattern=""sql-quote"" token=""' or "” Parameter=""address"" value=""hi' or 1=1--""]" POST 192.168.132.211/cgi-bin/process.cgi HTTP REQ-0+RES-0 "Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US; rv:1.8.1.20) Gecko/20081217 Firefox/2.0.0.20" 192.168.128.7 39661 Kevin http://192.168.132.211/cgi-bin/1.pl 11956 ATTACK_CATEGORY_INJECTION
Sumo Logic Confidential7
Business Metrics
ApplicationPerformance
Operations SLA
Security Posture
CustomerSupport Business
ServiceAnalytics
Characteristics of a Machine Data Solution
Sumo Logic Confidential8
Cloud
Dat
a va
lue
Time
Real-time windowSc
ale
+
Machine learning
Human expert
Applications
Mobile
Internet of Things
Network and Server
Detect
Visualize
Search
Transforming Machine Data Into Meaningful Insights
May-12
Jun-12Jul-1
2
Aug-12
Sep-12
Oct-12
Nov-12
Dec-12Jan
-13
Feb-13
Mar-13
Apr-13
May-13
Jun-13Jul-1
3
Aug-13
Sep-13
Oct-13
Nov-13
Dec-13Jan
-14
Feb-14
Mar-14
Apr-14
May-14
Jun-14Jul-1
40
2,000
4,000
6,000
8,000
10,000
12,000
0
100,000
200,000
300,000
400,000
500,000
600,000
500,000+ Queries per day
4+ PB of data scanned per day
15+ Trillion records scanned per day
500,000+ Events per Second Received
System Growth – Customer Usage
Sumo Logic Confidential10
GB/day Searches / day
The Holy Grail of IT – Anomaly Detection
Sumo Logic Confidential11
Unknown
Known Known
Unknow
n
Known
LogReduce™ - Transform Logs Into Meaningful Patterns
Sumo Logic Confidential12
We Identify the SimilaritiesMachine Data has PatternsThen Mask the Differences
2014-04-06 23:52:37 10.20.11.105 GET /Trade/StockTrade.aspx action=sell&symbol=s:156&holdingid=9875 80 Jayden 214.115.233.69 Mozilla/5.0+(Macintosh;+Intel+Mac+OS+X+10_7_3)+AppleWebKit/536.5+(KHTML,+like+Gecko)+Chrome/19.0.1084.54+Safari/536.5 200 0 0 333
2014-04-07 00:30:23 10.20.11.101 GET /Trade/StockTrade.aspx action=sell&symbol=s:142&holdingid=9066 80 Lily 219.5.73.118 Mozilla/5.0+(Macintosh;+Intel+Mac+OS+X+10_7_3)+AppleWebKit/536.5+(KHTML,+like+Gecko)+Chrome/19.0.1084.54+Safari/536.5 200 0 0 206
2014-04-06 21:23:56 10.20.11.103 GET /Trade/StockTrade.aspx action=sell&symbol=s:126&holdingid=4867 80 Hayden 233.134.69.149 Mozilla/5.0+(Macintosh;+Intel+Mac+OS+X+10_7_3)+AppleWebKit/536.5+(KHTML,+like+Gecko)+Chrome/19.0.1084.54+Safari/536.5 200 0 0 237
2014-04-06 20:58:23 10.20.11.102 GET /Trade/StockTrade.aspx action=sell&symbol=s:168&holdingid=9932 80 Thomas 150.205.10.108 Mozilla/5.0+(Macintosh;+Intel+Mac+OS+X+10_7_3)+AppleWebKit/536.5+(KHTML,+like+Gecko)+Chrome/19.0.1084.54+Safari/436.5 200 0 0 110
2014-04-06 13:33:20 10.20.11.103 GET /Trade/StockTrade.aspx action=sell&symbol=s:189&holdingid=3802 80 Harper 33.168.5.129 Mozilla/5.0+(Macintosh;+Intel+Mac+OS+X+10_7_3)+AppleWebKit/536.5+(KHTML,+like+Gecko)+Chrome/19.0.1084.54+Safari/536.5 200 0 0 398
2014-04-06 21:30:43 10.20.11.105 GET /Trade/StockTrade.aspx action=sell&symbol=s:175&holdingid=4147 80 Parker 120.22.112.139 Mozilla/5.0+(Macintosh;+Intel+Mac+OS+X+10_7_3)+AppleWebKit/536.5+(KHTML,+like+Gecko)+Chrome/19.0.1084.54+Safari/537.5 200 0 0 398
2014-04-06 08:31:03 10.20.11.101 GET /Trade/StockTrade.aspx action=sell&symbol=s:186&holdingid=4576 80 Maya 11.208.155.200 Mozilla/5.0+(Macintosh;+Intel+Mac+OS+X+10_7_3)+AppleWebKit/536.5+(KHTML,+like+Gecko)+Chrome/19.0.1084.54+Safari/536.5 200 0 0 168
2014-04-06 19:47:23 10.20.11.103 GET /Trade/StockTrade.aspx action=sell&symbol=s:158&holdingid=3051 80 Lillian 77.167.50.152 Mozilla/5.0+(Macintosh;+Intel+Mac+OS+X+10_7_3)+AppleWebKit/536.5+(KHTML,+like+Gecko)+Chrome/19.0.1084.54+Safari/546.5 200 0 0 206
2014-04-06 16:05:48 10.20.11.105 GET /Trade/StockTrade.aspx action=sell&symbol=s:155&holdingid=8506 80 Anthony 213.173.122.155 Mozilla/5.0+(Macintosh;+Intel+Mac+OS+X+10_7_3)+AppleWebKit/536.5+(KHTML,+like+Gecko)+Chrome/19.0.1084.54+Safari/536.5 200 0 0 623
2014-04-06 23:52:37 10.20.11.105 GET /Trade/StockTrade.aspx action=sell&symbol=s:156&holdingid=9875 80 Jayden 214.115.233.69 Mozilla/5.0+(Macintosh;+Intel+Mac+OS+X+10_7_3)+AppleWebKit/536.5+(KHTML,+like+Gecko)+Chrome/19.0.1084.54+Safari/536.5 200 0 0 333
2014-04-07 00:30:23 10.20.11.101 GET /Trade/StockTrade.aspx action=sell&symbol=s:142&holdingid=9066 80 Lily 219.5.73.118 Mozilla/5.0+(Macintosh;+Intel+Mac+OS+X+10_7_3)+AppleWebKit/536.5+(KHTML,+like+Gecko)+Chrome/19.0.1084.54+Safari/536.5 200 0 0 206
2014-04-06 21:23:56 10.20.11.103 GET /Trade/StockTrade.aspx action=sell&symbol=s:126&holdingid=4867 80 Hayden 233.134.69.149 Mozilla/5.0+(Macintosh;+Intel+Mac+OS+X+10_7_3)+AppleWebKit/536.5+(KHTML,+like+Gecko)+Chrome/19.0.1084.54+Safari/536.5 200 0 0 237
2014-04-06 20:58:23 10.20.11.102 GET /Trade/StockTrade.aspx action=sell&symbol=s:168&holdingid=9932 80 Thomas 150.205.10.108 Mozilla/5.0+(Macintosh;+Intel+Mac+OS+X+10_7_3)+AppleWebKit/536.5+(KHTML,+like+Gecko)+Chrome/19.0.1084.54+Safari/436.5 200 0 0 110
2014-04-06 13:33:20 10.20.11.103 GET /Trade/StockTrade.aspx action=sell&symbol=s:189&holdingid=3802 80 Harper 33.168.5.129 Mozilla/5.0+(Macintosh;+Intel+Mac+OS+X+10_7_3)+AppleWebKit/536.5+(KHTML,+like+Gecko)+Chrome/19.0.1084.54+Safari/536.5 200 0 0 398
2014-04-06 21:30:43 10.20.11.105 GET /Trade/StockTrade.aspx action=sell&symbol=s:175&holdingid=4147 80 Parker 120.22.112.139 Mozilla/5.0+(Macintosh;+Intel+Mac+OS+X+10_7_3)+AppleWebKit/536.5+(KHTML,+like+Gecko)+Chrome/19.0.1084.54+Safari/537.5 200 0 0 398
2014-04-06 08:31:03 10.20.11.101 GET /Trade/StockTrade.aspx action=sell&symbol=s:186&holdingid=4576 80 Maya 11.208.155.200 Mozilla/5.0+(Macintosh;+Intel+Mac+OS+X+10_7_3)+AppleWebKit/536.5+(KHTML,+like+Gecko)+Chrome/19.0.1084.54+Safari/536.5 200 0 0 168
2014-04-06 19:47:23 10.20.11.103 GET /Trade/StockTrade.aspx action=sell&symbol=s:158&holdingid=3051 80 Lillian 77.167.50.152 Mozilla/5.0+(Macintosh;+Intel+Mac+OS+X+10_7_3)+AppleWebKit/536.5+(KHTML,+like+Gecko)+Chrome/19.0.1084.54+Safari/546.5 200 0 0 206
2014-04-06 16:05:48 10.20.11.105 GET /Trade/StockTrade.aspx action=sell&symbol=s:155&holdingid=8506 80 Anthony 213.173.122.155 Mozilla/5.0+(Macintosh;+Intel+Mac+OS+X+10_7_3)+AppleWebKit/536.5+(KHTML,+like+Gecko)+Chrome/19.0.1084.54+Safari/536.5 200 0 0 623
$DATE 10.20.11.10* GET /Trade/StockTrade.aspx action=***&symbol=s:**** 80 *****Mozilla/5.0+(*****)+AppleWebKit/536.5+(KHTML,+like+Gecko)+Chrome/**0.1***+Safari/***** 0 *
$DATE 10.20.11.10* GET /Trade/StockTrade.aspx action=***&symbol=s:**** 80 *****Mozilla/5.0+(*****)+AppleWebKit/536.5+(KHTML,+like+Gecko)+Chrome/**0.1***+Safari/***** 0 *
$DATE 10.20.11.10* GET /Trade/StockTrade.aspx action=***&symbol=s:**** 80 *****Mozilla/5.0+(*****)+AppleWebKit/536.5+(KHTML,+like+Gecko)+Chrome/**0.1***+Safari/***** 0 *
$DATE 10.20.11.10* GET /Trade/StockTrade.aspx action=***&symbol=s:**** 80 *****Mozilla/5.0+(*****)+AppleWebKit/536.5+(KHTML,+like+Gecko)+Chrome/**0.1***+Safari/***** 0 *
$DATE 10.20.11.10* GET /Trade/StockTrade.aspx action=***&symbol=s:**** 80 *****Mozilla/5.0+(*****)+AppleWebKit/536.5+(KHTML,+like+Gecko)+Chrome/**0.1***+Safari/***** 0 *
$DATE 10.20.11.10* GET /Trade/StockTrade.aspx action=***&symbol=s:**** 80 *****Mozilla/5.0+(*****)+AppleWebKit/536.5+(KHTML,+like+Gecko)+Chrome/**0.1***+Safari/***** 0 *
$DATE 10.20.11.10* GET /Trade/StockTrade.aspx action=***&symbol=s:**** 80 *****Mozilla/5.0+(*****)+AppleWebKit/536.5+(KHTML,+like+Gecko)+Chrome/**0.1***+Safari/***** 0 *
$DATE 10.20.11.10* GET /Trade/StockTrade.aspx action=***&symbol=s:**** 80 *****Mozilla/5.0+(*****)+AppleWebKit/536.5+(KHTML,+like+Gecko)+Chrome/**0.1***+Safari/***** 0 *
$DATE 10.20.11.10* GET /Trade/StockTrade.aspx action=***&symbol=s:**** 80 *****Mozilla/5.0+(*****)+AppleWebKit/536.5+(KHTML,+like+Gecko)+Chrome/**0.1***+Safari/***** 0 *
Sumo Logic Confidential
Anomaly Detection – Expose Unknown Events In Real-Time
Log Signatures follow patterns
Sumo Logic Confidential
Anomaly Detection – Expose Unknown Events In Real-Time
Changes in patterns indicate an anomaly
Change in Signature pattern
A new signatureemerges
LogReduceTM and Anomaly Detection Analytics
16
Search
Aggregate
LogReduce™
Anomaly
500,000
50,000
20
1
Sumo Logic Confidential
Global Intelligence
Sumo Logic Confidential17
20122010 2015Log Collection
SearchDashboards
Apps SDK
LogReduceScale
Cloud
Dashboards
Anomaly Detection
Applications
Expert Community
Collaborative Analytics
Early Warning System
Cloud?
Correlation
2004
Competition