• Home

  • Technology

  • Five key learnings from building a large scale anomaly detection system
12
1 Page Number www.anodot.com |Confidential Eliminating Insight Latencies… One Anomaly at a Time Ira Cohen, Chief Data Scientist [email protected]

Five key learnings from building a large scale anomaly detection system

Tags:

Embed Size (px)

Citation preview

Page 1: Five key learnings from building a large scale anomaly detection system

1Page Numberwww.anodot.com |Confidential

Eliminating Insight Latencies…One Anomaly at a Time

Ira Cohen, Chief Data [email protected]

http://www.anodot.com/
Page 2: Five key learnings from building a large scale anomaly detection system

2Page Numberwww.anodot.com |Confidential

What is the problem?

Business Insight Latency

Delayed business insights cost companies millions of dollars

http://www.anodot.com/
Page 3: Five key learnings from building a large scale anomaly detection system

3Page Numberwww.anodot.com |Confidential

Business insights at any granularity

Publishers Ad exchanges DevicesConversion Ad types X XX

User types Countries RegionPurchases Item categories X XX

OS Browsers Software verPage views Pages X XX

Sheet type Locations CustomerPrint errors Printer type X XX

Measuring KPIs with many dimensions

Millions of time series metrics to track

http://www.anodot.com/
Page 4: Five key learnings from building a large scale anomaly detection system

4Page Numberwww.anodot.com |Confidential

How to Track the Millions and Get the Insights?

Automated Anomaly Detection

Aggregate, Detect and Group

http://www.anodot.com/
Page 5: Five key learnings from building a large scale anomaly detection system

5Page Numberwww.anodot.com |Confidential

Realtime Anomaly Detection: Design Principals

• Collect metrics from ANY source Universal• Realtime tracking of millions of metricsScale• Unique IP for anomaly detection that eliminates FP Accurate• Multivariate anomalies for swift problem isolation Concise

http://www.anodot.com/
Page 6: Five key learnings from building a large scale anomaly detection system

6Page Numberwww.anodot.com |Confidential

Automatic Anomaly Detection in Five Steps

Normal behavior learning

Abnormal behavior learning

Behavioral Topology Learning

Detection, scoring and grouping anomalies

Feedback based learning

http://www.anodot.com/
Page 7: Five key learnings from building a large scale anomaly detection system

7Page Numberwww.anodot.com |Confidential

Automatic Anomaly Detection in Five Steps

Normal behavior learning

- Adaptive- Detects and learns seasonal patterns + trends- Auto-configuration- Model selection algorithm

http://www.anodot.com/
Page 8: Five key learnings from building a large scale anomaly detection system

8Page Numberwww.anodot.com |Confidential

Automatic Anomaly Detection in Five Steps

P(Anomaly | Duration, Deviation)

Bayesian model combining statistics of:- Abnormal durations- Abnormal intensity- Abnormal pattern

Abnormal behavior learning

http://www.anodot.com/
Page 9: Five key learnings from building a large scale anomaly detection system

9Page Numberwww.anodot.com |Confidential

Automatic Anomaly Detection in Five Steps

- Normal behavior relationship learning- Abnormal behavior relationship learning- Meta-data based relationship learning

Behavioral Topology Learning

http://www.anodot.com/
Page 10: Five key learnings from building a large scale anomaly detection system

10Page Numberwww.anodot.com |Confidential

Automatic Anomaly Detection in Five Steps

Detection, scoring and grouping anomalies

Feedback based learning

90

http://www.anodot.com/
Page 11: Five key learnings from building a large scale anomaly detection system

11Page Numberwww.anodot.com |Confidential

Anodot - Summary

• Founded June 2014• Beta since Jan 2015, GA in Sep• Over 20 million metrics tracked• Over 12,000 real-time insights provided to over 200 users • To try us out, contact: [email protected]• Follow us on twitter: @TeamAnodot

http://www.anodot.com/
mailto:[email protected]
mailto:[email protected]
Page 12: Five key learnings from building a large scale anomaly detection system

12Page Numberwww.anodot.com |Confidential

THANK [email protected]@TeamAnodot

http://www.anodot.com/

CUSUM Anomaly Detection - Measurement Lab · Abstract Introduction Background Anomaly Detection in Network Traffic Flows CUSUM Charts CUSUM Anomaly Detection (CAD) Applying CUSUM

CUSUM Anomaly Detection - Measurement Lab · Abstract Introduction Background Anomaly Detection in Network Traffic Flows CUSUM Charts CUSUM Anomaly Detection (CAD) Applying CUSUM

Documents
Science of Anomaly detection - v4 Updated for HTM for IT · The#Science#of#Anomaly#Detection!! ! Page 3! Anomaly!Detection!inStreamingData" Early anomaly detection in streaming data

Science of Anomaly detection - v4 Updated for HTM for IT · The#Science#of#Anomaly#Detection!! ! Page 3! Anomaly!Detection!inStreamingData" Early anomaly detection in streaming data

Documents
Anomaly Detection in Predictive Maintenance - KNIME · 1 Anomaly Detection in Predictive Maintenance Time Alignment and Visualization for Anomaly Detection Rosaria Silipo Rosaria.Silipo@knime.com

Anomaly Detection in Predictive Maintenance - KNIME · 1 Anomaly Detection in Predictive Maintenance Time Alignment and Visualization for Anomaly Detection Rosaria Silipo [email protected]

Documents
Comparison of Unsupervised Anomaly Detection Techniques · a RapidMiner [10] Extension Anomaly Detection was developed that contains several unsupervised anomaly detection techniques

Comparison of Unsupervised Anomaly Detection Techniques · a RapidMiner [10] Extension Anomaly Detection was developed that contains several unsupervised anomaly detection techniques

Documents
Anomaly Detection Tutorial

Anomaly Detection Tutorial

Documents
Survey Anomaly Detection

Survey Anomaly Detection

Documents
Spacecraft anomaly detection via transformer ...€¦ · measures, anomaly detection is important and necessary to warn operation engineer of anomalies. Current anomaly detection

Spacecraft anomaly detection via transformer ...€¦ · measures, anomaly detection is important and necessary to warn operation engineer of anomalies. Current anomaly detection

Documents
Anomaly Detection: Principles, Benchmarking, Explanation ...tgd/talks/dietterich-anomaly... · Anomaly Detection: Principles, Benchmarking, Explanation, and Theory Tom Dietterich

Anomaly Detection: Principles, Benchmarking, Explanation ...tgd/talks/dietterich-anomaly... · Anomaly Detection: Principles, Benchmarking, Explanation, and Theory Tom Dietterich

Documents
Online Nonparametric Anomaly Detection in High-Dimensional ... · 13/2/2018 · Online Nonparametric Anomaly Detection in High-Dimensional Datasets Online Nonparametric Anomaly Detection

Online Nonparametric Anomaly Detection in High-Dimensional ... · 13/2/2018 · Online Nonparametric Anomaly Detection in High-Dimensional Datasets Online Nonparametric Anomaly Detection

Documents
09 anomaly detection

09 anomaly detection

Education
Real-Time Anomaly Detection in Water Distribution Networks ... Anomaly Detection in Water Distribution Networks using Spark Streaming ... 2.3 Anomaly Detection in Water Distribution

Real-Time Anomaly Detection in Water Distribution Networks ... Anomaly Detection in Water Distribution Networks using Spark Streaming ... 2.3 Anomaly Detection in Water Distribution

Documents
Anomaly Detection and Attribution Using Bayesian … · UNCLASSIFIED DSTO{TR{2975 Anomaly Detection and Attribution Using Bayesian Networks Executive Summary Anomaly detection techniques

Anomaly Detection and Attribution Using Bayesian … · UNCLASSIFIED DSTO{TR{2975 Anomaly Detection and Attribution Using Bayesian Networks Executive Summary Anomaly detection techniques

Documents
Anomaly detection- Credit Card Fraud Detection

Anomaly detection- Credit Card Fraud Detection

Science
Anomaly Detection Petty

Anomaly Detection Petty

Documents
Anomaly Detection Technology Using BigGraph€¦ · Anomaly Detection Technology Using BigGraph ... detection Several anomaly detection techniques have been developed and applied

Anomaly Detection Technology Using BigGraph€¦ · Anomaly Detection Technology Using BigGraph ... detection Several anomaly detection techniques have been developed and applied

Documents
Anomaly Detection Systems

Anomaly Detection Systems

Documents
L14. Anomaly Detection

L14. Anomaly Detection

Data & Analytics
BGP Anomaly Detection

BGP Anomaly Detection

Internet
Deep Learning based Anomaly Detection in€¦ · Deep Learning based Anomaly Detection in Multivariate Time-Series Data 1. Background Anomaly detection, also known as outlier detection,

Deep Learning based Anomaly Detection in€¦ · Deep Learning based Anomaly Detection in Multivariate Time-Series Data 1. Background Anomaly detection, also known as outlier detection,

Documents
Anomaly detection final

Anomaly detection final

Engineering
Anomaly Detection Techniques

Anomaly Detection Techniques

Documents
Anomaly Detection Survey

Anomaly Detection Survey

Documents
Network Traffic Anomaly Detection and Characterizationcs.northwestern.edu/~ajb200/anomaly detection paper 1.0.pdf · Network Traffic Anomaly Detection and Characterization Aaron Beach

Network Traffic Anomaly Detection and Characterizationcs.northwestern.edu/~ajb200/anomaly detection paper 1.0.pdf · Network Traffic Anomaly Detection and Characterization Aaron Beach

Documents
Anomaly Detection RapidMiner

Anomaly Detection RapidMiner

Documents
Anomaly detection char10

Anomaly detection char10

Technology
Anomaly Detection using Outlier Detection Schemes

Anomaly Detection using Outlier Detection Schemes

Documents
Intrusion & Anomaly Detection & Preventiongauss.ececs.uc.edu/Courses/c6056/pdf/intrusion.pdfIntrusion & Anomaly Detection & Prevention Intrusion Detection: Monitor events, analyze

Intrusion & Anomaly Detection & Preventiongauss.ececs.uc.edu/Courses/c6056/pdf/intrusion.pdfIntrusion & Anomaly Detection & Prevention Intrusion Detection: Monitor events, analyze

Documents
Chap10 Anomaly Detection

Chap10 Anomaly Detection

Technology
Anomaly Detection

Anomaly Detection

Documents
Anomaly Detection in Predictive Maintenance - KNIME · Anomaly Detection in Predictive Maintenance Anomaly Detection with Time Series Analysis

Anomaly Detection in Predictive Maintenance - KNIME · Anomaly Detection in Predictive Maintenance Anomaly Detection with Time Series Analysis

Documents