Upload
john-lau
View
201
Download
1
Tags:
Embed Size (px)
Citation preview
John K.S. Lau & Assoc Prof C.K. Tham , ECE, NUSE-mail: {elelksj, eletck}@nus.edu.sg
Hidden Markov Models for Abnormal Event Processing in Transportation Data Streams
IEEE ICPADS’12 International Workshop on Scalable Computing for Big Data Analytics (SC-BDA)
Hidden Markov Models for Abnormal Event Processing in Transportation Data StreamsJKS Lau & CK Tham 2
Motivation
How to leverage computationally expensive model in detecting abnormal events in real-time?
Hidden Markov Models for Abnormal Event Processing in Transportation Data StreamsJKS Lau & CK Tham 3
Contributions
•Application of Hidden Markov Models (HMM) in identifying not only hidden meaningful events but also abnormal events. The focus in this work is on abnormal events.
•Event-Driven Architecture (EDA) for decoupling of HMM computations from real-time event processing.
Hidden Markov Models for Abnormal Event Processing in Transportation Data StreamsJKS Lau & CK Tham 4
Event-Driven Architecture (EDA)
Hidden Markov Models for Abnormal Event Processing in Transportation Data StreamsJKS Lau & CK Tham 5
Abnormal Event Processing (AEP)
Hidden Markov Models for Abnormal Event Processing in Transportation Data StreamsJKS Lau & CK Tham 6
Abnormal Event Processing (AEP)
Hidden Markov Models for Abnormal Event Processing in Transportation Data StreamsJKS Lau & CK Tham 7
Abnormal Event Processing (AEP)
•SELECT tmin, tmax FROM [table name] WHERE [unique identifier]–tmin and tmax pre-computed via HMM based on most
recent data. Normal event is within tmin <= t <= tmax.
•SELECT * FROM [stream name] HAVING duration < tmin OR duration > tmax
–Any incoming event is flagged abnormal in real-time.
Hidden Markov Models for Abnormal Event Processing in Transportation Data StreamsJKS Lau & CK Tham 8
Experimental Results
HMM
Find highest probability
Hidden Markov Models for Abnormal Event Processing in Transportation Data StreamsJKS Lau & CK Tham 9
Experimental Results
Hidden Markov Models for Abnormal Event Processing in Transportation Data StreamsJKS Lau & CK Tham 10
Experimental Results
Hidden Markov Models for Abnormal Event Processing in Transportation Data StreamsJKS Lau & CK Tham 11
Experimental Results
Hidden Markov Models for Abnormal Event Processing in Transportation Data StreamsJKS Lau & CK Tham 12
Experimental Results
Hidden Markov Models for Abnormal Event Processing in Transportation Data StreamsJKS Lau & CK Tham 13
Future Works
•Ability to detect more meaningful hidden states such as accident, heavy rain, flood, etc. besides just general anomaly.
•Application of Map-Reduce in HMM computations.
Hidden Markov Models for Abnormal Event Processing in Transportation Data StreamsJKS Lau & CK Tham 14
THANK YOUQuestions?