PbSbi/hl bi & Publish/Subscribe & Big Event Data for …vleung/CVWS2012/Toronto/Presentations/H...• Materialized views Static queries View M Filters Notifications Hybrid data store

P bli h/S b ib & Publish/Subscribe & Big Event Data for gSmart Traffic Management

Hans-Arno Jacobsen

MIDDLEWARE SYSTEMSRESEARCH GROUP

MSRG.ORG

Connected Vehicles for Smart Transportation (CVST)

MIDDLEWARE SYSTEMSRESEARCH GROUPMSRG.ORG

2

Escaping Gridlock

Smart Traffic Management

Escaping GridlockWhat’s the solution to

Toronto’s traffic problems?By John Lorinc

• Traffic monitoring▫ Highly dynamic systemsHighly dynamic systems▫ Detailed data from different sources▫ Need notifications, filtering, and analytical processing

• Traffic monitoring queries▫ What is the traffic density at College & Spadina?▫ How many passengers are in the Spadina street cars?▫ What is the road condition on Dundas?▫ How to reroute traffic in an incident situation?

Workshop on Connected Vehicles – Middleware Systems Research Group, msrg.org

How to reroute traffic in an incident situation? ▫ How to adjust the timing of traffic lights during rush hours?


3

CVST Project Scope & Our Objectives

• Build an open & flexible applications platform for connected vehicles & smart for connected vehicles & smart transportation systems

• Focus on the following tasks▫ Open, secure, and privacy preserving event p p y p g

processing & storage platform▫ Real-time event management capabilities

E d i i d fil i


▫ Event detection, aggregation, and filtering


4

Smart Traffic Management

LTE 4G

G hi

Internet

DSRCGeographic Information

l i l

Filter


Traffic DatabaseAnalytical

QueriesNotification


5

Traffic Data (Toronto)( )• 2.000 signalized intersections1

• 20.000 street intersections1Data sources

• GPS – rate up to 1/sec• Crowd sourcing• 18.000 accidents with injuries

20102

• 1.5 million TTC trips daily3

illi ( %

• Crowd sourcing• Position, Time, Speed

• Road sensor info• Surface temperature

Vi ibilit• 2.4 million commuters (70 % car)4

• Up to 30% cyclists and rising4

• Visibility• Water film height• Freezing temperature• ...

ffi

• Big data challenge• Traffic cameras

• Red light cameras


1Conelly et al.: http://www.energydigital.com/company-reports/stacey-electric-company2Campbell et al.: Road to Health: A Healthy Toronto by Design Report. Toronto, 20123TTC General Information - http://www3.ttc.ca/Routes/General_Information/General_Information.jsp4Majority of Toronto commuters still get in cars to get to work: census - http://www.cbc.ca/news/canada/toronto/story/2008/03/04/car-toronto.html


6

Performance Requirementsq• High event rates▫ Millions events / sec

S d▫ GPS, sensor data• High query rates▫ Thousands queries / secQ i ( li i i li i )• Queries (explicit & implicit)▫ Filtering▫ Notifications

Analytical Source: http://theroadtochangeindia wordpress com/2011/01/13/better roads/▫ Analytical• Big data▫ ~ 250 Bytes per record (location, speed, weather, …)▫ ~ 250 MB / sec

Source: http://theroadtochangeindia.wordpress.com/2011/01/13/better-roads/


▫ ~ 250 MB / sec▫ ~ 600 TB / month


7

Our Research Questions in CVST

• How to cope with the enormous data rates?▫ Need for a highly scalable architecture▫ Need for a highly scalable architecture

• How to answer frequently asked queries efficiently?efficiently?▫ In memory storage and materialized views

• Which queries to materialize?q▫ Dependent on the query frequency

• How often to update materialized views?


▫ Dependent on the response time requirement


8

Directions & ChallengesgMADES (event storage) PADRES (event processing)

• Multi-layered Adaptive • Publish/Subscribe platform • Multi-layered, Adaptive, Distributed Event Storage

• Storage of high volume, high insertion rate measurement

Publish/Subscribe platform for event dissemination

• Aggregation in Publish/Subscribeinsertion rate measurement

event data▫ Fast access to recent event

data

▫ Minimum, maximum, average of matching events in given time frame

Stable communication overlay data▫ Analytical queries on

historic event data▫ Adaptive storage in cloud

• Stable communication overlay in unstable environment▫ Clients (e.g., cars, mobile

phones) continuously join


Adaptive storage in cloud environment

p ) y jand leave


9

MADES



10

MADES Project (Event Storage)j ( g )• Current systems’ performance▫ TPC-C results ~ 500K tx/ sec

KV lt 200K / 1▫ KV results ~ 200K ops / sec1

• Need for a new architecture▫ Multi-layered, Adaptive, Multi layered, Adaptive,

Distributed Event Storage▫ Highly scalable▫ High write throughput

St ti iModern data store performance for write-heavy workloads 2• 200K inserts on 12 Cassandra nodes1

Hi h f l t▫ Static queries▫ Analytical queries

Hybrid key-value store

• High performance cluster•Min 60 nodes to sustain 1 M inserts

• without any analysis• Netflix: 300 AWS nodes for 1 M writes2


Hybrid key value store

1Cockcroft: Global Netflix – Replacing Datacenter Oracle with Global Apache Cassandra on AWS. HTPS 20112Rabl et al.: Solving Big Data Challenges for Enterprise Application Performance Management. VLDB 2012

3• 280$ / h → 2.5M$ / y


11

MADES Architecture• Short-term data layer (on-line data)• Long-term data layer (historical data)

Raw Data Stream

•GPS Info•Traffic sensorsView1

(1st Replica)

Raw Data Stream

CompressedData Stream

StoreAdaptive ResourceAllocationView2

(2nd Replica)On-Line Stores

Event


Clients VisualizationDissemination


12

MADES Architecture• Materialized views▫ Static queries View

MStatic queries▫ Filters▫ NotificationsHybrid data store

ManagerNotifications

• Hybrid data store▫ All nodes are equal▫ DHT style inserts In-Memory

Storage

MessageBroker

Filter

▫ Replication for static data▫ Current data in-memory▫ Aggregated data in disk storage

EntryLog

Storage

Disk Storage

Incremental TransformationAnalytical


Aggregated data in disk storage▫ Asynchronous processing

Disk StorageQueryEngine


13

Schema Excerpt• Measurements▫ Has type (e.g., numeric value)

( ffi li h )

p

Measurementvalue

▫ Has source (e.g., a traffic light)▫ Can be aggregated

M t i

min_valuemax_valueno_pointsstart_timeend time

Sourcesource_id

source_namesource_type

• Metric▫ Type of measurement▫ Defines threshold

end_timemetric_id

Metric

Source_Typeid

source_id

location

• Source▫ Generator of measurement▫ Has a type (e g road sensor)

Metricmetric_id

metric_type

name

threshold


▫ Has a type (e.g., road sensor)▫ Can be aggregated


14

Materialized View

What is the average traffic at light XY?

SELECT source_name, AVG(value), dayFROM Measurement ms, Metric mt, Source s

WHERE ms metric id mt metric idWHERE ms.metric_id = mt.metric_idAND ms.source_id = s.source_idAND mt.metric_type = “traffic”AND ms.start_time BETWEEN “27/10/2011” AND “28/10/2011”AND s.source_name = “XY”



15

Materialized Views

What is the average traffic at light XY?

Measurementvalue

min_valuemax_value

Sourcesource_id

source_nameAVG T ffino_points

start_timeend_timemetric_idsource id

source_typelocation

AVG_Trafficsource_id

source_nameavg_value

time frame

Metricmetric_id

metric type

Source_Typeid

name

source_id _fmetric_type


metric_typethreshold


16

PADRES



17

PADRES Project (Event Processing)j ( g)• Current system▫ Allows clients to publish events & subscribe to eventsAllows clients to publish events & subscribe to events▫ Offers a rich subscription language enabling Fine-grained filtering Event correlation Event correlation Detection of composite events

• Need for extensionsC i f ▫ Continuous streams of events

▫ Aggregation of events over time & space▫ Time-sensitive processing & real-time


p g▫ Security and privacy


18

PADRES ESB for Event Managementg• Enterprise Service Bus (Events and Services Bus)• First generation of students▫ Peng, Alex, David, aRno, Eli, Serge

• Padres is Publish/subscribe Applied to Distributed Resource SchedulingW b t t d d l d• Web start and download▫ padres.msrg.org▫ Implemented in Java

Open so rce nder EPL 1 0▫ Open-source under EPL 1.0• Acknowledgements (2004-2010):



19

Publish/Subscribe Abstraction

DDPublish/SubscribeBDB

Publish/Subscribe


Traffic Database


20

P bli h

PADRES ESB is an Overlay of P/S Brokers

P

S

= publisher

= subscriber

SP/S Brokers

Matching

S

B

Matching Engine

Routing +Publications

BBP

input queue

output queue dest2

output queue dest3

dest1 subscription dest

Routing Table

service time < 3s dest2

output queue dest1

BS

service time < 2s dest3

service time = 2.5sservice time = 1sservice time = 3s



21

Innovative Features HistoricAccess

CompositeEvents

Access

CSRG TR 2009ACM DEBS’2007

A

B CD

EF

Events

ManagementACM Middleware’2004IEEE ICDCS’2005

ACM Middleware’2007

IEEE ICDCS’2009ACM Middleware’2008

ACM Middleware 2007

ACM DEBS’2007

Robustness

Load

SecurityIEEE ICDCS’2010ACM Middleware’2006


LoadBalancing


22

CVST – Big Pictureg• Communication▫ PADRES

View1(1st Replica)

• Storage▫ MADES

• Data sourceGPS d

On-Line Stores

View2(2nd Replica)

▫ GPS data• Analysis▫ Predict traffic

conditionsPADRES

conditions• Prediction▫ Autonomic

transportation

Clients


transportation management

AnalysisPrediction

HistoricStore


23

Team

• Dr Tilmann Rabl • Chen Chen Ph D candidateDr. Tilmann Rabl▫ MADES & materialized

views• Kaiwen Zhang, Ph.D.

Chen Chen, Ph.D. candidate▫ PADRES & overlay

construction• Kianoosh Mokhtarian, Ph.D. g

candidate▫ PADRES & aggregation in

pub/subM S dh i Ph D did t

candidate▫ PADRES & event

disseminationR Sh f t Ph D did t• Mo Sadhogi, Ph.D. candidate

▫ PADRES & MADES • Young Yoon, Ph.D. candidate▫ PADRES & overlay re

• Reza Sherafat, Ph.D. candidate▫ PADRES & reliability

• Rija Javdi, M.A.Sc. Candidate▫ MADES & materialized


▫ PADRES & overlay re-configuration

▫ MADES & materialized views


24

Thank youy

Questions?


Documents

PbSbi/hl bi & Publish/Subscribe & Big Event Data for …vleung/CVWS2012/Toronto/Presentations/H...• Materialized views Static queries View M Filters Notifications Hybrid data store