Complex Event Processing - A Survey

7/30/2019 Complex Event Processing - A Survey

1/7

Complex Event Processing - A Survey

V. Govindasamy and Dr. P. Thambidurai

Abstract

Complex Event Processing (CEP) deals with generating event notifications by deriving complex events fromprimitive events. The primitive events are from multiple distributed sources. In this paper, we present a survey of the recent

publications with respect to Complex Event Processing in Distributed Event Processing Systems. A comparative study of all the

major approaches are listed and discussed. Major contribution of this paper is i)Identification eight trends in CEP systems in

recent years ii) Detailed description of existing approach and iii) Comparative study of existing approaches.

Index TermsComplex Event Processing, Recent Work

1 INTRODUCTION

Database Management System (DBMS) is used for storingpersistent data and indexing data before the process of

the information. DBMS returns the result by processingthe query when explicitly asked by users. Data StreamManagement system (DSMS) is capable of performingdetection, analysis, manipulation and storage of complexevents. DSMS is developed as special DBMS. DBMSworks with persistent data, where updates are infre-quent; DSMS deals with transient data where updates arecontinuous. DBMS runs queries just once, to return acomplete answer, DSMS executes standing queries, whichruns continuously and provides updated answers whennew data arrives. DSMS system can handle both relation-al data and stream data. Windowing is a technique used

to process stream data.Publisher/subscriber systems allow users to express state-less subscriptions. They are evaluated over each eventthat arrives at the system. With the advent of event basedsystems that are highly distributed, loosely coupled com-ponents, where the publisher/subscriber middleware sys-tem is responsible for interpreting event notification.Event Stream Processing (ESP) deals with continuousqueries over data streams, often with a focus on high fre-quency events and scalability. In DSMS, the transient dataare stored, while in ESP, queries are stored instead of da-ta. The publisher events are data and subscriptions arequeries on data. The subscription is matched with thepublished events either by topic, type or content.ComplexEvent Processing (CEP) [11] is about inferring patternsfrom continuously arriving data. The CEP engine ex-amines continuously arriving events from various sourcesand detects patterns of interest and notify to the end us-ers. Event patterns are detected by combining simpleevents to composite or complex events. A simple event isa set of attributes with timestamp. The complex eventquery or pattern is represented as a SQL like la nguage

which is transformed into CEP operators.CEP differs from DSMS in their dedication to detect event

patterns rather than process streams in general. CEP sys-tems focused on optimizing parameters, like bandwidthutilization and end-to-end latency, which were usuallyignored in DSMSs. CEP systems are in relation with pub-lisher subscriber system in terms of scalability and per-formance.

1.1Features Of CEP SystemsInput: The data streams are continuous, infinite, and vola-tile. Data Streams exhibits strong temporal qualities. DataStreams are from more than one sourceQueries: The queries are in real time and process large

number of events. They are interested in newly arrivedevents and not on historical data.

1.2 Different Types of ApplicationsThe types of applications are i) Responsive Application,ii) Reactive Application, iii) Proactive Application. A res-ponsive application is the one where a student is query-ing the web in order to get material needed for writing aschool assignment. Many consumer and enterprise appli-cations fall under this category where users get quotefrom insurance company, query a data warehouse for asummary of sales in a certain segment.A reactive application detects a traffic jam, either by asingle camera observation (simple event), or by summingup the number of vehicles that enter a road segment with-in a certain time interval (a complex event), that reacts bychanging a street light schedule.The third is an example of a proactive application: a tech-nician is delayed at a customers house, and because thereis also a traffic jam, the technician is expected to miss thenext scheduled customer he should visit. While this hasnot happened yet, we wish to eliminate this anticipatedevent, by rescheduling the planning of all the techniciansin a team.

V. Govindasamy is with Department of Computer Science and Engineer-ing, Pondicherry Engineering College,Pondicherry-605011,India.

Dr. P. Thambidurai is Principal of Perunthalaiver Kamarajar Institute ofTechnology, Karaikal-609603,Pondicherry,India

JOURNAL OF COMPUTING, VOLUME 5, ISSUE 4, APRIL 2013, ISSN (Online) 2151-9617

https://sites.google.com/site/journalofcomputing

WWW.JOURNALOFCOMPUTING.ORG 1


2/7

1.3 Generic Architecture of CEP System

Fig.1

The Fig.1 highlights the generic architecture of CEP sys-tem where in EPA means Event Processing Agent. TheEPA may be a Filter agent, Transformation agent or Pat-tern Detection Agent.

2 RECENT WORKS IN DATA EVENTPROCESSING SYSTEMS

2.1 Rule ManagementHannes Obweger et al[2] has proposed a rule manage-ment framework which provides distinction between in-frastructure rules and respond rules in the business logicto provide flexibility and agility. The requirements forinfrastructural rules are

1. Expressiveness,2. Efficiency of use,3. Full and system-wide access.

The requirement for sense and respond rules are,

1. Decoupling,2. Reusability,3. Ease of use,4. Hot deployment,5. Security.

Yulia Turchin et al[3], has presented a framework for au-tomating domain expert rule specification using intelli-gent techniques for specification of rule parameters. Adiscrete Kalman filter algorithm is designed to determineand tune the parameter values.

2.2 Geospatial Distributed Event Based SystemsGeospatial distributed event based systems [12] provides

a general model for detection of events with network ofsensors during disaster.Michael Olson et al [13] analyzed streams of data fromlarge numbers of heterogeneous sensors to detect raregeospatial events.

2.3 Event Based SystemCordiesCordies [16] is a distributed system for the detection ofcorrelated events. CORDIES is designed for the operationin large-scale, heterogeneous networks. It adapts dynami-cally to changing network conditions.Cayuga

Cayuga [19] is a newer event processing system in devel-opment at Cornell University. The system can be used todetect event patterns in event streams. The Cayuga sys-tem is designed to leverage traditional publica-tion/subscription techniques to allow for high scalability.CoDA

Context Delivery Architecture (CoDA) [20] is aimed forservice oriented or event-driven applications, where thefunctioning of the software elements (services or eventhandlers) is determined by the input to the element, andthe context.

2.4 LanguagesTESLA

TESLA [18] [1] is a CEP language. TESLA provides a highdegree of expressiveness to users while keeping a simplesyntax with a rigorously defined semantics. In particular,TESLA provides selection operators, parameterization,

negations, aggregates, sequences, iterations, and fullycustomizable event selection and consumption policies.Further TESLA supports reactive and periodic rule evalu-ation within a common syntax.CUDA

Compute Unified Device Architecture (CUDA) [8] is ageneral purpose parallel computing programming. It of-fers a new parallel programming model and instructionset for general purpose programming on GraphicalProcessing Units (GPUs). CUDA Content-based Matcher(CCM) algorithm has been composed of two phases: i) aconstraint selection phase and, ii) a constraint evaluationand counting phase. When an event enters the engine, thefirst phase is used to select each attribute of the eventwith all the constraints having the same name. These con-straints are evaluated in the second phase, using the valueof the attribute.

2.5 Proactive Event Based SystemProactivity refers to the ability to mitigate or eliminateundesired future events, or to identify the advantage offuture opportunities by applying prediction and auto-mated decision making technologies. We investigate anextension of the event processing conceptual model andarchitecture with respect to proactive event-driven appli-

cations, and propose the main building blocks of a novelarchitecture. We first describe several extensions to theexisting event processing functionality that is required tosupport proactivity. In the next step, we extend the event





3/7

processing agent model to include two more type ofagents: i) Predictive agents that may derive future uncer-tain events based on prediction models, and ii) Proactiveagents that compute the best proactive action that shouldbe taken.Vinod Muthusamy et al [4] in their paper has designed apublisher/subscriber system that will predict an eventwill match in future. Using these knowledge applicationscan take proactive steps to prevent the situation or initiatesteps to react to the situation. A new algorithm based onMarkov model has been designed in this regard.Yagil Engel et al [5] introduce a high level conceptual ar-chitecture for proactive event driven computing.K. R. Jayaram et al [17] proposes a event based distributedsystem that analyze and separates related and unrelatedevents.

2.6 Uncertainty in Event Based SystemWaldemar et al [26] designed a model which captures

specifics of different variants of distributed event-basedsystems. In particular, the model is derived from variousprevious publications in five sub-areas such as (1) event-driven interaction paradigms (EDIP), (2) event streamprocessing (ESP), (3) complex event processing (CEP), (4)event-driven monitoring networks and wireless sensornetworks (WSN), and (5) event-driven business processmanagement (EDBPM).Alexander Artikis et al [27] describes that events may oc-cur during an interval, or more generally inside a unionof intervals. Second, the time in which the event occurredor even the fact that it occurred at all might be uncertain.

Amirhossein Malekpour et al [23] proposed a publish-er/Subcriber API that recovers the frequently lost messag-es. Diallo et al [24] the subscriber meets the requirementsof the publisher.Dan OKeeffe et al *28+ proposes a mechanism to detect amissing events, that the publishers to attach sequencenumbers to each event that they send. By examining thesequence numbers of events received, the mediator candetermine when events sent by a publisher have beenlost.Gabriella Tth et al [9] proposed for a synergy betweenCEP and Predictive Analysis. The Predictive Analysis part

contains a machine learning component.Segev et al [25] define the representation and semantics ofevent composition for probabilistic settings and hasshown how to apply these extensions to the quantifica-tion of the occurrence probability of events. These resultsenable any active system to handle such uncertainty.

2.7 Semantics and Context based EventProcessing SystemKia Teymourian et al [21] proposes huge amounts of do-main background knowledge stored in external know-ledge bases can be used in combination with eventprocessing in order to achieve more knowledgeable com-plex event processing.The ALERT [29] system is an active collaboration plat-form, in which a virtual actor that interacts with the de-velopers, processes and recognizes various kinds of inte-ractions and suggests actions on the basis of these, thusenabling developers to work better together.

2.8. Quality MeasuresScalability

Ella Rabinovich et al [6], presents a novel approach forpattern rewriting that aims at efficiently processing pat-terns which comprise all levels of complexity A cost mod-el based on balancing processing latency and eventthroughput according to user's preference is given. Pat-tern cost is then estimated using simulation-based tech-niques.SangJeong Lee et al [7], advocates for an event centricsharing approach to handle scalability issues for multiple

queries and multiple event sources. They have proposedfor a novel data structure Event-Centric ComposableQueue (ECQ) to build a single shared network wheremultiple queries are evaluated. Zhaoran Wang et al [10],shows near-linear scalability on a commodity multi-coremachine based on DFGS mechanism.Flexibility

Naomi Seyfer et al [14] present the stream-relational lan-guages, both visual and textual, modular abstraction ben-efits from type parameterization, the ability to capturetypes from the inputs, and the ability to manipulate un-expected fields opaquely.

3. SUMMARY OF VARIOUS TECHNIQUES

Paper Topic Technique Data Set Metric FutureEnhancements

1 Complex EventProcessing withTREX

TESLA Events Throughput,Completeness, lowprocessing time

Distributed eventprocessing, sca-lability

2 Rule Managementframework

SeparatesInfrastructurerules and

sense andrespond rules

NA NA NA

3 Automated ruleSpecification

DiscreteKalmann

DARPA-Intrusion

NA Find good initialparameter values





4/7

filter to de-termine andtune parame-ter values

detectionsystem

4 Prediction ofmatching sub-

scription in future

Algorithmbased on

Markovmodel

Syntheticdata set

Precision To handle impre-cisely defined

subscription

5 Prediction inevent processingsystem

High levelconceptualarchitecture

NA NA NA

6 Pattern rewritingfor agility andcalability

Policy basedpattern re-writing

Simulation Throughput, laten-cy, tradeoff,sensitivity analysis

Heuristic basedapproach for se-lection of rewrit-ing alternative

7 Event centric

sharing approachfor scalability

Event centric

composablequeue datastructure

Synthetic

data set

Processing and

storage cost, scala-bility

Nested queue

monitoring

8 Use of CUDAarchitecture

CUDA con-tent basedmatchingalgorithm

NA Event processingtime, Speed up

Probalistic Inexatmatching

9 Synergy betweenCEP and PA

PA usingmachinelearning

NA NA NA

10 Multi-Core BasedMessage Broker

Deadline-aware Fine-

GrainedSchedulingmechanism

Predicatesand

attributes

Processing time,event throughput,

resource allocation

NA

11 Complex EventProcessing

XChange NA NA Formal, languageexpressiveness,query optimiza-tion

12 Towards a Discip-line of GeospatialDistributed EventBased Systems

With clus-tered sensorsand MonteCarlo Simula-tion

Clustereddata set

Time, speed andaccuracy

NA

13 Rapid Detectionof Rare GeospatialEvents: Earth-quake WarningApplications

Sensor-sidePicking Algo-rithms:STA/LTA(Short TermAverage overLong TermAverage)

Simulation NA NA

14 Capture FieldsModularity in aStream-RelationalEvent Processing

Langauge

Stream BaseTrading Sys-tem Frame-work

NA Flexibility Schema Inherit-ance, SeparateCompilation,Runtime Type

Parameterization,Down casting inExtension Points

15 Controlled Eng- event vocabulary Flexibility NA





5/7

lish Language forProduction andEvent ProcessingRules

processingnetwork

16 Cordies:Expressive event

correlation in dis-tributed

Correlationdetection

algorithm

Syntheticdata

Latency, efficiencyand resource con-

sumption

NA

17 Program Analysisfor Event-basedDistributed Sys-tems

The EventJa-va runtimeframework

analysis Delay, throughput,latency, spuriousevents.

NA

18 TESLA: A Formal-ly Defined EventSpecification Lan-guage

Event detec-tion algo-rithm basedon automata

attributes Expressive, flexible combine lan-guage expres-siveness withsystem scalability

19 Cayuga: A Gener-al Purpose Event

Monitoring Sys-tem

Query engine Syntheticdata

Consume lessmemory

NA

20 Context AwareComputing andits utilization inevent-based sys-tems

NA NA NA NA

21 Fusion of Back-ground Know-ledge and Streamsof Events

NA Syntheticdatasets

Throughput Plan based ap-proach, optimiza-tion

22 Safe Autonomous

Transport Ve-hicles

Bayesian Fil-

ter

Occupancy

grid

Safety NA

23 Content-BasedPublish/SubscribeNetworks

Messageorientedmiddleware,end-to-endmethod

NA Reliability,throughput, delayreduction,recovery.

NA

24 Information Over-load-Aware Con-tent-BasedRouting

IOA-CBR Datasets Better trade-off NA

25 A Model for Rea-

soning with Un-certain Rules inEvent Composi-tion Systems

Probabilistic

Reasoning

Datasets Active Prediction of fu-

ture events

26 Deriving a Uni-fied Fault Tax-onomy for Event-Based Systems

Taxonomy Events,classes

Fault tolerance Machine Learn-ing techniquesfor monitoring

27 Event ProcessingUnder Uncertain-ty

ProbLog ECProgram

Variables Weight, noise NA

28 Reliable Complex

Event Detectionfor PervasiveComputing

NFP Detector events Correctness

ALERT: Semantic BTS Sensors events NA NA





6/7

29 event-driven col-laborative plat-form for softwaredevelopment

4. DISCUSSIONS AND CONCLUSION

In this paper, we have examined the research

area of CEP and other related works such as DSMS,

ESP, from which the CEP has been derived from. In

addition, we have studied various techniques, lan-

guages used by various researches. We have discussed

in detail how they were implemented in CEP. In con-

clusion, it is evident that this area is going to play a

vital role in future in the fields of software engineering,

software maintenance and IT operations.Rule Management [2] provides a framework

of tools and workflows for modeling processing logicand business logic based on a unified rule-evaluationmodel. Also the degree of control and complexity isadjusted to the needs and skills of the different usergroups in an enterprise. Training [3] is beneficial andcontinuous tuning can compensate for changes in dy-namic environments. Automatic tuning becomes bene-ficial in settings where manual tuning of a complex setof parameters becomes a cognitive challenge.

The algorithms [12] for rapid detection ofgeospatial events which is used on Cloud computingarchitectures and many servers collaborate to detect

events by analyzing data streams from large numbersof sensors. It [13] is a community-based event detec-tion system, in which individual members of the com-munity can install and operate sensors and responders.It can help society to deal with disasters collective-ly.The event based systems [16, 19, 20] has been devel-oped to detect the correlated events based on inputand output. Also languages [1, 8, and 18] provide bet-ter expressiveness and computing process of events.

REFERENCES[1] GianpaoloCugola, Alessandro Margara, Complex Event

Processing with T-REX, Journal of Systems and Software,

Volume 85 Issue 8, Pages 1709-1728, ACM, 2012,

[2] HannesObweger, Josef Schiefer, SzabolcsRozsnyai, User-Oriented Rule Management for Event-Based Applica-

tions,Proceedings of the 5th ACM international conference

on Distributed event-based system , Pages 39-48, ACM,

2011.

[3] YuliaTurchin, Avigdor Gal, SegevWasserkrug, TuningComplex Event Processing Rules using the Prediction Cor-

rection Paradigm, Proceedings of the Third ACM Interna-

tional Conference on Distributed Event-Based Systems, Ar-

ticle No. 10, ACM, 2009.

[4] VinodMuthusamy, Haifeng Liu and Hans-Arno Jacobsen,Predictive Publish/Subscribe Matching, Proceedings of

the Fourth ACM International Conference on Distributed

Event-Based Systems, Pages 14-25, ACM, 2010.

[5] Yagil Engel, OpherEtzion, Towards Proactive Event-Driven Computing, Proceedings of the 5th ACM interna-

tional conference on Distributed event-based system, Pages

125-136,ACM, 2011.

[6] Ella Rabinovich, OpherEtzion, Avigdor Gal, Pattern Re-writing framework for Event Processing Optimization,

Proceedings of the 5th ACM international conference on

Distributed event-based system, Pages 101-112,ACM, 2011.

[7] SangJeong Lee, Youngki Lee, Byoungjip Kim, K. Seluk-Candan, Yunseok Rhee, Junehwa Song, High-

Performance Composite Event Monitoring System Support-

ing Large Numbers of Queries and Sources, Proceedings of

the 5th ACM international conference on Distributed event-

based system, Pages 137-148,ACM, 2011.

[8] Alessandro Margara, GianpaoloCugola, High PerformanceContent-Based Matching Using GPUs, Proceedings of the

5th ACM international conference on Distributed event-

based system,Pages 183-194,ACM, 2011.

[9] Gabriella Tth, LajosJenoFlp, L{szlVid{cs, ComplexEvent Processing Synergies with Predictive Analytics, Pro-

ceedings of the Fourth ACM International Conference on

Distributed Event-Based Systems,Pages 95-96,ACM, 2010.

[10] Zhaoran Wang, Yu Zhang, Xiaotao Chang, Xiang Mi, YuWang, Kun Wang,Huazhong Yang,

Pub/Sub on Stream: A Multi-Core Based Message Broker

with QoS Support, Proceedings of the 6th ACM Interna-

tional Conference on Distributed Event-Based Sys-

tems,Pages 127-138,ACM,2012.

[11] Michael Eckert and FrancoisBry, Complex EventProcessing (CEP), Informatik-Spektrum, Volume 32, Issue

2, pages 163-167, Springer, 2009.[12] Annie Liu, Michael Olson, Julian Bunn, Towards a Discip-

line of Geospatial Distributed Event Based Systems, Pro-

ceedings of the 6th ACM International Conference on Dis-

tributed Event-Based Systems,Pages 95-106,ACM,2012.

[13] Michael Olson, Annie Liu, Matthew Faulkner, Rapid De-tection of Rare Geospatial Events: Earthquake Warning Ap-

plications, Proceedings of the 5th ACM international confe-

rence on Distributed event-based system,Pages 89-

100,ACM, 2011.

[14] Naomi Seyfer, Richard Tibbetts, Nathaniel Mishkin, Cap-ture Fields Modularity in a Stream-Relational Event

Processing Langauge, Proceedings of the 5th ACM interna-

tional conference on Distributed event-based system,Pages15-22,ACM,2011.

[15] Mark H. Linehan, Sylvain Dehors, Ella Rabinovich, FabianaFournier, Controlled English Language for Production and

Event Processing Rules roceedings of the 5th ACM interna-

tional conference on Distributed event-based system,Pages

149-158,ACM, 2011.

[16] Gerald G. Koch, Boris Koldehofe, Kurt Rothermel, Cordies:Expressive event correlation in distributed systems, Pro-

ceedings of the Fourth ACM International Conference on


[17] K. R. Jayaram, Patrick Eugster, Program Analysis forEvent-based Distributed Systems, Proceedings of the 5th

ACM international conference on Distributed event-basedsystem,Pages 113-124, ACM, 2011.

[18] GianpaoloCugola, Alessandro Margara, TESLA: A Formal-ly Defined Event Specification Language, Proceedings of





7/7


Event-Based Systems,Pages 50-61,ACM, 2010.

[19] Alan Demers, Johannes Gehrke, Biswanath Panda, Mire-kRiedewald, Varun Sharma, and Walker White, Cayuga: A

General Purpose Event Monitoring System, ACM, 2007.

[20] OpherEtzion, YonitMagid, Ella Rabinovich, Inna Skar-bovsky, NirZolotorevsky,, Context Aware Computing and

its utilization in event-based systems, Proceedings of theFourth ACM International Conference on Distributed


[21] Kia Teymourian, Malte Rohde and Adrian Paschke, Fusionof Background Knowledge and Streams of Events, Pro-

ceedings of the 6th ACM International Conference on Dis-

tributed Event-Based Systems,Pages 302-313,ACM, 2012.

[22] Christian Kuka, Sebastian Gerwinn, Daniela Nicklas,Demo: Context-Model Generation for Safe Autonomous

Transport Vehicles, Proceedings of the 6th ACM Interna-

tional Conference on Distributed Event-Based Sys-

tems,Pages 365-366,ACM,2012.

[23] AmirhosseinMalekpour, Antonio Carzaniga, Fernando Pe-done, End-to-End Reliability for Best-Effort Content-BasedPublish/Subscribe Networks, Proceedings of the 5th ACM

international conference on Distributed event-based sys-

tem,Pages 207-218, ACM, 2011.

[24] Mohamed Diallo, Serge Fdida, IOA CBR:InformationOverload Aware Content-Based Routing, Proceedings of



[25] SegevWasserkrug, Avigdor Gal, OpherEtzion, A Model forReasoning with Uncertain Rules in Event Composition Sys-

tems, proceeding of: UAI '05, Proceedings of the 21st Con-

ference in Uncertainty in Artificial Intelligence,IBM.

[26] Waldemar Hummer, Christian Inzinger, Philipp Leitner,Benjamin SatzgerSchahramDustdar, Deriving a UnifiedFault Taxonomy for Event-Based Systems, Proceedings of

the 6th ACM International Conference on Distributed


[27] Alexander Artikis, OpherEtzion, Zohar Feldman, FabianaFournier, Tutorial: Event Processing Under Uncertainty,

Proceedings of the 6th ACM International Conference on


[28] Dan OKeeffe, Reliable Complex Event Detection for Per-vasive Computing, Proceedings of the Fourth ACM Inter-

national Conference on Distributed Event-Based Sys-

tems,Pages 73-84,ACM, 2010.

[29] LjiljanaStojanovic, SinanSen, Jun Ma, DominikRiemer,ALERT: Semantic event driven collaborative platform forsoftware development,Proceedings of the 6th ACM Inter-

national Conference on Distributed Event-Based Systems,

Pages 385-386, ACM, 2012.

V. Govindasamy is a research scholar in Departmentof CSE, Pondicherry Engineering College, Puducherry,India. He is currently working as Assistant Professorin Department of IT, Pondicherry Engineering College,Puducherry, India.

Dr. P. Thambidurai is at present working as Principaland Professor of Computer Science & Engineering,

Perunthalaivar Kamarajar Institute of Engineering andTechnology (PKIET), Karaikal, Union Territory of Pu-ducherry, India. He has completed his M.E.(CSE) fromAnna University, Chennai in 1984 and Ph.D.(CSE) from

Alagappa University, Karaikudi in 1995. He is FellowInstitute of Engineers, Life Member in CSI and ISTE.He has published more than hundred research papersin National and International Journalsand Conferences. His area of interest includes Net-working, Image Processing, Natural LanguageProcessing, and Information Security.




Documents

Complex Event Processing - A Survey