EPTS DEBS2011 Event Processing Reference Architecture and Patterns Tutorial v1 2

epts event processing technical society

eptsevent processing technical society

Adrian Paschke

(Freie Universitaet Berlin)

Paul Vincent

(TIBCO Software)

Catherine Moxey

(IBM)

Alex Alves

(Oracle)

Event Processing

Reference Architecture

and Design Patterns

Tutorial v1.2

Content by: members of the

EPTS Reference Architecture Working Group


Tutorial

• Event Processing is an increasingly “mainstream”

view in IT, if not widely in production

• Event Processing Functional Reference Architecture

(FRA) describes the functions involved in event

processing

• Functions in the FRA can themselves be described as

event processing patterns

• Most technology providers can support the FRA and

its associated design patterns

2


3

Agenda

• Introduction to the EPTS Functional Reference

Architecture, and what it contains

• Mapping Functions to Patterns: some sample

Event Processing Patterns

• EPTS member pattern implementation examples

• Summary and future work on EP Patterns


Introduction to Event Processing

• Event-centric view of IT

• Events are

– Sent and Received

– Aggregated, Transformed into Data, Deleted

– Processed in queries, rules etc

– Cause actions like processes, service invocations, etc

4


Event Processing, Complex Event Processing, Event Stream Processing

Event Abstraction, Event Pattern Detection, Event Composition

etc. etc.

Introduction to Event Processing

Event

Managem

ent

Event Consumer

(Event sink, event handler, event

listener)

Run time Administration

Event

Type D

efinitio

ns,

Event

Pro

cessin

g R

ule

s,

etc

.

Design time

Events Derived

Events

Event Producer

(Event Source, Event Emitter)


Motivation and Benefits

• Motivation

– Event Processing is evolving and exploiting many technologies

– Potential adopters (stakeholders) need a reference to understand suppliers‟ architectures and solutions.

• Benefits

– a Reference Architecture predefines customizable abstract frames of reference for specific stakeholder concerns and application domains.

• Aids reuse of successful EP architectures for frequently occurring EP design problems

– Underlying Architecture is a Reference Model that defines the terminology and components in Event Processing architectures


10

Reference Architecture Viewpoints

Viewpoint

Element

Viewpoint

Engineering EP

Architecture

Managing EP

Architecture

Business with EP

Architecture

Concepts How to

implement?

How to apply? How to utilize / sell /

own?

Stakeholders Architects /

Engineers

Project Manager Decision Maker,

Customer, Provider

Concerns Effective

construction and

deployment

Operational

Management

Strategic and

tactical

management

Techniques /

Languages

Modeling,

Engineering

IT (service/appl)

management,

project

management

Monitoring,

Enterprise Decision

Management,

Governance


11

Stakeholders and Viewpoints covered

Viewpoints (for CEP / stateful EP)

Stakeholder Component Functional

Decision Maker

/ End User

[Not applicable] Inputs, Outputs and

Processing Requirements

View

Architect /

Designer

Solution Components

View

Functions carried out in

CEP View

TBD (2012?) Focus so far


Functional Reference Architecture

• Architect and Developer perspective

– includes the 3 main functions

(development, run-time and administration),

– targets primarily the automated event processing

operations

• Run-time functions in 2 main groups:

– the event infrastructure (sources and consumers)

external to the event processor under consideration,

– the event processor.


Event Processing, Complex Event Processing, Event

Stream Processing

Event Abstraction, Event Pattern Detection, Event Composition

etc. etc.

Functional View source: definitions of EP

Event

Managem

ent

Event Consumer

(Event sink, event handler, event

listener)


Event

Type D

efinitio

ns,

Event

Pro

cessin

g R

ule

s,

etc

.

Design time

Events Derived

Events

Event Producer

(Event Source, Event Emitter)


Reference Architecture: Functional View

Event Production Publication, Retrieval

Even

t Process M

on

itorin

g, C

on

trol

Event PreparationIdentification, Selection, Filtering,

Monitoring, Enrichment

Complex Event DetectionConsolidation, Composition,

Aggregation

Event ReactionAssessment, Routing, Prediction,

Discovery, Learning

Event Consumption Dashboard, Apps,External Reaction


Even

t a

nd

Co

mp

lex E

ven

t(P

att

ern

, Contr

ol, R

ule

, Q

uery

, RegEx.e

tc)

Defi

nit

ion

, M

od

elin

g, (con

tin

uou

s) I

mp

rovem

en

t

Design time

Event AnalysisAnalytics, Transforms, Tracking, Scoring, Rating, Classification

0..*

0..*

0..*

0..*

Sta

te M

an

ag

em

en

t


Reference Architecture: Functional View / Runtime





Aggregation


Discovery, Learning


Run time


0..*

0..*

0..*

0..*

Sta

te M

an

ag

em

en

t





Event Production: the source of events for event processing.

• Event Publication: As a part of event production, events may be published

onto a communication mechanism (eg event bus) for use by event

consumers (including participants in event processing). This is analogous to

a "push" system for obtaining events.

• Event Retrieval: As a part of event production, events may be explicitly

retrieved from some detection system. This is analogous to a "pull" system

for obtaining events.





Event Consumption: the process of using events from event publication and

processing. Event processing itself can be an event consumer, although for

the purposes of the reference architecture, event consumers are meant to

indicate downstream consumers of events generated in event processing.

• Dashboard: a type of event consumer that displays events as they occur to

some user community.

• Applications: a type of event consumer if it consumes events for its own

processes.

• External Reaction: caused through some event consumption, as the result

of some hardware or software process.





Event Preparation: the process of preparing the event and associated payload and

metadata for further stages of event processing.

• Entity Identification: incoming events will need to be identified relative to prior events, such

as associating events with particular sources or sensors.

• Event Selection: particular events may be selected for further analysis. Different parts of

event processing may require different selections of events. See also event filtering.

• Event Filtering: a stream or list of events may be filtered on some payload or metadata

information such that some subset is selected for further processing.

• Event Monitoring: particular types of events may be monitored for selection for further

processing. This may utilise specific mechanisms external to the event processing such as

exploiting event production features.

• Event Enrichment: events may be "enriched" through knowledge gained through previous

events or data.




Event Analysis: the process of analysing suitably prepared events and their

payloads and metadata for useful information.

• Event Analytics: the use of statistical methods to derive additional information about an

event or set of events.

• Event Transforms: processes carried out on event payloads or data, either related to

event preparation, analysis or processing.

• Event Tracking: where events related to some entity are used to identify state changes in

that entity.

• Event Scoring: the process by which events are ranked using a score, usually as a part of

a statistical analysis of a set of events. See also Event Analytics

• Event Rating: where events are compared to others to associate some importance or

other, possibly relative, measurement to the event.

• Event Classification: where events are associated with some classification scheme for

use in downstream processing.




Aggregation

Complex Event Detection: the process by which event analysis results in the

creation of new event information, or the update of existing complex events.

• Event Consolidation: combining disparate events together into a "main" or "primary"

event. See also event aggregation.

• Event Composition: composing new, complex events from existing, possibly source,

events.

• Event Aggregation: combining events to provide new or useful information, such as trend

information and event statistics. Similar to event consolidation.




Discovery, Learning

Event Reaction: the process subsequent to event analysis and complex event

detection to handle the results of analysis and detection.

• Event Assessment: the process by which an event is assessed for inclusion in some

process, incorporation in some other event, etc.

• Event Routing: the process by which an event is redirected to some process, computation

element, or other event sink.

• Event Prediction: where the reaction to some event processing is that some new event is

predicted to occur.

• Event Discovery: where the reaction to some event processing is the disclosure of a new,

typically complex, event type.

• Note that event prediction is predicting some future event, usually of a known type, whereas event

discovery is the uncovering of a new event type. See also event-based learning.

• Event-based Learning: the reaction to some event processing that uses new event

information to add to some, typically statistical-based, understanding of events.

• Note that event-based learning is a specialisation of

general machine learning and predictive analytics.


Reference Architecture: Functional View / Design time

Even

t a

nd

Co

mp

lex E

ven

t(P

att

ern

, Contr

ol, R

ule

, Q

uery

, RegEx.e

tc)

Defi

nit

ion

, M

od

elin

g, (con

tin

uou

s) I

mp

rovem

en

t

Design time

Covers the definition, modeling, improvement /

maintenance of the artifacts

used in event processing:

• event definitions, including event metadata and payloads,

• event and event object organisations and structures,

• event processing transformations / queries / rules /

procedures / flows / states / decisions /

expressions (although these can sometimes be

considered as administrative updates in some situations)


Reference Architecture: Functional View / AdministrationEven

t Process M

on

itorin

g, C

on

trol

Administration

Administrative concepts of

monitoring and control. This may involve

• starting and stopping the application and event

processing elements, including application monitors

• providing and updating security levels to event inputs and

outputs (also can design-time)

• management of high availability and reliability resources,

such as hot standby processes

• resource utilisation monitoring of the event processing

components

• process updates, such as how-swapping of event

processing definitions to newer versions.


24

Agenda









• Functions from Reference Architecture are a

guide to possible low-level patterns

25




Aggregation


Discovery, Learning



Which Classification of Patterns to Use?

• Common EP Functions?

As described in the Reference Architecture

• Other sources?

Recent literature

26


Comparison of EPTS RA Fns and EPIA EP Agents

27




Aggregation


Discovery, Learning


EPA

Filter Transformation

Translate

Enrich Project

Aggregate Split Compose

Pattern detect


Comparison of EPTS RA Fns and EPIA EP AgentsEvent Preparation

28


Comparison of EPTS RA Fns and EPIA EP AgentsEvent Analysis

29


Comparison of EPTS RA Fns and EPIA EP AgentsComplex Event Detection

30


Comparison of EPTS RA Fns and EPIA EP AgentsEvent Reaction

31


What do we want to cover in an EP Pattern?

32

Event Filtering: a stream or list of events may be filtered

on some payload or metadata information

such that some subset is selected for further processing.

Type /

synonym

Operation /

Operators

Operands

Rationale /

business ruleMapping to

EPL template


EP MetaPattern version 1

• Name & Alternative Names

• Classification versus other references

– EPTS RA and other references e.g. “EP in Action” book

• Description

– Role, and Business Rule type specification / requirement

• Structure

– EPTS Glossary terms

• Implementations

33


EP MetaPattern future versions?

• Name & Alternative Names

• Classification versus other references

– EPTS RA and other references e.g. “EP in Action” book

• Description

– Role, and Business Rule type specification / requirement

• Structure

– EPTS Glossary terms

• Implementations

34

may use EPL classification

per EPTS Languages WG

may use revised EP Fns

classification / ontology


EP Pattern Implementations

• Differ in Implementation type (Event Processing

Language, Execution semantics, etc)

• Sampled systems we cover:

1.TIBCO BusinessEvents Rete-production-rule-engine

(with continuous query and event-expression options)

2.Oracle CEP Stream-processing engine

3.Prova Logic Programming Semantic CEP Rule Engine

4.IBM WebSphere Decision Server event-driven

production-rule-engine

& IBM Infosphere Streams stream processing engine

35


36

Agenda








Some EP Patterns to review in more detail

1. Filter: filter out all events that have some property in their

payload / data

– E.g. customer purchases: filter out those with values < $100

2. Enrichment: add some information based on prior events

– E.g. customer purchase: add the customer history to the

purchase event

3. Aggregation: add prior events together for new information

– E.g. customer purchase history: track the sum total of order

values for a customer

4. Routing: based on the purchase type pass the event on to the

appropriate service

– E.g. customer purchase event: pass on to a provisioning service

based on the type of product / product classification 37




payload / data




purchase event

3. Aggregation: add some information based on prior events



4. Routing: based on the purchase type pass the event on to

the appropriate service

– E.g. customer purchase event: pass on to a provisioning

service based on the type of product / product classification 38


Filter Pattern: Classification (1)

• EPTS Reference Architecture:

– "During event preparation, a stream or list of events may

be filtered on some payload or metadata information

such that some subset is selected for further processing."

• EPIA:

– Filter (EPA or Event Processing Agent) performs filtering

only and has no matching or derivation steps, so it does

not transform the input event.

39


Filter Pattern: Description (2)

• Role / Explanation

– Comparing some property of the event (an attribute or

metadata) with some other value (from some other event, or

data)

• Associated Business Rule Specification

– As a constraint: a selection (to be enforced during the

processing of events):

All <event entities>

that have

<filter expression>

must have

<some status>. 40

eventFilter

f(event)event

discarded

event


Filter Pattern: Structure (3)

• EPTS Glossary comparison

– A filter pattern can relate to several terms in the EPTS

Glossary

• can be specified in an event pattern (A template containing

event templates, relational operators and variables...)

• .. by an event pattern constraint (A Boolean condition that

must be satisfied by the events observed in a system...)

• .. as part of an event processing rule (A prescribed method

for processing events.)

• .. or as part of event stream processing (Computing on inputs

that are event streams.)

–

41


Filter Pattern: Implementations (4)

• TIBCO, Oracle, Prova and IBM implementations for

this pattern

42

eventFilter

f(event)event

discarded

event


Filter Pattern: Implementation 1aTIBCO BusinessEvents: rules

• General pattern: <declare> event

<if> event-boolean-expression

<then> update-event-with-fact

OR

action-for-filtered-event

• Effect: ignore filtered events

on operations on non-filtered events

43

event

Filter

f(event)

event

Operation on

event


Filter Pattern: Implementation 1bTIBCO BusinessEvents: event removal rules

• Implementation type: Rete-based rule-driven system

• Alternate pattern: <declare> event

<if>

NOT event-boolean-expression

<then>

consume-event

• Effect: removed filtered events from

system (and ALL subsequent operations!)

44

event

NOT Filter

!f(event)

event


Filter Pattern: Implementation 1c TIBCO BusinessEvents: State transitions

• General pattern: <state models for concept C>

<route from State n to n+1>

<on> event

<where> event-filter

• Effect: change state for some concept

• Notes: this is a specialisation of the

general rule pattern!

45

event

Filter

query(event)

State

transition


Filter Pattern: Implementation 1d TIBCO BusinessEvents: Queries

• General pattern: <select> output

<from> event

<where> event-filter

<policy> notification

• Effect: collect filtered events for

event operations per some policy

• Note: the pattern matching framework

has a similar notion

but a different surface syntax46

event

Filter

query(event)

Event

subset

Operations on

event subset

on policy event

Policy

event


Filter Pattern: Implementation 2aOracle CEP: queries

• General pattern:

SELECT <property>

FROM <STREAM>

WHERE <predicate-condition>

• Effect: discard events whose predicate

evaluate to false

47

event

predicate

event

Operation on

event


Filter Pattern: Implementation 2bOracle CEP: model

48

• Scenario:

– Select only stocks from the stock tick stream whose

symbols are „AAA‟.

• Application Model


Filter Pattern: Implementation 2cOracle CEP: query

49

Specify STREAM source

Define predicate for

filtering

SELECT *

FROM StockTickStream

WHERE symbol = „AAA‟


Filter Pattern: Implementation 3aProva: rules

• Implementation type: backward reasoning logical filter

• General pattern template: rcvMsg / rcvMult <Event Msg Pattern(s)> :-

<logical filter condition(s)>,

... .

<logical derivation rule 1> :-

<filter conditions>.

<logical derivation rule 2> :-

<filter conditions>.

...

• Effect: Unify received events with

event pattern definition and apply logical

filter rules (derivation rules with rule chaining)50

event

Event pattern +

logical rule filter

event

Operation on

event

event

Backward

Resoning

Rule

chaning

…


Filter Pattern: Implementation 3bProva: Messaging Reaction Rules in Prova

• Send a message

sendMsg(XID,Protocol,Agent,Performative,[Predicate|Args]|Context)

• Receive a message

rcvMsg(XID,Protocol,Agent,Performative,[Predicate|Args]|Context)

• Receive multiple messages

rcvMult(XID,Protocol,Agent,Performative,[Predicate|Args]|Context)

Description:

– XID is the conversation identifier

– Protocol: protocol e.g. self, jms, esb etc.

– Agent: denotes the target or sender of the message

– Performative: pragmatic context, e.g. FIPA Agent Communication

– [Predicate|Args] or Predicate(Arg1,..,Argn): Message payload


Filter Pattern: Implementation 3cProva: Example

% Filter for stocks starting with „A“ and price > 100

rcvMult(SID,stream,“S&P500“, inform, tick(S,P,T)) :-

S = "A.*",

P > 100,

sendMsg(SID2,esb,"epa1", inform, happens(tick(S,P),T).

% Filter for stocks starting with „A“ and price > 100


filter(S,P),

sendMsg(SID2,esb,"epa1", inform, happens(tick(S,P),T).

filter(Symbol,Price) :-

Price > 100,

Symbol = "A.*".

Example with rule chaining


Filter Pattern: Implementation 3dProva: Semantic Event Processing Example

Event Stream{(Name, “OPEL”)(Price, 45)(Volume, 2000)(Time, 1) }

{(Name, “SAP”)(Price, 65)(Volume, 1000) (Time, 2)}

Semantic Query Filer:Stocks of companies, which have production facilities in Europe andproduce products out of metal andHave more than 10,000 employees.

{(OPEL, is_a, car_manufacturer),(car_manufacturer, build, Cars),

(Cars, are_build_from, Metall),(OPEL, hat_production_facilities_in, Germany),

(Germany, is_in, Europe)(OPEL, is_a, Major_corporation),

(Major_corporation, have, over_10,000_employees)}

Semantic Knowledge Base

rcvMult(SID,stream,“S&P500“, inform,

tick(Name^^car:Major_corporation,P^^currency:Dollar,

T^^time:Timepoint)) :- … <semantic filter inference> .


Filter Pattern: Implementation 4aIBM WebSphere Decision Server: Business Events component

55

event

Filter

event

Operation on

event

• Example:

Define a filter to only process events with a

purchase value greater than £100

• Effect: Only process events

which match the filter

event


• Example:

composite Main {

graph

stream<rstring name, uint32 age> Beat = Beacon() {}

stream<rstring name, uint32 age> Youngs = Filter(Beat)

{

param filter : age < 30u;

}

(stream<Beat> Younger; stream<Beat> Older) = Filter(Beat)

{

param filter : age < 30u;

}

}

Filter Pattern: Implementation 4bIBM InfoSphere Streams: Streams Processing Language

56

• Effect: filter out into streams of older and younger ages




payload / data




purchase event









Enrichment Pattern: Classification (1)

• Alternative Names:

– Event Annotation, Semantic Event


– "During event preparation, events may be "enriched"

through knowledge gained through previous events or

data. "

• EPIA:

– Enrich (EPA): a translate EPA that takes a single input

event, uses it to query data from a global state element,

and creates a derived event which includes the attributes

from the original event, possibly with modified values...58


Enrichment Pattern: Description (2)


– Updating some attribute or metadata of the event with some

other values, possibly computed, from some other event or

data


– a selection (to be enforced during the processing of events):

All <event entities> that are also <enriched data

expression>

must have <some status>.

59

eventEnrichment

f(event)event

Other data

or event


Enrichment Pattern: Structure (3)


– Enrichment can be specified in an event pattern (pattern

definition containing additional meta data, semantic

knowledge)

– .. as part of an event processing rule (A method for

enriching events during processing, e.g., by analysing the

meta data and querying external data sources or

inferencing semantic knowledge bases)

– .. or as part of event stream processing (additional

additional information to event stream data, e.g. by

querying external data sources)

60


Enrichment Pattern: Implementations (4)


this pattern

61

eventEnrichment

f(event)event

Other data

or event


Enrichment Pattern: Implementation 1 TIBCO BusinessEvents: Rules

• General pattern: <declare> event, data

<if> event-data-match

<then> update-event-with-data

• Effect: find appropriate data for

events and apply enrichment

operations on them

62

event

Identify

enrichment data

Event

Enrich

Operation

on event

Enrich

data


Enrichment Pattern: Implementation 2aOracle CEP: queries

• General pattern template:

SELECT <property>

FROM <STREAM>[WIN],<RELATION>

WHERE <join-condition>

• Effect: enrich events from stream

with contextual data from relation

where condition applies.

63

event

join-condition

event

Operation on

event


Enrichment Pattern: Implementation 2bOracle CEP: model

64

• Scenario:

– For every stock event from stock tick stream whose bid is

greater than 5.0, find its company‟s location.



Enrichment Pattern: Implementation 2cOracle CEP: queries

65

SELECT event.symbol, location

FROM

StockTickStream [NOW] AS event,

StockTable AS data

WHERE event.symbol = data.symbol AND

event.lastBid > 5.0

No need for window

operator on a „relation‟


Enrichment Pattern: Implementation 2dOracle CEP: table metaphor

66

Time Input (StockTickStream) Output (QUERY)

1

2

3

Symbol Full-name Location

AAA The AAA company San Francisco

BBB The BBB company San Jose

{“AAA”, 10.0, 10.5} {“AAA”, “San…”}

{“BBB”, 11.0, 12.5} {“BBB”, “San Jose”}

{“AAA”, 4.0, 4.5}

StockTable


Enrichment Pattern: Implementation 3aProva: rules

• General pattern template: rcvMsg / rcvMult <Event Msg Pattern(s)> :-

<query built-in(s)>,

<enrichment condition(s)>

... .

<logical enrichment rule 1> :-

<enrichment operations>.

• Effect: query data from external source

and enrich the event

67

event

Query

event*

Operation on

enriched event

External

data

Enrichment

rules


• Java..., L=java.util.ArrayList(),, ..., java.lang.String(“Hello”)

• File Input / Output ..., fopen(File,Reader), ...

• XML (DOM)document(DomTree,DocumentReader) :-

XML(DocumenReader),

• SQL …,sql_select(DB,cla,[pdb_id,“1alx“],[px,Domain]).

• RDF ...,rdf(http://...,"rdfs",Subject,"rdf_type","gene1_Gene"),

• XQuery ..., XQuery = ' for $name in StatisticsURL//Author[0]/@name/text() return $name', xquery_select(XQuery,name(ExpertName)),

• SPARQL...,sparql_select(SparqlQuery,name(Name),class(Class),definition(Def)),

Enrichment Pattern: Implementation 3bProva: External Data and Object Integration


Enrichment Pattern: Implementation 3cProva: Example with SPARQL Query

% Filter for car manufacturer stocks and enrich the stock tick

event with data from Wikipedia (DBPedia) about the manufacturer

and the luxury cars


carManufacturer(S,Man), % filter car manufacturers

findall([luxuryCar|Data],luxuryCar(Man,Name,Car),Data), % query

EnrichedData = [S,Data], % enrich with additional data

sendMsg(SID2,esb,"epa1", inform, happens(tick(EnrichedData,P),T).

% rule implementing the query on DBPedia using SPARQL query

luxuryCar(Manufacturer,Name,Car) :-

Query="SELECT ?manufacturer ?name ?car % SPARQL RDF Query

WHERE {?car <http://purl.org/dc/terms/subject>

<http://dbpedia.org/resource/Category:Luxury_vehicles> .

?car foaf:name ?name .

?car dbo:manufacturer ?man .

?man foaf:name ?manufacturer.

} ORDER by ?manufacturer ?name“,

sparql_select(Query,manufacturer(Manufacturer),name(Name),car(Car)).


Enrichment Pattern: Implementation 4IBM WebSphere Decision Server: Business Events component

70

event

Mapped Key

look up

Enriched

event

• Example: Define a Mapped Key expression, where a

field in the event is used to look up objects in a

database table

• Effect: required enrichment data is obtained

from the data base and added to event




payload / data




purchase event









Aggregation Pattern: Classification (1)


– Event Summarization, Composite event, Complex event


– "During complex event detection, combining events to

provide new or useful information, such as trend

information and event statistics. Similar to event

consolidation "

• EPIA:

– Aggregate (EPA): a transformation EPA that takes as

input a collection of events and creates a single derived

event by applying a function over the input events.72


eventevent

Aggregation Pattern: Description (2)


– Aggregation is used to describe several events as a single

composite event, generally summarizing a metric of a set of

component events.

– For example: generate a (composite) event that contains the

medium price of a stock over a 10 minute stream of events.


– a summarisation (to be enforced during event processing):

The <aggregation fn> of <event entities> that

are <selection constraint>

must have

<some constraint>.

73

eventAggregation

f(events)event


Aggregation Pattern: Structure (3)


– The term aggregate event is sometimes used for some

forms of composite or derived event.

– Composite event: a derived, complex event

• is created by combining base events using a specific set of

event constructors such as disjunction, conjunction,

sequence, etc.

• always includes the base (member) events from which it is

derived.

– Derived event (/synthesized event): an event that is

generated as a result of applying a method or process to

one or more other events.

74


Aggregation Pattern: Implementations (4)


this pattern

75

eventevent

eventAggregation

f(events)event


Aggregation Pattern: Implementation 1TIBCO BusinessEvents: Rules

• General pattern: <declare> event,

aggregation-object

<if>

event-member-of-aggregation

<then>

update-aggregation-object

with event-data

• Effect: construct aggregation

(event) object for each incoming event76

event

Identify

event for

aggregation

Updated

Aggregation

Object

Aggregate

Operation

on event

Aggregation

object


Aggregation Pattern: Implementation 2aOracle CEP: queries


SELECT <summary-property>

FROM <STREAM>[WIN]

WHERE <predicate>

GROUP BY <aggregation-identity>

• Effect: summarizes the properties of

several simple event into a new

complex event

77

event

Identify

event for

aggregation

Updated

Aggregation

Object

Aggregate

Operation

on event

Aggregation

object


Aggregation Pattern: Implementation 2bOracle CEP: model

78

• Scenario:

– Output the average bid and ask price of a stock in the

last 10 seconds.



Aggregation Pattern: Implementation 2cOracle CEP: queries

79

SELECT symbol, AVG(bid), AVG(ask)

FROM

StockTickStream [RANGE 10 SECONDS]

GROUP BY

symbol


Aggregation Pattern: Implementation 2dOracle CEP: table metaphor

80

Time Input Output

1s

4s

9s

15s

20s

{“AAA”, 10.0, 12.0} {“AAA”, 10.0, 12.0}

{“AAA”, 12.0, 14.0} {“AAA”, 11.0, 13.0}

{“BBB”, 4.0, 5.0}{“AAA”, 11.0, 13},

{“BBB”, 4.0, 5.0}

{“BBB”, 8.0, 10.0} {“BBB”, 6.0, 7.5}

{“BBB”, 8.0, 10.0}

> 10s


Aggregation Pattern: Implementation 3aProva: rules


<rule_head> :-

<Create Aggregator>,

@group(<reaction group>)

@timer|size(Start,Interval,Aggregator)

rcvMsg <EventPattern> [<Aggregate Operation>].

<rule_head> :-

@or(<reaction group>)

rcvMsg <Aggregator>,

... <consume Aggregator>.

• Effect: Repeated incremental

aggregations over new events

81

event

Check & Update

Timer / Size Count

Updated

Aggregation

Object

Aggregate

Operation

on event

Aggregation

object

Identify event

for aggregation

Consume

Aggregation

Object

or


Aggregation Pattern: Implementation 3bProva: Example with Time Counter

% This reaction operates indefinitely. When the timer

elapses (after 25 ms), the groupby map Counter is sent

as part of the aggregation event and consumed in or

group, and the timer is reset back to the second

argument of @timer.

groupby_rate() :-

Counter = ws.prova.eventing.MapCounter(), % Aggr. Obj.

@group(g1) @timer(25,25,Counter) % timer every 25 ms

rcvMsg(XID,stream,From,inform,tick(S,P,T)) % event

[IM=T,Counter.incrementAt(IM)]. % aggr. operation

groupby_rate() :-

% receive the aggregation counter in the or reaction

@or(g1) rcvMsg(XID,self,From,or,[Counter]),

... <consume the Counter aggreation object>.


Aggregation Pattern: Implementation 4aIBM WebSphere Decision Server: Business Events component

83

• General Pattern:

Define how the

events should be

accumulated

... then when

processing the

events, use the

relevant

aggregator

• Effect of the example: sum up the value of a

customer‟s purchases over a week

Add to array

Intermediate

Object Array

Operation

on aggregate

event event


Aggregation Pattern: Implementation 4bIBM InfoSphere Streams: Streams Processing Language

84

• Example:// sliding window

stream<Beat, tuple<int32 maxSalary, uint32 ageOfMaxSalary> >

Agg4 = Aggregate(Beat)

{

window

Beat : sliding, time(10.5), count(10);

output

Agg4 : maxSalary = Max(salary), ageOfMaxSalary = ArgMax(salary, age);

}

• Effect of example: Aggregate the salary values in the

stream within a time-based window.




payload / data




purchase event









Routing Pattern: Classification (1)


– Event Summarization, Composite event, Complex event


– "During event reaction, event routing is the process by

which an event is redirected to some process,

computation element, or other event sink. "

• EPIA: no direct analogy, possibly maps to split /

compose / project

– Project (EPA): a translate EPA that takes an input event

and creates a single derived event containing a subset

of the attributes of the input event.86


Routing Pattern: Description (2)


– Routing is used to describe the process of adding one or

more new destinations to an event.

– For example: route an input event to the appropriate

specialist agent for that event type.


– a constraint (to be enforced during event processing):

The <event> that

satisfies <selection constraint>

must be assigned to

<destination>.

87

eventRouting

dest = f(event+data)event

Other data

or event


Routing Pattern: Structure (3)


– Routing (a process on events) is not defined, but is

related to associating an event to an event sink.

– Event sink (event consumer) is an entity that receives

events.

• Examples:

• ƒ Software module

• ƒ Database

• ƒ Dashboard

• ƒ Person

88


Routing Pattern: Implementations (4)


this pattern

89

eventRouting

dest = f(event+data)event

Other data

or event


Routing Pattern: Implementation 1TIBCO BusinessEvents: Rules

• General pattern: <declare> event,

routing-data

<if>

event-matches-routing-data

<then>

create-new-routed-event and

send-new-routed-event

• Effect: match events to routing rule

and construct / send new event90

event

Identify

event for

routing

event

Create routed

event

Routing

data


Routing Pattern: Implementation 2aOracle CEP: query pattern


Route 1: <filtering pattern> query

Route 2: <filtering pattern> query

Route n: <filtering pattern> query

• Effect: Different predicates

associated to different queries

evaluate and select appropriate

destinations

91

event

Identify

event for

routing

event

Select destination

Routing

data


Routing Pattern: Implementation 2bOracle CEP: model

92

• Scenario:

– Output stocks whose symbols start with „a-m‟ to

destination 1, and whose symbols start with „n-z‟ to

destination 2.



Routing Pattern: Implementation 2cOracle CEP: queries

93

Destination-1:

SELECT *


WHERE symbol.matches(“^[a-m]”)

Destination-2:

SELECT *


WHERE symbol.matches(“^[n-z]”)


Routing Pattern: Implementation 3Prova: rules


rcvMsg <Event Msg Pattern> :-

<routing decisions>,

sendMsg <route event >.

<routing decision rule 1> :-

<decision conditions>.

<routing decision rule 2> :-

<decision conditions>.

...

• Effect: Events are routed

according to the routing decision

rules94

event

Identify

event for

routing

event

Create routed

event

Routing

data

Routing

Decisions

Route

event


Routing Pattern: Implementation 3bProva: Example with Agent (Sub-) Conversations

rcvMsg(XID,esb,From,query-ref,buy(Product) :-

routeTo(Agent,Product), % derive processing agent

% send order to Agent in new subconversation SID2

sendMsg(SID2,esb,Agent,query-ref,order(From,

Product)),

% receive confirmation from Agent for Product order

rcvMsg(SID2,esb,Agent,inform-ref,oder(From, Product)).

% route to event processing agent 1 if Product is

luxury

routeTo(epa1,Product) :- luxury(Product).

% route to epa 2 if Product is regular

routeTo(epa2,Product) :- regular(Product).

% a Product is luxury if the Product has a value over …

luxury(Product) :- price(Product,Value), Value >=

10000.

% a Product is regular if the Product ha a value below

…


Routing Pattern: Implementation 4IBM: Business Events component of WebSphere Decision Server

• Example: Define an interaction set,

which will send the request to supplier A,

when the purchase indicates supply

from Supplier A

•

• Effect of example:

Events are routed as required

96

event

Identify

event

routing

event

Routing

information

Consume

event


97

Agenda








Another Pattern Type

Other patterns may be required for

implementation, such as handling

produced-consumer interactions


Application-Time Pattern

• Problem:

– Need to use application‟s view of time, instead of CPU wall-clock.

– This is particularly useful when events originate from different machines and need some way of synchronizing.

• Scenario:

– Again, consider bid and ask stream…

– However, bid and ask requests are time-stamped on trader at the time that the order is placed.

99



• Scenario:

– Seller places two ask requests respectively at time

8:00:00 and 8:00:12

– Buyer places one bid request at time 8:00:11

100



• Scenario:

– Remember that we want to correlate using a 10 seconds

window, as anything older is stale…

– Hence first event from seller should not be considered,

and we should correlate the ask price of 11.0 with the bid

price of 9.5

101



• Scenario:

– However, the reality is that the first two events could

arrive together in a burst in the exchange, and the third

could be delayed in the cloud…

102



• Scenario:

– In this case, bid price of 9.5 would correlate to ask price of 10.0

and the exchange would loose money as the spread is lower…

103


Application Time-stamped Pattern: Implementation 1Oracle CEP

• Solution:

– The query does not change…

– However STREAMS must be configured to use

application time-stamps based upon some event

property, instead of having events being system-

timestamped…

104


Application Time-stamped Pattern: Implementation 2IBM Business Monitor

• Example: Define a correlation expression

• Effect of example: The duration of a customer order can be obtained from

the timestamps on start and end events

105

„Order Complete‟

event

Correlate

Calculate

Duration

„Order Placed‟

event


Application Time Pattern: Implementation 3 Prova: rules


rcvMsg <Event Msg Pattern(s)> :-

<add application timestamp to event>,

sendMsg <Adjusted Event>.

• Effect: Any timestamp can be adjusted

(for time computations)

106

Event

adjust event

Updated

Event


Application Time-stamped Pattern: Implementation 4TIBCO BusinessEvents: PreProcessor Pattern

• General Pattern: adjust Event metadata (e.g.

Timestamp) in preprocessor for that event

– events are generally non-monotonic!

– PreProcessor applies regardless of engine used!

• Effect: Any custom timestamp can be adjusted (for time

computations)

107

Event

Preprocess:

adjust event

Updated

Event

Note: If the problem is “out of order” events

(rather than synchronizing clocks) a

different pattern may be better!


Use Cases

Illustrating using some of the patterns


IBM Use Case: Utilizing Smart MetersLarge energy company improved customer responsiveness using event processing

Use Case

• Utilized events (and lack of events) from

smart meters to detect service outages and

disruptions

• Empowered operations team to define

outage detection and response patterns

without coding or scripting

Smarter Business Outcomes

• Automated detection and response to service

outages

• Improved customer satisfaction by delivery of

estimated time to recovery alerts based on

customer preferences

Challenge: Reliance on customer calls

to detect and respond to outages and

reduce time to recovery while providing

more information to the customer


110

Customer Care

Manager(s) with IBM

Business Monitor

WebSphere

Business Events

Business

Events

Power

Disruption

3

Data Collection

Engine (DCE)

Outage

Management

System (OMS)

Tivoli OMNIbus,

Network Manager,

and Impact

Customer

Data

1

1

2

4

Location, Estimated Duration

of DisruptionAlert and Assist Impacted

Customers

Energy Company Leveraging Smart Meters to Improve Customer Responsiveness


Oracle Use-case: Capital Markets

• Low-Latency Trading

– Algorithm Trading

• Trade Matching

– High throughput

– In-flight fraud

• Position Capture and Aggregation

• Risk and Analytics

– Pre/post trade analytics

– On-demand risk management

111


TIBCO Use Case: Application Service GatewayTelco high performance service policies using event processing

112

Mobile Services

difficult to monitor

end to end

Auto detect and

fixing of issues

for better

user service

Excellent service quality

needed to avoid

customer churn

Service quality affects

take-up of new services


• Standardized channel for monitoring subscribers

• Standardized access to services (e.g. Facebook, Google, etc)

• Consolidation of existing multiple gateways

• Allows analysis of service usage, subscriber behavior, service performance, marketing campaign success, etc

• Creates events for revenue generation and reconciliation

• Handles Common Telco Notifications - change-of-device, contract, service- allowing UpSell/Cross sell etc

Service Gateway architecture exploits event processing


• Standardized channel for monitoring subscribers

• Standardized access to services (e.g. Facebook, Google, etc)

• Consolidation of existing multiple gateways

• Allows analysis of service usage, subscriber behavior, service performance, marketing campaign success, etc

• Creates events for revenue generation and reconciliation

• Handles Common Telco Notifications - change-of-device, contract, service- allowing UpSell/Cross sell etc

Service Gateway architecture exploits event processing

Routing Pattern

Transformation

Patterns

Decision / Policy

Pattern

Security Pattern


115

Agenda








Summary: Realization Patterns / Idioms

Event Production Application Time

Publication, Retrieval




Aggregation


Discovery, Learning



0..*

0..*

0..*

0..*

Sta

te M

an

ag

em

en

t


Categorization of Event Processing Patterns

Source:Paschke, A.: Design Patterns for Complex Event Processing, DEBS'08, Rome, Italy, 2008

http://arxiv.org/abs/0806.1100v1

http://arxiv.org/abs/0806.1100v1

epts event processing technical society Categorization of Patterns

• Categorization according to Good and Bad Solutions– CEP Patterns

– CEP Anti-Patterns

• Categorization according to the Abstraction Level– Guidelines and Best Practices

– Management patterns

– Architecture patterns

– Design patterns

– Mapping patterns

– Idioms / Realization patterns

– Smells / Refactoring patterns

• Categorization according to the Intended Goal– Adoption patterns

– Business patterns

– Integration patterns

– Composite patterns:

– …

• Categorization according to the Management Level– Strategic patterns

– Tactical patterns

– Operational patternsSource:Paschke, A.: A Semantic Design Pattern Language for Complex Event Processing,

Intelligent Event Processing, Papers from the 2009 AAAI Spring Symposium (2009) , p. 54-60.

http://www.aaai.org/Papers/Symposia/Spring/2009/SS-09-05/SS09-05-010.pdf











Next Steps

• Continue evolution of EP Reference Architecture

– Eg Logical Architecture vs Systems Description

• Evolve the EP Pattern <<metapattern>> to be more useful for developers and tools

• Expand coverage to all Functions as Patterns, + extend Functions as necessary

• Planned Outputs:

– EPTS-RA Description white paper

– Wiley Book "Pattern Oriented Software Architecture: Architectures, Models and Patterns for Event Processing" (already in preparation)

– Input for Event Processing Metamodels and associated standards


Summary

• Reference Architecture provides a common set of

EP Functions that may be mapped

to other EP ontologies / taxonomies

• Provides the basis for discussing EP Patterns

• EP Patterns can be semantic, design or runtime

focused

• Further evolution of EP Pattern and examples is

likely!

120


121

Thank you !

Acknowledgment to the Event Processing Technical Society

Reference Architecture working group members


References

• EPTS Glossary

http://www.ep-ts.com/component/option,com_docman/task,doc_download/gid,66/Itemid,84/





Technology

EPTS DEBS2011 Event Processing Reference Architecture and Patterns Tutorial v1 2