69
Complex Event Processing Tutorial Paul Vincent TIBCO Software Inc.

Complex Event Processing Tutorial - Object Management Group

  • Upload
    others

  • View
    9

  • Download
    0

Embed Size (px)

Citation preview

Page 1: Complex Event Processing Tutorial - Object Management Group

Complex Event Processing Tutorial

Paul VincentTIBCO Software Inc.

Page 2: Complex Event Processing Tutorial - Object Management Group

© 2008 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.2

workshop: DOC4RTES

Presenter

� Presenter:Paul Vincent, CTO Business Rules and CEP, TIBCO Software

� Member OMG PRR and W3C RIF rules standards bodies

� Co-author CEP Blog http://tibcoblogs.com/cep

� TIBCO Software Inc.:

� Provides enterprise software that helps companies achieve service-oriented

architecture (SOA) and business process management (BPM) success

� Headquartered in Palo Alto, California

� Over 3,000 customers and offices in 40 countries

� CEP product is TIBCO BusinessEvents

� Developed from a customer solution and launched 2005

� Currently at Release 3.0

Page 3: Complex Event Processing Tutorial - Object Management Group

© 2008 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.3

workshop: DOC4RTES

Agenda

� Introducing CEP

Page 4: Complex Event Processing Tutorial - Object Management Group

© 2008 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.4

workshop: DOC4RTES

Real-world Events

CustomerLogon

Fed Base RateIncrease

ProductionItem

Arrives at Store

CustomerChecks

“Close Account”Web Page

NewLiabilityAdded

RentalCar

Returned

NewOrder

ContractSubmitted

ContractReturnedthru EDIRental

CarCrashed

Mobile Callfrom [email protected]

Page 5: Complex Event Processing Tutorial - Object Management Group

© 2008 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.5

workshop: DOC4RTES

Where is the information?

CustomerLogon

Fed Base RateIncrease

ProductionItem

Arrives at Store

CustomerChecks

Close AccountWeb Page

NewLiabilityAdded

RentalCar

Returned

NewOrder

ContractSubmitted

ContractReturnedthru EDIRental

CarCrashed

Mobile Callfrom [email protected]

Page 6: Complex Event Processing Tutorial - Object Management Group

© 2008 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.6

workshop: DOC4RTES

Where is the information?

Fraud Risk!

Risk of CustomerDefection

EmployeeOver hours

CustomerCrossSell

Opportunity

ComplianceLimit

Approached

Customernow rated

Gold

Change inProduct Sales

Trend

ContractValidated

ContractValid

RentalContract Complete

Cell phonefraud alert

Page 7: Complex Event Processing Tutorial - Object Management Group

© 2008 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.7

workshop: DOC4RTES

Business Processes:BPM & Workflow Processes

IT Services: SOA & traditional Data Processing

PersistenceServices

Conventional Event Processing

NewOrderNew

Order(Lots ofEvents)

SynchronousSynchronousSynchronousSynchronousEventsEventsEventsEvents

EventEventEventEvent----atatatat----aaaa----timetimetimetime

Aggregation asAggregation asAggregation asAggregation asdatadatadatadata

Simple event Simple event Simple event Simple event processingprocessingprocessingprocessing

Page 8: Complex Event Processing Tutorial - Object Management Group

© 2008 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.8

workshop: DOC4RTES

Simple EP = default IT Model, 1950-now

� Based on “human workflow”: one thing at a time

� Processes handle cases 1 at a time � office clerk

� Use database and refer to it where necessary � card index

� Provide some service flexibility with middleware � internal mail

� Use BPM to document / manage / automate processes

� Use SOA to distribute / manage / automate services

This model does NOT exploit

ALL the information / data / eventsALL the time

Behaviour (and business logic) is silo’d

T h e r e i s a b e t t e r w a y !

Page 9: Complex Event Processing Tutorial - Object Management Group

© 2008 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.9

workshop: DOC4RTES

Business Processes:BPM & Workflow Processes

IT Services: SOA & traditional Data Processing

PersistenceServices

Complex Event Processing

continuousEventProcessing

NewOrderNew

Order

EventStore

Complex Patterns of Events

AsynchronousAsynchronousAsynchronousAsynchronousEventsEventsEventsEvents

(Lots ofEvents)

High performanceHigh performanceHigh performanceHigh performancepersistencepersistencepersistencepersistence

High performanceHigh performanceHigh performanceHigh performancepatternpatternpatternpattern----matching matching matching matching via rules, states, via rules, states, via rules, states, via rules, states,

& queries& queries& queries& queries

Page 10: Complex Event Processing Tutorial - Object Management Group

© 2008 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.10

workshop: DOC4RTES

CEP Terminology

� CEP (technology) applies pattern detection & filtering to the event clouds & streams and their histories

� Multiple modelling / execution paradigms are available for pattern detection

Event Streams

CEP

Event Cloud

Page 11: Complex Event Processing Tutorial - Object Management Group

© 2008 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.11

workshop: DOC4RTES

What does CEP cover?

“CEP applies to a very broad spectrum of

challenges in information systems. A short list includes:”

� Business process automation

� Computer systems to automate scheduling and

control network-based processes and

processing

� Identifying when complex contracts are fulfilled

� Detection intrusion, fraud and other network

attacks

� C3I

The Power of Events, Addison Wesley, ISBN: 0-201-72789-7, 2002

Page 12: Complex Event Processing Tutorial - Object Management Group

© 2008 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.12

workshop: DOC4RTES

What does CEP Solve?

CEP raison CEP raison CEP raison CEP raison dddd’’’’etreetreetreetre: : : : quickly respond to quickly respond to quickly respond to quickly respond to detected variancesdetected variancesdetected variancesdetected variances

Usually: Usually: Usually: Usually: warnings precede threatswarnings precede threatswarnings precede threatswarnings precede threats

Valu

e o

f acti

on

the the the the ““““ Latency ProblemLatency ProblemLatency ProblemLatency Problem““““

Page 13: Complex Event Processing Tutorial - Object Management Group

© 2008 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.13

workshop: DOC4RTES

What CEP provides

“Situational Awareness”

“Sense and Respond”

“Track and Trace”

Page 14: Complex Event Processing Tutorial - Object Management Group

© 2008 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.14

workshop: DOC4RTES

Agenda

�History

Page 15: Complex Event Processing Tutorial - Object Management Group

© 2008 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.15

workshop: DOC4RTES

Command and Control

� Command and Control

� Correlate all available information

� Determine tactics based on strategy and up-to-date information

-- from RAF Battle of Britain Fighter Control System 1940http://www.raf.mod.uk

Page 16: Complex Event Processing Tutorial - Object Management Group

© 2008 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.16

workshop: DOC4RTES

Data Fusion

24

EVENTSOURCES

EXTERNAL

.

.

.

Level 1

Processing

ObjectAssessment

Human/ComputerInterface

Data Fusion Domain

Database ManagementSystem

SupportDatabase

FusionDatabase

DISTRIBUTED

LOCAL

Sensors

Documents

People...

Data Sources

Level 2

Processing

Situation Assessment

Level 3

Processing

Impact Assessment

Level 4Processing

Process Refinement

-- Revised JDL data fusion model, 1998Steinberg, A., & Bowman, C., Handbook of Multisensor Data Fusion, CRC Press, 2001

Level 0

Processing

Sub-ObjectAssessment

Page 17: Complex Event Processing Tutorial - Object Management Group

© 2008 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.17

workshop: DOC4RTES

Condition Based Maintenance

Monitored System

Fe

atu

re E

xtr

ac

tio

n

Cla

ss

ific

ati

on

Fu

sio

n

Reas

on

ing

Well informed Decision

History

Mission

-- from “Data Fusion for Developing Predictive Diagnostics for Electromechanical Systems”Steinberg, A., & Bowman, C., Handbook of Multisensor Data Fusion, CRC Press, 2001

Page 18: Complex Event Processing Tutorial - Object Management Group

© 2008 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.18

workshop: DOC4RTES

Agenda

�Events and CEP

Page 19: Complex Event Processing Tutorial - Object Management Group

© 2008 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.19

workshop: DOC4RTES

Complex Business Problems

� Fraud / Theft� Thousands-to-millions of high-value small-size product

items or transactions

� How do you identify known patterns of “suspicious”behavior?

� Logistics / Scheduling� Raw material, production & delivery scheduling and

resources are complex and prone to change

� How do we reallocate resources to handle business and production changes?

� Activity Monitoring� Complex production and supply process with multiple

actors

� How to measure and action Key Performance Indicators?

Product location change

Relevant event of interest

Resource, requirement change

Process data change

Page 20: Complex Event Processing Tutorial - Object Management Group

© 2008 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.20

workshop: DOC4RTES

Associated Events

� Positive Events

� Product item X arrives at Production station S from Store T

� Production worker Y arrives at Production station S

� Production contract for item Z by time T is posted

� Negative Events

� Product item X has been in transit to Store T for >15 minutes

� Subcomponent Y hasn’t arrived at the Production station by the ETA

� Delivery of contract Z has not taken place

� Sets of Events

� 5+ items of Product item type Y failed to arrive at destination

� Supplier Y was 5 mins late for 1 delivery, but made it early to the next

� Return rate on component Z exceeds SLA %

Page 21: Complex Event Processing Tutorial - Object Management Group

© 2008 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.21

workshop: DOC4RTES

Significant features of these Events

� Time Sensitivity

� A thief may leave the building at the same time as stolen product

� A product should take 40 minutes to travel a given production line

segment

� Distributed Event Sources

� A series of produced items fails at various QA stages, and their

common attribute was a storage location

� Multiple suppliers for a subcomponent are reporting delivery delays

15:39:26.413

Dec 24

ERP Supplier

Page 22: Complex Event Processing Tutorial - Object Management Group

© 2008 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.22

workshop: DOC4RTES

What *is* an “event”?

� Change of state in some entity

� Customer call

� Bank debit

� Aircraft movement

� Observation of some entity

� CRM record of a customer call

� ATM report of debit transaction success

� Radar plot update of an aircraft

� IT Message

� Queued point-to-point message

� Publish / subscribe message

“Happening”

Observation

IT Message

Page 23: Complex Event Processing Tutorial - Object Management Group

© 2008 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.23

workshop: DOC4RTES

Sample Event Metamodel

� Event contains data + timestamp

� Various classifiers (int/ext, transport, lifecycle, …)

SimpleEvent

GeneralEvent

Inherits from

CustomAttributes

XML

Timestamp

TimeToLive

ExpiryAction

Channel

DestinationDefault

Destination

Page 24: Complex Event Processing Tutorial - Object Management Group

© 2008 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.24

workshop: DOC4RTES

Event-driven vs Event Processing

Event Sources

Event Consumers

EventBus

Eve

nt

Bu

s

SimpleEvent

Processing

Services

Data Services

Invoke / single event

Business ProcessManagement

ComplexEvent

Processing

Event Store

Inform / multiple events

Event ProcessingAgents

Page 25: Complex Event Processing Tutorial - Object Management Group

© 2008 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.25

workshop: DOC4RTES

Event Driven Architecture

S O A

Assumptions…

E D A

Synchronous Events

Asynchronous Events

Request Request Request Request ---- ReplyReplyReplyReply TransmitTransmitTransmitTransmit----ListenListenListenListen

Page 26: Complex Event Processing Tutorial - Object Management Group

© 2008 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.26

workshop: DOC4RTES

CEP in the Event Driven Architecture

S O A

Assumptions…

E D A

Synchronous Events

Asynchronous Events

Complex Event Processing

ContinuousContinuousContinuousContinuousEventEventEventEvent

MonitoringMonitoringMonitoringMonitoring

Page 27: Complex Event Processing Tutorial - Object Management Group

© 2008 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.27

workshop: DOC4RTES

Complex Event Processing

Processing type:Processing type:Processing type:Processing type:

DecisionDecisionDecisionDecisionProcessingProcessingProcessingProcessing

EventEventEventEventProcessingProcessingProcessingProcessing

Sense and Respond / Sense and Respond / Sense and Respond / Sense and Respond / Track and Trace / Situational AwarenessTrack and Trace / Situational AwarenessTrack and Trace / Situational AwarenessTrack and Trace / Situational Awareness

TIBCOTIBCOTIBCOTIBCOReference ArchitectureReference ArchitectureReference ArchitectureReference Architecture

Page 28: Complex Event Processing Tutorial - Object Management Group

© 2008 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.28

workshop: DOC4RTES

Analysts on CEP

Automated Operational Decisions

Automated Predictive Decisions Human Predictive Decisions

Human Operational Decisions

Decision Latency

Event Complexity

Pro

cess

Co

mp

lex

ity

Pa

ttern

Ma

tch

ing

an

d I

nfe

ren

cin

g

Anti-Money Laundering

Credit-Card Fraud

Exchange Compliance

Database Monitoring

Algorithmic Trading

Trade Desk Monitoring

Customer Interaction

Order Routing

RFIDTariff Look-Up

Rail Networks

Search & Rescue

Baggage Handling

Liquidity Management

--- Bloor Reporton Event Processing

Nov 2006

Page 29: Complex Event Processing Tutorial - Object Management Group

© 2008 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.29

workshop: DOC4RTES

Why CEP?

� Detecting event patterns across multiple event types + time is difficult for simple event processing solutions

� Computers can correlate across large volumes of events at high speed, identifying patterns that are not conventionally visible

� The architecture pattern of “continuous event processing”applies to many business domains such as BAM

� Examples in use:

� Track and Trace of RFID data

� Situation Assessment of airline operational delays (+ their causal events)

� Sense and Respond to fraud indicators in internet transactions

Page 30: Complex Event Processing Tutorial - Object Management Group

© 2008 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.30

workshop: DOC4RTES

Agenda

�CEP Technologies

Page 31: Complex Event Processing Tutorial - Object Management Group

© 2008 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.31

workshop: DOC4RTES

CEP = an Event-Decision Architecture

EVENTSOURCES

EXTERNAL

.

.

.

User

Interface&

Systems

Event-Decision Architecture

DISTRIBUTED

LOCAL

EVENTSERVICES

.

.EVENT

PROFILES..

DATABASES

.

.OTHER DATA

EVENT PRE-PROCESSING

LEVEL ONE

Event Tracking & Refinement

DATA MANAGEMENT

HistoricalData

Profiles &Patterns

LEVEL TWO

Situation Detection & Refinement

LEVEL THREE

Impact Assessment & Predictive

Analysis

LEVEL FOUR

Process Refinement &

Adaptation

-- Adapted from JDL Data Fusion Model by Tim BassSteinberg, A., & Bowman, C., Handbook of Multisensor Data Fusion, CRC Press, 2001

Proposed for EPTS Reference ArchitectureProposed for EPTS Reference ArchitectureProposed for EPTS Reference ArchitectureProposed for EPTS Reference Architecture

EventBehavior

EventBehavior

EventBehavior

EventBehavior

EventBehavior

Page 32: Complex Event Processing Tutorial - Object Management Group

© 2008 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.32

workshop: DOC4RTES

Requirements for CEP Technology

� Access and Monitor the “Event Cloud”

� JMS, RV, MQ, TCP/IP, etc…

� Timers to detect lack of events

� Determine event state changes

� Match Patterns, Apply Business Logic

� Detect events

� Detect event patterns

� Maintain State and Facts over time

� Update Detection algorithms as events change

Pattern Detection Model

for recognizing patterns,defining actions

Temporal Model

for determining time-dependent info

Event and Data Model

for information modeling

Event Bus or Source

Low-latency reliable message delivery

His

tory

an

d C

ac

he

Page 33: Complex Event Processing Tutorial - Object Management Group

© 2008 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.33

workshop: DOC4RTES

CEP-Related Standards

History and Cache

Pattern Detection Model

for recognizing patterns,defining actions

Temporal Model

for determining time-dependent info

Event and Data Model

for information modeling

Event Bus or Source

Low-latency reliable

message delivery

OMG EMP(RFP in progress)

OMG UML2

Class Models

OMG UML2

State Models

JMS, (DDS)

OMG PRR, W3C RIF

Production Rules

* = in devt

ANSI SQL-based continuous queries

Page 34: Complex Event Processing Tutorial - Object Management Group

© 2008 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.34

workshop: DOC4RTES

Example CEP Technologies

Event Sources

Event Consumers

EventBus

Eve

nt

Bu

s

Event ServicesEvent ServicesEvent ServicesEvent Services Types of CEP ProcessingTypes of CEP ProcessingTypes of CEP ProcessingTypes of CEP Processing

Rule-Driven CEP

Inference

Rule EngineRulebase

Event Stream Query CEP

Query Engine Stream Queries

Neural Net CEP

Neural Net Network

Orchestrated Event Stream CEP

Flow Engine Flow Model

Page 35: Complex Event Processing Tutorial - Object Management Group

© 2008 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.35

workshop: DOC4RTES

Sample Event Processing Metamodel

Event-drivenBehavior

Inherits from

GeneralBehavior

StateManager

QueryProcess

Procedural

Process

Input

Output

Event

Inherits from

RuleProcess

Timer

Event

TimeOut

Data

State

Chg

External

Internal

Page 36: Complex Event Processing Tutorial - Object Management Group

© 2008 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.36

workshop: DOC4RTES

State Model

CEP Behavior: State-oriented

History and Cache

Pattern Detection Model

for recognizing patterns,defining actions

Temporal Model

for determining time-dependent info

Event and Data Model

for information modeling

Event Bus or Source

Low-latency reliable

message deliveryNew

OrderNew

OrderNew

Order

Condition /TimeEvent

State 1 State 2

Page 37: Complex Event Processing Tutorial - Object Management Group

© 2008 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.37

workshop: DOC4RTES

State Model / Process Flow CEP Agent features

1. Visual modeling metaphor

� State diagram / flow diagram is simple to follow

2. State / flow transitions can be time-related

� Can model missing events through time-outs etc

Page 38: Complex Event Processing Tutorial - Object Management Group

© 2008 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.38

workshop: DOC4RTES

Rule

CEP Behavior: Rule-oriented

History and Cache

Pattern Detection Model

for recognizing patterns,defining actions

Temporal Model

for determining time-dependent info

Event and Data Model

for information modeling

Event Bus or Source

Low-latency reliable

message delivery

Complex Patterns

NewOrderNew

OrderNew

Order

(Lots ofEvents)

Reaction

Page 39: Complex Event Processing Tutorial - Object Management Group

© 2008 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.39

workshop: DOC4RTES

Inference Rule CEP Agent features

1. High performance pattern matching

� Rete algorithm determines rules that are executable based on underlying data

changes

2. Declarative + Inferencing

� Rules defined in terms

of classes:

can be relevant

for any # instances

� Rules’ actions can cause

other rules to fire

automatically

3. In-memory

� Limited only by

JVM / process memory

Page 40: Complex Event Processing Tutorial - Object Management Group

© 2008 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.40

workshop: DOC4RTES

Query

CEP Behavior: Query-oriented

History and Cache

Pattern Detection Model

for recognizing patterns,defining actions

Temporal Model

for determining time-dependent info

Event and Data Model

for information modeling

Event Bus or Source

Low-latency reliable

message delivery

Complex Query

NewOrderNew

OrderNew

Order

(Lots ofEvents)

Results Event

Page 41: Complex Event Processing Tutorial - Object Management Group

© 2008 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.41

workshop: DOC4RTES

Query CEP Agent features

1. Common query language

� Usually SQL-based – widely used language

� May be in-memory, in-file or both

� Can include query optimizers

2. Continuous

� Extensions usually support time windows for the query to operate over

Page 42: Complex Event Processing Tutorial - Object Management Group

© 2008 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.42

workshop: DOC4RTES

Agenda

�Examples

Page 43: Complex Event Processing Tutorial - Object Management Group

© 2008 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.43

workshop: DOC4RTES

Typical Business Situations for CEP

Alert Customer AgentCustomer Y payment for policy P is

very late

Alert manufacturing and marketingOrders for product Z are up >20%

Issued an automated reminder to

supplier

Subcomponent delivery Y is slightly

late

Investigate for fraud manuallyUser X is behaving suspiciously (high

likelihood of fraud)

Resulting Situation-DecisionDetected Business Situation

Page 44: Complex Event Processing Tutorial - Object Management Group

© 2008 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.44

workshop: DOC4RTES

CEP in Action: Investment Banking

Front Office Middle Office Back Office

Data Feeds (events)

Archival etcDatabase

SOAServices

Algorithmic Trading CEP Application

Query

EngineQuerybase

Compliance CEP Application

State Engine

Inference

Rule EngineRulebase

State Model

Page 45: Complex Event Processing Tutorial - Object Management Group

© 2008 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.45

workshop: DOC4RTES

CEP

Rulebase State Model

Event Cache Other Data

NewOrder channelled

CEP Processing

Update

& Track

StatesClassification

& filter

(Re)Assess Order

vs other orders

FSA compliance

check required?

1. Order event arrives over JMS / EMS or RV on a particular channel

2. Order event classified into appropriate class, stored in cache for future reference

3. Order state assessed and updated due to event details

4. Comparison rules run against order by rule engine, new information inferred

5. State transition rules executed to check for any state change due to new inferred data.

6. If order can be completed, run FSA compliance rules

Page 46: Complex Event Processing Tutorial - Object Management Group

© 2008 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.46

workshop: DOC4RTES

Example CEP Product Architecture

CEP engine(s)

Query Agent(s)

Rule Agent(s)Rulesets

State Models

Functions

ES

B /

Messag

e B

us /

Even

t B

us

Queries

Event Preprocessor

Function

Distributed cacheof events etc

Cache Node(s)

BackupDB

Conceptual Model

Channels

Destinations

ReteWorking Memory

Events

Conceptual Model

Page 47: Complex Event Processing Tutorial - Object Management Group

© 2008 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.47

workshop: DOC4RTES

Agenda

�“Advanced” CEP

Page 48: Complex Event Processing Tutorial - Object Management Group

© 2008 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.48

workshop: DOC4RTES

“Advanced” CEP defined in many ways

� Intelligent CEP

� Adaptive

� Learning

� Logic

� AI

� Semantic CEP

� Ontologies + Logic

� Text / language interpretation

� Multiple CEP

� Including all types of data processing paradigm

(transactional, CLP, inference, mathematical methods, …)

Page 49: Complex Event Processing Tutorial - Object Management Group

© 2008 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.49

workshop: DOC4RTES

Advanced CEP Infrastructure

Event Sources

Event Consumers

EventBus

Eve

nt

Bu

s /

Sto

re

Event Processing CEP Agents

State Engine

Inference

Rule Engine

State Model

Rulebase

BI / Analytics tools

Graphical

Pattern DetectionDashboard

Graphical

ReportsOptimization & Machine Learning

Ontologies,

Logic MetaRulesBPM / SOA

Business User Interface

CEP

Editors

QueriesStreaming

Query Engine

Statistical

Pattern Detection

CLP

Engine Constraints

Page 50: Complex Event Processing Tutorial - Object Management Group

© 2008 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.50

workshop: DOC4RTES

Advanced Patterns & Event Behaviors

� Many EP apps fit the standard CEP patterns:

� Filter interesting rules

� Detect predefined patterns / state changes

� Update data / invoke processes and

services based on business rules and

high level events

� Advanced EP:

� Apply interesting statistical functions

to event data to detect new / complex

trends

� Apply different algorithms to event data

� Modify parameters used in other rules

(“metarules”)

Page 51: Complex Event Processing Tutorial - Object Management Group

© 2008 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.51

workshop: DOC4RTES

“Event-Decision” Architecture

24

EVENT PRE-PROCESSING

EVENTSOURCES

EXTERNAL

.

.

.

LEVEL ONE

Event Tracking & Refinement

User

Interface&

Systems

Event-Decision Architecture

DB MANAGEMENT

HistoricalData

Profiles &Patterns

DISTRIBUTED

LOCAL

EVENTSERVICES

.

.EVENT

PROFILES..

DATABASES

.

.OTHER DATA

LEVEL TWO

Situation Detection & Refinement

LEVEL THREE

Impact Assessment & Predictive

Analysis

LEVEL FOUR

Process Refinement &

Adaptation

-- Adapted from JDLSteinberg, A., & Bowman, C., Handbook of Multisensor Data Fusion, CRC Press, 2001

Page 52: Complex Event Processing Tutorial - Object Management Group

© 2008 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.52

workshop: DOC4RTES

Self-Modifying “Event-Decision” Rules

24

EVENT PRE-PROCESSING

LEVEL ONE

Event Tracking & Refinement

Event-Decision Architecture

DB MANAGEMENT

HistoricalData

Profiles &Patterns

LEVEL TWO

Situation Detection & Refinement

LEVEL THREE

Impact Assessment & Predictive

Analysis

LEVEL FOUR

Process Refinement &

Adaptation

� What are

the variablesthat can

be adjustedin real-time

to optimize

system performance?

Page 53: Complex Event Processing Tutorial - Object Management Group

© 2008 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.53

workshop: DOC4RTES

Pre-Processing Event Filtering Rules

24

EVENT PRE-PROCESSING

LEVEL ONE

Event Tracking & Refinement

Event-Decision Architecture

DB MANAGEMENT

HistoricalData

Profiles &Patterns

LEVEL TWO

Situation Detection & Refinement

LEVEL THREE

Impact Assessment & Predictive

Analysis

LEVEL FOUR

Process Refinement &

Adaptation

If RFID event for

product X

Then

monitor, else

ignore

Becomes

If RFID event for

product in list Y

where cost > Z

Then monitor, else

ignore

Updated by

If average loss increase

for all products in Y > 2%

Then reduce Z by 5%

Page 54: Complex Event Processing Tutorial - Object Management Group

© 2008 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.54

workshop: DOC4RTES

Event Tracking and Refinement Rules

24

EVENT PRE-PROCESSING

LEVEL ONE

Event Tracking & Refinement

Event-Decision Architecture

DB MANAGEMENT

HistoricalData

Profiles &Patterns

LEVEL TWO

Situation Detection & Refinement

LEVEL THREE

Impact Assessment & Predictive

Analysis

LEVEL FOUR

Process Refinement &

Adaptation

If drug class X

and dose > 200ml

Then

move to monitored

drug state

Becomes

If drug class X

and dose > Y ml

Then

move to monitored

drug state

Updated by

If clinical negative events for

drug class X increase

Then reduce Z by 10ml

Page 55: Complex Event Processing Tutorial - Object Management Group

© 2008 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.55

workshop: DOC4RTES

Event Tracking and Refinement Rules

24

EVENT PRE-PROCESSING

LEVEL ONE

Event Tracking & Refinement

Event-Decision Architecture

DB MANAGEMENT

HistoricalData

Profiles &Patterns

LEVEL TWO

Situation Detection & Refinement

LEVEL THREE

Impact Assessment & Predictive

Analysis

LEVEL FOUR

Process Refinement &

Adaptation

If bag X is not on

prescribed flight

at (depart – 20)

Then

move X state to

MissedFlight

BecomesIf bag X is not on

prescribed flight

at lastBagTime

Then

move X state to

MissedFlight

Updated byIf flight NOT international

Then set lastBagTime to

carrier’s min( DoorCloseTime)

Page 56: Complex Event Processing Tutorial - Object Management Group

© 2008 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.56

workshop: DOC4RTES

Event Tracking and Refinement Rules

24

EVENT PRE-PROCESSING

LEVEL ONE

Event Tracking & Refinement

Event-Decision Architecture

DB MANAGEMENT

HistoricalData

Profiles &Patterns

LEVEL TWO

Situation Detection & Refinement

LEVEL THREE

Impact Assessment & Predictive

Analysis

LEVEL FOUR

Process Refinement &

Adaptation

If

product.ShipDelay

> 1 days

Then

contactLegal

BecomesIf

product.ShipDelay

> contract.SLA

MaxDelay -

AllowedShipLag

Then

warnLegal

Updated byIf contract.customer.status = Hi

Then set AllowedShipLag to 2

days

Page 57: Complex Event Processing Tutorial - Object Management Group

© 2008 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.57

workshop: DOC4RTES

Issues

� Needs constraints

� Eg Cannot reduce discount to <0 or increase above>25

� Can handle as “change events” and rules to test…

� Difficult to test

� May be based on statistical functions – implies complex test regimes (or test-

specific rules)

� Complex to prove ROI / value

� End-user may not be able to source or validate the advanced rules

� Requires statistical function libraries / analytics

Page 58: Complex Event Processing Tutorial - Object Management Group

© 2008 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.58

workshop: DOC4RTES

Other sources for “advanced rules”

� Uncertainty

� Scoring

� Generating rules

� Machine learning

� Predictive Analytics

� Reasoning + Ontologies

� Other types of rules

� Constraint Logic Programming

Page 59: Complex Event Processing Tutorial - Object Management Group

© 2008 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.59

workshop: DOC4RTES

Scoring

� Simple technique to handle “variable” decisions

� Rules update a score

� Example: insurance scoring

� Typically handled in aspecial ruleset(or decision table)

� Good as a KPI in ascorecard

� Typically used with anaggregation rule

Score

Effect

ConditionObject

Property

-570+Age

050 to 69Age

+527 to 49Age

-1519 to 26Age

-10<18Age

Page 60: Complex Event Processing Tutorial - Object Management Group

© 2008 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.60

workshop: DOC4RTES

Machine Learning

� Given a set of data, deduce classification patterns and hence rules

� Requires sophisticated algorithms

4otherotherotherother

3ABAB

2BABA

1AAAA

resultdata

Page 61: Complex Event Processing Tutorial - Object Management Group

© 2008 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.61

workshop: DOC4RTES

Predictive Analytics

� Analyze data to deduce segmentation breaks for tasks like customer classifications

� Eg: Which customers should be offered what interest rate to maximise profit?

� Typically using specialist data mining tools

� Exports decision tree, rules etc in varieties of PMML

� Overlaps with BI (eg custom reports on historic data)

� Analytical functions may also be mapped to a ruleset in CEP for real-time analytics

Page 62: Complex Event Processing Tutorial - Object Management Group

© 2008 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.62

workshop: DOC4RTES

Reasoning + Ontologies

� “Semantic Event Processing”

� Use Semantic Web technologies to augment CEP

� Textual news etc analysis

� Use of deeper ontology relationships

� Example components

� OMG Ontology Definition Metamodel ODM

joins W3C OWL to UML concept models

� OWL, RDF, RDFS for terminology, relationships

� Logic languages/rules to reason about truth over event types and metadata

Page 63: Complex Event Processing Tutorial - Object Management Group

© 2008 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.63

workshop: DOC4RTES

Example: Semantic Technology to Refine CEP

event event event… … …

CEP + Production Rule reasoning

Notifications of Anomalies,

Course Corrections,

Instructions for Retraining & Recharacterization,

Actions, Queries

Sense events �

Assess conditions �

take action

• Features include:• Production rule reasoning can

• report up (situation assessment & current course of action)

• report laterally (situation assessment, & sensed changes, etc.)

• report down (initiating actions, querying)• Adaptive capabilities are possible at all levels

Semantic Networking Event Monitoring Architecture

-- courtesy of Sandpiper Software

CEP Engine+ Production Rules

Operational Decisions

Event Source / Bus

Event Classification & Aggregation

SPARQL/Pellet/OWL

+ Description Logics

Reasoning

+ Production Rule

ReasoningEvent

KB

event

SPARQL/Pellet/OWL

+ Description Logics

Reasoning

+ Probabilistic & Special Purpose ReasoningPattern &

Trend

KBLonger-Term Reasoning

Trend Analysis, Contextual Learning

Page 64: Complex Event Processing Tutorial - Object Management Group

© 2008 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.64

workshop: DOC4RTES

Constraint Logic Programming

� Constraint rules for systems

� Constraint solver to find best values (eg optimize price)

� With response time as a system constraint!

� Goal-driven

� Uses:

� Maximizing value of inventory

� Scheduling the best routes for trucks

� Maximizing probability for SLA achievement

Page 65: Complex Event Processing Tutorial - Object Management Group

© 2008 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.65

workshop: DOC4RTES

Agenda

�The End

Page 66: Complex Event Processing Tutorial - Object Management Group

© 2008 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.66

workshop: DOC4RTES

Agenda

�Q & A

Page 67: Complex Event Processing Tutorial - Object Management Group

© 2008 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.67

workshop: DOC4RTES

APPENDICES

A. Appendices & Back-up Information

Page 68: Complex Event Processing Tutorial - Object Management Group

© 2008 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.68

workshop: DOC4RTES

Appendix: Generalized Architecture for CEP

Event Sources

CEP Development

Event Consumers

Event Processing Application / Agent

CEP Test

CEP Monitor

BusinessUser Interface

VerificationTools

ValidationTools

Event Monitor

Event Visualization

Event Analytics

ModelRepository

Event HistoryWarehouse

DistributedEvent StoreEvent

Bus

Eve

nt

Bu

s

EventBus

Co

ntr

ol

Bu

s

BusinessModel

Event ProcessEditors

EP Control & Update

Event Processing components

CE

P P

rocessin

g

Engin

es

Page 69: Complex Event Processing Tutorial - Object Management Group

© 2008 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.69

workshop: DOC4RTES

Appendix: Useful web resources

� Event Processing Technical Society EPTSwww.ep-ts.com

� Luckham’s web sitecomplexevents.com

� Various vendor blogs (reference from complexevents.com)