Commercial Trends in Complex Event Processing
© 2009 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.
Processing
Paul Vincent,
CTO Business Rules and CEP, TIBCO
http://tibcoblogs.com/cep
We help customers…
Who is TIBCO?
We provide enterprise infrastructure software that
enables real-time awareness, understanding and action.
© 2009 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.
2
Improve operational visibility, collaboration and ability to be
proactive
Increase operational efficiency and effectiveness
Accelerate IT projects
What are we helping Customers handle?
Perform Trading Analytics
Handle Call Center Inquiries
Airline Operations
Track Financial/Risk Position
Supply Chain Updates
30 minutes
20 minutes
8 hours
1 day
1 day
5 seconds
30 seconds
10 seconds
5 minutes
15 minutes
Process Cycle-Time Improvements
© 2009 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.
3
Document Transfer
Settle Trade/Transaction
Build-to-Order Product
Refresh Data Warehouse
Supply Chain Updates
Phone/Service Activation
1 month
3 days
6 weeks
5 days
1 day
3 days
1 hour
Mail… express… fax… e-mail20 seconds
1 day
1 day
15 minutes
1 hour
Source: Gartner
Who TIBCO helps
47 of the World’s 100 Largest Companies are TIBCO Customers
Retail Banking — 17 of top 20
Hi-Tech Manufacturing — 15 of top 20
Investment Banking — 9 of top 10
Telecommunications — 8 of top 10
© 2009 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.
4
* By annual revenues except for investment banking which is measured by assets
Consumer Package Goods — 5 of top 10
Energy — 5 of top 10
Hi-Tech Manufacturing — 15 of top 20
Manufacturing (non High-tech) — 5 of top 10
Pharmaceutical — 6 of top 10
Transportation — 4 of top 10
How TIBCO Organizes IT Solutions
Increase operational
efficiency and effectiveness.
Improveoperational visibility,
collaboration and ability to
be proactive
BusinessOptimization
BusinessProcess
Management
© 2009 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.
5
Accelerate projects, initiatives, and
go-to-market cycles
Service-Oriented
Architecture
1. From Messaging to Event Processing
2. Event Processing vs CEP
3. Complex Event Processing – Who and When
© 2009 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.
6
4. CEP Evolution and Trends
5. Some Customer Examples
6. Future research topics
Networking – a brief history
Military, large systems
High costLow re-use
Financial investment etcsystems
High costHigh value
Internet era starts,Pub-sub
Low costCommodity
© 2009 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.
7
Custom Messaging/ communications
Generic Messaging+ application communications
1960s 1970s 1980s
Messaging – a brief history
1968 -Arpanet
1984 –Internet to TCI/IP
1974 -TCP
1985 –Teknekron
1986 –Banking
Middleware
1991 –CORBA
2001 –JMS
© 2009 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.
8
Custom Messaging/ communications
Generic Messaging+ application communications
1960s 1980s
JMS
1990s
TIBCO MOM – example of commercial MOM
� TIBCO Enterprise Message Service (EMS)
�Server-based store and forward
�J2EE and .NET
� TIBCO Rendezvous (RV)
�Peer-to-peerdistributed
�Multicast
© 2009 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.
9
�Multicast
� TIBCO SmartSockets
�Server-based messaging used in exchanges
� TIBCO Hawk
�Management framework for systems
1998•Portal•XML Schema Mgmt
1999•Full-featured Workflow•Single Solution for EAI & B2B
•Wrote the Book on RTB
2002•BAM•HIPAA and HL7 Support
•ebXML
1987EnterpriseMessaging
2003• EM Advisor
TIBCO – Evolution of a Middleware Company
1985Teknekron Software Systems
1994•Teknekron acquired by Reuters PLC; expanded use
2004• ENS• EMS• ESB• TIBCO
2005• Over
1000 ESB customers
1980s: commercial m/w
2000s: standards1990s: distributed systems
© 2009 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.
10
1985Publish/
Subscribe
1996•ApplicationMonitoring
•1st to support HTTP, FTP, SMTP, SNMP
2001SophisticatedXML & Web
Services,BusinessWorks
•Wrote the Book on RTB•1st in Production with RosettaNet PIPs
•OpenSourced PGM Messaging
•UML Modeling•1st to Support JDBC, Corba, JRMI, IIOP, COM Support
•ebXML Support
Messaging • EM Advisor• Enterprise
Messaging• BW Workflow• 1st integration
company w.> 2,000 customers
• Roadmap to Predictive Enterprise
Systems founded
2000EJB Adapter and full J2EE Support
expanded use in financial services clients
•EAI
• TIBCO revises Reuters relationship
customersNew products: • GI (AJAX)• Data
Exchange• Business
Events• RFID
Interchange
2000s: m/w exploitation eg SOA, BPM, CEP
1. From Messaging to Event Processing
2. Event Processing vs CEP
3. Complex Event Processing – Who and When
© 2009 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.
11
4. CEP Evolution and Trends
5. Some Customer Examples
6. Future research topics
Simple Event Processing
NewOrder
Business Process(BPM)
© 2009 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.
12
1
IT Services(SOA)
Database
Multiple Simple Event Processing
NewOrder
Cancel OrderProcess
New OrderProcess
etc etc
CancelOrder
NewSupplier
SLAChk Req
© 2009 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.
13
1
Process
IT Services …
Database
Process
Complex Event Processing
Continuous
NewOrderNew
OrderNew
Order
(Lots ofEvents)
(Complex) Event
© 2009 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.
14
1
ContinuousEventProcessing
(Complex) EventPatterns
Workflow Processes (BPM)
IT Services (SOA)
EventStore
Where is the information?
CustomerLogon
Fed Base RateIncrease
CustomerChecks
Close AccountWeb Page
NewOrder
Contract
© 2009 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.
15
1
ProductionItem
Arrives at Store
NewLiabilityAdded
RentalCar
Returned
ContractSubmitted
ContractReturnedthru EDIRental
CarCrashed
Mobile Callfrom [email protected]
Where is the information?
Fraud Risk!
Risk of CustomerDefection
CustomerCrossSell
Opportunity
Change inProduct Sales
Trend
Contract
© 2009 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.
16
1
EmployeeOver hours
ComplianceLimit
Approached
Customernow rated
Gold
ContractValidated
ContractValid
RentalContract Complete
Cell phonefraud alert
Complex Business Problems
� Fraud / Theft
� Thousands-to-millions of high-value small-size product items or transactions
� How do you identify known patterns of “suspicious” behavior?
� Logistics / Scheduling
� Raw material, production & delivery scheduling and resources are complex and prone to change
How do we reallocate resources to handle
Product location change
Relevant event of interest
Resource,
© 2009 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.
17
1
� How do we reallocate resources to handle business and production changes?
� Activity Monitoring
� Complex production and supply process with multiple actors
� How to measure and action Key Performance Indicators?
Resource, requirement change
Process data change
Associated Events
� Positive Events
� Product item X arrives at Production station S from Store T
� Production worker Y arrives at Production station S
� Production contract for item Z by time T is posted
� Negative Events
� Product item X has been in transit to Store T for >15 minutes
© 2009 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.
18
1
� Subcomponent Y hasn’t arrived at the Production station by the ETA
� Delivery of contract Z has not taken place
� Sets of Events
� 5+ items of Product item type Y failed to arrive at destination
� Supplier Y was 5 mins late for 1 delivery, but made it early to the next
� Return rate on component Z exceeds SLA %
Significant features of these Events
� Time Sensitivity
� A thief may leave the building at the same time as stolen product
� A product should take 40 minutes to travel a given production line segment
� Distributed Event Sources
� A series of produced items fails at various QA stages, and
15:39:26.413
Dec 24
© 2009 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.
19
1
� A series of produced items fails at various QA stages, and their common attribute was a storage location
� Multiple suppliers for a subcomponent are reporting delivery delays
ERP Supplier
TIBCO Software Stack
BPM / WorkflowEg TIBCO iProcess
Visual BIEg TIBCO Spotfire
© 2009 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.
20
Middleware / Enterprise Service BusEg TIBCO Rendezvous, TIBCO EMS (JMS)
SOA SupportEg TIBCO ActiveMatrix, BusinessWorks, …
Eg TIBCO iProcessCEP
Eg TIBCO BusinessEvents
Complex Event Processing
© 2009 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.
21
2
1. From Messaging to Event Processing
2. Event Processing vs CEP
3. Complex Event Processing – Who and When
© 2009 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.
22
4. CEP Evolution and Trends
5. Some Customer Examples
6. Future research topics
CEP – a brief history
Military, Logistics etcsystems
High cost / analogLow re-use
Financial investment etcsystems
High costHigh value
Commercialsystems
Lower costROI
© 2009 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.
23
Custom CEPimplementations
Generic CEP tooling
1960s 1990s 2000s
Complex Event Processing
Join
Decide
React
© 2009 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.
24
2
Filter
Join
Military Sensor Data Fusion as Custom CEP
EVENT PRE-PROCESSING
EVENTSOURCES
EXTERNAL
LEVEL ONE
EVENTREFINEMENT
User
Event-Decision Architecture
DISTRIBUTED
LOCAL
EVENTSERVICES
.
LEVEL TWO
SITUATIONREFINEMENT
LEVEL THREE
IMPACT ASSESSMENT
Filter
Join
React
© 2009 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.
25 24
.
.
.
UserInterface
DB MANAGEMENT
HistoricalData
Profiles &Patterns
.
.EVENT
PROFILES..
DATABASES
.
.OTHER DATA
LEVEL FOUR
Process Refinement
Adapted from JDL: Steinberg, A., & Bowman, C., Handbook of Multisensor Data Fusion, CRC Press, 2001
Decide
Generic CEP Architecture
Event and Data Model
for information modeling
Event Bus or Source
Low-latency reliable message delivery
© 2009 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.
26
History and Cache
Pattern Detection Model
for recognizing patterns,defining actions
Temporal Model
for determining time-dependent info
2008
2009
CEP – Vendor Offerings
1st
© 2009 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.
27
199920002001
2005
2006
2007
20032004 2002
CEP - Market Players
© 2009 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.
28
DroolsFusion
Oracle CEP
CEP - Marketshare, by #Customers
300
400
500
600
Nu
mb
er
of
Cu
sto
mers
Reported CEP Customers to 2009
IBM
Oracle
© 2009 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.
29
0
100
200
Nu
mb
er
of
Cu
sto
mers
Year
Oracle
Streambase
Aleri&Coral8
Apama
TIBCO
©TIBCO Software Inc
1. From Messaging to Event Processing
2. Event Processing vs CEP
3. Complex Event Processing – Who and When
© 2009 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.
30
4. CEP Evolution and Trends
5. Some Customer Examples
6. Future research topics
2008
2009
Example CEP Evolution: TIBCO BusinessEvents
BE 1.4+ AIX deploy
BE 2.1+ Persistence control
BE 2.2+ BW runtime integration
BE 3.0+ Business User Interface+ Query support+ Shareable Cache+ Distributed model
BE 4.0?????
© 2009 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.
31
2005
2006
2007
20032004 2002
BE 2.0+ Debugger+ Analyser+ API+ Exceptions
BE 1.3+ BW Designer
(SOA co-development)
BE 1.0+ Complex Event Proc’ing+ Win / Sol deploy
BE 1.1+ Rule fns+ XPath+ Linux deploy
BE 1.2+ Tester+ HPUX deploy
+ AIX deploy + Persistence control
Customer Project+ High performance event patterns
� Event-driven and continuous processing of events
Basics for CEP: Events, Data
Concept Model
- Data Model (eg UML Class model for
fixed relationships)
Event Model
- Events read from Channels(eg custom, RV, EMS)
- Payload, Metadata (eg
© 2009 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.
32
Event Processing Application
Eve
nt
So
urc
es
Event Bus
Eve
nt
Sin
ks
Event Model Class Model Libraries
Local Event Store
fixed relationships)- Payload, Metadata (egTimeToLive, expiry action)
� Complex event processing models: continuous correlations of events
Basics for CEP: Event Processing behaviour
Event Processing
High Performance CEP
- Event Processing models (eg code generation)
- “Event processing” includes detecting patterns across
© 2009 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.
33
CEP Tool
Eve
nt
So
urc
es
Event Bus
Eve
nt
Sin
ks
Event Model Class Model Libraries
Local Event Store
detecting patterns across events, past events (data,
event history), time, state…
Low Latency CEP
- In-Memory event and data store for maximum
performance
CEP Model 0: Procedural Rule
Event and Data Model
for information modeling
Event Bus or Source
Low-latency reliable message delivery
NewOrderNew
OrderNew
OrderTrigger
© 2009 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.
34
Rule Function
History and Cache
Pattern Detection Model
for recognizing patterns,defining actions
Temporal Model
for determining time-dependent info Invoked / Programmed Reaction
Supported thru shared libraries, etc
CEP Model 1: Production Rule-oriented
Event and Data Model
for information modeling
Event Bus or Source
Low-latency reliable message delivery
NewOrderNew
OrderNew
Order
(Lots ofEvents)
© 2009 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.
35
Rule
History and Cache
Pattern Detection Model
for recognizing patterns,defining actions
Temporal Model
for determining time-dependent info Patterns Reaction
Rule-driven CEP for complex patterns, decisions
CEP Rule ElementsKnowledge-driven CEP
- Inference Rule Engine for- event-condition-action- Infering new events
- Working across multiple events and data in memory
© 2009 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.
36
CEP Tool
Eve
nt
So
urc
es
Event Bus
InferenceRule Engine
Rulebase
Eve
nt
Sin
ks
Event Model
Local Event Store
Class Model Libraries
Inference Rule CEP Agent features
1. High performance pattern matching
� Eg Rete algorithm determines rules that are executable based on underlying data changes
2. Declarative + Inferencing
� Rules defined in terms of classes: can be relevant
© 2009 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.
37
can be relevant for any # instances
� Rules’ actions can cause other rules to fire automatically
3. In-memory
� Limited only by JVM / process memory
CEP Model 2: State-oriented
Event and Data Model
for information modeling
Event Bus or Source
Low-latency reliable message delivery
NewOrderNew
OrderNew
Order
Condition /TimeEvent
© 2009 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.
38
State Model
History and Cache
Pattern Detection Model
for recognizing patterns,defining actions
Temporal Model
for determining time-dependent info State 1 State 2
Modeling complex event lifecycles with states
CEP State Elements Dynamic State for CEP
- State Machine for event lifecycle modeling via states
and state transition rules
© 2009 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.
39
CEP Tool
Eve
nt
So
urc
es
Event Bus
State Engine
State Model
Eve
nt
Sin
ks
Event Model
Local Event Store
Class Model Libraries
State Model / Process Flow CEP Agent features
1. Visual modeling metaphors
� Eg State diagram / flow diagram is simple to follow; UML standard
2. State / flow transitions can be time-related
� Can model missing events through time-outs
© 2009 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.
40
through time-outsetc
CEP Model 3: Query-oriented
Event and Data Model
for information modeling
Event Bus or Source
Low-latency reliable message delivery
NewOrderNew
OrderNew
Order
(Streams ofEvents)
© 2009 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.
41
Query
History and Cache
Pattern Detection Model
for recognizing patterns,defining actions
Temporal Model
for determining time-dependent info
Continuous Query
Results Event
Query-based Patterns for Event Streams
Event Stream Processing
- Continuous queries fordetecting patterns across
event streams with SQL-type constructs
Query Elements
© 2009 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.
42
CEP Tool
Eve
nt
So
urc
es
Event Bus
Eve
nt
Sin
ks
Event Model
Local Event Store
Query Engine
Query List
Class Model Libraries
SQL/OQL-based Continuous Query Languages
� Extensions allow for:
�Streaming Events
�Continuous Queries
� Notifications or events when changes occur in resultset
© 2009 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.
43
Query Language – Time Window Example
� Retrieve certain tick info from cached trade data over some period
�select … from …{policy: maintain last … } where … group by …
© 2009 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.
44
Runtime: Distribute for resilience, scalability
CEP Agent
State Engine
Inference
State Model
Scalable CEP
- Replicated CEP Agents forload-balancing and
hot-standby
CEP Agent
State Engine
Inference
State Model
© 2009 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.
45
CEP Tool
Eve
nt
So
urc
es
Event Bus
InferenceRule Engine
Rulebase
Eve
nt
Sin
ks
Event Model
Shared Event Store
Class Model Libraries
InferenceRule Engine
Rulebase
Distributed Support
� Distribute event processing agents to load-balance across enterprise
� All agents / engines may share / co-operate on the same
Primary Engine(s) / Rule Agent(s)
Events
Secondary Engine(s)
Query Agent(s)
© 2009 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.
46
operate on the same event store
� Distributed cache etc used for high performance event stores
� Allows exceeding of single process limits
RecoveryDatabase
Fault Tolerant Data Cluster
Cache Node Cache Node(s)
Rule Management, Multiple Event Processing paradigms
CEP Rule Agent
State Engine
Inference
State Model
CEP Query Agent
Query Engine
DecisionManagement
CEP Editors Managed Entities
RealtimeDashboard
Views
Managed Decisions
- workflow-enabled Decision Management
for Business control of CEP Elements
CEP Framework for Multiple
© 2009 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.
47
Distributed CEP Framework
Eve
nt
So
urc
es
Event Bus
InferenceRule Engine
Rulebase
Data Grid
Eve
nt
Sin
ks
Query Engine
Query List
Event Model Class Model Libraries
CEP Framework for Multiple Paradigms / EPLs
- Event Processing Elementof multiple types
Example: Decision Manager
© 2009 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.
48
Decision Lifecycle: BRMS and BPMS features…
� Decisions managed by end-users need controlled lifecycles
� Control process can be managed by external applications
� Cover also simple event pattern specifications (etc)
© 2009 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.
49
Complex Event Processing Platform and Data Grid
CEP Rule Agent
State Engine
Inference
State Model
CEP Query Agent
Query Engine
DecisionManagement
CEP Editors Managed Entities
RealtimeDashboard
Views
CEP Platform
- distributed Event Processing Agents
gives scalability in volume + complexity
Shared Low-Latency Data
© 2009 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.
50
Distributed CEP Framework
Eve
nt
So
urc
es
Event Bus
InferenceRule Engine
Rulebase
Data Grid
Eve
nt
Sin
ks
Query Engine
Query List
Event Model Class Model Libraries
Shared Low-Latency Data
- distributed event history Data Grid
for Low Latency High Volumes information
Co-operative Distributed Event Processing Agents
Data Grid
Same-MemorySame-Process
Agents
Same-MemoryDifferent-Process
Agents
RuleAgent3
Rule
Different-MemoryDifferent-Process
Agents
Different-MemorySame-Process
Agents
RuleAgent6
RuleAgent7
© 2009 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.
51
MultipleData spaces
RuleAgent1
RuleAgent2
Load Balancing
RuleAgent4
Query Agent5
Co-Operative Processing
RuleAgent8
StandaloneProcessing
PartitionedData
Processing
Co-operative Distributed Event Processing Agents
Same-MemorySame-Process
Agents
Same-MemoryDifferent-Process
Agents
RuleAgent3
Rule
Different-MemoryDifferent-Process
Agents
Different-MemorySame-Process
Agents
RuleAgent6
RuleAgent7
Project Resources
ReplicatedRuleset
& State Model F
Ruleset B
Ruleset C
Queryset D
De
cis
ion
Ta
ble
X
De
cis
ion
Ta
ble
Y
© 2009 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.
52
RuleAgent1
RuleAgent2
Event Filtering &Tracking
RuleAgent4
Query Agent5
SituationAssessment
RuleAgent8
SLA & Health
Monitoring
ProcessRefinement & Executive
Ruleset andState Model G
ReplicatedRuleset
& State Model A
DynamicQueryset E
MultipleSharedModels
PredictiveAnalysis
Real-timeAnalytics
Real-timeDecisions
Decis
ion
Managem
ent
De
cis
ion
Business Control ofCEP
Summary of Trends
1. Automated Trading was but 1 CEP use case (+ 4 CEP vendors) – Rest Of World now moving to CEP
2. CEP exploiting / bringing together other s/w development best practices eg MDA
3. CEP – processing of events is “a process” –business process automation
© 2009 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.
53
business process automation
4. EDA vs SOA
5. Advanced software techniques for boundary cases eg pattern discovery via analytics
6. Exploiting faster networking, cache event storage, etc
1. From Messaging to Event Processing
2. Event Processing vs CEP
3. Complex Event Processing – Who and When
© 2009 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.
54
4. CEP Evolution and Trends
5. Some Customer Examples
6. Future research topics
Commercial CEP – creating Use Cases
Customer, Vendor discuss problem areas
Vendor propose solution using CEP technology
© 2009 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.
55
Customer agrees to solution
Development (in-house / vendor / SI)
Production success = use case
CommercialCEP Use Cases
from TIBCO
Logistics – Track and Trace
© 2009 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.
56
Logistics – Track and Trace
Transport – Situation Awareness
Finance – Sense and Respond
Logistics – Example#1
� Problem: Timeliness of reports for border-crossings
�Report data is event-driven, asynchronous(manifests, load data, forwarded after loads picked up)
�Border crossings required compliance to
© 2009 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.
57
�Border crossings required compliance to regulations, leading to delays in crossing / non-compliance fines / missed schedules
�Requirement: means of processing data “just in time” and handling missing information correctly
� Wider Problem: handling process exceptions
Logistics Transaction Monitoring = Track and Trace
� Goal: models that provide end-to-end visibility of business processes
�System-to-system, manual and semi-automated processes
�Need to track missing events, SLA violations, and
© 2009 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.
58
58
�Need to track missing events, SLA violations, and then re-sequence activities based on context
� Identify failures, and re-route according to data-driven workflow rules
Monitoring Processes and Systems: CEP as BAM
BusinessWorks – Service layer
TIBCO General Interface – BAM UI Monitoring Actions
Integration services layer: allows exposure of KPIs via WebServices
Task & Alert MgtReal-time KPI dashbordKPI AnalyzerGateways to BI apps…
© 2009 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.
59
BusinessEvents - CEP
TIBCO Enterprise Message Service
Working MemoryKPI, thresholds
and alerts
Cache ServerStores the concepts that
are less frequently accessed
Event filtering and correlation,KPI calculation, alert generation
I.S.
Captures and transports events comingfrom the Information System
Finance – Example#2
� Problem: Personalization of marketing and CRM
�Customer “lifecycle” events drive needs and interest in financial services
�Awareness of customer situation can be implied from data mining, analysis, etc
© 2009 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.
60
implied from data mining, analysis, etc
�Requirement: means of providing customer-relevant communications, assistance, offers etc
� Wider Problem: intelligent CRM
Bank Customer Interaction Management - Sense & Respond
� Goal: to provide relevant advice to customers during multiple interactions across various + multiple channels
� Relevant
� Requires knowledge of customer, current financial status, history of interactions, soft-data (e.g. life events), in real-time or as batch
� Advice
© 2009 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.
61
61
� Advice
� Offers/promotions/campaigns that are targeted at a particular customer (e.g. travel insurance promotion, session w/ financial advisor)
� During interactions / across channels
� Whatever the interaction is (service call or request for info)
� Relevant advice must be generated during the customer’s attention span and presented in real-time in the relevant medium
� Supporting multiple channels
� Web, phone, ATM, agent, etc.
NewRules/ Patterns
Transactions
Event Channels
CEP Rules
CEP Patterns
CEP State Models
Sense
Channels
Customer Interaction
Pluggable Algorithms
t0 tn
Events Responses
Customer Interaction Management Example
© 2009 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.
62
62
Products
Customers
Data Mining
Event Channels
Internal
Knowledge
External
Knowledge
Respond
Knowledge
Customer Interaction
Pluggable Algorithms
DW /Customer
Integration Architecture & Couplings
Integration layer
To
uch
po
int
sBusiness Users
Desktop
CEP Integration Integration
1. Mkt defines situation patterns
3. Combined with master and historical data
4. Patterns are applied,
Offers created/modified
6. Users monitor measure, optimize
© 2009 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.
63
CustomerAccounts /
Relationships/ CampaignsT
ou
ch
po
int
Transactional Banking Systems
?
CEP Business
Events
IntegrationServices
& Adapters
IntegrationServices
& Adapters
IntegrationServices / Adapters
Log / Exp
Log / Exp
Log / Exp
Log Rec
MailEmailPhoneStatementingWebATM
2. Trigger Events are received
data
5. Track Customer Communication
Sample for Pattern-Rule: Mortgage offer
Marketing defines campaign rules:
IF Customer has large funds-in transaction (> 10 K)
AND Customer also uses mortgage calculator on web-site
AND both events happen within 1-2 days
AND customer profile matches basic eligibility criteria
© 2009 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.
64
AND customer profile matches basic eligibility criteria (delinquency, blocked, est. affluence, opt-out)
THEN
Create “Mortgage offer” for customer in central palette
with highest priority
available through channels Web, Call-Center, Sms, ATM, Telemarketing
Additional Banking CEP-CRM Uses
� New Mover detected ���� Personal loan
� Flight ticket purchase ���� Travel insurance
� Outbound Contact failed ���� Service offer to supply contact details
� Major credit card transaction ���� Conversion into
© 2009 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.
65
� Major credit card transaction ���� Conversion into loan installment
� Significant Funds-In transaction detected ���� time
deposit offer
� ...
Transport – Example#3
� Problem: Information loads on gate agents
�Multiple sources of real-time information and problems
�Responses to “problem scenarios” (late flight, weather, ...) require full information
© 2009 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.
66
flight, weather, ...) require full information awareness
�Requirement: means of providing all required event data
� Wider problem: Management information in real time and optimizing responses to situations
Airlines Operations - Situational Awareness
� Operational Efficiency and Problem Detection
� Goal: Monitor operational data/events and communicate “problem events” to recovery engines and other systems
� Listen to multiple sources of data (dispatchers, maintenance control, customer service, ATC/weather specialists, etc.).
� Pass data through a series of dynamic rules, and generate action events.
� Sample rules include:
© 2009 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.
67
67
� Sample rules include:
� 1. Aircraft forecast late coming out of MX (maintenance).
� 2. Flight delayed over 30 minutes.
� 3. MX Item exceeds its die date.
� 4. Flight estimated to arrive during airport curfew.
� 5. A flight had an out-gate time 30 minutes ago: no OffGroundmessage has been received.
� 6. MX station is over capacity for scheduled work.
� 7. Taxi time from gate to takeoff has been greater than 30 minutes
Sample Architecture
Rules and Patterns related to State Change, Validation, Optimization, Situational Awareness
OptimizationLP/MIP ModelsAircraft and Crew
SolverProblem Detected
Situational Awareness(Present Problems so that user can understand the situation at a glance)
Set of Solutions
Solutions
CEP - TIBCO BusinessEvents
© 2009 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.
68
Operational Data: Aircraft, Crews, Passenger, Maintenance, Ground, etc.
TIBCO Messaging
Other Data Sources
Events
Operational Events: Aircraft State, Passenger State, Weather, Airport, ATC…
NotificationsAggregate Problems By TypeFilter By SeverityString Graphs
Problem Store
Transport – Example#4
� Problem: Management of complex railway network resources
�32K miles of track, 48K employees, 7.5K locos, 87K wagons
�Network schedules difficult to keep due to 3rd
© 2009 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.
69
party events, staffing regulations etc, track performance: 1 kph = $MM per month
�Requirement: situation awareness
Issue: Reporting and Response Latency
Evolving from an
“after the fact”
© 2009 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.
70
“after the fact”
awareness to
Real time action.
Railways - Dynamic Resource Management
� Each train demands the 5 critical resources over time
� For a train plan to be effective, all trains must have all resources allocated
Time ����
Train Schedule
Crew Assignment
Train Demand Dimension ����
Resource Supply Dimension ����
© 2009 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.
71
all resources allocated
� As the train schedule moves, the resource demand moves
Network Capacity Allocation
Car Assignment
Terminal Resource Allocation
Locomotive Assignment
Crew Assignment
Dynamic Resource Management Requirements
� Provide a Train Lineup with the dynamic updating of train schedules
� Associate trains on Train Lineup with crews
� Improve train movement projections
� Detect opportunities to improve quality of decisions.
© 2009 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.
72
� Detect opportunities to improve quality of decisions.
� Assess the quality of the Operating Train Plan
Dynamic Resource Management IT Requirements
� Model and Manage Hundreds of Events, Data Values and Relationships Between Individual Crews, Trains, Terminals, and Network Segments, Points, and other entities
� Model and Manage Hundreds of Rules depending on
© 2009 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.
73
above relationships and data values
� Enable Dynamic Allocation of hundreds of Resources separated by hundreds of miles of track
How Did CEP Help? Event, Data Models
© 2009 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.
74
How Did CEP Help? Schedule Lifecycles and States
State Machine to model and monitor a train’s progress over a route
© 2009 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.
75
Track Segment Resource Configuration
© 2009 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.
76
Specific
Certification
Event Visualization via AJAX
AJAX BASED IDE FOR EVENT & SITUATION VISUALIZATION(RICH CLIENT WEB APPLICATION)
© 2009 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.
77
Telco – Example#5
� Problem: Fulfillment of complex product orders liable to change
� Processes are dynamic
� Events cause process changes
© 2009 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.
78
Process Component
Library
Goal Driven BPM
For Each Request
Order Plan Plan
© 2009 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.
79
Repeating Process ComponentsRepeating Process Components
Order Plan Development
Order Plan Execution
Order Request
4. Order Plan ExecutionOrders
Installed BaseTech Product
Cat
3. Order Plan Development
1. Store Order Request
5. Order Complete
Goal-Driven BPM example
3 types…• Manual• Template• Automatic
Order Entry
2 types available…• BPM-centric• Straight-through
© 2009 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.
80
2. Identify & Start
Fulfillment Process
RulesOrder Plan
Order Plan Designer
�Uses enhanced GANTT chart notation
�Displays run-time status of plans
�Capable of grouping plan tasks
� GANTT summary task notation
� Possible to define dependencies from/to groups
© 2009 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.
81
� Nesting possible to any level
� Possible to import order plan templates as groups
�Browser based
�Used for Manual Order Plan Development
An example Order Plan
Process Component
This maps to underlying BPM process
© 2009 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.
82
The Orchestration component is agnostic of the underlying process
engine.
Milestone
The synchronisation points within process
componentsDependency
The linkage between process components. Information may
flow through dependencies
Critical path
The longest path through a plan. Defines the projected end date/time of the plan
Goal-Driven Process Components
StartEvents
Change
DecisionProcessing
Plan Selectionrules
Goal-drivenPlans
Plans
TemplatesDynamic
PlansManualPlan Building
© 2009 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.
83
Change Events
BusinessProcesses
andServices
Business ProcessEngine
workflow flows
SOA Services
service flows
TemplatesPlans
ManualPlan Buildingrules
Rule-basedautomatedprocesses
iProcess Conductor + Rule-driven Events
iProcess Conductor
Metadata TIBCO
SemanticModeler
TIBCO BusinessEvents Workbench
StateModeler
Event Management,Correlation, Aggregation,Inferencing and Analysis
Rules, Knowledge, Patterns, Models
Activity
Optimised Order PlanOptimised Order Plan
© 2009 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.
84
Product Definitions Service Image Order Content
Collection
Normalized Data
Metadata Repository
TIBCO BusinessEventsInference Engine
Rule EditorFormulaEditor
CollaborativeIntegration
ContextualIntegration
Normalization
ActivityHistory
TIBCO iProcess
Conductor+ BE
iProcess Conductor in Operation: Fulfilment
Order Request
Orchestrator
Order PlanTemplates
Orders Rules
Order stored in Order stored in Order stored in Order stored in Order repositoryOrder repositoryOrder repositoryOrder repository
Static BPM e.g. Static BPM e.g. Static BPM e.g. Static BPM e.g. feasibilityfeasibilityfeasibilityfeasibility
Process decides that it Process decides that it Process decides that it Process decides that it is ready for dynamic is ready for dynamic is ready for dynamic is ready for dynamic
part (Order Plan part (Order Plan part (Order Plan part (Order Plan
Template chosen Template chosen Template chosen Template chosen using selection Ruleusing selection Ruleusing selection Ruleusing selection Rule
Instantiate selected order plan Instantiate selected order plan Instantiate selected order plan Instantiate selected order plan and adds process components and adds process components and adds process components and adds process components
to BPM layerto BPM layerto BPM layerto BPM layer
Order line status updated Order line status updated Order line status updated Order line status updated as Process Components as Process Components as Process Components as Process Components
completecompletecompletecomplete
Using a Rule, identify and Using a Rule, identify and Using a Rule, identify and Using a Rule, identify and start correct fulfilment start correct fulfilment start correct fulfilment start correct fulfilment
processprocessprocessprocess
© 2009 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.
85
TIBCO BPM
Fulfilment Process
ProcessComponent
Versions
DynDynDynDyn
feasibilityfeasibilityfeasibilityfeasibilitypart (Order Plan part (Order Plan part (Order Plan part (Order Plan Development)Development)Development)Development)
to BPM layerto BPM layerto BPM layerto BPM layer
Consult Order Plan and starts Consult Order Plan and starts Consult Order Plan and starts Consult Order Plan and starts first Process Component in first Process Component in first Process Component in first Process Component in
BPM layerBPM layerBPM layerBPM layer
BPM engine executes Process BPM engine executes Process BPM engine executes Process BPM engine executes Process Component. Last step is status Component. Last step is status Component. Last step is status Component. Last step is status
messagemessagemessagemessage
Consult Order Plan again and Consult Order Plan again and Consult Order Plan again and Consult Order Plan again and issue next start message, until issue next start message, until issue next start message, until issue next start message, until
Plan is completePlan is completePlan is completePlan is completeAlternatives are Alternatives are Alternatives are Alternatives are
Manual or AutomaticManual or AutomaticManual or AutomaticManual or Automatic
1. From Messaging to Event Processing
2. Event Processing vs CEP
3. Complex Event Processing – Who and When
© 2009 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.
86
4. CEP Evolution and Trends
5. Some Customer Examples
6. Future research topics
Research Topics for PhDs: technology
1. Event and Data representationse.g. RDF, ontologies for CEP
2. Event Pattern Discovery e.g. algorithms for detecting new types of pattern
3. Event-based Analyticse.g. Event-at-a-time statistics etc
© 2009 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.
87
e.g. Event-at-a-time statistics etc
4. New visualizations for event processese.g. 3D representations
5. Merger of decision / BPM technologiese.g. BEMN vs BPMN2, constraint solvers vs EP
6. Exploiting novel h/we.g. GPU processing for fast event processing
Research Topics for PhDs: use cases
1. Social Event Processinge.g. Personal Event Management and awareness
2. Novel Sensors e.g. Exploiting personal weather stations, cellphones
3. Socially Beneficial Applications
© 2009 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.
88
3. Socially Beneficial Applicationse.g. Bio- and Medical-statistics / CEP in healthcare
Research Topics for PhDs: caveats
1. CEP industry is v dynamic=> don’t follow industry, lead it
2. Build-on / exploit prior art, COTS technologies=> vendors encourage academic research
3. Exploit vendor relationships=> join EPTS, use Glossary, help standards
© 2009 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.
89
=> join EPTS, use Glossary, help standards
The End
� TIBCO view has been that CEP is simply a solution to real-time business optimization problems
� Multiple CEP examples
© 2009 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.
90
outside of investment banking now operational (TIBCO market share >40% per IDC)
� Interest from IBM, Oracle, Microsoft will increase awareness in the benefits of CEP
ComplexEvent
Processing