Proactive eth talk

© 2011 IBM Corporation

IBM Haifa Research Lab

Proactive event-driven computing

Towards Proactive Event-Driven Computing Talk in ETH – March 2012

Opher Etzion ([email protected])


© 2012 IBM Corporation2IBM Haifa Research Lab

The source code movie (spoiler)

The hero of the story is sent to the occupy theBody of a dead person during the last 8 minutes of

His life, trying to find out who put dirty bomb insideA train so he’ll be stopped from doing the next attack-

In an unexpected turn of events he succeeds toEliminate the attack and change the past



The proactive event-driven principle

time

Proactive action

Forecast

Real-time decision

Detect

now



Proactive traffic management system



The proactive pattern

DetectForecast

Act(proactive)

Monitoring streams of events from sensor in highway and leading ways, from mobile devices, and from accidents reports

Monitoring streams of events from sensor in highway and leading ways, from mobile devices, and from accidents reports

Forecasting that at some point in 15 minute a traffic congestion of certain size will occur in probability of 0.6

Forecasting that at some point in 15 minute a traffic congestion of certain size will occur in probability of 0.6

Taking proactive actions in setting up entry and exit traffic lights durations and speed limit in highway segments

Taking proactive actions in setting up entry and exit traffic lights durations and speed limit in highway segments

Decide(RT)



Proactive energy use for home consumers/producers




DetectForecast

Act(proactive)

Monitoring sun, wind and demand on power grid Monitoring sun, wind and demand on power grid

Forecasting that in the morning hours the household will not produce any energy and the power grid’s price will be high

Forecasting that in the morning hours the household will not produce any energy and the power grid’s price will be high

Using RT optimization – schedule appliances use and apply through actuators

Using RT optimization – schedule appliances use and apply through actuators

Decide(RT)



Proactive post-earthquake disaster management system




DetectForecast

Act(proactive)

Monitoring earthquake, spread by sensors, and citizen reports

Monitoring earthquake, spread by sensors, and citizen reports

Forecasting that at some point in the next hour there is going to be a a potential damage in a certain location

Forecasting that at some point in the next hour there is going to be a a potential damage in a certain location

Taking proactive actions in notifying and performing actions like – close roads, reduce speed of trains, turn off gas and water supply…

Taking proactive actions in notifying and performing actions like – close roads, reduce speed of trains, turn off gas and water supply…

Decide(RT)



The evolution towards proactive computingThe evolution towards proactive computing

Typically people employ computing in responsive way: the person makes decisions and the computer assists in data, knowledge, advice

The vision is to move to proactive computing: (Detect-Derive-Predict-Decide now-Do)

X

Recently, there is more employment of computers in reactive way: events drive decisions (Detect-Derive-Decide-Do)

The initiative remains in human hands;most persons are not proactive by nature

The initiative moves to the computer; reactions to events that already occurred

The initiative moves to the computer; actions to events before they occur



Why it is difficult to create proactive solutions now ?

A way of thinking

Incompatible programming model of the moving parts, and gaps in each of them within the current product

Event Processing

Predictive analytics Optimization Decision models

Multiple skills are needed



Proactive computing as cultural change The culture in many organizations and personal behavior advocates a routine behavior governed by fixed set of rules

Many people are deterred from ad-hoc behavior even if it has relative benefit in specific case and prefer statistical metrics .

Current analytics tools are geared towards improving the “fixed set of rules ”

Proactive thinking is different – it provides exception behavior to mitigate or eliminate problems when current rules will not work



Scalable platform


uncertain events, future

events, correctness

issues

Event processing foundations

adaptive real time

optimization for proactive decisions

human interaction in

proactive systems

Paradigm: methodology, seamless programming model

Integrative platform and validation

event recognition,

expertforecasting

models,goal driven supervised

learning

Event recognition and forecasting

Event-based optimization

Human computer interaction

The Proactive pillars



Event-flow programming model: the EPN

Event Producer 1

Event Consumer 1

Event Consumer 2

Event Producer 2

Event Consumer 3

Agent 2

Channel

Agent 1

State

Agent 3



EPA

EPA

EPA

Producer

Producer

PRA

State

Consumer

Actuator

e1

e2

d1

d2

d4A1

OK

State

Context

PRA can send events to

EPAs, e.g., “emergency generator fix”

PRA can be defined per

context segment, and receive

events only from EPAs in the same

context

EPAs may need to consult the current state of the

PRA

Actuator may respond immediately, or send acknowledgement via

an event

Messages to and from EPAs are

(potentially uncertain) events,

with present or future time

interval

EPA

e3d4

d3

dB

Enrichfrom dB



Proactive Agent (PRA) PRA

Input: forecasted events + state information

Output: Action – recommendation, activation, command to actuator

Process: real-time decision making

Input: forecasted events + state information

Output: Action – recommendation, activation, command to actuator

Process: real-time decision making

Real time decision making

Spectrum from

Trivial: decision tree

Basic: basic conditions - MDP

Advanced: simulation based optimization, advanced modeling

Real time decision making

Spectrum from

Trivial: decision tree

Basic: basic conditions - MDP

Advanced: simulation based optimization, advanced modeling



Enhancing the current event processing technology

Events may be uncertain: uncertainty about their occurrence ,occurrence time, and any of their attribute values; furthermore there may be uncertainty about relation between derived event and Situation, and propagation of uncertain values to derived events

Derived event may occur in the future)using predictive models – (

Running future time window in the presentFurthermore the semantics of derived event

Changes from virtual event to raw event

Applying event processing abstractions tostates – and use hybrid model



Glo

bal

Dat

a V

olu

me

in E

xab

ytes

Sens

ors

(Inte

rnet

of T

hing

s)

Multiple sources: IDC,Cisco

100

90

80

70

60

50

40

30

20

10

Agg

rega

te U

ncer

tain

ty %

VoIP

9000

8000

7000

6000

5000

4000

3000

2000

1000

0

2005 2010 2015

By 2015, 80% of all available data will be uncertain

Enterprise Data

Data quality solutions exist for enterprise data like customer, product, and address data, but

this is only a fraction of the total enterprise data.

By 2015 the number of networked devices will be double the entire global population. All

sensor data has uncertainty.

Social Media

(video, audio and text)

The total number of social media accounts exceeds the entire global

population. This data is highly uncertain in both its expression and content.



The dimensions of “BIG DATA”Data has to be processed in higher Velocity

Data has high variability: poly-structured. Many sources: sensors, social media, multi-media…

Volumes of data are constantly growing

Veracity: Data has inherent uncertainty associated with it



Uncertainty aspects

Meta-data representationof uncertainty

Removalof uncertainty

Propagation of uncertainty

Real-time decision underuncertainty

ByThresholds

Semantic propagation

ByRobust determination

Bayesian Nets

Monte-Carlo methods



Adding canonic representation for uncertainty handling:

Uncertain whether an reported event has occurred (e.g. accident)

Uncertain whether an reported event has occurred (e.g. accident)

Uncertain what really happened. What is the type and magnitude of the accident (vehicles involved, casualties)

Uncertain what really happened. What is the type and magnitude of the accident (vehicles involved, casualties)

Uncertain when an event occurred (will occur): timing of forecasted congestion

Uncertain when an event occurred (will occur): timing of forecasted congestion

Uncertain where an event occurred (will occur): location of forecasted congestion

Uncertain where an event occurred (will occur): location of forecasted congestion

Uncertain about the level of causality between a car heading towards highway and a car getting into the highway

Uncertain about the level of causality between a car heading towards highway and a car getting into the highway

Uncertain about the accuracy of a sensor input: count of cars, velocity of cars…

Uncertain about the accuracy of a sensor input: count of cars, velocity of cars…

The pattern: more than 100 cars approach an area within 5 minutes after an accident derives a congestion forecasting

The pattern: more than 100 cars approach an area within 5 minutes after an accident derives a congestion forecasting

Uncertain about the validity of a forecasting pattern Uncertain about the validity of a forecasting pattern

Uncertain about the quality of the decision about traffic lights setting

Uncertain about the quality of the decision about traffic lights setting



Real-time decision under uncertainty

Stochastic RT

optimization

Simulation-base RT

optimizationSimulation-

base RToptimization

Robust RTOptimization

Stochastic RTOptimization

Simulation-based RT optimization



Learning patterns and causalities

EventPatterns

Pattern and causality acquisition

This is a direction to reduce the complexity of application development

There are challenges in doing it – since “detected situations ”are “inferred events” and may not be reflected in past events



Summary

Proactive event driven computingis a new paradigm with potentialbig impact on society as well as future IT

There is an ecosystem of external collaborators mainly working on proposed EU project

The aim is to combine science and engineering to create a generic software platform

Technology

Proactive eth talk