Upload
others
View
15
Download
0
Embed Size (px)
Citation preview
DETECTION AND CLASSIFICATION OF TRANSIENT ASTRONOMICAL OBJECTS IN REAL
TIME USING COMPLEX EVENT PROCESSING
Veljko Vujcic, researcher at Astronomical Observatory of Belgrade
Brno, april 2016.
Contents
● Complex event processing
● Is CEP suited for astronomy?
● Building blocks
● Our work
● Perspectives
Complex event processing (CEP)
● CEP is a technology for extracting higher level knowledge from situational information abstracted from processing business-sensory information. Business-sensory information is represented in CEP as event data, or event attributes, transmitted as messages over a digital nervous system, such as an electronic messaging infrastructure.
● A component of Event driven architecture
● CEP products offer languages with high level of abstraction for describing inference mechanisms
CEP vs common applications
Is CEP suited for astronomy?
● No great solution for handling astronomical events (yet!)
● Performance (can handle high throughput with low latency)
● Declarative language (high level of abstraction)● Use cases from various industries● Open-source implementations● Conceptual clarity
CEP target scenarios
● High volume of events but only a few of real interest
● Strong temporal connection between events
● Individual events are rarely important but composition and aggregation of events is important
● Events are immutable
When to use event processing?
● When application is naturally centered around events
● When application needs to identify and react to certain situations (either good or bad) as they occur. An event-driven approach, where changes in state are monitored as they happen, lets an application respond in a much more timely fashion than a batch approach where the detection process runs only intermittently.
● When intermediary event processing logic needs to be separated out from the rest of the application. This can allow quick adaptation to meet new requirements, sometimes by the application business users themselves.
● When application involves analysis of a large amount of data in order to provide an output to be delivered to a human user or another application. This data can be organized into streams of events which are then distributed to multiple computing nodes allowing separate parts of the analysis to be performed in parallel.
What is an event?
“Anything that happens, or is contemplated as happening”
“A notable occurrence at a particular point of time”
Event object, event message, event tuple
“An observable occurrence”
A discrete incidence inside of a domain that system is capable of detecting
Simple → Complex events
Simple (raw) events
We define granularity inside scope of our system● A mouse click
● A financial trade
● Plane landing
● RFID reading
● Detection of an object in the sky
Complex (derived) events
Abstraction of simple events, its members.
An event that summarizes, represents, or denotes a set of other events.
● A completed stock sale
● Detection of a credit card fraud
● Violation of authorization in a manufacturing plant
● Network attack
● Supernova discovery
Building blocks of event processing
● Event producers● Event consumers● Event types● Event channels● Event processing agents● Context● Global state
What's going on in an EPA?
EPA: Transformation● Translate: applies a derivation
function on an input event
● Compose: derives events from a number of streams
● Aggregate: applies an aggregate function on a collection of events (SUM, AVG, COUNT, MAX...)
● Enrich: adds data from a global state
● Split: creates a collection of events from a single event
● Project: takes a subset of event attributes to create a derived event
EPA: Pattern matching
● Key element of CEP, outputs complex events ● Performs a matching function on one or more
input streams● Capable of applying context
EPA: Pattern matching
Basic patterns● Logical (all, any, absence)
● Threshold (aggregate functions)
● Subset selection (relative n highest values)
● Modal (always, sometimes)
Dimensional patterns● Temporal (instances in a
sequence, trends)
● Spatial (distances)
● Spatiotemporal (moving in constant / mixed direction)
LSST alerts
alertId Unique id identifying this alert
Level 1 database id
Corresponding database id
Science data • DiaSource record that triggered the alert
• DiaObject record• Historical instances of DiaSources for same DiaObject
Centered image cutouts 30x30px
• Difference image cutout
• Template image cutout
Event processing engine in LSST era
● ~10k alerts per visit – up to 40k ● ~10M alerts per night● Alerts will be transmitted with historical
DiaSources attached for the same DiaObject ● Event processing engines (“event brokers”)
will be developed by 3rd parties, LSST will offer a simple filtering broker
VOEvent processing network
LSST feed Simple filters (mag_diff < n)
Patternmatching
querySDSS()
FollowupPool
requestFollowup()
VOEvent VOEvent
VOEvent
Emit results
External service
realOrBogus()
External DB
VOEvent – format for interoperability
Our work - AlertSim
● Part of LSST simulations group
http://www.lsst.org/scientists/simulations
● End-to-end simulation of LSST data – to be replaced with real data, software stays the same
● AlertSim = python module inside of lsst_sims
https://github.com/lsst-sims/sims_alertsim
● Transforms simulated data to VOEvents
● Used to test formats, protocols, load, add noise, complicate things
Our work - AlertSim
● A python module inside of lsst_sims
alertsim
catsim opsim
VOEventslsst_sims(subset of lsst stack)
“Broker” #1
“Broker” #2
outside world
Our work – CEP engine for astronomy
● Using Esper libraries http://www.espertech.com/esper/
● Parse input XML● Apply filtering, pattern matching,
transformation● Define inference mechanisms with SQL-
based language
Our work – perspectives
● Construct DSL for astronomy which can be formally translated to EPL
Our work – perspectives
● Memory-tests, alternative memory solutions
Stock = few objects, frequent ticks
Astro = M's of objects, ticks every few days● Collaborations with other event-oriented
scientific fields (e.g ionosphere data)
References
● Event Processing in Action, Opher Etzion, Peter Niblett, Manning 2010
● www.thecepblog.com
● www.complexevents.com
● www.tibco.com/blog
● www.rulecore.com/CEPblog
● http://epthinking.blogspot.com
● www.ivoa.net
● http://hotwireduniverse.org/