UrbanFlood WP5 Common Information Space

(CIS)after Year 1

Marian Bubak, Bartosz Baliś

and WP5 teamACC Cyfronet AGH, Kraków, Poland

{bubak,balis}@agh.edu.pl

PlanMotivation, goals, tasks, and the

teamObjectives for Year 1Results of requirement analysisCommon Information Space

◦architecture◦ implementation status

CIS-based Flood EWSSummary of achievementsPlans for Year 2

Motivation & GoalsFacilitate creation, deployment and

robust operation of Early Warning Systems

Early Warning System: any system working according to four steps:◦ Monitoring◦ Analysis◦ Judgment◦ Action

Example: environmental monitoring Also: cloud infrastructure monitoring,

EWS self-monitoring

Tasks and the Team WP5 leader: Marian Bubak T5.1: Execution framework

◦ Integration platform: Marek Kasztelnik, Bartosz Balis◦ Data access: Piotr Nowakowski, Tomasz Gubala◦ Resource allocation: Tomasz Bartynski◦ Development of CIS-based EWSs: Bartosz Balis, Marek

Kasztelnik, Jeroen Broekhuijsen, Artem Ozhigin T5.2: Event infrastructure

◦ Communication Bus: Bartosz Balis, Marek Kasztelnik, ◦ Self-monitoring: Grzegorz Dyk, Bartosz Balis

T5.3: Provenance logging (Metadata)◦ Generic metadata service: Tomasz Gubala, Piotr

Nowakowski, Adam Belloum T5.4: User & developer GUIs

◦ Jeroen Broekhuijsen, Tomasz Gubala

Milestones: M12: The CIS integrates the (distributed) software

modules of the EWS

M24: The CIS manages (online) computing resources

M33: The CIS has been demonstrated to SEIS and /or INSPIRE representatives

Timeline

Requirement analysisExisting sources of live and archive sensor

dataExisting legacy applications

◦ Heterogeneous platformsComplex scenarios emerge, involving:

◦ Real-time processing of sensor data◦ Decision-making, automatic or by human

interaction◦ Compute-intensive simulations◦ What-if scenarios

High priority (urgent) computingSharing of limited resources

System requirementsApplication integration: enable data

exchange between diverse components◦ Legacy, heterogeneous

Orchestration of components into composites◦ Scientific / business workflows◦ Integration patterns

Dynamic resource allocation managementMetadata management

◦ Domain (applications, data)◦ System information (resources, data sources,

running tasks)◦ Provenance

Overview of CIS architecture

PlatIn: the Integration PlatformStart and stop EWS

capability implemented (Every EWS part delivers management WS)

Integrated with UfoReg (receiving EWS metadata)

EWS parts can be created using BPEL or Camel integration technology

UFoReg: the UrbanFlood Registry

Stable prototype deployed at Cyfronet and populated with initial sensor/EWS/Cloud data

Basic GUI available RESTful API interfaces are in place to enable integration

of UFoReg with DyReAlla and visualization components

Currently includes three data models:◦ Sensor metadata

model◦ EWS model

(describing EWSs and their appliances)

◦ Cloud model (describing the available hardware resources and virtual system images, and recent cloud state)

UfoReg Service

Integration platform UI

Applicationmanager

Administration Tool

Rule engineRoute engineBPEL EngineUser Tool

Provenance API

log provenance and historical data

Search API

search configuration

search domain and historical metadata

search appliance

REST API

Storage

Persistence

Domain Model

Deployed schemalessdatabase populateddynamically with domainmodel objects

Flexible, extensible modelnotation to cover growingnumber of use cases andvarious types of metadatato store and publish

DyReAlla: Dynamic Resource Allocation service Integrated with

cloud infrastructure (receiving reports about resource availability and load metrics)

Integrated with UfoReg (storing cloud status data)

Not in the first milestone / release of CIS

Paradigms and technologies usedEnterprise Service Bus (Open ESB)Business Process Management

(BPEL)Enterprise Application Integration

framework (Apache Camel)Message-Oriented communication

middleware (ActiveMQ)Robust NoSQL database (MongoDB)

Early Warning System – the CIS view

Each EWS is composed of: (1) Appliances, (2) Parts, (3) External components

Appliance: application component exposed as a service and wrapped into a virtual image◦ Typically legacy EWS-specific application

EWS Part: composite integrating and orchestrating control and data flow between appliances (and possibly other parts)◦ Also can be exposed as a high-level service

External component: external producer or consumer of data, not managed by CIS

Generic operations: ◦ Start, stop◦ Change alert

levelInvokes

appliancesFulfills well-defined high-level service exposed via part-specific interface

Sends messages to and receives from a message bus

External Component

EWS Part

start stop

Invoke<<service bus>>

part-specific interface

Message Bus

send message

receive message

Appliance

alert levelEWS Part

Loose coupling

Parts are loosely coupled Communicate via the message busNo direct dependencies between partsFacilitates extendibility of the EWS

◦ Addition of new parts that further process already published data

◦ Connecting new visualization front-ends◦ Can even be done at runtime

EWS PartEWS Part Message Bus

Attention (alert) level

General state of every EWS shared by all its partsA Part may send ’alert level change’ (raise or

decrease) messageAlert Level Manager (general-purpose

specialized EWS part) receives the messages, calculates a new alert level, and sends ’alert level set’ message

Other EWS parts may adjust their operation depending on the alert level

Attention Level Manager

Other EWS Part Message Bus

Receive attention level

change messages

Send attention level

change messages

Send attention level set

messages

Receiveattention level set

messages

Flood Early Warning SystemMonitoring of dikes using wireless

sensorsAI-based detection of sensor signal

anomaliesDike failure predictionSimulation of inundation due to failureVisualization and user interactions on

Multi-touch Tables

Flood EWS implemented with CIS

AnySense

Storage (AnySense)

Multi-Touch Table

HRW DRFSM

HRW Hydrograph

FloodSimulation

Part

Simulation commands

Simulation results

Reliable

AI-based Monitoring Part

Sensordata /

Anomaly probability

AIFiltered

sensor data

Attention level

change

Sensordata

Reliable Monitoring Part

Filtered sensor data

Attention level

change

Simulation commands

Simulation results

Sensordata

Anomaly probability

CIS Message Bus

Archiver Part

Attention level set /

Dike failure prob.

Dike failure probability

Attention Level Manager Part

Attentionlevel set

Attention level

change

EWS Part

ApplianceExternal

componentCIS

Technology

UFoReg (CIS metadata registry)

Legend:

Self-monitoring & cloud monitoring as Early Warning Systems

Each EWS has: Its own instance of

PlatIn, the CIS integration platform,

Specific appliances in the cloud

EWS Parts: Composites which orchestrate the execution of appliances

Multiple EWSs: EWS1, EWS2: two instances of Dike Monitoring EWS EWS3: Monitoring and management of other EWSs

(DyReAlla and UFoReg) EWS4: Cloud monitoring and management

Deliverables, milestones & meetings in Y1Deliverables:

◦ D5.1 Common Information Spaces (description of state of the art and future developments) (M8)

◦ D5.2 Specification of the architecture and interfaces of the Common Information Space (M9)

◦ D5.3 Orchestrating the information flow in a Common Information Space (M12)

Milestones:◦ M12: The CIS integrates the (distributed) software

modules of the EWSMeetings:

◦ Kick-Off meeting in Groningen◦ Consortium meeting in Wallingford (definition of first

EWS)◦ Integration meeting in Cracow (first EWS protoptype)◦ Integration meeting in Amsterdam (first EWS

production run)◦ Teleconferences: every Wednesday (current issues

discussions, progress reports)

Publications & presentations B. Balis, T. Bartynski, M. Bubak, M. Kasztelnik, and P. Nowakowski: The

UrbanFlood Common Information Space. Poster and presentation at CGW10 – Cracow Grid Workshop 2010, Krakow, October 11-13, 2010, http://www.cyfronet.pl/cgw10

M. Bubak: Towards Collaborative Workbench for Science 2.0 Applications. Presentation at HPC 2010 – High Performance Computing, GRIDS and clouds, An International Advanced Workshop, June 21 – 25, 2010, Cetraro, Italy, http://www.hpcc.unical.it/hpc2010

B. Balis, T. Bartynski, M. Bubak, M. Kasztelnik, P. Nowakowski: Common Information Space, a framework for creating and hosting Early Warning System. Poster and presentation at Joint UrbanFlood & SSG4Env Workshop, 11-12 November 2011, Amsterdam, http://urbanflood.eu/urbanFloodWorkshop2010.aspx

B. Balis, M. Kasztelnik, M. Bubak, T. Bartynski, T. Gubala, P. Nowakowski, J. Broekhuijsen: The UrbanFlood Common Information Space for Early Warning Systems. Submitted to the ICCS 2011 Conference; to be published (if accepted) in Elsevier Procedia Computer Science, http://www.iccs-meeting.org/

V.V. Krzhizhanovskaya, G.S. Shirshov, N.B. Melnikova, R.G. Belleman, F.I. Rusadi, B.J. Broekhuijsen, B. Gouldby, J. L'Homme, B. Balis, M. Bubak, A.L. Pyayt, I.I. Mokhov, A.V. Ozhigin, B. Lang, R.J. Meijer. Flood early warning system design and implementation. Submitted to the ICCS 2011 Conference; to be published (if accepted) in Elsevier Procedia Computer Science, http://www.iccs-meeting.org/

Plans for Y2Task 5.1 Improvement of functionality of all modules

(e.g. support for self-monitoring) Implementation of the dynamic resource

allocation mechanismFurther implementation and refinement of

components of the CIS-based flood EWS. Task 5.2Development of a robust software sensor

network for self-monitoring Improvement of scalability and flexibility of

self-monitoring infrastructure Improvement of robustness of the

communication services.

Plans for Y2 (ctd)Task 5.3 Continuous optimization of storage, gathering

and publishing techniques inside UFoReg Support for the provenance model Continuous extending domain models when

requested by partners and required by the CIS/EWS design

Task 5.4 Implementation of graphical dashboards for

users and developers for monitoring status and health of CIS and EWSs

Integration of CIS services and CIS-based EWS services with graphical user interfaces of the Decision Support System

Development of auxiliary developer tools for automation of CIS-based EWS development.

Means to high-quality softwareAdoption of mature, industry-quality,

standards-based approaches and technologies ◦ Apache, MongoDB, OpenESB, BPEL, Passenger

At the same time adoption of platform and technology-independent design in order to avoid vendor lock-in◦ SOA, loose coupling between components ◦ Communication based on well-defined protocols

Self monitoring inherent part of the CIS design

Test driven development

Hardware contributed by CyfronetComputer host (Y1)

◦ 2 dualcore processors Intel Xeon CPU 5150 @2.66GHz, 4 GB RAM, 500 GB storage

◦ Deployed services CIS (ActiveMQ JMS, Glassfish server, UFoReg,

DyReAlla, Health monitoring) EWS parts (message filters, converters etc)

4 nodes (from February 2011)◦ 2 quadcore processors Intel Xeon CPU

L5420 @2.50GHz, 16 GB RAM, 120 GB local storage, access to shared storage over iSCSI

◦ Dedicated for hosting UF appliances (Flooding simulator,Reliable, Hydrograph, DRFSM, AnySense, AI)

www.urbanflood.eudice.cyfronet.pl/cis