19
Diamon/Laser Design proposal Marek Misiowiec BE/CO/AP May 2010

Diamon/Laser Design proposal Marek Misiowiec BE/CO/AP May 2010

Embed Size (px)

Citation preview

Page 1: Diamon/Laser Design proposal Marek Misiowiec BE/CO/AP May 2010

Diamon/Laser Design proposal

Marek Misiowiec BE/CO/AP

May 2010

Page 2: Diamon/Laser Design proposal Marek Misiowiec BE/CO/AP May 2010

Core OverviewMulti-tier, distributed, fail-safe system processing diagnostic data

collected repeatedly from a variety of sources. Raw data of

different shapes is transformed into unified formats and further

analyzed on a middle layer according to well-defined rules.

Actions may be taken, in form of alarms or monitoring information

delivered to clients. Processed data is made available as JAPC

parameters or for dedicated GUIs. It is stored in long term archive

too. Downward communication with the source devices is possible.2

devs

apps DAQ

raw data

unifieddata

SRV

clients

client view

Page 3: Diamon/Laser Design proposal Marek Misiowiec BE/CO/AP May 2010

Core Overview Diamon/Laser core responsibilities:

• data acquisition for various sources – push & pull modes

• communication through layers, publishing

• data transformation, processing, analysis, conditioning

• failover, database separation, scaling

complementary functionalities:• rules engine

• logging, archiving, configuration updates

non-core:• definitions & rules, data provider’s workflow

• GUIs, dedicated client applications

3

Page 4: Diamon/Laser Design proposal Marek Misiowiec BE/CO/AP May 2010

Importance of a change

Nowadays: upgrades mess

• stale sources, legacy constraints, dusty components

meeting only part of the requirements• partial data, simplistic thresholds, no conditioning

monitoring based on alarms - Diamon on Laser• misuse of the system, overhead

• mediocre scaling

• badly-behaving sources/clients cannot be detached/isolated

legacy solutions hiding in Laser world• SonicMQ, OC4J, EJBs, Hibernate, Sleepycat

...and no standard CO components in place

4

Page 5: Diamon/Laser Design proposal Marek Misiowiec BE/CO/AP May 2010

Importance of a change

We would like to: widely use CO solutions and help to establish others worth it better drive interactions with the bottom and upper tiers consolidate logics in system core be able to draw more conclusions on collected data sustain upgrades, updates, dependency changes

5

Page 6: Diamon/Laser Design proposal Marek Misiowiec BE/CO/AP May 2010

Transition

6

JMSNotifier

oc4j-diamon

AlarmArchive

Agent Messages

Checks surveillance

Alarms, filters non registered

Alarms

Result Messages

Sends update/Requests config

CLIC Agents

Central Agents

CMWJMXWIE TWIDO SNMP

PING

jdaemon

Configuration

Agentsto be

configured

configures

commands

commands

Router

CCC Consoles

LASER

ALARMS

YAMI protocol

JMS

Page 7: Diamon/Laser Design proposal Marek Misiowiec BE/CO/AP May 2010

Transition

7

JMSNotifier

oc4j-diamon

AlarmArchive

Agent Messages

Checks surveillance

Alarms, filters non registered

Alarms

Result Messages

Sends update/Requests config

CLIC Agents

Central Agents

CMWJMXWIE TWIDO SNMP

PING

jdaemon

Configuration

Agentsto be

configured

configures

commands

commands

Router

CCC Consoles

LASER

ALARMS

YAMI protocol

JMS

OC4J, EJBs, Hibernate,

BerkeleyDB

OC4J, EJBs, Hibernate,

BerkeleyDB

secondSonicMQ

firstSonicMQ

In-house DB

In-house DB

SNMP, CMW, variety of data

formats

directly to devices

not enough data

YAMI specific protocol

Page 8: Diamon/Laser Design proposal Marek Misiowiec BE/CO/AP May 2010

Transition

8

JMS

spring AlarmArchive

Agent Messages

Result Messages

Sends update/Requests config

CLIC Agents

Central Agents

CMWJMXWIE TWIDO SNMP

PING

jdaemon

Configuration

Agentsto be

configured

configures

commands

commands

CCC Consoles

YAMI protocol

JMS

LASERPROXY

LASERCMW

Page 9: Diamon/Laser Design proposal Marek Misiowiec BE/CO/AP May 2010

Transition

9

spring

AlarmArchive

Result MessagesSends update/

Requests config

CLIC Agents

Central Agents

CMWJMXWIE TWIDO SNMP

PING

jdaemon

Configuration

configures

commands

CCC Consoles

YAMI protocol

JMS

LASERPROXY

LASERCMW

Page 10: Diamon/Laser Design proposal Marek Misiowiec BE/CO/AP May 2010

Transition

10

spring

AlarmArchive

Result Messages

Data Acquisition Layer

CLIC AGENTS

CMWJMXWIE TWIDO SNMP

PINGConfiguration

commands

CCC Consoles

JMS

LASERPROXY

LASERCMW

Rules, thresholds, distribution, analysis, publishing...

JAPC

other applications

Page 11: Diamon/Laser Design proposal Marek Misiowiec BE/CO/AP May 2010

Technical outcomeThen: unified view of complete data sound construction on CO materials

• OC4J -> Spring, Hibernate -> JDBC Templates, EJBs -> POJOsSonicMQ -> ActiveMQ, multitude of data formats -> JAPCin-house database -> CCDB + Logging

• wider choice of middleware: CMW, YAMI, JMS

focusing logics in one place• rules, thresholds, alarm generation

• conditioning: machine mode, working hours, state of subsystems

driving the whole process • detachable sources, clients

• less restart/reboot downtime

• convergence with similar efforts

11

Page 12: Diamon/Laser Design proposal Marek Misiowiec BE/CO/AP May 2010

information exchange

12

japc-ext-yami japc-ext-cmw japc-ext-snmp jms-to-japc proxy japc-ext-?

Core Services

rules thresholds

alarm generator

externalsconditions

publisher w-flow

consoles

client middleware: JMSclientinter

actions

other apps

JAPCprovider

ConfigDB

archiverArchive

DB

Core

DAQ [1] DAQ [2] DAQ [n]

tackleproblematic

sources

Clicagents

CMWsrcs

LaserWie, Twi

?other

sources

source middleware protocols

Page 13: Diamon/Laser Design proposal Marek Misiowiec BE/CO/AP May 2010

Available technologies

Building blocks: CO standard components eg. JAPC CO less standard, but demanded eg. japc-ext-yami Laser dev code base, ready to reuse eg. caching, distribution TIM being refurbished and struggling with similar future other CO projects: INCA

13

Page 14: Diamon/Laser Design proposal Marek Misiowiec BE/CO/AP May 2010

Lessons learned from Laser eradication of legacy concepts & technologies pays off to scale well, we need to distribute processing power

• a supervisor over multiple workers

• dividing is difficult

separation of concerns single processing unit as a set of loosely coupled actions

• patterns: chain of responsibilities, observable

• upward flow: source input -> processing -> publishing

cater for database outages through local caching JMS+RMI as a middleware for all parties involved

• more advanced features of JMS channel help

...but no JAPC concepts were introduced at all

14

Page 15: Diamon/Laser Design proposal Marek Misiowiec BE/CO/AP May 2010

Device/property model to clarify the current incoherent view strong incentives:

• speak one language

• let everyone fit (Logging)

• common tools, well-established in CO world

migration of legacy sources• expose properties

• eradicate laser-source as it is

interaction with devices through JAPC calls domain entities:

• Alarm, Alarms, Diagnostic,...

15

Page 16: Diamon/Laser Design proposal Marek Misiowiec BE/CO/AP May 2010

JAPC extensions to unify view on a variety of devices/applications

available now:• CMW (japc-ext-cmwrda) and JMS (remote) as most popular

we need some more:• YAMI through japc-ext-yami is already available,

used in new Clic agent

• SNMP through japc-ext-snmp under development

16

Page 17: Diamon/Laser Design proposal Marek Misiowiec BE/CO/AP May 2010

Bottom tier

Experience gained in: current Diamon and Laser

• eg. CMW Alarm Monitor as a huge data acquisition process

LHC Concentrators , similar problems INCA data acqusition layer

Progress:

17

source type current planned/done

Clic Agents JMS, YAMI japc-ext-yami

CMW Alarm Monitor Laser gateway JAPC device

Laser sources C/Java laser-source Laser-to-JAPC proxy

Wiener, Twido SNMP central agents japc-ext-snmp

Ping central agent JAPC device

Page 18: Diamon/Laser Design proposal Marek Misiowiec BE/CO/AP May 2010

Already developed JAPC extensions current version of YAMI meets our requirements Diamon Clic agent

• fully YAMI-based, JAPC values

CMW Alarm Monitor refurbished• 6000 subscriptions to GM and FESA devices

• will be incorporated into new core

proxy between Laser production alarms and JAPC values• for the legacy sources

18

Page 19: Diamon/Laser Design proposal Marek Misiowiec BE/CO/AP May 2010

Open questions

Still to think over: best components and collaborations to choose level of distribution and scaling shared memory, distributed cache storage: short-term, long-term, Logging DB rules handling: homemade, proprietary ...and many others

19