LHCb ECS Integration Guidelines

LHCB Technical NoteIssue:  Draft Revision:  4

Reference:  EDMS 732486Created:  27 March 2006 Last modified:  20 July 2018

Prepared By: LHCB Online GroupC. Gaspar/EditorLuis Granado Cardoso

LHCb ECS Integration Guidelines Reference: EDMS 732486LHCB Technical Note Revision: 4Issue: Draft Last modified: 20 July 2018Appendix A

Abstract

This document is intended to guide sub-detector control system designers and developers with the integration of their systems into the final LHCb Control System.

We would like to thank ATLAS and CMS for allowing us to use their documents as a starting point.

Document Status Sheet

Table 1 Document Status Sheet

1. Document Title: [Project Name Qualification] User Requirements Document

2. Document Reference Number: [Document Reference Number]

3. Issue 4. Revision 5. Date 6. Reason for change

Draft 1 07 March 2006 First version

2 12 June 2006 Changed System Numbers definitionUpdated the TFC partitioning table (it was incorrect)

3 27 April 2007 Added a new chapter on “Accounts, Groups and File System Conventions”

4 20 July 2018 Updated document according to new conventions for the LHCb Upgrade systems and moved previous conventions to appendixes

page 2ii

Table of Contents

LHCB TECHNICAL NOTE..............................................................................................................................................I

ISSUE:  DRAFT.............................................................................................................................................................I

ABSTRACT.........................................................................................................................................................................I

DOCUMENT STATUS SHEET........................................................................................................................................I

TABLE 1 DOCUMENT STATUS SHEET......................................................................................................................I

1. INTRODUCTION......................................................................................................................................................2

2. TOOLS: WINCC OA AND THE JCOP FRAMEWORK.....................................................................................3

3. NAMING CONVENTIONS......................................................................................................................................4

3.1. SUB-SYSTEM NAMES............................................................................................................................................43.2. DOMAIN NAMES:..................................................................................................................................................53.3. COMPUTER NAMES...............................................................................................................................................53.4. OTHER ETHERNET EQUIPMENT NAMES (CREDIT-CARD PC, CAEN, ETC.)..........................................................63.5. WINCC OA SYSTEM NAMES & PROJECT NAMES................................................................................................63.6. WINCC OA SYSTEM NUMBERS............................................................................................................................73.7. FSM NAMING CONVENTIONS...............................................................................................................................8

3.7.1. FSM Object Names.......................................................................................................................................83.7.2. FSM Object & Device Type Names............................................................................................................10

3.8. WINCC OA PANEL NAMES................................................................................................................................10

4. ACCOUNTS, GROUPS AND FILE SYSTEM CONVENTIONS.......................................................................11

4.1. FILE SYSTEM AREAS...........................................................................................................................................114.2. USER GROUPS.....................................................................................................................................................114.3. INSTALLING WINCC OA PROJECTS AT THE PIT................................................................................................12

5. APPENDIX A..............................................................................................................................................................2

5.1. SUB-SYSTEM NAMES............................................................................................................................................25.2. DOMAIN NAMES:..................................................................................................................................................25.3. COMPUTER NAMES...............................................................................................................................................35.4. OTHER ETHERNET EQUIPMENT NAMES (CREDIT-CARD PC, CAEN, ETC.)..........................................................35.5. WINCC OA SYSTEM NAMES & PROJECT NAMES................................................................................................45.6. WinCC OA System Numbers...............................................................................................................................4

1. Introduction

LHCb's Experiment Control System is in charge of the configuration, control and monitoring of all the components of the online system. This includes all devices in the areas of: data acquisition, detector control (ex slow controls), trigger, timing and the interaction with the outside world.

The central ECS team, with the help of the Joint Controls Project (JCOP), will provide a framework to the sub-detectors groups which will need to configure and/or develop the specific control functions for their sub-detector. 

The control framework is based on a SCADA (Supervisory Control and Data Acquisition) system called PVSSII and it will have to be interfaced to all experimental equipment.

Useful Links

LHCb’s Experiment Control System WEB pagehttp://lhcb-online.web.cern.ch/lhcb-online/ecs

LHCb’s Experiment Control System (paper)http://lhcb-online.web.cern.ch/lhcb-online/ecs/pdf/RT_2003.pdf

ECS Tutorial, March 2006http://lhcb-online.web.cern.ch/lhcb-online/ecs/powerpoint/LHCb_ECS_Tutorial.ppt

WinCC OA Introductionhttp://lhcb-online.web.cern.ch/lhcb-online/ecs/PVSSIntro.htm

WinCC OA Home Pagehttp://itcobe.web.cern.ch/itcobe/Services/Pvss/welcome.html

LHCb Framework pagehttp://lhcb-online.web.cern.ch/lhcb-online/ecs/lhcb-fw/default.htm

JCOP Framework pagehttp://itcobe.web.cern.ch/itcobe/Projects/Framework/welcome.html

LHCb Upgrade Redmine Forge:https://lbredmine.cern.ch/

LHCb Software GIThttps://gitlab.cern.ch/lhcb-amc40firmware-mng/lhcb-amc40software

page 22

2. Tools: WinCC OA and the JCOP Framework

The JCOP Framework (based on WinCC OA) is an integrated set of guidelines and software tools which is used by Developers to develop their part of the Control System application.

Why you should use WinCC OA and the JCOP Framework

LHCb will implement a homogeneous and coherent Experiment Control System. For this reason the different parts of the system should look and work in a similar way, independently of who developed them, so that they can be integrated into the complete final system. And so that operators can be provided with a homogeneous view of all the systems.

How you should use WinCC OA and the JCOP Framework

It is important to distinguish between a development system and a production system. Working with a WinCC OA project in one way or the other can be quite different.

A Development System is where the developer creates, modifies, debugs and tests the project.

A Production System is where the final developed project is installed to be used by operators and experts.

Tools used in each system (development or production) can be different. The way WinCC OA is configured in each case can also be different. The development system application must be tested before progressing to the production system. If you have to modify your application, the modification must be done in the development system, tested again and brought back to the production system. Modifications must never be made in a production system.

A development project must be packaged as a FW component in order to be installed as a Production system. Please contact the central team when you are ready to move a project into production.

Typical mistakes and misunderstandings using WinCC OA and the JCOP Framework

A few things to remember and understand when using WinCC OA and the JCOP framework are:

The Device Editor and Navigator is not a user tool. This tool will not be available in a production system. The Device Editor and Navigator is only used in development systems by developers.

The WinCC OA console should not be available in a production system. LHCb Operators do not start or stop projects or WinCC OA managers. This will be automated.

LHCb Operators will operate the system through the FSM panels and the WinCC OA alarm screen.

3. Naming Conventions

When setting up PCs and WinCC OA projects for LHCb, naming conventions must be respected in order to coherently integrate the different sub-detectors and sub-systems.

Please consult the LHCb ECS team when adding any new production system, as it is necessary to keep track of the whole.

Why you should use naming conventions

There are several obvious reasons why one should use naming conventions. Naming conventions are intrinsically necessary for communication. Using these conventions we uniquely identify applications, systems, FSM nodes, FSM types or files. Still, uniquely identifying each system is not enough. The identification names also need to be coherent so that Operators can easily learn how to operate all parts of the system.

When and how you should apply LHCb naming conventions

It is important do distinguish between a development and a production system. In principle, in a development system one would not need naming conventions. However, since many of the names used in a development system will be exported to a production system, the conventions should be applied already during development. Also, if you are planning to test your development system in a distributed way, you can benefit from using naming conventions.

3.1. Sub-System Names

The following names have been defined per Sub-system:

Table 1: List of Sub-Detector Names (2 letters):

Vertex Locator VEUpstream Tracker UTRICH1 R1RICH2 R2SciFi Tracker SFElectromagnetic Calorimeter

EC

Hadron Calorimeter HCMuon MU

page 44

And for Central Systems:

Table 2: List of central Sub-Systems name

Timing and Fast Control TFCHigh Level Trigger HLTInfrastructure INFCentral Control LB

3.2. Domain Names:

The following Domain Names have been defined:

Table 3: List of Domain names

DAQ and Trigger domain DAQData Acquisition Infrastructure DAIDetector Infrastructure DCSHigh Voltage HV

3.3. Computer Names

Computer names (i.e. network name) of Control PCs are made up using the following convention:

The following fields are used: <Sub-system name>[<Domain name>]<Sub-system part>[<W>]

Where:

Sub-system name is one of the above Sub-Detector/Sub-System names

Domain name is one of the above Domain names

o Central Sub-System PCs do not need the Domain name. Ex.: TFC01W, HLT01

Sub-system part name is composed of 3 letters or digits specifying either a geographical division, like side A or C, U(p) or D(own) etc. and/or a sequence number(with leading ‘0’s)

The letter ”W” if the PC runs Windows

Examples: R1HV01W, VEDAQA01

Note: During the commissioning phase, the computer names should be appended with the letter ‘u’ at the end. This is to avoid clashes with the computer names of the pre-2019 LHCb experiment. At some point, when the old machines are decommissioned, this ‘u’ will be removed.

3.4. Other Ethernet Equipment Names

Other Ethernet equipment names are made up using the following convention:

The following fields are used: <Sub-system name><Equipment type><Sub-system part>

Where:

Sub-system name is one of the above Sub_detector/Sub-System names

Equipment type or Board type (for CC-PCs) can be for example: TELL10, CAEN, Wiener, etc

Sub-system part name is composed of 1 letter specifying either a geographical division, like side A or C,U(p), D(own), etc. and a sequence number with 2 digits and leading ‘0’s:

Examples : VETELL40A03, SFCAEN02

3.5. WinCC OA System Names & Project Names

A WinCC OA project must be given a name at creation time, and a system name which is used (in conjunction with a system number) to identify it in a distributed system The system name and number may be changed using tools supplied by ETM after creation, and they must be unique throughout the distributed system. The project name can only be changed by copying the project to a new name. The following convention has been defined for the system and project names:

The following fields are used: <Sub-system name><Domain name><Sub-system part>

Where:

Sub-system name is one of the above Sub-detectors/Sub-System names

Domain name is one of the above Domain names

Sub-system part name is composed of 1 letters specifying either a geographical division, if needed (e.g. side A or C,U(p), D(own), etc.). and/or a sequence number (no leading ‘0’s)

Examples : R1HV1, VEDAQA1

• All UPPER case

• The WinCC OA system name is exactly the same as the project name

In order to change the name of a project, you can copy the project to a new name. This allows project and system names to be kept the same.

page 66

3.6. WinCC OA System Numbers

Each WinCC OA project is required to have a unique system number in order to allow connection to the distributed system for the LHCb ECS. Tables 5 and 6 show the range of system numbers attributed to each sub-system.

Table 4: List of WinCC OA System numbers attributed to Sub-Detectors

Vertex Locator VE 1100 – 1119RICH1 R1 1130 – 1149RICH2 R2 1160 - 1179Upstream Tracker UT 1190 – 1209SciFi Tracker SF 1220 – 1239Electromagnetic Calorimeter

EC 1250 – 1269

Hadron Calorimeter HC 1280 – 1299Muon MU 1310 – 1329

Table 5: List of WinCC OA System numbers attributed to central Sub-Systems

Timing and Fast Control TFC 710 – 719Infrastructure INF 720 – 729Central Control LB 730 – 749High Level Trigger HLT 750 – 799Reserved 1 – 709

800 – 10991340 - 2048

In a hierarchical structure a certain node has to be connected to all the nodes below, so that one can navigate and view information from the other nodes. The best solution to set up a distributed system is that the top-level system acts as a client for all other systems, the bottom-level systems act as servers to all other systems and the medium-level systems act as clients for all systems below and servers to all system above (see Figure 1).

Figure 1: Setting up the distributed system

Thus, in the “config” file of a certain node a line “distPeer” needs to be added per system below. For example if a middle system called “VEUDAQ1” contains two children: VEDAQA1 and VEDAQA2, the config file of system VEUDAQ would contain:

[general] distributed = 1

[dist] distPeer = "VEUDAQA01" 1105 #Connect to the VELODAQA1 PC, system number 1105distPeer = "VEUDAQA02W" 1107 #Connect to the VELODAQA2 PC, system number 1107

3.7. FSM Naming Conventions

Detailed information about FSM naming conventions and definitions is available at:https://edms.cern.ch/document/655828/3

Below is a summary of the naming conventions for completeness.

Table 6: List of FSM Names per Sub-Detector

Vertex Locator VE VELORICH1 R1 RICH1RICH2 R2 RICH2Upstream Tracker UT UTSciFi Tracker SF SFElectromagnetic Calorimeter

EC ECAL

Hadron Calorimeter HC HCALMuon MU MUON

Table 7: List of FSM Names for Central Sub-Systems

Timing and Fast Control TFC TFCHigh Level Trigger HLT HLTInfrastructure INF INFCentral Control LB ECS

3.7.1. FSM Object Names

All FSM names are UPPER case.

The top nodes of a sub-detector correspond to the Domain names. <Sub-system name>_<Domain name> Ex: ECAL_DAQ, RICH1_HV

o If a Sub-detector has more then one TFC partition, The Sub-detector top nodes should represent the partitioning (for all domains). For example side A or C. Ex.: VELOA_DAQ and VELOC_DAQ. Sub-detectors allocated two TFC partitions are: VELO (A/C?), and MUON (A/C?).

o Complete list of Top level nodes for Sub-detectors proposed in Table 8.

The children of the sub-detector domain node can be either a functional division or a geographical division. Ex.: VELOA_DAQ_FEE, RICH1_DCS_LV or IT_DAQ_CHAMBER1

page 88

The next level can again be either a functional or a geographical division. Ex:RICH1_DCS_LV_Q1, IT_DAQ_CHAMBER1_FEE

The next children will normally be a DU. However, if it is not the case, one could follow with the same convention: Ex: RICH1_DCS_LV_Q1_SECTOR1

Table 8: List of Top level domains per Sub-detector

Sub-detector Part DCS DAI HV DAQVertex Locator VELOA_DCS VELOA_DAI VELOA_HV VELOA_DAQ

VELOC_DCS VELOC_DAI VELOC_HV VELOC_DAQRICH1 RICH1_DCS RICH1_DAI RICH1_HV RICH1_DAQRICH2 RICH2_DCS RICH2_DAI RICH2_HV RICH2_DAQUpstream Tracker UTA_DCS UTA_DAI UTA_HV UTA_DAQ

UTB_DCS UTB_DAI UTB_HV UTB_DAQSciFi Tracker SFA_DCS SFA_DAI SFA_HV SFA_DAQ

SFB_DCS SFB_DAI SFB_HV SFB_DAQElectromagnetic Calorimeter

ECAL_DCS ECAL_DAI ECAL_HV ECAL_DAQ

Hadron Calorimeter HCAL_DCS HCAL_DAI HCAL_HV HCAL_DAQMuon MUONA_DCS MUONA_DA

IMUONA_HV

MUONA_DAQ

MUONC_DCS MUONC_DAI

MUONC_HV

MUONC_DAQ

Table 9: List of Possible/Example Sub-system Functional Parts per Domain

DCS Temperatures TEMP Cooling System COOLING Gas System GAS Low Voltages LVDAQ Front-End Electronics FEE TELL40 TELL40DAI Crates CRATES

3.7.2. FSM Object & Device Type Names

Logical Object Types: <Sub-system name>_<function_name>. Examples: VELO_TELL40, RICH1_TEMP, UT_COOLING

If the same type is repeated in different places (for different geographic parts, etc.) one needs to define it only once.

Device Unit Types <Device base type>_<sub-system name>_<sub-system part name> Examples: fwAi_RICH1_TEMP

3.8. WinCC OA Panel Names

Each Control Unit (CU), Logical Unit (LU) or Device Unit (DU) has a WinCC OA panel associated to it. The name of this panel should be the same as the FSM object.

Example: if there is a CU with name “VELOA_DAQ“, then the panel will be called “VELOA_DAQ.pnl”.

A convention for panel names is important for integration into the LHCb final distributed system.

Note: It is important to verify that your panels function correctly in different systems. For example, the system name will be different in your development project compared to your production project(s).

3.8.1. System scaling

The PCs for the control system will be hosted in the LHCb Virtual Machines infrastructure. This provides a big flexibility in what concerns the number of PCs available and their distribution of resources, which means that there is not a hard limit on the number of PCs to be used.

There may be some constraints concerning the performance of each project, mainly due to the number of devices to be monitored and controlled, which

The FSM tree control units are implemented as separate processes. Some optimization may be used by implementing some as nodes logical units (LUs) associated to a control unit. Nevertheless it is necessary to have an idea on the number of needed nodes to plan their distribution on the various detector control PCs.

page 1010

4. Accounts, Groups and File System Conventions

4.1. File System Areas

The areas defined on the online cluster at the PIT are as follows:

Areas Linux Windows

home directories /home/<user id> U:\

group areas /group/<group id> G:\<group id>

LHCb software (Moore, etc.) /sw/lib/lhcb; /sw/lib/lcg; etc. P:\lib\*

WinCC OA components (unzipped) /sw/wincc P:\wincc

cluster-wide scratch area /scratch S:\

local scratch area /localdisk/scratch L:\scratch

WinCC OA Projects /localdisk/wincc L:\wincc

WinCC OA Project installed components /group/<group id>/wincc G:\<group id>\wincc

4.2. User Groups

All User accounts belong to Groups. Groups are the same on Windows and Linux. The group defines what privileges the users may have and Users can be members of more than one group. Group membership gives access to the group area in the online cluster.

The groups defined on the online cluster (and the corresponding areas) at the PIT are as follows:

admin

online

trg /group/trg; /group/trg/l0; etc.

hlt /group/hlt

velo /group/velo

ut /group/ut

sf /group/sf

rich /group/rich; /group/rich/rich1; /group/rich/rich2

calo /group/calo; /group/calo/ecal; /group/calo/hcal;

muon /group/muon

infra for Infrastructure

gcs for Gas Control System

lb Central accounts for shifters and operators

4.3. Installing WinCC OA Projects at the PIT

Recommendations for Installing and Running WinCC OA projects at the PIT:

Try to start with WinCC OA 3.15

Make sure all your components and/or applications are packaged as FW type components (Containing ONLY your DPs, scripts, panels, etc.)

Create a new project

o In /localdisk/wincc

o With correct project name, system name and system number as described in sections 3.5 and 3.6 above.

Install ALL Components (FW/LHCb and your specific components)

o In /group/<group id>[/<sub group>]/wincc/fwComponents_<yourProjectName>

o All FW/LHCb Components are available for installation (unzipped)

In /sw/wincc/

Note: It is very important that all components are installed in the group area, as described above, so that WinCC OA user interfaces have access to the panels and scripts of the various Sub-detectors.

page 1212

5. Appendix A

This appendix contains the conventions which should be followed for setting up PVSS/WinCC OA systems for the LHCb experiment before the 2019 LHCb upgrade.

5.1. Sub-System Names

The following names have been defined per Sub-system:

Table 1: List of Sub-Detector Names

Vertex Locator VE - Pile-Up PU*RICH1 R1RICH2 R2Outer Tracker OTSilicon Tracker ST - Inner Tracker IT* - Trigger Tracker TT*Electromagnetic Calorimeter

EC

Hadron Calorimeter HCPre-Shower PS - Scintillator Pads SP*Muon MU

* If needed

And for Central Systems:

Table 2: List of central Sub-Systems name

Level 0 Trigger TRG - L0 Muon TMU - L0 Calorimeter TCA - L0 Pile-Up TPUTiming and Fast Control TFCHigh Level Trigger HLTInfrastructure INFCentral Control LB

5.2. Domain Names:

The following Domain Names have been defined:

Table 3: List of Domain names

DAQ and Trigger domain DAQData Acquisition Infrastructure DAIDetector Infrastructure DCSHigh Voltage HV

5.3. Computer Names

Computer names (i.e. network name) of Control PCs are made up using the following convention:

The following fields are used: <Sub-system name>[<Domain name>]<Sub-system part>[<W>]

Where:

Sub-system name is one of the above Sub-Detector/Sub-System names

Domain name is one of the above Domain names

o Central Sub-System PCs do not need the Domain name. Ex.: TFC001W, HLT001

Sub-system part name is composed of 3 letters or digits specifying either a geographical division, like side A or C, U(p) or D(own) etc. and/or a sequence number(with leading ‘0’s)

The letter ”W” if the PC runs Windows

Examples: R1HV001W, VEDAQA01

5.4. Other Ethernet Equipment Names (Credit-Card PC, CAEN, etc.)

CC-PC and other Ethernet equipment names are made up using the following convention:

The following fields are used: <Sub-system name><Equipment type><Sub-system part>

Where:

Sub-system name is one of the above Sub_detector/Sub-System names

Equipment type or Board type (for CC-PCs) can be for example: TELL1, UKL1, CAEN, etc

Sub-system part name is composed of 3 letters or digits specifying either a geographical division, like side A or C,U(p), D(own) etc. and/or a sequence number(with leading ‘0’s)

page 22

Examples : R1UKL1011, VETELL1A03, STCAEN002

5.5. WinCC OA System Names & Project Names

A WinCC OA project must be given a name at creation time, and a system name which is used (in conjunction with a system number) to identify it in a distributed system The system name and number may be changed using tools supplied by ETM after creation, and they must be unique throughout the distributed system. The project name can only be changed by copying the project to a new name. The following convention has been defined for the system and project names:

The following fields are used: <Sub-system name><Domain name><Sub-system part>

Where:

Sub-system name is one of the above Sub-detectors/Sub-System names

Domain name is one of the above Domain names

Sub-system part name is composed of 3 letters or digits specifying either a geographical division, like side A or C,U(p), D(own) etc. and/or a sequence number (no leading ‘0’s)

Examples : R1HV1, VEDAQA1

• All UPPER case

• The WinCC OA system name is exactly the same as the project name

In order to change the name of a project, you can copy the project to a new name. This allows project and system names to be kept the same.

5.6. WinCC OA System Numbers

Each WinCC OA project is required to have a unique system number in order to allow connection to the distributed system for the LHCb ECS. Tables 5 and6 show the range of system numbers attributed to each sub-system.

Table 4: List of WinCC OA System numbers attributed to Sub-Detectors

Vertex Locator VE 110 – 119Pile-Up PU 120 – 129RICH1 R1 130 – 139Silicon Tracker ST - Inner Tracker IT 140 – 149 - Trigger Tracker TT 150 – 159Outer Tracker OT 160 – 169RICH2 R2 170 – 179Pre-Shower/ Scintillator Pads

PS/SP 180 – 189

Electromagnetic EC 190 – 199

CalorimeterHadron Calorimeter HC 200 – 210Muon MU 210 – 219

Table 5: List of WinCC OA System numbers attributed to central Sub-Systems

Level 0 Trigger TRG 250 – 254 - L0 Muon TMU 220 – 229 - L0 Calorimeter TCA 230 – 239 - L0 Pile-Up TPU 240 – 249Timing and Fast Control TFC 10 – 19Infrastructure INF 20 – 29Central Control LB 30 – 49High Level Trigger HLT 50 – 99Reserved 1 – 9, 100 –

109In a hierarchical structure a certain node has to be connected to all the nodes below, so that one can navigate and view information from the other nodes. The best solution to set up a distributed system is that the top-level system acts as a client for all other systems, the bottom-level systems act as servers to all other systems and the medium-level systems act as clients for all systems below and servers to all system above (see Figure 1).

Figure 1: Setting up the distributed system

Thus, in the “config” file of a certain node a line “distPeer” needs to be added per system below. For example if a middle system called “VEDAQ1” contains two children: VEDAQA1 and VEDAQA2, the config file of system VEDAQ would contain:

[general] distributed = 1

[dist] distPeer = "VEDAQA01" 5 #Connect to the VELODAQA1 PC, system number 5 distPeer = "VEDAQA02W" 7 #Connect to the VELODAQA2 PC, system number 7

page 44