CO Support proposal for 2014

Marine Pace Technical Committee -12 Dec 2013

Context

• Detailed proposal for the organisation and the process of the CO support in 2014

• Main focus on PS Complex as subject to major controls changes & to a new support model

2

3

Recap of phases from Eugenia’s presentation

DRY RUNS COMMISSIONING & EARLY BEAM OPERATION

STABLE OPERATION

CO Support phaseswith respect to re-commissioning schedule 2014

(Latest Official schedule still being discussed EN/GS/OP)

July3 4 5 6 7

Jan Feb Mar Apr May Jun198 9 10 11 12 13 14 15 16 17 18 20 21 22 23 24 25 26 27 28 29

# # # 3 17 # 3 # 17

PSB

# # # 7 #5 #

L2

# # 2 9# # 7 # # ##

LEIR

PS

AD

Services

Ventilatin commissioning

Electrical distribution(EL, EPC)

Controls PSB/TT2

LEIR - L2-L3 PSB Chilled W.

Compressed air

Bake-out

Hardware tests

Hardw. tests

PS main magnets

POPS MPS

Hardw.

AD target SPS last circuits

Hardw.

ADEast area

Glob. Acc. Test

Access N.C.

Glob. Acc. Test

& DSO

Commissioning BEAMSource

Cold Check-out Beam set-up BEAM

Cold Check-out Beam set-up BEAM

HW TEST period : CO is involved in many DRY RUNS with EQP GP + OPAim : TEST as much as possible the controls infrastructure

PHASE 1 : CO DRY RUNS -> all experts required for the dry run will be present

CO Support phaseswith respect to re-commissioning schedule 2014

(Latest Official schedule still being discussed EN/GS/OP)

July3 4 5 6 7

Jan Feb Mar Apr May Jun198 9 10 11 12 13 14 15 16 17 18 20 21 22 23 24 25 26 27 28 29

# # # 3 17 # 3 # 17

PSB

# # # 7 #5 #

L2

# # 2 9# # 7 # # ##

LEIR

PS

AD

Services

Ventilatin commissioning

Electrical distribution(EL, EPC)

Controls PSB/TT2

LEIR - L2-L3 PSB Chilled W.

Compressed air

Bake-out

Hardware tests

Hardw. tests

PS main magnets

POPS MPS

Hardw.

AD target SPS last circuits

Hardw.

ADEast area

Glob. Acc. Test

Access N.C.

Glob. Acc. Test

& DSO

Commissioning BEAMSource

Cold Check-out Beam set-up BEAM

Cold Check-out Beam set-up BEAM

PHASE 2 -> Commissioning & Early Operation phase -> all experts on best effort and selected experts on special shift work

Phase 1: HW Tests

• CO Dry Runs• Involved: CO + EQP Groups + OP • Working hours• BE/CO support organization -> next slide• MCCs are responsible for the dry-runs

6

7

Dry-runs Organization

• 1 Dry run expands over one week • Several dry-runs per week • Groups involved: CO + EQP Groups + OP • Each dry-run deals with a system (ex: Matrix 10

MHz)

Claude’s courtesy

Dry-runs Insertion in OP Schedule - example

List of DRY RUNSMany in parallel

Dry-runs Support Team

MONDAY TUESDAY WEDNESDAY THURSDAY FRIDAY

AM PM AM PM AM PM AM PM AM PM

Preparation 1

BE-CO Test 1

Preparation 2

BE-CO Test 2Debriefing/Debugging/Consolidation Debriefing

5 working days Period

A specific team of 5 experts is assigned to each dry-run by the MCC

CO experts for all systems involved are available on-call

Claude’s courtesy

Next Steps

1. The dry Run planning (duration, date, content) will be consolidated by Claude (as planning officer) with input from MCCs

2. Then will be discussed with EQP groups + OP + Katy Foraz

3. The final agreed planning will be incorporated in the 2014 OP Schedule

Milestone: these steps must be completed by end January 2014

Next Steps for the MCC

• Prepare for each dry-run : • List of pre-requisites defined as Jira issues• List of checks defined as Jira issues• Wiki page centralizing all information• List of experts to be present during the whole dry-run and

those to be available on-call.

12

Phase 2: Cold Checkout

Can be during night time• 24-hour planning of checks defined by OP • CO INTENSE validation phase• Experts on best effort• Experts PRESENT in the CCC (full availability expected) during day

time• Some experts placed under special shift work conditions • Organization under MCC responsibility

13

Phase 2: Beam Set up & Early Operations phase

BEAM SETUP & OPERATIONS• CO is considered operational by OP but is NOT • Experts on Best Effort but high availability is expected• Requested experts placed under special shift work

conditions • Organization under EXM responsibility

14

Special Shift work for phase 2• Motivation for shift work: speed up CO issues resolution • Inspired from BI Commissioners organization at LHC startup• Role of expert on shift

– Offer support to OP on CO problems to diagnose and solve the problem– Report feedback on DIAG tools as will be heavily used – If time left during shift, work on DIAG tools

• Will be a natural trigger to improve diagnostics tools to avoid constant calls to CO experts

• Organization– 3 shifts per day – Max 2 shifts / week per person and typically only one out of normal

working hours– Compensation: experts on shift are required to take the morning off

after an evening shift and the complete day off after a night shift

Special Shift work • Current estimate of total shift period:– Max 3 months to cover all accelerators

• To be adjusted according to real time requirements• AD case (starting later, will benefit from debugging done on previous

accelerators -> no shift work may be required)

15

Start CCO Lin2/PSB

CO system STABLE

Special Shift work : status

• The detailed organization will be presented to the group early 2014

16

17

Phase 3: Beam Operations

•

Beam OPERATION • Controls stabilized• Upgrades only for justified requests and under

OP authorization• BEST EFFORT support ONLY

Support model in stable phase• BE/CO support is based on Best Effort specialists teams

– For all CO Hardware and Software – We reach a unique model for all accelerators PSX SPS LHC– Based on 2014 interventions statistics, we’ll evaluate and tune our

support strategy• Previous ‘Piquet’ suppression motivated by :

– 1st line FE support transfer to EQP Groups– New HW -> less failures (teething problems solved in previous phase) – Many problems reported to the PIQUET were outside its scope of

responsibility (wrong timing config….)• Pre-requisites for Best Effort mode to perform well:

– The responsibility transfer of first line support for FEs from CO to EQP is effective

– Strong diagnostics infrastructure -> Vito’s talk

19

First line support for Front ends 1/2• Responsibility is transferred to EQP Groups – Except for pure BE/CO systems (LTIM, OASIS, …)– Policy agreed between CO an EQP GP – Special case of legacy systems (ex. CPS/PSB BLM)

• Responsibility share between CO and EQP GP• Case by case study to be done

– To minimize intervention time, the EQP group specialist will have access to centralized stock of BE/CO HW modules to perform the HW change if (s)he is already on site • BE-BI is on board• Training of EQP experts on procedure to extract HW modules from CO

stock

• CCC will call directly the FRONT END specialist in the EQP GP

First line support for Front ends 2/2

• CO requirements– Every FE is described in the CCDB with its responsible

group and its operational support team– Information is propagated from CCDB to DIAMON and

the OP WEB Support page ‘webPiquet’:• DIAMON should display the operational support team when

clicking on the front end• WEBPIQUET should display the phone number(s) related to

the operational support team

• Actions on MCC– Ensure interfaces and data propagation are available– Check that operational support team is filled

INCA /LSA

.

OASIS TIMING

CMW/JAPC DRIVER

HW SPECIALISTS

CONSOLE + SERVER

... .

FRONT END SYSTEMS

SIS Logging

FEC OS…

EQP GP FRONT-END(Ex: TE/EPC FEC) BECO BEST EFFORT SERVICES

DIAMON/LASER

…...

OTHER SERVICES.

..

EQP GP FRONT-END(Ex: TE/EPC FEC)

EQP GP FRONT-END(Ex: TE/EPC FEC)

FRONT END INSTALLATION

Organization of expert teams under discussion in CO- Team membership - Generic phone number - Anonymous phone number …

BEST EFFORT Expert Teams– not exhaustive nor correct list

OPTeams likely often contacted > hence the importance of good diagnostics

Scenario for a problem

• A problem manifests in a high-level application or on DIAMON screen1. OP will have improved error reporting from application and core

diagnostic tools to identify the source of problem2. Possible recovery solutions will be available via DIAMON (repair,

restart process, reboot FE, etc. ) to be done by OP• Requisite: DIAMON will display the right status of all FEs at all times

3. If no recovery was possible by available actions, OP will call the best effort expert from the OP Web support list & record the intervention

4. In case of a FE problem: EQP GP specialist is called and runs his expert diagnostics

5. if it turns to be a CO FE problem, he will ask the CCC to call a CO specialist from the Best Effort team identified as concerned.

Intervention Recording• The recording of each problem and intervention should be done by OP using

eLogbook– This is critical to ensure the compensation of the experts on best effort when they are

called by the CCC• The best effort specialist will ensure the information is updated at the end of his own

intervention

– A technical report of the intervention will be written by the specialist• This will be used by the second expert and ease the hand over

• This means extensions on e-logbook & Jira– e-logbook: Creation of FAULT sub-categories under ‘ CONTROLS FAULTS’ category,

basically one per BEST EFFORT system– e-logbook: Write access mode from home– …JIRA issues automatic creation: to be generalized to all issues & promoted to LHC &

SPS• OP agreed on all of these, design to be finalized for elogbook + JIRA

• CO recorded interventions will be monitored by EXM to ensure any persisting problems are properly followed up and resolved inside the group

Perspectives 2014 …… for LHC !

24

Early 2015 schedule “Nominal" 25 ns Physics from July’15

b* = 19 m

vdM

LHCf

Work in progress!

RECOMMISSIONING WITH BEAM

Preliminary Planning – Commissioning starts in 2014

• 2014

26

Month Activity2014

Jan

Feb

Mar

Apr

May

June

July

Aug

Sept

Oct

Nov

Dec

2015

Jan

Feb

Mar

Dry Run

Dry Run

Dry Run

Dry Run

Dry Run

Dry Run

Dry Run

Sector Test

Check-out

Beamcomm.

Dry Run,Settings,

Cycle

LBDS Prep

LBDS reliability run: CCC: 6 weeks June/July

From April:

Every 2nd week: dry run

October:

Every week: dry run + sector test preparation

Beginning of Nov:

Sector Test

January:

Machine Check-Out

Feb + Mar:

Beam Commissioning

LBDS

+LBDS Prep

+LBDS Prep

Additional slides

27

Additional slides

28

OP SUPPORT Web page webPiquet

Tentative Re-commissioning Schedule(still being discussed EN/GS/OP)