24/02/2011 Muon Barrel Workshop 1
Gianni Masetti & Ignacio Redondo (and others)
Shutdown interventions and operation in 2011
24/02/2011 Muon Barrel Workshop 2
Outline• Operations in 2011
– New oncall support structure– Comment on Interfill Cosmics
• DocDB interlude• Attemp to summarize shutdown interventions
– Everything touched,…but MCs & chambers– Several crucial soft points reinforced.– 2011 DT is a stronger system
• Thanks to everyone involved
24/02/2011 Muon Barrel Workshop 3
Operations in 2011• Special runs in first fills under discussion (only RPC in trigger for
instance) • New oncall support scheme: 1. DT DOC (Detector On Call) is our first defense line:
– 1 responsible per week– Reports in DT weekly meeting/daily run meeting– Earns central credit –points (21) – and 0.25 ManMonth of ESP for the training week?– More exposure (4 weeks min) and longer shift periods (1-2 week)– First discussion session held yesterday– Practical trainings to be organized– Population: 10/15 old DT oncall; 2/15 old DTFM; 3/15 newcomers
2. Infrastructure oncall support down to minimum (2)3. DT expert cloud
24/02/2011 Muon Barrel Workshop 4
Le XV de DT (DOCs 2011) Zoltan Szillasi
Noemi Beni Mario Pelliccioni Marco Dallavalle
Daniele Fasanella Jesus Puerta Pelayo
Daniel Dominguez Vazquez Juan Pablo Fernandez
Alicia CalderonShan-Huei Chuang
Isidro González Caballero Massimo Nespolo
Paolo Bellan Antonio BrancaDaniel Teyser
24/02/2011 Muon Barrel Workshop 5
DT interfill Operations
• No Heavy Stable Particle data this year• Remaining HV ON ~1 h after the fill dump was and is
not a problem• Probably cosmics
(What flavour of cosmics?)(How many?)
• Expectacion is interfill time will narrow down• System operation will need (part of) this time for
mantainance – MC/Mezzanine reconfiguration– System changes tests– Routine Test pulse runs– …
24/02/2011 Muon Barrel Workshop 6
CMS DocDB: (Yet another) CMS database CMS DocDB: (Yet another) CMS database
DB documental dropbox: designed to host any kind of documents except:•Talks already included in Indico•Engineering docs (EDMS)•Official papers (CDS)
Any other document can (must?) be uploaded to DocDB
Open to any CMS member (NICE login)
Still preliminary version (some issues with permissions)
https://cms-docdb.cern.ch/cgi-bin/DocDB/DocumentDatabase
Jesus Puerta
24/02/2011 Muon Barrel Workshop 7
A single «document» can contain several files
Any entry is updateable a posteriori (versioning)
Uploaded document data to be included:TitleDescriptionPrivilegesTopics/Tags (to be chosen from a predefined list)
Several tags for same document can be used. Documents can be browsed
by tags/topics!
Any document or list of docs worth being kept, feel free to give DocDB a try.
Let Jesús know about any other tag structure
Preliminary list of DT-related tags(Cristina/Alberto/Jesús)
Jesus Puerta
24/02/2011 Muon Barrel Workshop 8
Readout/Control– New VME PC crate control board: Linco
• SC power cycle required vmepc reboot• Sometimes Linco Driver reload results in blocked PC • More frequent after general PC reinstallation by system
administrators• Exchange two damange fibres helps• Need long term monitoring• SC power cycle does not require to reboot PCs ! • Still have two remember to hit 2 buttons in two programs
– Thinking how to integrate this
– 10 5 DDU configuration• Minor FW upgrade• Spares!• As a subproduct, get faster configuration (30 %)
A. Triossi et al.
V. Monaco et al.
24/02/2011 Muon Barrel Workshop 9
ROS FW upgradeVME FPGA cannot be remotely programmed: delicate intervention
Main changes:
GOL does not power cycle on each configuration (to minimize stress), it will only power cycle at power boot (or at demand with expert stand alone program)
Maximum Event Size register can be reseted (manually and with a TTC command):
TTC BGo Channel 12 (command 0x30) performs Event Size reset, so we can choose the frequency we want to monitor max event size (once per orbit, etc)
TTC Hard reset implemented (BGo Channel 6, command 0x18):– Reload the firmware of CEROS and ROSCTRL FPGAs (radiation SEE)– Drawbacks:
• HU &TO resynch indep registers will give error after reload• ROS configuration will be erased (if something comes as mandatory it
will have to be hardcoded in the firmware)
DAQ software has been changed to make it faster (~50 % )
Cristina Fernandez Bedoya
24/02/2011 Muon Barrel Workshop 10
New MC Secondary link
• Granularity increased from half wheel to one sector (maximum achievable)
• Each sector chain completely independent (in the old board only 4 rs485 ports are available)
• FW remotely upgradable• the old board was upgraded adding some protection
circuitry, this circuitry will be integrated in the new one• Fibre connectors inside the crate, so more protected• Added an 1-wire bus for linco monitoring• More robust link to USC: rs422 via linco (fast), 1 fiber
compatible mode (same as now), 1 fiber fast mode• Power supply independent from trg/ro crate
Franco Gonella
24/02/2011 Muon Barrel Workshop 11
Primary link fiber 5 sec3(new fast mode)
UXCUSC
PIC32 Ethernet Starter KitWith optical interface
Ethernetconnection
DCS PCs rack
Primary link fiber 5 sec2(compatible mode)
DCS fibers rack
DCS in0 YB+1
USB
I connector
Secondary link board prototype installed on YB+1 TRG/RO upper crate
(the old board was left in site for quickly restoring in case of problems)
newold
Franco Gonella
SC crate•the prototype seemsto work properly•10 boards (+ 5 spares) will be ready at the end of the year
24/02/2011 Muon Barrel Workshop 12
Trigger
• See Session with specific talks• Janos installed new VME Control FW in DTTF crates
– Common effort: DT/DTTF, Bolonga/Padova (linco was very helpful), others,…
– Enables optoRX monitoring– Enables automatic link synchronization
• New crate tested (and removed) in sectors #1 # 2• New Barrel Sorter FW (under test)• New Wedge sorter FW (under test)• DT technical trigger (to be commisioned)
24/02/2011 Muon Barrel Workshop 13
Infrastructure
• Power Supply system• HV• LV
• Gas
24/02/2011 Muon Barrel Workshop 14Alberto Benvenuti
Chambers with Problems Since January 24 HV ON
All problems but for YB+2 MB2 S03 SL2 L2 occurred at HV ON Six chambers had problems at HV ON in 2010 and were quiet afterwards. Most probably
due to A877 vagaries In addition the A876 for YB0 S09 failed to turn ON and was replaced No HV problems occurred during HV ON from STANDBY after ramp to 3.8 T
24/02/2011 Muon Barrel Workshop 15Alberto Benvenuti
• Mapping of the smoke detectors DSS alarms was checked and several wrong paths were corrected. It was not easy but now the map is correct for all tower racks. (D. Fasanella)
•Several temperature sensors were replaced by David and Cristina, all sensors are now operational
•The grounding of the cable braids was checked and standardized at all tower racks (Franco,Barcellan):
• Braids connected to ground for FE LV cables• All MC LV cables are floating
Maintenance Report (NOT a complete list)
24/02/2011 Muon Barrel Workshop 16
One major incident:Oct 5 post-mortem
• During fill T sensor in turbine rack (wheel/4) fails, triggering DSS interlock. • Unfortunately, safety system was wrongly cabled, switching off wrong
rack, bringing down wheel -2• Problem understood (and short-term fixed) before the end of the fill.
Interlock expert had to drive in (1/2h).• Final fix done and tested during last technical stop.
• ~1 h to change conf to get CMS running (without wheel -2).– Good part of it because we needed trigger to change configuration accordingly– Prompt DT response: The “right” experts where involved.– Some time lost because power cycling crates involve rebooting control PCs to
load control drivers, sometimes PCs block at load.» Upgrade control boards this Christmas shutdown so no reboot is required.
– Would all DT experts have been equally fast ?• Documentation written• Need to spread the ability to change configuration among the DT FM pool
Reminder
Zurich Run Coordination Workshop
24/02/2011 Muon Barrel Workshop 17
LV VCON errors
• Several cable connectors were checked/replaced that showed anomalous temperatures at the A3009 modules
– No clear defects (L. Barcellan) were visible
• Several LV modules have been exchanged thanks to David and Daniele:
– MAO for YB0 S10, A3100 for YB0 Top Sector Collector– A3050 for YB0 MB2 S10– NOT obvious what to do with these modules since there is no
specific fault apart for repeated VCON VCC/VDD errors.• All connectors have been lubricated but those in
YB0+SCs (already done in summer 2010)
Alberto Benvenuti
24/02/2011 Muon Barrel Workshop 18Alberto Benvenuti
30°
25°
New LV Connector failuresThe temperature of one A3100 connector powering YB+2 SC 1-7 jumped to 32° and then dropped to 27°Checking the temperature after power cycle it dropped a bit. There was another sudden jump above 30° on 18-feb and then it dropped to a normal value on 17-feb
YB0 MB4L S10 Vcon from February 1 to 20
This episode occurred all by itself.The Voltage is still higher than it should be albeit below the alarm threshold
Drops when “touching” PS Positive polarity connector(failure not inside the crate)
24/02/2011 Muon Barrel Workshop 19
YB-1 MB3 S12 Vcon ErrorNew LV Connector
failuresFirst alarm 21-feb at 22:52 MC switched OFF at 0:42. Problem persists after MC power up.UX access needed to fix it.
Problems are no new in the sense that the pattern is the same as last year (positive polarity is warmer through an increase of resisitivity) It is a bit unsettling that there are still these problems after changing cables, reinforcing contacts, adding lubricant….
Maybe we are seeing effects due to the magnetic field, we could try to block the cables as was done before the connector exchange
At any rate, I suspect we have to live with this, since changing the connectors on the A3050 looks like mission impossible
Alberto Benvenuti
24/02/2011 Muon Barrel Workshop 20
Gas Analysis
A rather painful restart:•Software updates blocked the data storage/acquisition•The refresh rate of the measurements was changed and the time stamp is not useful anymore, still to be fixed•O2 Analysis cells damaged by leak (will be replaced ASAP)•H2O analysis cells to be replaced with a new model (proto in YB0)•Current O2 measurements are 20 to 50 ppm lower than in 2010 probably due to analysis cells•The system is still leaking 1000 liters/hour, ongoing since July
Issues were discussed with gas group (Haider, Guida…) suggestions for a more efficient operation should be implemented ASAP
Online gas chromatograph should be operational by end of march.
Alberto Benvenuti
24/02/2011 Muon Barrel Workshop 21
DT shutdown highlights• Readout/Control
– New VME PC crate control board: LincoNo need to reboot PCs after crate Power cycle
– 10->5 DDU configurationSpares!
• As a subproduct get faster configuration (30 %)– New ROS FW (delicate intervention)
FW reloaded at TTCHardReset• As a subproduct get faster configuration (50 %)
– Prototype Secondary link to MC installed • Trigger (see session with specific talks)
– Janos installed new VME Control FW in DTTF crates • Common effort: DT/DTTF, Bologna/Padova (linco was very helpful), others
Enables optoRX monitoringEnables automatic link synchronization
• Infrastructure: – Extensive DSS tests
Several mapping errors corrected– HV– LV
All connectors lubricated– Gas
VDC system installed (following talk)
24/02/2011 Muon Barrel Workshop 22
The End
24/02/2011 Muon Barrel Workshop 23Franco Gonella
sec1
sec2sec3
sec4sec5
sec6
TSB backup link
sec1
sec2
sec3
sec4
sec5
sec6
Foreseen mapping
node1
node2
node3
node4
node5
node6
node7
Present mapping
Fiber1 (compatible mode)
Fiber2 (new fast mode)RS4221-Wire
RS422 VME crate Primary link 5 sec2
Primary link 5 sec3
PIC
Protection circuitry
PIC
PIC
PIC
PIC
PIC
PIC
DS1820 temp sensor
PIC
PIC
PIC
Can bus
node0
node8
node9
RS422 link 230400 bps
Optical link 230400 bps
Optical link 38400 bps
485port0
485port1
485port2
485port3
485port4
485port5
485port6
Prototype of the new Secondary Board installed on YB+1
With this mapping the board is fullycompatible with the old one
-the prototype seemsto work properly
-10 boards (+ 5 spares)will be ready at theend of the year
Recommended