Upload
ashley-allen
View
218
Download
0
Embed Size (px)
DESCRIPTION
Geneva CMS Week, RPC Operation Review Anton Dimitrov DETECTOR AND SERVICE EQUIPMENT BARRELENDCAPTRIGGERSPARE UXC Spare USC Spare UXCUSC Spare UXCUSC Trigger EndcapBarrel EQUIPMENT ALREADY UPLOADED in EMDb EQUIPMENT TO BE UPLOADED in EMDb – till end of June 2010
Citation preview
Geneva 15.06.2010 CMS Week, RPC Operation Review
Anton Dimitrov
RPC Detector RPC Detector ReviewReview
Hardware Status and New Hardware Status and New Technical ProceduresTechnical Procedures
Geneva 15.06.2010CMS Week, RPC Operation Review
Anton Dimitrov
CMS RPC EQUIPMENTCMS RPC EQUIPMENT
Detector and Service EQUIPMNETDetector and Service EQUIPMNET Known ProblemsKnown Problems ProceduresProcedures
Geneva 15.06.2010CMS Week, RPC Operation Review
Anton Dimitrov
DETECTOR AND SERVICE EQUIPMENT
BARREL ENDCAP TRIGGER SPARE
UXC
Spare
USC
Spare
UXC USC
Spare
UXC USC
Trigger
EndcapBarrel
EQUIPMENT ALREADY UPLOADED in EMDb
EQUIPMENT TO BE UPLOADED in EMDb – till end of June 2010
Geneva 15.06.2010CMS Week, RPC Operation Review
Anton Dimitrov
KNOWN PROBLEMS
Chambersin SG Slow Control Masked Rolls
Threshold Control TPH Sensors
PROBLEMS by TYPE
PROBLEMS by SEVERITY
1) Problems for long shutdown/CMS opening2) Problems for NEXT UXC ACCESS3) Reappearing Problems
Geneva 15.06.2010CMS Week, RPC Operation Review
Anton Dimitrov
Technical ProceduresTechnical Procedures How to deal with a new equipment How to deal with a new equipment
problems? problems? How to create a work package when How to create a work package when
access is announced?access is announced? How to replace equipment in UXC?How to replace equipment in UXC? How to replace equipment in USC?How to replace equipment in USC?
How to deal with an equipment problem
Operation Manager
P5 SHIFTER RPC Equipment Manager
HARDWAREEXPERT
elog Savannah
Sav
anna
hA
utom
atic
mai
l
Automatic mail
Sav
anna
h
ACCESS CMS RC&TC, GLIMOSCMS RC&TC, GLIMOSSLIMOS, RFM, CSLSLIMOS, RFM, CSL
Subdetector RFM(Operation ManagersOperation Managers)
WP
EMDB System Experts
EMDB Manager
RP Technicalshifter
BUFFERZONE
EMDBEQUIPMENT
UXC
AnnounceAccess
RP
Sw
eep
Access O
pen
Authorize
Req
uest
/Mod
ify
Req
uest
/Mod
ifyO
pen/
Clo
se
Ope
n/C
lose
ControlGive Access/Key
Change D
ata (Full System
Rights)
Supervise (Full R
ights)
Operational SpareOperational
Defective
Che
ck E
quip
men
t
Repair
Still Defective Tested
WasteBroken
Take after RP check
Use WP name to check in equipment
Geneva 15.06.2010CMS Week, RPC Operation Review
Anton Dimitrov
Work Package ListWork Package List## WP nameWP name DateDate DuratioDuratio
nnTask DescriptionTask Description probprob
lllocatiolocatio
nn11 RB LV board RB LV board
inspectioninspection14.06.114.06.1
001.1h1.1h W+1/S7 board replacedW+1/S7 board replaced 11 X2A11X2A11
22 RB HV testRB HV test 31.05.131.05.100
1.9d1.9d RPC-DT noise testRPC-DT noise test 11 X1 PPX1 PP
33 RE HV checkoutRE HV checkout 31.05.131.05.100
2d2d Investigate RE+1/2/21_22Investigate RE+1/2/21_22 11 X1 PPX1 PP
44 RE LV connect. RE LV connect. replacementreplacement
31.05.131.05.100
2.1h2.1h RE-3/2,3/25 Anderson RE-3/2,3/25 Anderson changedchanged
11 X3S51X3S51
55 RB LV connector RB LV connector checkcheck
31.05.131.05.100
2.2d2.2d W-1/S5/RB3-W-1/S5/RB3- 11 X4S11X4S11
66 RB LV module checkRB LV module check 20.05.120.05.100
1.1h1.1h W-1/S5/RB3-W-1/S5/RB3- 11 X4S11X4S11
77 RE LV module RE LV module replacementreplacement
20.05.120.05.100
1.1h1.1h Due to channel RE-Due to channel RE-3/2,3/253/2,3/25
11 X3S51X3S51
88 RPC LV connector RPC LV connector checkcheck
19.05.119.05.100
39min39min RE-3/2,3/25 & W-1/5/RB3-RE-3/2,3/25 & W-1/5/RB3- 22 X4S11, X4S11, X3S51X3S51
99 RB HV check outRB HV check out 04.05.104.05.100
2.1h2.1h W+2/8/RB2in & W+2/8/RB2in & W+1/7/RB3+W+1/7/RB3+
22 X1PPX1PP
1100
RPC HV system RPC HV system inspectioninspection
26.04.126.04.100
2.1d2.1d 4 spare RB cables for 4 spare RB cables for SVmaxSVmax
3 RE probl (2 3polar 3 RE probl (2 3polar changed)changed)
77 X1 PPX1 PP
1111
RB HV check outRB HV check out 08.04.108.04.100
5.2h5.2h Trip channel at W0 PPTrip channel at W0 PP 11 X2N39X2N39
1122
RB HV chain check RB HV chain check outout
16.03.116.03.100
1.9d1.9d 7 HV prob(W+2,W+1,W-7 HV prob(W+2,W+1,W-2,RE-1)2,RE-1)
77 X1 PPX1 PP
Geneva 15.06.2010CMS Week, RPC Operation Review
Anton Dimitrov
Work Packages by Work Packages by systemsystem
SystemSystem # WP# WP DurationDuration # Problems# Problems InsideInsideMTAMTA
OutsideOutsideMTAMTA
RB HVRB HV 55 6.2 days6.2 days 1313 1010 33RB LVRB LV 44 2.3 days2.3 days 22 11 11RE HVRE HV 33 6 days6 days 66 66 00RE LVRE LV 33 2.2 days2.2 days 11 11 00
RG RG TriggerTrigger
0 (2)0 (2) sharedshared 22 22 00
TotaTotall
1515 RB 8.5d RB 8.5d RE 8.2dRE 8.2d
2424 2020 44
Geneva 15.06.2010CMS Week, RPC Operation Review
Anton Dimitrov
BARREL HV BARREL HV INTERVENTIONSINTERVENTIONS
WheelWheel No. ProblemsNo. Problems No. No. Components Components
changedchangedRB+2RB+2 77 66RB+1RB+1 88 33RB0RB0 11 22RB-1RB-1 00 00RB-2RB-2 11 11
TOTALTOTAL 1717 1212
Geneva 15.06.2010CMS Week, RPC Operation Review
Anton Dimitrov
Barrel HV System: Summary Barrel HV System: Summary by Components II Trimesterby Components II Trimester
WheWheelel
HV HV BoardBoardA3512A3512
NN
Short red Short red cablecable
Brd -> YBrd -> Y
YY Long Long black black
cable US-cable US-UXUX
BeyoBeyond PPnd PP
SVMSVMax < ax < 96009600
i1 i1 settings settings > 10uA> 10uA
Rare Rare ProbleProble
mm
RB+RB+22 -- 11 33 22 -- -- -- --
RB+RB+11 11 -- 11 11 44 11 --
RB0RB0 -- -- 11 11 -- 11 -- --RB-1RB-1 -- -- -- -- -- -- -- --RB-2RB-2 11 -- -- -- -- -- -- --TotTotalal
22 11 44 44 11 55 11 00
Geneva 15.06.2010CMS Week, RPC Operation Review
Anton Dimitrov
Barrel HV System Status II Barrel HV System Status II trimestertrimester
Broken Y in Distributor Boxes: 6/480; ~ 1.2% Broken Y in Distributor Boxes: 6/480; ~ 1.2% (0.6%*)(0.6%*)
Broken long black cables (USC – UXC): 11 Broken long black cables (USC – UXC): 11 (7*)(7*) Spare black cables used 17/480; ~ 3.5% Spare black cables used 17/480; ~ 3.5% (2.7%*)(2.7%*) 473** RPC in DG mode 473/480; ~ 98.54% 473** RPC in DG mode 473/480; ~ 98.54%
(98.75%*)(98.75%*) 5 RPC in SG mode 10/1020; ~ 1% 5 RPC in SG mode 10/1020; ~ 1% (1%*)(1%*) 1 RPC totally dead 1/480; ~ 0.21% 1 RPC totally dead 1/480; ~ 0.21% (0.21%*)(0.21%*)N.B. N.B. ** => => Barrel HV system status for I trimester Barrel HV system status for I trimester
20102010**** => => 1 RPC OFF1 RPC OFF due to LV problem (next slide)due to LV problem (next slide)
Geneva 15.06.2010CMS Week, RPC Operation Review
Anton Dimitrov
Barrel Low Voltage/Slow Barrel Low Voltage/Slow Control Status II TrimesterControl Status II Trimester1 RPC LV OFF 479/480 ~ 1 RPC LV OFF 479/480 ~ 0.21%0.21% (0%*)(0%*)18 RPC I2C lines (in 17 chambers) not 18 RPC I2C lines (in 17 chambers) not working in the Barrel 18/800 ~ working in the Barrel 18/800 ~ 2.25% 2.25% (2.13%*)(2.13%*)Redundant DT lines working for 8/18 ~ Redundant DT lines working for 8/18 ~ 44%44% (9/17~53%*)(9/17~53%*)10/800 I2C lines in the Barrel not 10/800 I2C lines in the Barrel not controlled at all ~ controlled at all ~ 1.25% 1.25% (8/800 ~ 1%*)(8/800 ~ 1%*) Vth or Vmon values not settable on chip Vth or Vmon values not settable on chip basis in 4 cases in the Barrel basis in 4 cases in the Barrel (no change*)(no change*)
Geneva 15.06.2010CMS Week, RPC Operation Review
Anton Dimitrov
Barrel Threshold Control Barrel Threshold Control HistoryHistory
DateDate ProblematiProblematic FEB/chipc FEB/chip
No THR No THR ControlControl
DT line DT line WorkingWorking
No No redundantredundant
ControlControl22.01.20022.01.200
88 66 1212 77 5521.05.20021.05.200
99 88 1313 88 5523.11.20023.11.200
99 66 1414 88 6616.03.20116.03.201
00 44 1717 99 8815.06.20115.06.201
00 44 1818 88 1010
Geneva 15.06.2010CMS Week, RPC Operation Review
Anton Dimitrov
RE+2/2, 3/11- TrippingRE+2/2, 3/11- Tripping: Problem localized in HV distributor so fixed it by using : Problem localized in HV distributor so fixed it by using
the spare channel. (On April08)the spare channel. (On April08) RE-1/3/31, 32- SpikesRE-1/3/31, 32- Spikes: Problem localized in RE-1/3/31 (TOP) so disconnect this : Problem localized in RE-1/3/31 (TOP) so disconnect this
gap and run rest of three gaps. (On April08)gap and run rest of three gaps. (On April08) RE-1/3/05, 06-TrippingRE-1/3/05, 06-Tripping: Localized ~ at PP so fixed it by changing the tripolar : Localized ~ at PP so fixed it by changing the tripolar
connector in UXC. (Reported on April10)connector in UXC. (Reported on April10) RE-3/2, 3/26-TrippingRE-3/2, 3/26-Tripping - Fixed by changing connector in UXC at PP. (On April 28)- Fixed by changing connector in UXC at PP. (On April 28) RE-1/3/35, 36-TrippingRE-1/3/35, 36-Tripping – Actually problem was in RE-1/2/01,02 because channels – Actually problem was in RE-1/2/01,02 because channels
were swapped. Put back the channels at original position. Problem do not were swapped. Put back the channels at original position. Problem do not reproduce. (On April 28 reproduce. (On April 28
RE+1/2/21, 22- TrippingRE+1/2/21, 22- Tripping – During TA31052010 switch on back this channel so it – During TA31052010 switch on back this channel so it start working without any intervention and it is working up to now.(Reported on start working without any intervention and it is working up to now.(Reported on May06)May06)
RE-2/2,3/11-TrippingRE-2/2,3/11-Tripping- During TA31052010 swapped the channel with RE-- During TA31052010 swapped the channel with RE-2/2,3/12 both are working without any problem. (Reported on June 02).2/2,3/12 both are working without any problem. (Reported on June 02).
During last 3 months total 7 (6 trpping+1spikes) HV problems were reported
Saleh Muhammad NCP Islamabad
Endcap HV Interventions and StatusEndcap HV Interventions and Status
Geneva 15.06.2010CMS Week, RPC Operation Review
Anton Dimitrov
Endcap LV Interventions and StatusEndcap LV Interventions and Status
RE+2/2, 3/31- Tripped (LV FEB)RE+2/2, 3/31- Tripped (LV FEB) Fixed the problem by changing the problematic Fixed the problem by changing the problematic module with good spare onemodule with good spare one. (. (On March 11).On March 11).
RE-3/2, 3/25 –Tripped (LV FEB)RE-3/2, 3/25 –Tripped (LV FEB) Localized the problem in the module channels. Localized the problem in the module channels. Fixed the problem by changing the module also by repairing the connector which Fixed the problem by changing the module also by repairing the connector which break during changing the module . (Reported on May19)break during changing the module . (Reported on May19)
LV modules serial no.47LV modules serial no.47 which was replaced form P5 test it in bld.904 testing setup. which was replaced form P5 test it in bld.904 testing setup. The board did not give the problem again so put it back into system at P5 on May31. The board did not give the problem again so put it back into system at P5 on May31.
On 1On 1stst of April the X2S32 rack (YE-1 far side, X2 level) all electronics become off due of April the X2S32 rack (YE-1 far side, X2 level) all electronics become off due to rising the temperature of MAO at 29C and LV boards at 40C. This happen due to to rising the temperature of MAO at 29C and LV boards at 40C. This happen due to power failure of the turbine. But after 4 hours power came back and just switching on power failure of the turbine. But after 4 hours power came back and just switching on the MAO and boards everything start working. the MAO and boards everything start working.
Saleh Muhammad NCP Islamabad
Geneva 15.06.2010CMS Week, RPC Operation Review
Anton Dimitrov
Endcap Summary (RPC TCM)Endcap Summary (RPC TCM) SystemsSystems HVHV LVLV Slow controlSlow control Gas Gas
RPC TCM RPC TCM New New pending pending Problem Problem
Spare Spare cables cables in usein use
DB spare DB spare ch. in use ch. in use
Single Single gap gap
RPCsRPCs
Full Full RPC RPC outout
New New pending pending problemproblem
Full Full RPCs RPCs
outout
TH TH setting setting
ProblemProblempendingpending
T and RH T and RH sensor sensor
problem problem pendingpending
Known big leaks Known big leaks channelschannels
April26, 2010April26, 2010 33 00 11 1818 11 00 11 77 00 22
May 03, 2010May 03, 2010 00 11 11 1717 11 00 11 77 00 22May 10, 2010May 10, 2010 11 00 11 1717 11 00 11 77 00 22May 17, 2010May 17, 2010 11 00 11 1717 11 00 11 77 00 22May 23,2010May 23,2010 11 00 11 1717 11 11 11 77 00 22June 02,2010June 02,2010 00 00 11 1717 11 00 11 77 00 22June 07, 2010June 07, 2010 00 00 11 1717 11 00 11 77 00 22June 14, 2010June 14, 2010 00 00 11 1717 11 00 11 77 00 22
Saleh Muhammad NCP Islamabad
Geneva 15.06.2010CMS Week, RPC Operation Review
Anton Dimitrov
Trigger USC electronicsTrigger USC electronics CAEN VME controller failedCAEN VME controller failed in one Trigger in one Trigger
Crate on night 18/19 April 2010. Crate on night 18/19 April 2010. Replaced by a spare on 19 April. It was Replaced by a spare on 19 April. It was checked that only mezzanine board with optics checked that only mezzanine board with optics was damaged (most probably optical was damaged (most probably optical transmitter).transmitter).
At the moment no other At the moment no other problems - 100% of the system problems - 100% of the system functional!!!functional!!!
Karol Buńkowski, CERN, UW
Geneva 15.06.2010CMS Week, RPC Operation Review
Anton Dimitrov
Trigger Trigger Link System Link System ~10 March: one ~10 March: one Half Box tripped during normal operationHalf Box tripped during normal operation (not (not
during power on), first we have replaced the LV module, but it did during power on), first we have replaced the LV module, but it did not helped, so then we have replaced all boards from that Half Box not helped, so then we have replaced all boards from that Half Box ((CBCB, , 22x x MLBMLB,,44xx SLB SLB), and this helped. Test showed that:), and this helped. Test showed that: CB Actel damagedCB Actel damaged, the CB itself is OK,, the CB itself is OK, GOLs on both MLB damagedGOLs on both MLB damaged,, SLBs are OK,SLBs are OK,
RB+1 far (24 April 2010)RB+1 far (24 April 2010) - - pproblems with the CCU ringroblems with the CCU ring:: FEC mezzanine was not well attached to the CCU board.FEC mezzanine was not well attached to the CCU board. When it was corrected CCU errors appeared again. So the FEC When it was corrected CCU errors appeared again. So the FEC
mezzanine was swapped with other one. After that both FECs were mezzanine was swapped with other one. After that both FECs were working correctly. Possible reason: working correctly. Possible reason: FEC and the ribbon fiber FEC and the ribbon fiber connectors “not compatible”???connectors “not compatible”???
RB+2RB+2__near near ((31 May 31 May 2010) - 2010) - the CCU errors become very the CCU errors become very frequentfrequent,, ~every second (before that ring had a few CCU errors ~every second (before that ring had a few CCU errors per day)per day), , The ring B for that tower was disconnected, after that no CCU errors.The ring B for that tower was disconnected, after that no CCU errors.
CCU errors appear only on one-two rings, with low rate. In most of CCU errors appear only on one-two rings, with low rate. In most of the cases are recovered by the reset_fec automatically executed by the cases are recovered by the reset_fec automatically executed by the online software. the online software.
The system is working stable - 100% of the system functionalThe system is working stable - 100% of the system functional
Karol Buńkowski, CERN, UW
Geneva 15.06.2010CMS Week, RPC Operation Review
Anton Dimitrov
Trigger FirmwareTrigger Firmware 22 April - CB firmware updated - 22 April - CB firmware updated -
problem with the LB firmware problem with the LB firmware loading from FLASH correctedloading from FLASH corrected
Karol Buńkowski, CERN, UW 14 June 2010
Geneva 15.06.2010CMS Week, RPC Operation Review
Anton Dimitrov
Trigger Online Software Trigger Online Software StatusStatusThe Trigger Supervisor control over the full system (Link The Trigger Supervisor control over the full system (Link
System, Trigger System, RBC/TTU) is working fine.System, Trigger System, RBC/TTU) is working fine. Latest improvements:Latest improvements:
Monitoring panels in TSMonitoring panels in TS(the data from the XDAQs to the TS are send with use of the flash lists)(the data from the XDAQs to the TS are send with use of the flash lists)
Panel in the TS LBox Worker:Panel in the TS LBox Worker: Disabling the LBox XDAQs from the TS controlDisabling the LBox XDAQs from the TS control Loading the LB firmwareLoading the LB firmware Presenting the LBox XDAQs statusPresenting the LBox XDAQs status
Panel in the TS Cell:Panel in the TS Cell: disabling the TC XDAQs from the TS controldisabling the TC XDAQs from the TS control Presenting the TC XDAQs statusPresenting the TC XDAQs status
Alarms in the HOTSOPTAlarms in the HOTSOPT GUI for disabling the Sorter Crate inputs (in case of the Trigger Crate GUI for disabling the Sorter Crate inputs (in case of the Trigger Crate
malfunction)malfunction) To do next:To do next:
Better management of the configuration data (different local config keys Better management of the configuration data (different local config keys for different parts of the system),for different parts of the system),
FEB configuration from the DBFEB configuration from the DB Storing monitoring data in the condition databaseStoring monitoring data in the condition database
Karol Buńkowski, CERN, UW
Ian Crotty 14 June 2010
RPC CoolingRPC Cooling
Two main changes have occurred this year• Regulation Valve change on the “Endcap cooling circuit
Week 17 2010
• The changes of set Point and regulation Parameters sinceWeek 20 2010
Continuing ConcernsLow contingency
Temps still high in EndcapsTemp gradients along Z in RE
Ian Crotty 14 June 2010
RB Temperature Log since April 2010RB Temperature Log since April 2010
Ian Crotty 14 June 2010
RE Temperature Log since April 2010RE Temperature Log since April 2010
Ian Crotty 14 June 2010
Endcap Circuit Set-point Changed to Endcap Circuit Set-point Changed to 16.5°C16.5°C
Ian Crotty 14 June 2010
X2 Power Cut & New Set-point Change X2 Power Cut & New Set-point Change
Ian Crotty 14 June 2010
RPC Temperature Distributions RPC Temperature Distributions
+Z -ZRB020
.33C
19.9
0C
19.7
9C
20.2
5C
20.2
5C
18.6
7C
19.4
1C
20.8
3C
21.3
3C
20.3
2C
21.9
2C
0.5
C 0.8
C 0.6
C0.7
C0.8
C0.75 C
Gains since April 2010
Geneva 15.06.2010CMS Week, RPC Operation Review
Anton Dimitrov
Gas StatusGas Status New gas leak in WP2_S05_RB2 notified to CERN New gas leak in WP2_S05_RB2 notified to CERN
gas group. gas group. W0 readout board replaced (middle of March). No W0 readout board replaced (middle of March). No
effect on RPC operationeffect on RPC operation Warning due to purifiers fixed.Warning due to purifiers fixed. New Flow cell threshold settings for warnings and New Flow cell threshold settings for warnings and
alarms uploaded into DB. This did not solve the alarms uploaded into DB. This did not solve the sporadic warnings due to spikes on cells readout. sporadic warnings due to spikes on cells readout. For this is needed a new approach in the warning For this is needed a new approach in the warning treatment to be implemented into RPC-DCS. treatment to be implemented into RPC-DCS. Discussion on this is ongoing with G.Polese. Discussion on this is ongoing with G.Polese.
L. Benussi
Geneva 15.06.2010CMS Week, RPC Operation Review
Anton Dimitrov
Gas Flow cell Reading Gas Flow cell Reading fluctuationsfluctuations
L. Benussi
Geneva 15.06.2010CMS Week, RPC Operation Review
Anton Dimitrov
ConclusionsConclusions Partitions that cannot be threshold controlled are Partitions that cannot be threshold controlled are
increasing in time, leading to the dilemma increasing in time, leading to the dilemma “noise “noise vs. masking”!vs. masking”!
Apart from that Detector Performance is stable!Apart from that Detector Performance is stable! For better Detector performance threshold For better Detector performance threshold
refinement is required => DB is highly needed!refinement is required => DB is highly needed! Temperatures in Endcap still higher than 20°C, but Temperatures in Endcap still higher than 20°C, but
decreasing after the change of coolant set-point. decreasing after the change of coolant set-point. Signals of after pulsing raised again the question of Signals of after pulsing raised again the question of
individual chamber HV working point definition.individual chamber HV working point definition. Still lot of effort to be put in order to update all the Still lot of effort to be put in order to update all the
instructions! instructions!