8 Hour Heat RunSequencer

• History of the test

• Analyze of the events

• Memory space used by the sequencer

• Questions in view of the future tests

History of the test

• The ultimate current had to be entered by hand (Iultimate instead of Imax). It had to be done twice because the console went down.

• The 8HourRunTest was started on 33 PCs. – Strange current shapes on the graphs

– Change of PC state on the ‘plateau’. The sequencer generated a test failure for the related PCs.

• The tests were aborted with ‘ABORT ALL’ button.

• The SDDS file was created and saved on the desktop. The journal file was not created.

• The response time of the sequencer was a bit slow.

History of the test…• The 8HrRunTest was restarted. The PCs were ramped ok.• The sequencer and the console ran out of memory space. The sequencer had to be killed and the console rebooted. The SDDS files

on the desktop was deleted by a new logging on opera.

• The sequencer was restarted. A new sequence to log the data (SDDS logging) till the end of the test was created and started.• At 19h00 a sequence to switch the PC to OFF was launched – end of the test.

Analyze of events

• The ultimate currents were not copied in the LSA database (Imax requested in the LHC-D-HCP-0005 document).

• The edited parameters are not persistent.

• There were strange currents on the graph because they were badly time stamped.

• The sequencer generates a test failure when the state of a PC changes (!=IDLE) when Iultimate is reached

• The journal file was not created because a bug was generated when moving from FGC V1 to FGC v2 (a non static variable was declared as static). It prevented the test from fully stopping. Some threads were still active and the memory was not released by the garbage collector. It explains why the sequencer became slower.

Analyze of events…

• When the 8HourRunTest was restarted the sequencer ramped correctly the PCs but it slowly took all the available memory space and neither windows nor the sequencer could respond to operator commands. The console had to be rebooted. The problem is under investigation (next slides).

• The sequencer was restarted and a new sequence was launched to start the SDDS logging. It worked fine.

• Finally a stop sequence was launched to put the PCs to OFF.

Sequencer outputs

• SDDS logging of all PCs from 13h30 to 19h00 (current, voltage, pc_state, faults and time)

• SDDS logging of PC ‘Switch OFF’• No journal of the starting and ramping of PCs

Memory use as a function of the number of tests run in sequence

The memory use when running in sequence a test on 33 PCs (values given by JProfiler)

(The bug was fixed before getting the numbers)

1. First run, memory use: between 100 and 140 MB

2. Second run, memory use: between 155 and 185 MB

3. Third run, memory use: between 200 and 245 MB

4. Fourth run , memory use: between 260 and 305 MB

Conclusion: ~50 MB of memory is not released between each test

This problem was solved yesterday using JProfiler

80 PCs

Memory problems

• A bug found

• Double buffering

• No Dispose() on frames

• Static vector

• Object not released in java data viewer

Memory use as a function of the number of PCs

• 10 PCs, memory use: between 57 and 80 MB

• 20 PCs, memory use: between 80 and 105 MB

• 30 PCs, memory use: between 100 and 130 MB

• 40 PCs, memory use: between 120 and 150 MB

• 80 PCs, memory use: between 190 and 240 MB

• Conclusion: ~ 2 MB per PC (even less with no double buffering option) starting at 40MB

Limits of the number of PCs tested in parallel

• Memory limit

• GUI limit

• Number of threads

• Program response time

Questions in view of the future tests

• Edited parameter persistence?• Imax, Iultimate, Inominal? Which one and when?• Synchronization of ramping (80 PCs started in 55

seconds) - Ramping synchronized by timing event?

• Max number of PCs in parallel?• Less graphs, more diagnostics?• Restart PCs which fail during the test? Not

feasible with current version of the sequencer

Remarks

• Two many things to get working in a short time (FCR, consoles, opera profile, ccm, sequencer modification for FGC version 2, logging)

• Key people on holiday• FCR too noisy• More accurate planning and ‘cahier des

charges’

Conclusions

• Memory leak problems are solved

• Max number of PCs per test mainly limited by GUI

• More dedicated diagnostics with limited GUI when testing numerous PCs in parallel

• Better planning and ‘cahier des charges’