Embed Size (px)
Citation preview
IMPROVEMENT OF THE ORACLE SETUP AND DATABASE DESIGN ATTHE HEIDELBERG ION THERAPY CENTER
K. Höppner∗, Th. Haberer, J. M. Mosthaf, A. Peters,HIT, Heidelberg Ion Therapy Center, University Hospital, Heidelberg, Germany
M. Thomas, A. Welde, Eckelmann AG, Wiesbaden, GermanyG. Fröhlich, S. Jülicher, V. RW Schaa, W. Schiebel, S. Steinmetz,
GSI Helmholtzzentrum für Schwerionenforschung, Darmstadt, Germany
Abstract
The HIT (Heidelberg Ion Therapy) center is an acceler-ator facility for cancer therapy using both carbon ions andprotons, located at the university hospital in Heidelberg.It provides three therapy treatment rooms: two with fixedbeam exit (both in clinical use), and a unique gantry witha rotating beam head, currently under commissioning. Thebackbone of the proprietary accelerator control system con-sists of an Oracle database running on a Windows server,storing and delivering data of beam cycles, error logging,measured values, and the device parameters and beam set-tings for about 100,000 combinations of energy, beam sizeand particle rate used in treatment plans. Since going op-erational, we found some performance problems with thecurrent database setup. Thus, we started an analysis in co-operation with the industrial supplier of the control system(Eckelmann AG) and the GSI Helmholtzzentrum für Schw-erionenforschung. It focused on the following topics: hard-ware resources of the DB server, configuration of the Ora-cle instance, and a review of the database design that under-went several changes since its original design. The analysisrevealed issues on all fields. The outdated server will be re-placed by a state-of-the-art machine soon. We will presentimprovements of the Oracle configuration, the optimizationof SQL statements, and the performance tuning of databasedesign by adding new indexes which proved directly visiblein accelerator operation, while data integrity was improvedby additional foreign key constraints.
THE HIT MEDICAL ACCELERATOR
The heavy ion accelerator at HIT is used for raster-scanning radiation of cancer patients (cf. [1, 2] for anoverview) with different types of ions from two sources(upgrade to three sources in progress [3,4]) in several treat-ment rooms, two with horizontal fixed beam exit and theheavy ion gantry with rotatable beam exit (under comis-sioning [5]), and a beam exit for experiments (see Fig. 1).Each combination of source and destination may be usedfor medical treatment, represented within the AcceleratorControl System (ACS) by the so-callled virtual accelera-tor number. A radiation plan consists of a series of beampulses chosen from a catalogue of 255 different energy
Figure 1: HIT accelerator facility with two ion sources, lin-ear accelerator, synchroton, two horizontal beam exits andgantry for medical treatment. The experimental area is notshown.
values (88–430 MeV/u for carbon, 48–220 MeV/u for pro-tons), 6 focus sizes, 15 intensity values (2× 106–5× 108
particles per second for carbon, 8× 107–2× 1010 pps forprotons), and 36 exit angles in case of the gantry. Thesetuples of beam settings are named the MEFI combinations.Both the virtual accelerator as the MEFI combination maybe changed from beam pulse to beam pulse (multi-plexedoperation).
DATABASE AS PART OF THEACCELERATOR CONTROL SYSTEM
The database is a core part of the accelerator control sys-tem. It is used for
• a list of executed beam cycles for about 6 weeks to-gether with measured values for e. g. currents of powersupplies, data beam instrumentation (like ionizationchambers), vacuum pressure, . . .
• list of devices that are active for a virtual accelerator,• settings for all accelerator devices to be written into
the device controllers,• alarm and log messages,
Proceedings of ICALEPCS2011, Grenoble, France MOPMS030
Upgrade of control systems 393 Cop
yrig
htc ○
2011
byth
ere
spec
tive
auth
ors—
ccC
reat
ive
Com
mon
sAtt
ribu
tion
3.0
(CC
BY
3.0)
• users and access rights,• computers and applications.
The device settings are generated as binary large ob-jects (BLOBs) by the DSM, the data supply module, fromthe physical beam properties optimized for a representa-tive number of MEFI values and interpolation over the fullMEFI range. Finally, the device blob per MEFI is storedwithin the Oracle database.
There exist three kinds of device blobs to combine theneed for adjusting the accelerator settings and test of dif-ferent beam optics with the quality requirements for stablesettings needed for medical treatment:
• flash blobs that are stored in non-volatile memory ofthe device controllers after verification for therapy,
• online blobs stored in the RAM of the device con-trollers that are changed during accelerator adjustment(without affecting the verified flash data), and
• offline blobs to save old device data, present in thedatabase, only.
The checksums of the blobs stored in the device controllersare checked against the blob checksums stored in databaseon any beam request to ensure the integrity of the settings.
DATABASE PROBLEMS
Since about 2010 the existing Oracle database (Oracle 9irunning on a server with a dual core Xeon CPU and 2 GBof RAM under Windows 2003) showed performance prob-lems. This was both visible during therapy (when checkingthe blob checksums, writing measured values) as during ac-celerator adjustment (when writing generated online blobsinto database, copying online blobs into flash blobs afterverification).
Obviously, a major reason was the outdated hardware,that was upgraded to a modern blade center during summer2011 (as presented in [6]), but also issues beyond hard-ware should be addressed. Thus, we established a work-ing group consisting of HIT controls group, EckelmannAG (as supplier of the proprietary control system) and thedatabase experts from the accelerator controls departmentof the GSI Helmholzzentrum für Schwerionenforschung inDarmstadt, Germany.
The workgroup analyzed both the setup of the currentOracle installation (e. g. by statspack1 reports of usual oper-ating tasks) as well as the structural design of the database(tables, indexes, constraints).
RESULTS, RECOMMENDATIONS, ANDREALIZATION
Hardware and Oracle SetupAs expected, both the CPU speed and the memory of
the database server were considered insufficient. It was re-1 statspack is a performance monitoring and reporting tool provided by
Oracle starting from version 8i, cf. http://www.oracle.com or http://www.orafaq.com/wiki/Statspack.
commended to upgrade the Oracle installation to version11g and use Oracle’s Automatic Storage Manager (ASM)as file system which is optimized for the data storage of anOracle instance. RMAN2 should be used as backup toolinstead of the previous daily tablespace exports.
With the installation of the new blade servers during thesummer shutdown period, these proposals were put intopractise.
Number of Commits
The statspack analysis showed a high number of about20 commits per second. This is caused by several data gate-way processes that insert measured values from the devicesinto the database. In the current realization of the controlsystem, a commit command is sent per insert statement. Itwas proposed to decrease the number of commits per sec-ond.
Eckelmann AG was checking the impact of groupingseveral insert statements per commit on the overall perfor-mance of the ACS. Obviously, the longer the delay be-tween insert and commit, the slower the GUIs will presentstate changes and measured values to the operator. Thus,a compromise had to be found. Additionally, since sev-eral applications are writing into the database, the effort toimplement and test a better commit mechanism was ratherhigh.
Finally, the changes will be installed at HIT in October.
Structure of Tables
Many requirements for accelerator operation changeduntil or even since the accelerator facility became opera-tional for medical treatment, inspired by the lessons learntduring commissioning. Thus, there was an evolution of theACS from its original design until its current state. Thedatabase design reflects this evolution.
It was the overall impression that the database designfor rather static data (like definition of ion types, beam-lines, list of devices and their assignment to beamlines) wasgood, with appropriate relations between tables defined byforeign key constraints. But in some cases, data columnswith dynamic, volatile data were added into these “static”tables due to changing or adding new features for accel-erator controls. Tables for dynamic data (beam cycle data,measured values, messages) were often lacking foreign keyconstraints.
The recommendations for an improvement of thedatabase design were as follows:
• Separate tablespaces for static (configuration) dataand dynamic (measured) data,
• avoid to mix columns with static and volatile data intables,
• define proper relations between tables by adding for-eign key constraints where missing,
2RMAN is a tool provided by Oracle to backup, restore and recoverOracle databases, cf. http://www.oracle.com
MOPMS030 Proceedings of ICALEPCS2011, Grenoble, France
394Cop
yrig
htc ○
2011
byth
ere
spec
tive
auth
ors—
ccC
reat
ive
Com
mon
sAtt
ribu
tion
3.0
(CC
BY
3.0)
Upgrade of control systems
Figure 2: Current design of the table with MEFI deviceblobs, with three blobs per record.
• add indexes to optimize database queries,• redesign “wide” tables to achieve a better in-memory-
caching of data records within the Oracle database in-stance.
A good example for the last item is the table of deviceblobs per MEFI combination. As mentioned above, thereexist online, offline and flash blobs. In the current tabledesign, the table for MEFI device blobs has three blobcolumns (as shown in Fig. 2), one per blob type, i. e. whenloading the data record for a MEFI combination and a de-vice, three blobs are read and have to be cached in mem-ory. Consider the workflow during accelerator adjustment,when online blobs are generated and written into databasefor all MEFI combinations and all devices. For download-ing the new online blobs into the RAM of the device con-trollers, many records from the table with MEFI deviceblobs have to be read, and though only the online blobs areneeded, the offline and flash blobs are read as well, con-suming I/O ressources and memory of the database cache.
It was recommended to keep the blobs itself in an owntable and replace the blob columns in the MEFI blob tableby references. This would increase the number of tables it-self, but keep the MEFI blob table compact. By appropriatejoins between the MEFI blob table and the table(s) keepingthe “real” blobs, the set of blobs that are read by Oraclewould be reduced to those that are really needed. Figure 3shows a possible redesign, where the blob columns are re-placed by references to another table. Since changes to thetable structure have impacts on the whole ACS, a redesignneeds to be thorougly planned and tested. This process willbe done until the end of this year.
Other recommendations listed above are already imple-mented or will be installed during the next ACS update inOctober 2011 (like adding foreign key constraints). Theseparation of static and volatile data in different tablespaceswas realized during the migration from Oracle 9i to 11g.
Figure 3: Possible change of table structure for MEFI de-vice blobs by replacing the blob columns with a referenceto another table keeping the blobs.
Several missing indexes were pointed out by the workinggroup, that were added in the meantime, and some indexeswere reorganized to achieve unique indexes as unique in-dex scans are more efficient than normal index scans.
FIRST RESULTS
In combination with the new hardware platform, thechanges to the Oracle database show already a positive per-formance effect.
Figure 4 compares the time between the stop and startbroadcast signals (as sent by the central timing system),i. e. the dead time between beam cycles needed to prepareand setup the next cycle, for a week of therapy in Septem-ber 2010 and September 2011, respectively. While the time
0 500 1000 1500 2000 2500 3000
num
ber
of c
ycle
s (a
rbitr
ary
units
)
time (ms)
Broadcast Stop-Start Sept. 2010Broadcast Stop-Start Sept. 2011
0 1000 2000 3000 4000
zoomed view
Figure 4: Comparison of dead time between beam cyclesneeded for preparation of the next beam pulse: September2010 (red) and September 2011 (green).
Proceedings of ICALEPCS2011, Grenoble, France MOPMS030
Upgrade of control systems 395 Cop
yrig
htc ○
2011
byth
ere
spec
tive
auth
ors—
ccC
reat
ive
Com
mon
sAtt
ribu
tion
3.0
(CC
BY
3.0)
0
500
1000
1500
2000
2500
3000tim
e (m
s)Broadcast Stop-Start Sept. 2010Broadcast Stop-Start Sept. 2011
Figure 5: Scatter plot of dead time between beam cyclesneeded for preparation of the next beam pulse: September2010 (red) and September 2011 (green).
needed to prepare the next beam cycle after the end of theprevious cycle is about 250 ms for most of the cycles, theaverage dead time was spoiled by many cycles with a muchlonger preparation time before the start broadcast signalwas sent. The analysis of traces of servers and device con-trollers showed, that this effect was mainly caused by
• a delay in the Oracle database when creating thedatabase record for the upcoming cycle and queryingthe list of devices that are active for the cycle,
• a read delay in the flash memory of the device con-trollers when checking the MEFI blob checksums.
After both problems were solved this year, it is immedi-ately visible from the plot that most delayed cycles wereeliminated, with the effect that the average preparation timewas decreased from 490 ms in September 2010 to 390 ms inSeptember 2011. The scatter plot in Fig. 5 also shows, thatthe number of cycles with preparation time above about400 ms is decreased dramatically.
We compared also the times between the broadcast sig-nals for cycle start and stop during therapy (based on medi-cal treatment with up to 5 s radiation of the tumor per beamcycle) for September 2010 and 2011. As seen in Fig. 6the cycle performance increased a lot, caused both by im-provements in the therapy control system (TCS) and thecommunication between ACS and TCS, as well as the per-formance tuning of the database. The analysis of the tracesshowed that the time needed for updating the cycle statusin the database was sometimes up to 200 ms.
CONCLUSION
As presented in the previous section, the accelerator per-formance was increased a lot during the last year. This isdue to many efforts to optimize the ACS, both by hardwareas well as by improvements to the database.
The improvements are also visible in GUIs for the oper-ator that profit from better database querying performance,
4000 6000 8000 10000 12000
num
ber
of c
ycle
s (a
rbitr
ary
units
)
time (ms)
Broadcast Start-Stop Sept. 2010Broadcast Start-Stop Sept. 2011
9000 10000 11000 12000
zoomed view
Figure 6: Comparison of beam cycle time (for up to 5 stumor radiation): September 2010 (red) and September2011 (green).
and time consuming tasks during accelerator adjustments,like interpolating and generating MEFI blobs or flashingare much faster than before. E. g., an interpolation of alldevices for the gantry needs about 1.5 hours per ion type,instead of 6 hrs. before.
The recommendations didn’t just address database per-formance, but data integrity, too. The realization of properrelations between the tables, as it is currently in progressor preparation, will help to avoid both redundant or eveninvalid data within the database.
REFERENCES
[1] Th. Haberer et al., “The Heidelberg Ion Therapy Center”,Rad. & Onc., 73 (Suppl. 2), 186–190, (2004).
[2] S. Combs et al., “Particle therapy at the Heidelberg Ion Ther-apy Center (HIT)”, Radiotherapy and Oncology, Vol. 95, 41–44, (2010).
[3] A. Peters et al., “Operational Status and Further Enhance-ments of the HIT Accelerator Facility”, IPAC 2010, Kyoto,Japan, May 2010, MOPEA006, http://www.jacow.org.
[4] R. Cee et al., “Status of the Ion Source and RFQ Test Bench atthe Heidelberg Ion Beam Therapy Centre”, IPAC 2011, SanSebastián, Spain, September 2011, WEPS044, http://www.jacow.org.
[5] M. Galonska et al., “Commissioning of the Ion Beam Gantryat HIT”, IPAC 2011, San Sebastián, Spain, September 2011,THOAB03, http://www.jacow.org.
[6] J. M. Mosthaf et al., “Upgrade of the Server Architecturefor the Accelerator Control System at the Heidelberg IonTherapy Center”, ICALEPCS 2011, Grenoble, France, Oc-tober 2011, MOMMU009, http://www.jacow.org.
MOPMS030 Proceedings of ICALEPCS2011, Grenoble, France
396Cop
yrig
htc ○
2011
byth
ere
spec
tive
auth
ors—
ccC
reat
ive
Com
mon
sAtt
ribu
tion
3.0
(CC
BY
3.0)
Upgrade of control systems
