G

F

M

Ta

b

a

ARAA

KDDDM

1

pr

•

•

•

Mne

pTtM

0h

ARTICLE IN PRESS Model

USION-7116; No. of Pages 3

Fusion Engineering and Design xxx (2013) xxx– xxx

Contents lists available at ScienceDirect

Fusion Engineering and Design

journa l h om epa ge: www.elsev ier .com/ locat e/ fusengdes

DSplus automated build and distribution system

. Frediana,∗, J. Stillermana, G. Manduchib

Massachusetts Institute of Technology, 175 Albany Street, Cambridge, MA 02139, USAConsorzio RFX, Euratom-ENEA Association, Corso Stati Uniti 4, Padova 35127, Italy

r t i c l e i n f o

rticle history:eceived 21 May 2013ccepted 27 November 2013vailable online xxx

eywords:ata acquisition systemsata managementata formats

a b s t r a c t

Support of the MDSplus data handling system has been enhanced by the addition of an automated buildsystem which does nightly builds of MDSplus for many computer platforms producing software packageswhich can now be downloaded using a web browser or via package repositories suitable for automaticupdating. The build system was implemented using an extensible continuous integration server productcalled Hudson which schedules software builds on a collection of VMware based virtual machines. Newreleases are created based on updates via the MDSplus cvs code repository and versioning are managedusing cvs tags and branches. Currently stable, beta and alpha releases of MDSplus are maintained foreleven different platforms including Windows, MacOSX, RedHat Enterprise Linux, Fedora, Ubuntu and

DSplus Solaris. For some of these platforms, MDSplus packaging has been broken into functional modules sousers can pick and choose which MDSplus features they want to install. An added feature to the latestLinux based platforms is the use of package dependencies. When installing MDSplus from the packagerepositories, any additional required packages used by MDSplus will be installed automatically greatlysimplifying the installation of MDSplus. This paper will describe the MDSplus package automated buildand distribution system.

. Introduction

MDSplus [1] is a collection of libraries and applications used forroviding access and storage of scientific data. It provides a wideange of functionality including:

Data acquisition from measurement devices; organization of thestorage of both experimental measurements, analysis results andvarious metadata associated with those data items.Access to the data and metadata from a wide variety of program-ming languages and utilities.Remote data access using a variety of transport mechanisms.

Originally developed on the OpenVMS platform in the 1980s,DSplus has since grown in functionality and has been ported to

umerous computing platforms. MDSplus is used by scientists andngineers worldwide mostly in the field of fusion energy research.

Maintaining the MDSplus software on a large number of com-

Please cite this article in press as: T. Fredian, et al., MDSplus autohttp://dx.doi.org/10.1016/j.fusengdes.2013.11.012

uting platforms and operating system versions is no easy task.o ensure that users of the software on each of these platforms canake advantage of bug fixes and new features in a timely fashion, the

DSplus developers constructed an automated build and release

∗ Corresponding author. Tel.: +1 617 253 7623.E-mail address: twf@psfc.mit.edu (T. Fredian).

920-3796/$ – see front matter © 2013 Elsevier B.V. All rights reserved.ttp://dx.doi.org/10.1016/j.fusengdes.2013.11.012

© 2013 Elsevier B.V. All rights reserved.

system. Each night the MDSplus software code management soft-ware is queried to determine if any changes have been made tothe software. If changes are found new releases of the MDSplussoftware are built for nearly a dozen different computer operat-ing systems. These new releases are made available to MDSplususers via the MDSplus web server via file downloads or packagerepositories making it possible to update MDSplus packages usingsome of the same tools used for updating operating system soft-ware. This paper will discuss how this build and distribution systemwas implemented.

2. Current computer platforms supported

MDSplus distribution kits are currently available for the follow-ing computing platforms:

• RedHat Enterprise Linux Versions 5 and 6, 32-bit and 64-bit(download and repository)

• RedHat Fedora Core Versions 15, 17 and 18, 32-bit and 64-bit(download and repository)

• MacOSX Lion 10.5 (has been known to work on 10.6 through 10.8)(download)

mated build and distribution system, Fusion Eng. Des. (2013),

• Ubuntu Versions 11 and 12, 32-bit and 64-bit (download andrepository)

• Microsoft Windows Windows 7, 32-bit and 64-bit (download)• Solaris Version 11 (download and repository)

ING Model

F

2 ering a

3

uIficowancdohpao

jm

4

aVs5aXbiCvpmnwt

5

htp

•

•

•

•

•

•

•

ARTICLEUSION-7116; No. of Pages 3

T. Fredian et al. / Fusion Engine

. Automated build control

To manage the nightly builds for the MDSplus software we aresing a software package called “Hudson Extensible Continuous

ntegration Server” [2]. This package provides the ability to con-gure build jobs to be dispatched to various server systems whichheck for MDSplus code updates and perform the software buildsn those servers. Hudson keeps track of the code updates foundhich triggered the builds, the log files associated with the builds

nd the completion status of the builds. The developers can beotified if any problems were detected during these builds via theontinuous integration server has many useful features such asefining dependencies between jobs so the jobs can be scheduled toccur at fixed intervals or after another job completes. Hudson alsoas the capability to start and stop virtual machines such as thoserovided by products such as VMware vSphere and Oracle Virtu-lbox. This enables the building of software for several differentperating systems using a single host computer.

Each nightly build of MDSplus is performed with over 30 buildobs spread over more than 10 different computers or virtual

achines.

. Hardware and virtual machine configuration

To perform MDSplus builds on multiple platforms we are using collection of servers in a virtualization environment utilizingMware vSphere 5.0 [3]. The current environment consists of twoervers, a free-standing Dell Precision Workstation 690 with a Xeon150 dual core processor with 18 GB of RAM. The second server is

rack-mounted Dell PowerEdge R410 server with dual quad coreeon X5672 processors and 32 GB of RAM. Two new servers areeing added to the configuration. This virtualization environment

s being used for many functions at the Plasma Science and Fusionenter at MIT in addition to the MDSplus build system. Each night,arious virtual machines are powered on to build MDSplus and thenowered back down after completion. Configuration of new virtualachines can be completed to add additional build platforms as

eeded with minimal effort. A backup product called Veeam [4]hich is optimized for backing up VMware virtual machines is used

o protect the contents of the virtual machines.

. Packaging and versions

For the Linux based distributions the MDSplus software packageas been split into functional pieces so you only need to installhe software modules that you wish to use. Currently the list ofackages include:

Camac – Libraries, device support and utilities used when doingdata acquisition with hardware based on the CAMAC data acqui-sition hardware standard.D3D – Scripts used at General Atomics in San Diego, CA for acces-sing data on the D3D fusion experiment.Devel – Object libraries and include files needed for develop-ing compiled libraries and utilities which call into the MDSplusinternal routines.Epics – Interface modules to link the Epics process control systemwith MDSplus.GSI – Libraries and configuration files to utilizing Globus SecurityInfrastructure in MDSplus client/server communications.HDF5 – Libraries and utilities for moving data between HDF5 and

Please cite this article in press as: T. Fredian, et al., MDSplus autohttp://dx.doi.org/10.1016/j.fusengdes.2013.11.012

MDSplus storage systems.IDL – Libraries and procedures to access MDSplus from theInteractive Data Language product distributed by Exelis VisualInformation Solutions.

PRESSnd Design xxx (2013) xxx– xxx

• JAVA – Libraries and utilities to enable access to MDSplus via Javabased applications.

• KBSIdevices – Device support modules developed and used bythe Korea Basic Science Institute.

• Kernel – MDSplus core libraries, utilities and configuration sett-ings used by all other packages.

• Labview – Libraries and modules providing an interface betweenthe National Instrument’s Labview product and MDSplus.

• Matlab – Libraries and modules providing an interface betweenMathworks’ Labview product.

• MITdevices – Device support modules developed and used atthe Massachusetts Institute of Technology’s Plasma Science andFusion Center.

• Motif – Libraries and applications based on the Motif X Windowsusers graphical interface.

• MsSql – Libraries to provide an interface between MDSplus andMicrosoft SQL Server databases.

• PHP – Libraries to access MDSplus data for producing web cgiscripts using the PHP cgi utility.

• Python – MDSplus module for the Python language.• RFXdevices – Device support modules developed and used at the

RFX experiment in Padova, Italy• MDSplus – Catch all package for installing the complete MDSplus

system.

When using the MDSplus software repositories, the newMDSplus packages contain module dependencies such that anyadditional software packages needed for a particular package willautomatically be installed along with the requested package.

The new MDSplus build and distribution system provides forthree distinct levels of releases: stable, beta and alpha. The stablereleases may incorporate bug fixes as soon as they are available butno new features are added until those features have been releasedin the beta and alpha versions and tested. Similarly the beta releasesmay incorporate bug fixes but no new features are added until theyare tested in the alpha releases. The alpha releases represent themost recent changes to any of the MDSplus code modules. Packagereleases are identified with the major classification of stable, beta oralpha as well as major, sub and minor release numbers. For instancea package might have the name: mdsplus-beta-kernel 5.0.0 i386indicating that it is a beta release of the kernel with version number5.0.0 for a 32-bit computer system.

New software build scripts written in Python were developed tobe used by the build system to automate the checking for code mod-ifications to either the stable, beta or alpha branches of the MDSpluscode version control system (currently CVS). If a change is detected,a new stable, beta, or alpha build is executed and tested and if suc-cessful a new release with the minor version number incrementedis automatically made available for download via the MDSplus website or through the installation repositories. On platforms wherethe MDSplus installation package has been broken into subpack-ages, only the affected subpackages will have their minor releaseversions incremented. Development scripts were also developed tosimplify the promotion of alpha releases to beta and beta to stableand incrementing the sub or major portions of the version num-bers. These operations are done manually by the developers whennew functionality has been added and tested.

6. Downloads and repositories

The MDSplus web site has been modified to provide access to

mated build and distribution system, Fusion Eng. Des. (2013),

all the installation kits being built by the automated build system.Users can download individual installation kits from the web site orinstall repository installation kits which will configure the systemto access MDSplus repositories directly. System administrators of

ING Model

F

ering a

stitMs

7

btbfflituarlu

so

ARTICLEUSION-7116; No. of Pages 3

T. Fredian et al. / Fusion Engine

ystems such as RedHat and Ubuntu are usually very familiar withhe use of repositories as this is how system updates or new packagenstalls are generally performed. Utilities such as yum (Redhat sys-ems) and apt-get (Ubuntu systems) can then install and/or update

DSplus packages using the same type of command that operatingystem installation and updates are performed.

. Conclusions

The new MDSplus automated build and distribution system haseen in service for nearly a full year now although some addi-ional platform support has been added more recently. The highlyuild feature reduces the workload on the developers and hope-ully provides the user community with quicker updates and neweature releases on all the platforms. There has been surprisinglyittle feedback from the user community on the increased availabil-ty of installation kits and updates. Hopefully this is an indicationhat things are working well and fulfilling the needs of MDSplussers. I suspect that many sites are still installing MDSplus pack-ges manually and do not routinely update their systems via theepositories. As users begin using these facilities perhaps we willearn more about how well the new system is functioning from a

Please cite this article in press as: T. Fredian, et al., MDSplus autohttp://dx.doi.org/10.1016/j.fusengdes.2013.11.012

ser point of view.There is always room for improvements. As each release is con-

tructed there is a set of regression tests done on the installationf the packages and the operation of several of the key functions

[[[

[

PRESSnd Design xxx (2013) xxx– xxx 3

provided by the MDSplus core system. More thorough test cov-erage would better guarantee the quality of the releases but asusual designing better test scenarios is often as difficult and timeconsuming as developing the functions that they are supposed totest.

Another area of improvement will likely come after we gainexperience in release management. Currently there is no formalprocedure for determining the appropriate time for major releaseupdates where the alpha and beta releases get moved on to thestable versions. This will be more critical as sites begin to performautomatic software updates on production systems.

The use of continuous integration technology in conjunctionwith virtual machine capabilities has greatly improved our abilityto produce MDSplus distribution kits for a wide variety of com-puting platforms. MDSplus bug fixes and new features can now bedistributed to the MDSplus community in a fraction of the time andwith much less developer effort than ever before.

The MDSplus Hudson setup can be viewed athttp://hudson.mdsplus.org.

References

mated build and distribution system, Fusion Eng. Des. (2013),

1] MDSplus, http://www.mdsplus.org2] Hudson Extensible Continuous Integration Server, http://hudson-ci.org3] VMware vSphere® , http://www.vmware.com/products/datacenter-

virtualization/vsphere/overview.html4] Veeam Modern Data Protection, http://www.veeam.com/