A

NdocM

scc©

K

1

tlccasatTfifr

dc

a

0d

Available online at www.sciencedirect.com

Fusion Engineering and Design 83 (2008) 520–524

Real-time communication for distributed plasma control systems�

A. Luchetta ∗, A. Barbalace, G. Manduchi, A. Soppelsa, C. TaliercioConsorzio RFX, Associazione Euratom-ENEA sulla Fusione, Corso Stati Uniti 4, Padova 35127, Italy

bstract

Real-time control applications will benefit in the near future from the enhanced performance provided by multi-core processor architectures.evertheless real-time communication will continue to be critical in distributed plasma control systems where the plant under control typically isistributed over a wide area. At RFX-mod real-time communication is crucial for hard real-time plasma control, due to the distributed architecturef the system, which consists of several VMEbus stations. The system runs under VxWorks and uses Gigabit Ethernet for sub-millisecond real-timeommunication. To optimize communication in the system, a set of detailed measurements has been carried out on the target platforms (MotorolaVME5100 and MVME5500) using either the VxWorks User Datagram Protocol (UDP) stack or raw communication based on the data link layer.

Measurements have been carried out also under Linux, using its UDP stack or, in alternative, RTnet, an open source hard real-time network protocol

tack. RTnet runs under Xenomai or RTAI, two popular real-time extensions based on the Linux kernel. The paper reports on the measurementsarried out and compares the results, showing that the performance obtained by using open source code is suitable for sub-millisecond real-timeommunication in plasma control.

2008 Published by Elsevier B.V.

inux

cmoa

tilfibonett

eywords: Real-time control; Real-time communication; Ethernet; VxWorks; L

. Introduction

In recent years, a large number of real-time control applica-ions have been developed in fusion experiments with decreasingatency time down to sub-millisecond range and growingomplexity. Control algorithms have become more and moreomplicated, using in some cases advanced techniques, suchs state-space representation and multiple-input multiple-outputchemes. The number of input/output signals of the controlpplications has grown significantly in time and the implemen-ation of distributed control systems has become the standard.o fulfil the stringent real-time control requirements in complexusion applications, the engineers’ efforts have been focusedn two directions, first to increase the system processing per-ormance and secondly to reduce data transmission latency ineal-time communication systems.

Complex, hard real-time applications have been generallyeveloped in fusion experiments by using general purpose pro-essors running real-time operating systems (OS) [1–5]. In some

� Presented at Sixth IAEA Technical Meeting on Control, Data Acquisition,nd Remote Participation for Fusion Research, 4–8 June 2007, Inuyama, Japan.∗ Corresponding author. Tel.: +39 049 829 5043; fax: +39 049 8700718.

E-mail address: adriano.luchetta@igi.cnr.it (A. Luchetta).

aeuntrt

R

920-3796/$ – see front matter © 2008 Published by Elsevier B.V.oi:10.1016/j.fusengdes.2007.12.026

; RTAI; Xenomai; RTnet

ases, real-time applications have also been implemented byeans of non-real-time OS, using Linux configured to minimize

r avoid non-deterministic activities, such as interrupt servicesnd paging [6,7].

The current trend of processors toward multi-core architec-ure [8,9], though intended to enhance performance withoutncreasing clock frequency and power dissipation, will, mostikely, help keeping low latencies in real-time applications. Theact that by multi-core technology processes can be executedn parallel on different cores sharing the same data cache, cane used in real-time to enhance task scheduling by reservingne or more cores to critical real-time tasks and letting theon-real-time processes run on a dedicated core. Despite thenhancement that multi-core technology will, most likely, bringo real-time application, communication remains a critical func-ion for real-time applications in fusion, as in this domain thepplications are typically distributed. Many diagnostics are gen-rally required to produce data to be transmitted in real-time andsed as input signals in real-time control algorithms. Commu-ication among nodes in distributed real-time control system is,o our experience, the most time-consuming function in the cur-

ent state-of-the-art technology and is responsible for most ofhe overall latency in distributed control systems.

This paper describes in Section 2 the experience gained inFX with real-time communication and reports, in Section 3,

ering

ou(4oRRt

2

u1[tttrcEJTcsnsfiwfpwSuilnwot[(cp

macc

mppmlMq

piatwt

dcatoEGis

3s

careawwnbwas responsible for the performance loss. Traces (a), (b) and (c)in Fig. 1 plot the transmission times achieved using the VxWorks5.5.1 native IP stack on either MVME5100 (100 Mbit/s) orMVME5500 (100 Mbit/s and 1 Gbit/s). The measurements have

A. Luchetta et al. / Fusion Engine

n a set of measurements on real-time communication executednder the VxWorks OS [10] on two Single Board ComputersSBC) that are very common in real-time applications. In Sectionmeasurements are reported, carried out on the same platformsf Section 3, but using the IP stack of Linux or, in alternative,Tnet [11,12], an open source hard real-time network protocol.Tnet runs over RTAI [13,14] or Xenomai [15], two popular real-

ime extensions for Linux. Section 5 presents some conclusions.

. The RFX experience with real-time communication

The first implementation of real-time communication at RFXsed proprietary reflective memories and was introduced in997. This technology is still used in some fusion experiments2,16] mainly because its use is quite simple. In our case, ashe reflective memory was implemented as a VMEbus board,he achievable data transmission latency was affected by theime needed to access the memory board through VMEbusead/write cycles. In 2001 we decided to enhance the real-timeontrol system, introducing real-time data exchange based onthernet with User Datagram Protocol (UDP). At that timeET had already experienced with success the Asynchronousransfer Mode technology (ATM) [17]. The reason why wehose to use plain Ethernet, instead of adopting ATM, was totay in the mainstream of the network technology, where tech-ological evolution is more rapid and upgrade is, generally,moother. In addition, Ethernet was moving toward applicationelds where determinism is required [21]. Cost and simplicityere also advantages considered for the technology choice. In

act this solution requires no additional hardware, as Ethernetorts are default components in any computer system, and net-ork components, such as switches, are common and cheap.ince UDP is an unreliable protocol, some precautions weresed in the implementation of the communication layer, includ-ng: (1) a timestamp was added at application level to detectost and mis-ordered packets, (2) the layout of the real-timeetwork was kept as simple as possible (one-switch-only net-ork) and (3) extensive data flow measurements were carriedut on the selected hardware architecture to characterize point-o-point transmission latency and jitter as function of packet size18]. The hardware architecture was the Single Board ComputerSBC) Motorola MVME5100 equipped with the MPC7410 pro-essor, 500 MHz clock frequency, two 10/100 Mbit/s Ethernetorts—running the VxWorks v. 5.4.2 operating system.

Using this technology it was possible to deploy the RFX-od MHD mode control system, that can process 192 + 192

nalogue input signals, producing 192 references to drive theontrol power amplifiers with a latency for the overall controlycle of less than 340 �s.

The requests for more complex control algorithms, whileaintaining the latency constraints of the real-time control,

ushed the upgrade of the hardware platform to achieve fasterrocessing and lower data transmission latency in real-time com-

unication. In fact, more than 150 �s out of the total 340 �s

atency were needed for real-time communication. The SBCotorola MVME5500 – MPC7455 processor, 1 GHz clock fre-

uency, one GBit/s Ethernet port, one 10/100 Mbit/s Ethernet

FaT

and Design 83 (2008) 520–524 521

ort – was the natural hardware upgrade, since its processors based on the same Single Instruction Multiple Data parallelrchitecture as the MVME5100 (AltiVec, 128 bit vector registersechnology). The porting of software was, in fact, straightfor-ard. Unfortunately, the VxWorks version had to be upgraded

o 5.5.1, as previous versions are not supported on MVME5500.Apart from the processors, there are some other hardware

ifferences between the two platforms, including the systemontrollers and PCI host bridges (Hawk ASIC for MVME5100nd Marvel GT-64260B for MVME5500) and the Ethernet con-rollers (Intel 82559ER fast Ethernet controller for MVME5100,ne Intel 82544EI Gigabit Ethernet controller plus one fastthernet controller integrated in system controller MarvellT64260B for MVME5500). These differences have an impact,

n some cases, on system performance, as shown in the nextection.

. Data communication under the VxWorks operatingystem

We were expecting a significant improvement in networkommunication speed, due to the fact that the new board hostsGigabit Ethernet port and has 1 GHz clock frequency. The

esults of the first tests were disappointing and we experiencedven worse performance for small size packets (<1 kbyte) thatre those of interest for our real-time applications. Further testsere required to determine whether the performance problemas caused by the operating system IP stack or was related to theetwork port driver. It turned out by measurements on local loop-ack data transmission that the IP stack management software

ig. 1. Comparison among data transmission performance in SBCs MVME5100nd MVME5500 using VxWorks 5.5.1, Linux 2.6, RTAI, Xenomai and RTnet.he average transmission times are plotted vs. data size.

5 ering and Design 83 (2008) 520–524

bMtctsamiaetpeonEMtaact

tMpbbMap

FM

Fi5

twhO6I

i

22 A. Luchetta et al. / Fusion Engine

een executed by using two pairs of equal SBCs (MVME5100-VME5100 and MVME5500-MVME5500) connected directly

hrough cross-cables on their own Ethernet ports. One packetontaining a timestamp identifier is sent from the sender tohe receiver board that, on reception, immediately returns theame packet to the sender. The measurements report the aver-ge transmission times obtained by dividing by two the timeeasured to transmit and receive a data packet with increas-

ng size of transmitted data. The data size refers to sheer datand does not include the UDP/IP and Ethernet protocol head-rs. In the case of the MVME5500 data are reported for theransmission on both 10/100 Mbit/s and Gigabit/s ports. Thelots show surprisingly that 100 Mbit/s transmission is morefficient on the MVME5100, despite the lower clock frequencyf the processor. This is to be attributed to the different inter-al organization of the two SBCs. Transmission using Gigabitthernet on MVME5500 is slightly faster than transmission onVME5100, but it is not satisfactory at all. The ratio between

he times achieved in transmission with Gigabit on MVME5500nd 100 Mbit/s on MVME5100 decreases with the packet size,s expected, but the enhancement is less than 30% in the rangeonsidered, despite the 10-fold increase of the physical mediumhroughput and the two-fold increase of clock frequency.

Traces (a), (b) and (c) in Fig. 2 show the overall datahroughput for MVME5100 and MVME5500. The values for

VME5500 are very far from the medium throughput. Theoor performance of the IP stack in VxWorks 5.5.1 is proba-ly due to the lack of optimization in the stack code. This cane clearly seen in Fig. 3 comparing UDP transmission on the

VME5100 under two different versions of VxWorks (5.4.2

nd 5.5.1). The plot shows a fixed delay (60–70 ms) superim-osed to the measurements under version 5.5.1 with respect to

ig. 2. Comparison among measured data throughput in SBCs MVME5100 andVME5500 using VxWorks, Linux, RTAI, Xenomai, RTnet.

(atl(wofituam

4

tBpim

erssl

ig. 3. Comparison between measured transmission times in MVME5100 show-ng the degradation of the IP stack performance from VxWorks 5.4.2 to VxWorks.5.1.

hose ones achieved under version 5.4.2. The supplier of the OSas notified with the results of the tests, but no improvementsave been introduced in the IP stack of VxWorks5.5.1 so far.ptimization is claimed to have been implemented in VxWorks.0, a fact that we cannot confirm as we decided not to test theP stack under that version.

To overcome the limitation of the VxWorks IP stack wemplemented the Ethernet communication at data link layerDLL) level. Traces (d), (e) and (f) in Figs. 1 and 2 show theverage transmission times and throughput in three types ofransmission on MVME5500 under VxWorks 5.5.1: data linkayer with and without socket Application Program InterfaceAPI) and UDP. In this case the Wind River Enhanced Net-ork Driver was compiled by the GNU C compiler using theptimization options “-O3-fno-volatile” that enhance the per-ormance [19]. In fact the measurements for UDP transmissionn trace (d) of Figs. 1 and 2 are better, though still not satisfac-ory, than those shown in trace (a) that was achieved withoutsing this optimization. In any case, even in the best case thechieved throughput is very far from the limit of the physicaledium (about six times lower).

. Data communication under Linux

POSIX-based, real-time Operating Systems are available dis-ributed under GNU General Public License and providing theoard Support Package for the MVME5500 single board com-uter [20]. Due to lack of resources, we did not investigaten this promising direction but preferred to accomplish some

easurements on Linux itself.In recent years, the growing penetration of Linux has gen-

rated much interest in the possibility of adapting Linux for

eal-time applications. Though Linux is the standard operatingystem in scientific experiments for all those tasks requiring notrict timing determinism, there are some aspects of Linux thatimit its usage in a real-time environment. The recent Linux ker-

ering

nmpptaCgrtanfdo

LVvrToccc

rnfdmfssotaRbEprct

wwoaditiArum

ca

ioaecjosntitnpnsatLXbutoXti

Etnet. Such discipline is optionally provided by RTnet and aims atforcing determinism in network access, avoiding possible col-lisions. Using a TDMA cycle time of 100 �s and a time slotof 40 �s for the data packet, we obtained an overall latency of

A. Luchetta et al. / Fusion Engine

el 2.6 provides solutions to overcome some of the problemsentioned above. It allows, in fact, the association of a fixed

riority with a subset of processes and the kernel has been madere-emptive by accurately defining un-interruptible segments inhe kernel code and protecting them with spin locks. Moreover,new O(1) implementation of the scheduler has been provided.onsidering also that swapping can be disabled and that, foriven level ranges, priorities can be made fixed, Linux can cur-ently be considered a soft real-time operating system and can,herefore, be used effectively in many applications which toler-te occasional delays in system response. However Linux 2.6 isot yet suitable for hard real-time applications, as required ineedback control for fusion devices. In this case, in fact, unpre-ictable response time may deteriorate the quality of the controlr, worse, lead to unrecoverable instabilities.

We tested the Ethernet communication performance underinux kernel 2.6 on the MVME5500 as previously done underxWorks. The communication latency and the data throughputersus packet size are shown in traces (g) of Figs. 1 and 2,espectively, that document the efficiency of the Linux IP stack.he performance is, in fact, the best one among those basedn the UDP stack and is overcome only by the data link layerommunication without socket API under VxWorks. This resultonfirms that the use of Linux as a soft real-time OS is a goodhoice also with reference to the real-time communication.

As mentioned before, Linux 2.6 can be considered a softeal-time system. To achieve a hard real-time behaviour weeed a way of adding to Linux the possibility of defining aew real-time tasks that are ensured to get control within aeterministic time as soon as they are ready to run. Strict deter-inism can be added to Linux by some real-time extensions;

or this we considered RTAI and Xenomai that are both openource. They share most concepts (they originated from theame project) and both represent, rather than a replacementf Linux, an additional component that works in conjunc-ion with Linux, handling the scheduling of real-time tasksnd letting Linux provide all the remaining functionality. BothTAI and Xenomai have an active developer community andoth may represent an open-source alternative to VxWorks.xtensive work has been carried out in RFX to compare theerformance of VxWorks, Linux, RTAI and Xenomai in a hardeal-time application very close to the typical RFX control appli-ations [22]. In this paper we focus on real-time communica-ion.

To enhance the deterministic response in communication,e considered RTnet, an open-source project for real-time net-ork communication. RTnet provides a new implementationf the IP stack up to UDP, where causes of non-determinismre accurately avoided. In particular: (1) memory allocation forata packets is handled by pre-allocating all the required buffersn advance in order to avoid dynamic memory allocation; (2)he dynamic protocol for address resolution has been convertedn a static procedure and the routing table contains also the

RP cache, i.e. the destination MAC addresses (3) the output

outing tables are optimized for the limited number of entriessed in RTnet. RTnet is available for both RTAI and Xeno-ai. The use of RTnet is intended to minimize the jitter in data

FM2p

and Design 83 (2008) 520–524 523

ommunication, avoiding, as far as possible, non-deterministicctivities.

We executed some measurements to get a figure of the jittern communication latency in the OS considered. The test wasrganized as follows: 64 input analogue channels are acquirednd one of the channels, connected to a waveform generator, ischoed on one output analogue channel. The input and outputhannels are acquired on a digital oscilloscope and the delay anditter are calculated. We executed the tests initially using onlyne single SBC. In this case the delay among the input/outputignals is pure system latency to acquire and echo out the sig-als. After this, we repeated the test using two SBCs connectedhrough the Ethernet ports, one SBC being in charge of acquir-ng the inputs and sending a data packet containing all input datao the second SBC that, on data reception, echoes out the chan-el. The delay in this case includes, beside the system latencyreviously computed, also the data communication latency. Theetwork communication tests have been executed using a fixedize for the UDP data packet (256 bytes corresponding to 64nalogue channels). The measured transmission times and jit-ers are displayed in Fig. 4, where we plot the performance ofinux and VxWorks, using their native IP stack, and RTAI andenomai, both using RTnet. The values of latency are obtainedy difference on the latencies in the two tests, whereas the val-es of jitter are computed by adding the jitter values in thewo tests. It is worth noting once again the good performancef RTnet that shows little latency and jitter on both RTAI andenomai. We observe a slight difference between the two real-

ime extensions probably due to the different execution paths innterrupts.

In the above tests we enabled no access control disciplines forthernet in RTnet. We repeated the test with RTnet by enabling

he Time Division Multiple Access (TDMA) discipline on Ether-

ig. 4. Comparison among measured transmission time and jitter in SBCsVME5500 using VxWorks, Linux, RTAI, Xenomai. Data size is fixed to

56 byte. Linux 2.6 shows a limited jitter. RTAI and Xenomai show an excellenterformance for both latency and jitter.

5 ering

ars

5

ptassrhatIsw

f

-

-

tnatf

A

t

s

R

[[[

[[

[[

[

[

[

[[

24 A. Luchetta et al. / Fusion Engine

pprox. 150 �s with a jitter around 50 �s. It is not possible toeduce the TDMA cycle time (and therefore latency and jitter)ince this causes an unacceptable CPU load.

. Conclusion

The aim of the reported work was the characterization of theerformance of real-time communication over Ethernet usinghe MVME5100 and MVME5500 platforms under VxWorksnd Linux. The measures achieved show that the VxWorks IPtack suite is not optimized for low latency data transfer of smallize packets (1 kbyte) that are those of interest in most fusioneal-time distributed control systems. Ethernet communicationas been then tested under other OS, in particular Linux, RTAInd Xenomai that have shown very good figures for Ethernet dataransfer latency and jitter. Linux has been tested using its nativeP stack, whereas in RTAI and Xenomai the IP stack has beenubstituted with RTnet. Both RTAI and Xenomai complementedith RTnet exploit very efficient network communication.Based on the reported performance measures, we observe the

ollowing facts:

The network performance represents a strong point in favourof a possible migration towards RTAI or Xenomai. UDP hasbeen successfully used for real-time network communication,and RTnet proved to be a very performing solution, especiallycompared with the poor performance we experienced withthe latest version of the VxWorks IP stack for the consideredboard.The best performance in RTnet is achieved without enablingthe TDMA access discipline, which appears to be best suitedto systems with a large number of nodes (and thereforehigher probability of access conflicts) but less stringent timingrequirements. This is not the case of the RFX-mod experimentwhich involves less than 10 synchronous control units. Pos-sible future work will be focused on multi-partner, real-timecommunication.

Based on [21] and the results presented in this paper, we canherefore state that both RTAI and Xenomai represent valid alter-atives to VxWorks in the real-time control system of RFX-mod,nd, more generally, for fusion devices. We can also state thathe performance obtained by using open source code is suitableor sub-millisecond real-time communication in plasma control.

cknowledgements

This work was supported by the Euratom Communities underhe contract of Association between EURATOM/ENEA.

The authors wish to thank Paolo Barbato for his preciousupport in the work.

[

and Design 83 (2008) 520–524

eferences

[1] R. Felton, E. Joffrin, A. Murari, L. Zabeo, F. Sartori, F. Piccolo, J. Farthing,T. Budd, S. Dorling, P. McCullen, et al., Real-time measurement and controlat JET experiment control, Fusion Eng. Des. 74 (2005) 561–566.

[2] K. Kurihara, I. Yonekawa, Y. Kawamata, M. Sueoka, H. Hosoyama, S.Sakata, T. Ohshima, M. Sato, K. Kiyono, T. Ozeki, Status and prospect ofJT-60 plasma control and diagnostic data processing systems for advancedoperation scenarios, Fusion Eng. Des. 81 (2006) 1729–1734.

[3] V. Vitale, C. Centioli, F. Iannone, G. Mazza, M. Panella, L. Pangione, S.Podda, L. Zaccarian, Real-time Linux operating system for plasma controlon FTU—implementation advantages and first experimental results, FusionEng. Des. 71 (2004) 71–76.

[4] A. Luchetta, G. Manduchi, General architecture, implementation and per-formance of the digital feedback control in RFX, IEEE Trans. Nucl. Sci.47 (2000) 186–191.

[5] W. Treutterer, K. Behler, R. Cole, J. Hobirk, M. Jakobi, A. Lohs, K.Luddecke, G. Neu, G. Raupp, W. Suttrop, et al., The new ASDEX upgradereal-time control and data acquisition system, Fusion Eng. Des. 66–68(2003) 755–760.

[6] J.A. Stillerman, M. Ferrara, T.W. Fredian, S.M. Wolfe, Digital real-timeplasma control system for Alcator C-Mod, Fusion Eng. Des. 81 (2006)1905–1910.

[7] B.G. Penaflor, J.R. Ferron, M.L. Walker, D.A. Piglowski, R.D. Johnson,Real-time control of DIII–D plasma discharges using a Linux alpha com-puting cluster, Fusion Eng. Des. 56–57 (2001) 739–742.

[8] O. Wechsler, Inside INTEL® Core Trade Mark microarchitecture,white paper, 2006, available at http://download.intel.com/technology/architecture/new architecture 06.pdf.

[9] D. Hamilton, Multicore microprocessor white paper, 2005, available athttp://www.freescale.com/files/32bit/doc/white paper/multicoreWP.pdf.

10] Wind River Home page [Online], http://www.windriver.com.11] RTnet Home Page [Online], http://www.rtnet.org.12] J. Kiszka, B. Wagner, Y. Zhang, J.F. Broenink, RTnet—a flexible hard real-

time networking framework, in: Proceedings of the 10th IEEE InternationalConference on Emerging Technologies and Factory Automation, Catania,Italy, 2005.

13] RTAI Home page [Online], http://www.rtai.org.14] P. Cloutier, P. Mantegazza, S. Papacharalambous, I. Soanes, S. Hughes, K.

Yaghmour, DIAPM-RTAI position paper, November 2000, RTSS 2000 -Real Time Operating System Workshop, 2000.

15] Xenomai Home Page [Online], http://www.xenomai.org.16] D. Moulin, B. Couturier, C. Grisolia, G. Martin, F. Saint Laurent, The real

time plasma control system of Tore Supra, Fusion Eng. Des. 43 (1999)371–376.

17] R. Felton, K. Blackler, S. Dorling, O. Hemming, P. Knight, M. Lennholm, F.Milani, F. Sartori, Real-time plasma control at JET using an ATM Network,in: Proceedings of the 11th IEEE-NPSS Real Time Conference, Santa FeNM, USA, 1999.

18] M. Cavinato, A. Luchetta, G. Manduchi, G. Marchiori, C. Taliercio, Firstoperation of RFX-mod real-time control system, Fusion Eng. Des. 81(2006) 1765–1770.

19] D. Krejsa, Improving the performance of Wind River Network StackDrivers and Applicatons, white paper, 2006, available from Wind River.

20] RTEMS Home Page [Online], http://www.rtems.com.21] A. Barbalace, A. Luchetta, G. Manduchi, M. Moro, A. Soppelsa, C. Tal-

iercio, Performance comparison of VxWorks, Linux, RTAI and Xenomaiin a hard real-time application, IEEE-Trans. Nucl. Sc., submitted for pub-lication.

22] S. Vitturi, On the use of ethernet at low level of factory communicationsystems, Comp. Stand Inter. 23 (4) (2001) 267–277.