ELSEVIER

Nuclear Instruments and Methods in Physics Research A 389 (1997) 51-55 NUCLEAR

INSTRUMENTS &METHODS IN PMYSICS RESEARCH

seC!lOrl A

Problem-solving environment for parallel computers

Kurt Appert”, Ralf Gruberbq*, Silvio Merazzic, Trach Minh Tran”, Serge Wiithrichd

a CRPP-EPFL, CH-1015 Lausanne, Switzerland ’ SIC-EPFL, CH- IO1 5 Lausanne. Switzerland

’ SMR Corporation, CH--7500 Birnne. Sw,it-_erland ’ PATP-Cra.v Research Inc.. EPFL. CH-1015 Lausannr. S~vitxrland

Abstract A program environment is presented that enables a co-operative design of high-technology products demanding

interdisciplinary computer simulations for their optimisation. The system includes a multi-site, multi-machine, distri- buted data manager accessible by a data monitor. A client-server concept supports the user of massively parallel machines. The exchange of data between different executable scientific application modules is performed through the

data base. Special modules to define the geometry, to decompose the domain in subdomains, to construct adaptive meshes in each subdomain, and to represent graphically the results run on workstations. Direct and iterative matrix

solvers are embedded in the system. They optimally run on massively parallel machines. As a demonstration of the system. the high frequency wave generator (Gyrotron) application is presented.

1. Introduction

Computer simulation is used to optimise products

with respect to production costs, running costs, quality. security, weight, form, pollution, or social acceptance. Computer simulation helps to improve their quality level, supports their advertisement by video films and

reduces the time to market and the design costs. Many tries can be done without constructing the object. Thus, expensive and time-consuming wind tunnel experiments

and crash tests can be replaced by cheap and rapid computer experiments, the design of new drugs and materials can be accelerated and pollution reduced by a better understanding of the combustion processes. Total- ly new innovative solutions can virtually be realised that would not have been considered before because of the high design and production costs. Weather forecasting or a study of the climate evolution are too big systems, molecules too small systems and reactor accidents too dangerous systems to be experienced on. They have to be studied by computer simulations.

An engineer who likes to design a new product or improve an existing one by means of computer simula-

* Corresponding author.

tion is today in an uncomfortable situation. For his interdisciplinary problem he has to find programs in all the domains relevant to his problem, has to link them

together, has to understand the different user interfaces that are based on input files, keyword driven languages, or graphical user interfaces. Each program has its own data formats, its own visualisation package, and its own data archiving facility. Often the software is old and

inadequate to run on modern powerful computers. As a consequence, the resolution and the complexity of the chosen model have to be reduced to be tractable on less powerful scalar machines. Maintainability of such sys- tems is very difficult. Some industries are aware of this software integration problem. Solutions chosen to tackle this problem are often based on metafile data transfer concepts [l] and adapted to specific needs. This situation demands a change.

In this paper, a program execution and program deve- lopment system [Z] is presented. This environment con- sists of a skeleton into which application modules are embedded. It has been strongly influenced by older simi- lar environments such as B2000 [3] and ASTRID [4]. Such an integrated computer simulation system for a co- operative design offers to the industry a number of clear advantages. The multi-site, multi-machine, distributed data and network management system [9] combines the work of all co-workers. A single script based user

0168-9002/97/$17.00 Copyright c: 1997 Elsevier Science B.V. All rights reserved

PII SO1 68-9002(97)00039-9 IIb. PARALLEL COMPUTING

52 K. Appert et al. / Nucl. Ins@. and Meth. in Phys. Res. A 389 (1997) 51-55

interface with a graphical support guarantees a common usage of the system following one strategy. The intelligent software layer helps users in defining

data and interpreting results. Each application

module can be executed on the most adequate machine. The integration of relevant application packages into one program environment enables resolving interdisci- plinary problems difficult to treat with present approaches. The program environment hides the under-

lying computer platforms, supporting those scientists who do not like to care about computer architectures. As a demonstration example, the high frequency wave

generator, called Gyrotron [S], is presented in more detail.

Several other applications have already been embed- ded in the program environment: a Laser optimisation

program [6], magnetohydrodynamic equilibrium and stability computation [7], the B2000 unstructured finite element package [S] for structural analysis, and the electrostatic precipitator application [S] realised co- operatively between industry and academia.

2. Towards a program environment for engineering applications and research

An engineering program environment consists of a program execution environment that eases execution of the interdisciplinary, integrated computer simulation

package, and a program development environment that supports the extension, the modification and the opti- misation of the system.

2.1. Functions of the program execution environment

2.1.1. Script based and graphical user interface (GUI) The GUI is adapted to the needs of the specific engi-

neering community. Scripts and templates enable defin- ing the case, its execution, and the interpretation of the results. In many important industrial applications, the most time-consuming task is the definition of the geo- metry and the generation of the adaptive numerical

mesh. The system includes easily usable script based geometry and mesh generators, called MINIM and MESH (Fig. l), respectively.

2.1.2. Rule based system A rule based system helps the engineer in the handling

of the non-linear dependencies between modules. Such “intelligent” software layers should be developed in common between computer and computational scientists and offered to the engineers. The present environment includes a command language that can be used to guide execution.

Fig. 1. Program environment for the Gyrotron application. The modules MINIM (minimodeller), MESH (mesh generator), PO- TENTIAL (electric field), and HEAT (beam collector) originated from the ASTRID project [3], the modules FIELD (magnetic

field), PATH (particle pusher) and CHARGE (charge density

distribution) have been developed in a co-operative manner and

embedded later on. The POTENTIAL module, PATH and

CHARGE have been optimised for parallel computers, The

target machines are High Performance Computers (HPC),

Workstation clusters (WS cluster), Workstations (WS), Personal

computers (PC), Graphical workstations, or Massively Parallel

Machines (MPM).

2.1.3. Execution control system A user should not care about where a module should

best be executed. Well defined rules, easy to change must

enable executing any module on the most adequate com- puter platform. Distributed computing minimises not only running costs but also engineer’s time. At present,

the user has to define himself the platforms he likes to compute on.

2.1.4. Visualisation package The engineer can mix and visualise all the data pro-

duced by modules and stored in a data base. The present graphics package includes many functionalities de- manded by the engineer.

2.1.5. Data handling The data is archived and easily accessible and modify-

able at any given time. A data monitor enables access to the data from the executable program, through the command language to control the execution pathes and directly by the user during run time.

2.1.6. On-line documentation and information system The engineer must have access to an on-line documen-

tation system, tightly coupled to an information system that includes all the relevant informations helping the

K. Appert et al. /Nucl. Instr. and Meth. in Phjs. Res. A 3X9 (1997) 51-55 5?

engineer to interpret the data, to define new simulation

cases and compare with older cases. The present environ-

ment does not include such functionalities.

that pumps energy out of the beam. The high frequency

wave is injected into a fusion experiment where its energy

is absorbed by resonant electrons, being one of the

important heating mechanisms of such machines. 2.2 Functions of a program development environment

3.2. Numerical model The program execution environment is based on

a data base and data management system, well defined

data structures and the possibility to easily embed new

and change old modules.

2.2.1. Data base and data manager All data reside in the data base [9]. Data that move

from one module to another have to flow through the data base. Modules can be executed on different com-

puters. The data manager handles all these data flows in a distributed computer environment thus making the

system independent of the compute platform. The data base is adapted to scientific applications and runs very efficiently in a modern, distributed high performance computing environment.

2.2.2. Data structures Well defined data structures ease data transfer between

the modules. By means of the data monitor, they are

easily accessible by the outside world. Typically, block structured and unstructured data arising from local func- tion approaches, from data structures for global function approaches, and from particle simulations have been implemented.

2.2.3. Module integration Customer modules can be imported into the environ-

ment by writing an interface to the data standards. Embedding a new module enables an immediate inter- action with all the previously added ones.

22.4. Client-server concept A client--server concept [lo] eases access to data on

massively parallel computers. The present concept has optimally been implemented on a Cray T3D, on the Intel Paragon and on the IBM SP-2.

3. An example: Gyrotron application

3. I. Problem description

The Gyrotron is an electron gun in which an electron beam is accelerated by a strong electric field to a velocity of half the speed of light. The beam is travelling in a strong magnetic field. about half of the energy is rota- tional energy. The electron cyclotron frequency of around 100 gigahertz corresponds to electron cyclotron frequencies in fusion reactor experiments. In a cavity, the electron beam enters in resonance with a standing wave

We consider a 2D axisymmetric geometry as shown in Fig. 2. The overall domain is decomposed in subdomains

using the module MINIM (Fig. 1). In each subdomain a structured mesh (using MESH) is built according to a mesh density function that measures the deposited charge of the electron beam. This enables mesh accumu- lation around the beam path.

The Poisson equation

A4= -p

is solved applying a finite element approach. The elec-

tropotential 4 is restricted by the Dirichlet boundary conditions. The right-hand side p represents the charge density distribution deposited by all the particles in the beam. The relativistic particles are pushed in the electro- magnetic field according to

su 4 -=-E+%xB, 2t m m)

?.u II _=_

?t ;”

where u. x and q are the particle velocity, its position and charge, respectively, E and B are the electric and mag- netic fields, and JJ is the correction due to the relativistic speed.

At the right end of the Gyrotron (Fig. 3) is placed a copper collector that absorbs the beam energy. It is

strongly heated. Its temperature is computed in HEAT. Due to the enormous heating, the gyrotron has to be pulsed.

In Fig. 2 we show the geometry of the gun with particle traces, the domain decomposition and the mesh of the emitter, and the adaptive mesh in the beam tunnel. The

beam is first accelerated to half of speed of light by the electric field applied in the gun emitter zone. compressed by the strong magnetic field, looses energy in the resona- tor to the standing wave, and is finally dispersed before being absorbed at the collector.

3.3. Data carrier

The different types of applications that have to be interlinked are all embedded in the program environ- ment shown in Fig. 1. The data carrier part is put in grey. Data are transferred between the data base and the different hardware platforms by a network manager. Any new module can be added, existing ones easily replaced.

IIb. PARALLEL COMPUTING

54 K. Appert et al. /Nucl. Instr. and Meth. in Phys. Rex A 389 (1997) 51-55

\ Gun collector

Fig. 2. Gyrotron geometry (above) with the gun emitter where particles are accelerated, the superconducting magnetic field coils, the resonator where beam energy is transferred to the wave and the gun collector where the energy is deposited at the copper wall. Below. at the left, is shown a cut of the gun emitter with its decomposition in subdomains, each one being meshed with a structured grid. At the right-hand side is represented the adaptive mesh in the region of beam compression.

Interactions between modules are controlled by the Data Manager.

The keyword-driven and graphical user interface con- nects the users to the program environment in a standar- dised manner. This enables replacing a program by another one without changing the input drastically.

The software layer helps users in the definition of the input and the interpretation of the results. Such

rule-based interpreters should in future be a help to the users.

An information system should in future be added to the environment. It will garantee the information flow be- tween co-workers at different sites.

A control system should interpret the input parameters to execute a module on the most adequate computer platform. This enables hiding the underlying computer architectures and garantees efficiency in the execution of the programs.

Data residing on the Data Base MEMCOM [9] can be queried by means of the MONITOR. This can be

done by the user through the user interface or by the software layer to guide the execution. Conver- gence parameters can be read and interpreted and a different execution path taken. A Network Manager is needed in a multi-site and multi-machine environ- ment.

3.4. Parallelised modules

MINIM, MESH and the integrated graphical package VIEW typically run on a graphical workstation.

The modules POTENTIAL, FIELD, PATH and CHARGE are executed on a parallel computer using PVM [ 1 l] for communication purposes. The magnetic FIELD has to be computed once per mesh generation and kept unchanged during the time-evolution.

The Laplace operator for the POTENTIAL is un- changed during the time evolution, only the right hand side changes. The matrix has to be computed once per mesh. It can be computed in parallel on all subdomains

K. Appert et al. /Nucl. Instr. and Meth. in Phw. Rex A 3X9 (1997) 51.-55 55

without any communications and is stored in com-

pressed, not totally assembled form.

A direct solver has been written [ 123 based on a color-

ing technique. After construction of the assembled matrix, factorisation is performed once. Backsubsti- tution is repeated for each time step. Intensive use of LAPACK [13] routines is made to remain portable from one machine to another without loose of perfor-

mance. An iterative solver [14] based on a conjugate gradient

method with a Jacobi preconditioning has been added to tackle big cases. The only operations that need com- municating between subdomains are the sparse matrix*

vector multiply and the scalar product. Locally, we first perform the matrix*vector multiply using the unassem- bled. compressed matrices. The results at the borders have then to be exchanged and added, needing communi- cations only on vectors. This is done with a master/slave concept. For the scalar product, the local products are

computed, the slave contributions deduced, and a global sum performed.

In PATH [I 51 the particles are pushed in the isoparametric plane of the structured subdomain mesh. This eases finding neighboring cells and subdomains. Particles move from one subdomain to the next one. No load balancing is performed. Since the particles are injec- ted constantly, about half of the processors are in average

active for particle pushing. The CHARGE density distri- bution is summed up during particle move.

The gyrotron simulation program has been success- fully compared with the EGUN code [16] and runs a few orders of magnitudes faster.

4. Outlook

In future, increase in computing power is reached by increasing the number of processing elements all linked to a virtually shared main memory. To reach

high performance in the execution of simulation pro- grams, special care has to be taken in the implement- ation of modules to future generation parallel computers. Co-operation between computational and computer scientists will be necessary to develop in common such modern program development and execu- tion environments that can hide the underlying computer architecture and increase user friendliness. Such

‘Scientific Macintosh” type engineering computing environments are clearly needed to make computer experimentation available to scientists who are non- specialists in computer science.

Acknowledgements

We would like to acknowledge the very fruitful dis-

cussions we had with Peter Stehlin and Edgar Gerteisen.

References

Cl1

PI

[31

[41

ISI

161

171

PI

[91

Cl01

Cl11

[l?l

Cl31

Cl41

ll51

Cl61

J.M. Maloney, Concurrent Engineering in an Integrated

Computing Environment. Boeing, Internal Report.

R. Gruber, E. Gerteisen. G. Jost and S. Merazzi, Speedup

8 (1) (1995) 41.

S. Merazzi, B2000 Data Access and Data Description

Manual, SMR Corporation. Bienne. Switzerland.

S. Merazzi and R. Gruber. ASTRID Reference Manual.

SMR SA. Bienne (1994);

E. Bonomi. M. Fltick, R. Gruber, R. Herbin, S. Merazzi, T.

Richner, V. Schmid and C.T. Tran. in: Scientific Comput-

ing on Supercomputers II, eds. J.T. Devreese and P.E. van

Camp (Plenum Press, NY, 1990) pp. 51 83.

T.M. Tran, D.R. Whaley, S. Merazzi and R. Gruber. in:

Proc. 6th Joint EPS-APS Int. Conf. on Physics Computing

(Lugano, Switzerland, 1994) pp. 491 494.

C. Pfistner, R. Weber. H.P. Weber. S. Merazzi and

R. Gruber. IEEE J. Quantum Electron. 30 (1994) 1605.

D.V. Anderson. W.A. Cooper, R. Gruber. S. Merazzi and

U. Schwenn. in: Scientific Computing on Supercomputers

II, eds. J.T. Devreese and P.E. van Camp (Plenum Press,

New York. 1990) pp. 159 174.

W. Egli, 0. Riccius, U. Kogelschatz. R. Gruber and

S. Merazzi. in: Proc. 6th Joint EPS-APS Int. Conf. on

Physics Computing (Lugano. Switzerland. 1994) pp.

535. 542 and

E. Gerteisen, W. Egli and B. Eliasson. Helv. Phys. Acta 67

(1994) 769.

S. Merazzi, The MEMCOM User Manual (Version 6.3).

SMR Corporation, Bienne, Switzerland (1995).

S. Wiithrich, S. Merazzi and T.M. Tran. A client&server

monitor data management environment for parallel com-

putation, LRP 537,‘96 (CRP-EPFL. Lausanne. Switzer-

land) 1996.

G.A. Geist, A.L. Beguelin. J.J. Dongarra. R.J. Mancheck

and VS. Sundaram, PVM 3.0 User’s Guide and Reference

Manual, ORNLiTM-12187. 1993.

Tran T.M., R. Gruber, K. Appert and S. Wiithrich. Com-

put. Phys. Commun. 96 (1996) 118.

E. Anderson et al.. LAPACK user’s guide (SIAM. Phila-

delphia. 1992).

R. Gruber. E. Gerteisen. S. Merazzi, T.M. Tran and

S. Wiithrich, in: Proc. Conf. in Physics Computing Pc’96.

Cracow, 1996, pp. 572- 579.

C.K. Birdsall and A.B. Langdon, Plasma Physics via Com-

puter Simulation (McGraw-Hill. New York. 1985).

W.B. Herrmannsfeldt. Electron trajectory program, SLAC Rep. 226. Standford Univ., 1979.

IIb. PARALLEL COMPUTING