Parallel Computational Steering and Analysis for HPC Applications using a ParaView Interface and the HDF5 DSM Virtual File Driver J. Biddiscombe, J. Soumagne,

1Parallel Computational Steering and Analysis for HPC Applications using a ParaView Interface

Parallel Computational Steering and Analysis for HPC Applications using a ParaView Interface and the HDF5 DSM Virtual File DriverJ. Biddiscombe, J. Soumagne, G. Oger, D. Guibert, J.G. Piccinali

Monday, 11 April 2011


Project/Funding


NextMuSE (http://nextmuse.cscs.ch)

Next generation Multi-mechanics Simulation Environment

European Community’s Seventh Framework Programme (FP7/2007-2013)

Information and Communication Technologies (ICT)

Small or medium-scale focused research project - Specific Targeted Research Project (STREP)

Future and Emerging Technologies Open Scheme (FET-Open)

http://nextmuse.cscs.ch/


Project : Naval Partners (branching out)



leafletshinges

closedopen

25 mm

Project : Mechanical heart valves

Reynolds number 6000

Design static since early 1980s



Project : TurboMachinery Partners



Requirements

In-Situ Visualization Computational Steering Two way exchange of potentially large data

(simulation has no meshing capability)

Multiple codes/languages Treat Cluster/Supercomputer like workstation extension

Constraints Partners who are far away ... and who can’t program their way out of a paper bag

(because they’re physicists really)



Ideal Solution

Sim Node

Sim Node

Sim Node

Sim Node

Vis Code

Vis Code

Vis Code

Vis Code

Sim Node

Sim Node

M Nodes

Vis Code

Vis Code

Visit/ParaView

Use this approach for in-situ visualization

Memory pressures

(especially for time-dependent data)

If Vis/Analysis code doesn’t scale as well as

simulation code, possible problem

Render Server

Render Server

Render Server

Render Server

Data Server



Parallel IO (c.f. Lustre)

Sim Node

Sim Node

Sim Node

Sim Node

OSS Node

OSS Node

OSS Node

OSS Node

Sim Node

Sim Node

M Nodes N OSS Nodes

All IO goes through OSS nodes

HDF5 Model already

looks like this



DSM Implementation

Sim Node

Sim Node

Sim Node

Sim Node

Vis Node

Vis Node

Vis Node

Vis Node

DSM

DSM

DSM

DSM

Sim Node

Sim Node

M Nodes N Nodes

DSM

DSM

HDF5/MPI

HDF IO intercepted and routed to DSM

HDF5/MPI

DSM Model could

look like this

Just replace the IO

layer in HDF5

IO Forwarding

c.f. ADIOS etc



Analysis/Vis

Machine

Simulation

Machine

(Infiniband)

Switch

Simulation

Nodes

Analysis/Vis

Nodes

DS

M

DS

M

DSM may be on

either side

Render

Workstation

GU

I

GUI may be

separate

Workstation

DS

M

Simulation

Machine

DS

M

DS

M

GU

I

Render

Render

GU

I

GU

I

Render

Analysis/Vis

Workstation

GU

I

Render

3

2

2


Configurations


H5FDdsm Bandwidth

J. Soumagne, J. Biddiscombe, “Computational Steering and Parallel Online Monitoring Using RMA through the HDF5 DSM Virtual File Driver”, International Conference on Computational Science, ICCS 2011

J. Soumagne, J. Biddiscombe, J. Clarke, “An HDF5 MPI virtual file driver for parallel in-situ post-processing”, EuroMPI 2010, Sep 2010. vol. 6305 of Lecture Notes in Computer Science. Springer Berlin/Heidelberg, 2010, pp. 62–71

Talk to Jerome for further info on numbersMonday, 11 April 2011


Our driver: H5FDdsm Data sent into HDF5 is then automatically redirected to this driver Currently uses flat memory model (may be improved by scattering)

more links = more BW

HDF5 – Virtual File Drivers


…start

end of file (eof)

local length 𝟐× ( local length )(𝑵 −𝟏 )×(local length )𝟎 𝑵 ×(local length )

eof + metadata =total length of DSMNetwork

network layer = plugin

TCP / MPI / MPI_RMA

(Gemini/DMAPP)


Performance – Write test with a 20GB DSM distributed among 8 post-processing nodes on a Cray XT5


16 32 64 128 256 5120

2000

4000

6000

8000

10000

12000

Parallel File System (Lustre)H5FDdsm (Socket over SeaStar2+ Interconnect)

Number of PE used for writing to the DSM

Dat

a w

rite

rate

(GB/

s)

Saturation of the network


Performance – Steering overhead

16 32 640

2000

4000

6000

8000

10000

12000

Write test with a 20GB DSM distributed among 8 post-processing nodes on a Regular cluster

Parallel File System (GPFS)H5FDdsm (Socket over Infiniband QDR 4x)H5FDdsm (MVAPICH2 over Infiniband QDR 4x)


Data

writ

e ra

te (G

B/s)


Series10

100

200

300

400

500

600

700

800

900

Steering overhead using a unique time step of an arbitrary simulation (here GADGET-2)

File System H5FDdsm (No Steering)H5FDdsm (With Steering)

Tim

e (m

s)


Current developments

Scalability on large systems Dynamic MPI communication not

supported Use MPMD job instead of two

separate SPMD jobs

Additional communicator designed for Gemini systems RMA here does not perform so well

DMAPP communicator (reachable bandwidth of 5GB/s between nodes)

16 64128

256512

10241536

20483072

0

5,000

10,000

15,000

20,000

25,000

30,000

35,000

H5FDdsm write test of 96GB to 176DSM PEs (44 nodes) using MPI RMA

on Cray XE6 (palu)


Da

ta w

rite

ra

te (

MB

/s)



In-Situ Visualization – How to use the driver

Create a file access property list

Instead of using the MPI-IO driver

Insert our DSM driver

Then carry on as usual

Comment out any MPIO_COLLECTIVE dataset transfer properties


H5Pset_fapl_dsm(fapl_id, MPI_COMM_WORLD, NULL);

H5Pset_fapl_mpio(fapl_id, MPI_COMM_WORLD, MPI_INFO_NULL);

file_id = H5Fcreate(filename, H5F_ACC_TRUNC, H5P_DEFAULT, acc_plist_id);H5Pclose(acc_plist_id);H5Fclose(file_id);

fapl_id = H5Pcreate(H5P_FILE_ACCESS);


In-Situ Visualization – Create Template


Built on top of Xdmf model and format

Generate Xdmf files to drive Xdmf reader in plugin

Uses in-memory H5dump to get the missing (metadata) information

Much simpler to write Allows variable number of

elements

<Grid> <Topology TopologyType="Polyvertex"> </Topology> <Geometry GeometryType="XYZ"> <DataItem>/fluid/position</DataItem> </Geometry> <Attribute> <DataItem>/fluid/density</DataItem> </Attribute> <Attribute> <DataItem>/fluid/velocity</DataItem> </Attribute> <Attribute> <DataItem>/fluid/vmob</DataItem> </Attribute> <Attribute> <DataItem>/fluid/w</DataItem> </Attribute></Grid>


In-Situ Visualization – Turbine



Selectable Arrays

ParaView doesn’t have ‘contracts’, so from the template, we can generate on/off switches for each exported dataset/array

Simulation can check if an array is ‘enabled’ and not send it. (better than turning it off in the XdmfReader)

We don’t use HDF5 for this, just a metadata flag maintained by the H5FDdsm library.


H5FD_dsm_steering_is_enabled(const char *name); // /grid/group/name


Note on Steering Architecture


ParaView

Servers

…

Set Steering Commands and Disabled Objects in

metadata section

Read simulation data – write user defined steering commands and data into HDF5 “file” section

Parallel

Simulation

Write output data – Read user defined steering commands and data

Get Steering Commands and Disabled Objects


Computational Steering

Each simulation requires custom controls

And different analysis requirements


Vis + GUI +

AnalysisHeart

Valve

Sloshing

Ship

Turbine

Deforming

Hull


Automatic GUI Generation : Interactions XML

<DataExportProperty

name="ModifiedBodyNodes"

command="SetSteeringArray"

label="Modified Body Node Data">

<DataExportDomain name="data_export"

full_path="/Mesh_DataSet"

geometry_path="/Mesh_Nodes#1/NewXYZ"

topology_path="/Mesh_Nodes#1/NewCo..."

command_property="ReloadFreeBodyMesh">

</DataExportDomain>

</DataExportProperty>


Reuse ServerManager syntax from ParaView

Added some extra Properties

Can also link GUI 3D widgets using ‘hints’


No ‘app-specific’ knowledge required

Generate ServerManager XML from template

parse hints/commands and other stuffRegister it internally on the flyNo need to (re)compile paraview plugin

Unless you have some custom heart valve adjustment widget that is specially designed for that app

Engineer only modifies simulation ‘Names’ and XML template



H5FDdsm Steering API

Initialize the steering extension (buffer is optional)

Get steering commands and disabled objects

Test if “name” is a disabled object

Get/Set user defined scalar parameter

Get/Set user defined vector parameter


H5FD_dsm_steering_init( MPI_Comm comm, void *buffer);

H5FD_dsm_steering_update();

H5FD_dsm_steering_is_enabled(const char *name);

H5FD_dsm_steering_scalar_get(const char *name, hid_t mem_type, void *data);

H5FD_dsm_steering_vector_get(const char *name, hid_t mem_type, hsize_t

number_of_elements, void *data);


H5FDdsm Steering API

Test if a user defined parameter is set

Wait for an incoming “play” command

Begin/End query and avoid multiple metadata accesses

Get/Free dataset handle for data specific parallel read


H5FD_dsm_steering_is_set(const char *name, int *set);

H5FD_dsm_steering_begin_query();

H5FD_dsm_steering_get_handle(const char *name, hid_t *handle);

H5FD_dsm_steering_free_handle(hid_t handle);

H5FD_dsm_steering_wait();


Can switch between

modes arbitrarily

(or every N iterations)

Calculation

Write

Initialization

Analysis

+ GUI

Interaction

CalculationWait

mode

Analysis

+ GUI

Interaction

File

Wait

mode

Wait Mode


Computation/Analysis not overlapped

Define some pipeline in the

ParaView GUI which does

something interesting

(to send data back)

H5FD_dsm_steering_wait();


GUI

Write

Initialization

Ana

lysi

s

File

Calculation

Ana

lysi

s

Ana

lysi

s

Free Mode


Computation/Analysis overlapped

Can switch between

modes arbitrarily

(or every N iterations)

File Create wipes DSM

clean

So may want to add more

sync optionsCan store N

time steps


Steering demo (recorded)


29Parallel Computational Steering and Analysis for HPC Applications using a ParaView InterfaceMonday, 11 April 2011

Steering demo (recorded)


Animated Wedge



Conclusion

Minimally intrusive way to steer applications Supports Parallel for Sim and Vis/Analysis Allows remeshing/exchange of large data

Can do stuff that we couldn’t do before!

Further work Now developing new MPI mode for large core counts

DMAPP/Gemini layer

Turbine mesh next target for manipulation



Software available on HPCforge


H5FDdsm: https://hpcforge.org/projects/h5fddsm

ICARUS: (ParaView plugin) https://hpcforge.org/projects/icarus

XdmfGenerator: https://hpcforge.org/projects/xdmfgenerator

Questions:

https://hpcforge.org/projects/h5fddsm

https://hpcforge.org/projects/icarus

https://hpcforge.org/projects/xdmfgenerator

Documents

Parallel Computational Steering and Analysis for HPC Applications using a ParaView Interface and the HDF5 DSM Virtual File Driver J. Biddiscombe, J. Soumagne,