Code Saturne a H.P.C point of view - Welcome to cfd…cfd.mace.manchester.ac.uk/twiki/pub/CfdTm/PresFile30/... · · 2008-07-01Presentation of Code_Saturne. June, 26th 2008 J. Bonelle

Code_Saturnea H.P.C point of

viewDaresbury meeting

June, 26th 2008

Presentation of Code_Saturne

June, 26th 2008J. Bonelle - EDF R&D - Code_Saturne : a H.P.C. point of view http://rd.edf.com/code_saturne3

Code_SaturneEDF General purpose CFD software


Open source since March 2007

http://rd.edf.com/code_saturne (source code and manuals)

[email protected] (support)

Basic capabilities :

Simulation of incompressible or expandable flows with or without heat transfer and turbulence (mixing length, 2-equation models, v2f, RSM, LES, …)

1998Prototype

20001.0

Basic modelling

Wide range of meshes

2001Qualification for nuclear application

20041.1

Parallelism

L.E.S

20061.2

State of the art in turbulence

2007Open source

20081.3

Massively parallel

ALE

Code coupling




Co-located Finite Volume approach

Meshes with any type of cells (tetrahedral to polyhedral)

PWR lower plenum

Example of mesh with stretched cells and hanging nodes

3D polyhedral cells

Example of composite mesh




Main application area :

Nuclear power plant optimisation in terms of lifespan, productivity and safety.

But also applications in :

Combustion (gas and coal)

Electric arc and joule effect

Atmospheric flows

Radiative heat transfer

Other functionalities :

Fluid structure interaction

Deformable meshes : Arbitrary Lagragian Eulerianmethod (ALE)

Dispersed particle tracking (Lagrangian approach)


Code_SaturnePortability and parallelism

Code_SaturnePortability and parallelism

Available on a wide range of operating systems : Linux, Unix (HP-UX, Solaris, Cray, OSF1, Mac OSX …)

Available on a wide range of architectures :

IBM BlueGene/L and BlueGene/P (PowerPC 4)

IBM Cluster – MareNostrum, Barcelona Supercomputing Centre (PowerPC 5)

Bull Novascale (Intel Itanium2)

Cray XT3 and XT4 – Hector, Edinburgh (AMD Opteron)

Cluster of PCs (Intel Xeon, AMD Opteron, …)

Parallel on distributed memory machines thanks to MPI implementations (OpenMPI, MPICH2, …)

Tested so far from 1 to 8192 processors

Periodicity capability : translation and rotation

Domain splitting thanks to METIS


Code_Saturne architectureCode_Saturne architecture

FVMlibrary

Code_Saturne

Preprocessor (Envelope)

mesh import mesh pasting

domain decomposition

Post-processing

output

Restart files

Meshes

Parallel Kernelghost cells creation

periodicityCFD Solver

code coupling

parallel treatmentCode_Saturne

SYRTHESCode_Aster

Salome platform...

GUI

Xml data file

External libraries (EDF, LGPL):• BFT: Base Functions and Types• FVM: Finite Volume Mesh

BFT library

Serial I/Omemory management

Parallel meshManagementParallel I/O


Code_SaturneCode coupling capabilities

Code_SaturneCode coupling capabilities

All these softwares are under Gnu GPL or LGPL version 2

SALOME : http://www.salome-platform.org

Code_Aster : http://www.code-aster.org

SYRTHES : soon available

Parallel code coupling capabilities thanks to FVM (EDF’s “Finite Volume Mesh” library)

SALOME Platform

Code_Saturne

Code_Saturne

SYRTHESC.A.D, mesh, visualization, coupling platform

CFD + conjugate heat transfer and radiation

CFD + CFD = RANS/LES or Chimera approach

CFD + structural mechanics analysis

Code_Saturne’sHPC performance


Code_SaturneBenchmark test cases

Code_SaturneBenchmark test cases

Father

200 iterations

Turbulence = L.E.S

Number of cells in the mesh in million

Hypi

200 iterations

Turbulence = L.E.S

GRILLE

50 iterations

Turbulence = k-ε

Industrial studies Exploratory studies

1 10 100


Code_Saturne’s benchmark on EDF’s HPC resourcesCode_Saturne’s benchmark on EDF’s HPC resources

47.7022.900.662.40Theoretical

performance in Tflops

InfinibandIBMMyrinetInfinibandInterconnect

552

1.8 and 2.6 GHz

AMD Opteron

Tantale

8192

0.7 GHz

IBM PowerPC 440

BlueGene/L

Intel Itanium2 dual-coreAMD OpteronProcessor

1.6 GHz2.6 GHzFrequency in GHz

7456128Number of

proc. or cores

PlatineCluster ChatouMachine’s name


Code_SaturneFATHER test case (1M)

Code_SaturneFATHER test case (1M)

10

100

1 000

10 000

100 000

1 10 100 1 000 10 000

Number of processors

Elap

sed

time

(s)

Opteron 2.6 GHz- Myrinet

Tantale :Opteron 1.8/2.6 GHz - Infiniband

Platine : Itanium2 1.6 GHz - Infiniband

BlueGene/L : PowerPC440 0,7 GHz - IBM


Code_SaturneHYPI test case (10M)


100

1000

10000

100000

1000000

1 10 100 1000 10000


Elap

sed

time

(s)






Code_SaturneGRILLE test case (100M)

Code_SaturneGRILLE test case (100M)

100

1000

10000

100000

100 1000 10000


Elas

ped

time

(s)



Cache effect

Code_Saturne’sHPC prospects


Code_SaturneWhy more and more flops ?

Code_SaturneWhy more and more flops ?

To make numerical studies with finer and larger meshes

Bigger computational domain (from a geometrical point of view)

Refined computational domain (simulation of very small scales)

To make “easily” many computations

“Monte Carlo” approaches : treatment of uncertainties

To increase the simulated physical time of phenomena

Computation of transitional phenomena

To take more physics into account

Use of LES turbulence modelling instead of k-εmodelling if necessary

Mixing different approaches : eulerian and lagrangian approaches

Multi-physics studies : fluid structure interactions, coupling codes

17

Code_Saturne – HPC Roadmap: application examples

2003 2010 201520072006Consecutive to the Civauxthermal fatigue event

Computations enable to better understand the wall thermal loading in an injection.

Knowing the root causes of the event ⇒ define a new design to avoid this problem.

Part of a fuel assembly3 grid assemblies

Computation with an L.E.S. approach for turbulent modelling

Refined mesh near the wall.

9 fuel assemblies

No experimental approach up to now

Will enable the study of side effects implied by the flow around neighbour fuel assemblies.

Better understanding of vibration phenomena and wear-out of the rods.

The whole vessel reactor

106 cells3.1013 operations

108 cells1016 operations




Fujistu VPP 5000

1 of 4 vector processors

2 month length computation

Cluster, IBM Power5

400 processors

9 days

# 1 Gb of storage

2 Gb of memory

IBM Blue Gene/L « Frontier »

8000 processors

# 1 month

30 times the power of


# 1 month

# 15 Gb of storage

25 Gb of memory

# 10 Tb of storage

25 Tb of memory

# 1 Tb of storage

2,5 Tb of memory

# 200 Gb of storage

250 Gb of memory

Power of the computer Pre-processing not parallelized Pre-processing not parallelized

Mesh generation

resolved

… ibid. …

… ibid. …

Visualisation

500 times the power of


# 1 month

Computations with smaller and smaller scales in larger and larger geometries ⇒ a better understanding of physical phenomena ⇒ a more effective help for decision making

⇒ A better optimisation of the production (margin benefits)

Power of the computer

resolved

… ibid. …

Scalability / Solver

Appendix


Code_SaturneBenchmark results (1/2)

Code_SaturneBenchmark results (1/2)

Interest of using BLAS library

Example on platine : local dot product for the HYPI test case

233,7143,026,619,7With BLAS library (GFlop/s)

5,56,94,03,6Without BLAS library (GFlop/s)

2561283216Number of processors

Performance of the local matrix vector product

Example on 32 processors for the FATHER test case

4.1

5.3

Platine(Opt. BULL)

1,34,03,45,3Performancein GFlop/s

1.03.12.64.1Ratio

PowerPC 440 0,7 GHzItanium2 1,6 GHzOpteron 1,8 GHzOpteron 2,6 GHzProcessor

BlueGenePlatineTantaleCluster ChatouMachine


Code_SaturneBenchmark results : I/O issue (1/2)

Code_SaturneBenchmark results : I/O issue (1/2)

Elapsed time in outputting post-processing files and restart files

Grille test case : comparison between Platine and BlueGene/L

54 s

156 s

X

256

99 s

169 s

556 s

1024

199 sBlueGene/L: After I/O upgrade

180 sPlatine

487 sBlueGene/L: Before I/O upgrade

2048Number of processors

Elapsed time in outputting post-processing files and restart files increases with the number of processors

Because of our algorithm = send all data to write to rank 0.

Solution : use of MPI I/O

Criteria to evaluate I/O quality:

Intrinsic quality of each I/O node and their number

Interconnect and MPI implementation

Filesystem (Lustre Vs GPFS)








0

10

20

30

40

50

60

1 10 100 1 000 10 000


Elap

sed

time

(s)

Documents

Code Saturne a H.P.C point of view - Welcome to cfd…cfd.mace.manchester.ac.uk/twiki/pub/CfdTm/PresFile30/... · · 2008-07-01Presentation of Code_Saturne. June, 26th 2008 J. Bonelle