Parallel MultiParallel Multi--Zone Methods for LargeZone Methods for Large--Scale Multidisciplinary Computational Scale Multidisciplinary Computational

Physics SimulationsPhysics Simulations

Ding Li, Ding Li, Guoping XiaGuoping Xia and Charles L. Merkleand Charles L. Merkle

Purdue UniversityPurdue University

The 6th International Conference on Linux ClustersThe 6th International Conference on Linux ClustersThe HPC Revolution 2005The HPC Revolution 2005

Chapel Hill, NC, April 25Chapel Hill, NC, April 25--28, 200428, 2004

Presentation OutlinePresentation Outline

MultidisciplinaryMultidisciplinaryNumerical Analysis SystemNumerical Analysis SystemGEMS codeGEMS codeGeneralized Equations of MotionGeneralized Equations of MotionLinux Cluster and Benchmarks Linux Cluster and Benchmarks Parallel ImplementationParallel ImplementationRepresentative ApplicationsRepresentative Applications

Multidisciplinary Computational Multidisciplinary Computational PhysicsPhysics

•Multi-physicsstructures, plasma dynamics, fluid dynamics,

electromagnetics, radiative energy transfer and neutron transport

•Different approaches

loosely coupled and individual codes

closely coupled and solved simultaneously

•Unified Framework General conservation law

Data Mining/Data Mining/VisualizationVisualization

CAD

GRID GENERATOR

PGRIDPGRID

Data Repository

G E

M S

G E

M S

G E

M S

PropertyPropertyModulesModules

Numerical Analysis SystemNumerical Analysis System

Purdue University - School of Mechanical Engineering

Fluid-Solid Model

GEMS codeGEMS code

∑∑ ΩΩ =−+∂

∂Γ

N

ii

Visivn

N

ii

Invin

p VSAFAFVt

Q,,

General EquationPreconditioned,Multiple-TimeAlgorithms

Preconditioned,Multiple-TimeAlgorithms

Structured-Unstructured Grids

Cluster ComputingCluster Computing

Multiple Physical Zones

G E

M S

G E

M S

G E

M S

στρρ rrrrrrr

•∇+•∇+−∇=•∇+∂∂ pVV

tV

−= )(

312 etraceSM

rrrrrrrrδµσ

ElectromagneticsElectromagnetics

Generalized EquationsGeneralized Equations

0=Φ∇+×∇+•∇+∂∂

+∂

∂Γ CD

p FFtQQ

τ

Generic set of partial differential equations:Generic set of partial differential equations:

0=∫ ΩΦ∇+∫ Ω×∇+∫ Ω•∇+∫ Ω∂∂

+Ω∂

∂Γ

ΩΩΩΩddFdFd

tQQ

CDp

τ

0=∫ ΣΦ+∫ Σ×+∫ Σ•+

∫ Ω

∂∂

+Ω∂

∂Γ

Ω∂Ω∂Ω∂ΩdndFndFnQd

tQ

CDp

τ

Normal Flux Tangential Flux Scale

Number of Partial Differential Number of Partial Differential Equations in Various FieldsEquations in Various Fields

MultiMulti--Physics Physics Zone MethodZone Method

Cluster2Cluster2Cluster2Cluster3Cluster3Cluster3

Cluster1Cluster1Cluster1

• Distinct Physics Zones•Different media•Different equations

• Parallel Processing

•Each zone divided into sub-clusters

• Load Balancing•Number of equations•Size of grids

Linux ClustersLinux ClustersSimbaSimba (2001)(2001)51 nodes:Single P4 1.8 51 nodes:Single P4 1.8 GhzGhz CPU:1Gb CPU:1Gb RAMsRAMs:10/100 :10/100 EthernetEthernetRedhat9.0, Redhat9.0, Lahey Lahey Fortran Compiler, MPICH1.2.4, PBSFortran Compiler, MPICH1.2.4, PBSMacbethMacbeth (2005)(2005)98 nodes: dual AMD 98 nodes: dual AMD OpteronOpteron 1.8 1.8 GhzGhz CPU: 4GB CPU: 4GB RAMsRAMs::InfinibandInfiniband interconnect:4X interconnect:4X InfinibandInfiniband network network fabric (10Gbps)fabric (10Gbps)Redhat Redhat Enterprise, Intel, PGI and Enterprise, Intel, PGI and PathscalePathscale Fortran Fortran Compiler, MPICH1.2.6,Compiler, MPICH1.2.6,PBSProPBSPro

Simba Simba vs. vs. MacbethMacbeth2D turbulent flow w/ 0.5 Million grid cells

Number of Processors

WTi

me/

cells

/iter

atio

ns

Wal

lTim

e(s

)

0 10 20 30 40

0.0E+00

1.0E-04

2.0E-04

3.0E-04

4.0E-04

0

500

1000

1500

2000

2500

wall time (Simba,Lahey)

wtime/cells/iterations (Macbeth.Intel)

wall time (Macbeth,Intel)

wtime/cells/iterations (Simba,Lahey)

Number of Processors

WTi

me/

cells

/iter

atio

ns(s

ec.)

0 10 20 30 40 50 60

6.0E-05

8.0E-05

1.0E-04

1.2E-04

1.4E-04wtime/cells/iterations/Processors

Pathscale

PGI

Intel

Intel, Intel, Pathscale Pathscale vs. PGIvs. PGI

3D flow with 3D flow with 3.375Million grid cells3.375Million grid cells

Parallel Computing & PartitioningParallel Computing & Partitioning

N01

N02

N03

N04…N10

10 Partitions

Parallel Data StructureParallel Data Structure

Definitions:Definitions:Interface:The face adjoined by two different partitionsSending Data:The cells of current partition adjoined to the interfaceReceiving Data:The cells of all partitions except current partition adjoined to the interface

Current partition

Receiving data

Sending data

Interface

Exchanging matrixExchanging matrix•Zero diagonal: no data exchange inside a partition

•Rows represent the number of sending data

•Columns represent the number of receiving data

•Sum of rows are total number of data received in the partition of the row

•Sum of columns are total number of data sent in the partition of the column

•The number of each element is the number of data sent by the column partition to the row partition

0280030000800300030500050

1 2 3 4 5

1

2

3

4

5

Sending partition (processor)Receiving partition (processor)

Cell list for sending and receivingCell list for sending and receiving

535251232221201818103

Cell list for sending in partition 3

Cell list of receiving in partition 5

Cell list of receiving in partition 2

87654321

10987654321

Partition 1

Partition 4

Representative ApplicationsRepresentative ApplicationsConstant Volume Combustion Turbine SystemConstant Volume Combustion Turbine System

Calculation in only one sector is necessary, since flow in other sectors experience same condition, at different time

Boundary condition at sector interface is provided by the solution in the same sector at earlier time step, which is determined by the firing order

Pulse Detonation Engine and Pulse Detonation Engine and Turbine Interaction ResearchTurbine Interaction Research

Tem

pera

ture

&

Tem

pera

ture

&

Pre

ssu

re C

on

tou

rsP

ress

ure

Co

nto

urs

SummarySummary

Unified parallel framework for dealing with multi-physics problemsParallel Computational implementation.Generalized form with divergence, curl and gradientPotential to apply in the fast growing grid computing.A variety of interesting physical phenomena and the efficacy of the computational implementation

Thanks

Any Questions?