Parallel Particle Simulation in Multiscale Fluid ... · Bart Verleye 8 July 2014,...

Technische Universitat Munchen

Parallel Particle Simulation inMultiscale Fluid Representations

SuperMUC Review Workshop

Philipp Neumann, Tobias Weinzierl, Denis Jarema,Bart Verleye

8 July 2014,neumanph@in.tum.de

P. Neumann et al.: Parallel Particle Simulation in Multiscale Fluid Representations

SuperMUC Review Workshop, 8 July 2014, neumanph@in.tum.de 1

Technische Universitat Munchen

Motivation

1 Multi-level fluid-particle treatment2 Many-particle systems:

Spacetree algorithms3 Outlook: Walking down the scales

P. Neumann et al.: Parallel Particle Simulation in Multiscale Fluid Representations

SuperMUC Review Workshop, 8 July 2014, neumanph@in.tum.de 2

Technische Universitat Munchen

Lattice Boltzmann–Navier-Stokes: ModelsNavier-Stokes Lattice Boltzmann

Mesh-based Mesh-based

Fluid in equilibrium Fluid close to equilibrium

DoF: velocities u, pressure p DoF: distribution functions fi

∇ · u = 0∂tu + u · ∇u = −∇p + 1

Re ∆u fi (x + cidt , t + dt) = fi (x, t) + ∆i (f − f eq)

Navier-Stokes

Lattice Boltzmann

ρ(x, t) =Q∑

i=1fi

p(x, t) = ρ(x, t)/c2s

u(x, t) = 1ρ(x,t)

Q∑i=1

fici

P. Neumann et al.: Parallel Particle Simulation in Multiscale Fluid Representations

SuperMUC Review Workshop, 8 July 2014, neumanph@in.tum.de 3

Technische Universitat Munchen

From Navier-Stokes to Lattice Boltzmann• Given: ρ, u, ∂xβuα + ∂xαuβ• Generally, it holds

Q > 1︸︷︷︸ + D︸︷︷︸ + D(D + 1)/2︸ ︷︷ ︸mass momentum stresses

• Split fi = f eqi (ρ,u) + f neq

i

• Ansatz1: continuum↔ non-equ. parts as small as possible

minf neq∈RQ

g(f neq) such that

∑i f neq

i = 0 mass∑i f neq

i ciα = 0 momentum∑i f neq

i ciαciβ = −c2s τ(∂xβuα + ∂xαuβ

)stresses

1 P. Neumann, H.-J. Bungartz, M. Mehl, T. Neckel, and T. Weinzierl. Commun. Comput. Phys. 12(1):65–84, 2012.

P. Neumann et al.: Parallel Particle Simulation in Multiscale Fluid Representations

SuperMUC Review Workshop, 8 July 2014, neumanph@in.tum.de 4

Technische Universitat Munchen

From Navier-Stokes to Lattice Boltzmann• Given: ρ, u, ∂xβuα + ∂xαuβ• Generally, it holds

Q > 1︸︷︷︸ + D︸︷︷︸ + D(D + 1)/2︸ ︷︷ ︸mass momentum stresses

• Split fi = f eqi (ρ,u) + f neq

i

• Ansatz1: continuum↔ non-equ. parts as small as possible

minf neq∈RQ

g(f neq) such that

∑i f neq

i = 0 mass∑i f neq

i ciα = 0 momentum∑i f neq

i ciαciβ = −c2s τ(∂xβuα + ∂xαuβ

)stresses

1 P. Neumann, H.-J. Bungartz, M. Mehl, T. Neckel, and T. Weinzierl. Commun. Comput. Phys. 12(1):65–84, 2012.

P. Neumann et al.: Parallel Particle Simulation in Multiscale Fluid Representations

SuperMUC Review Workshop, 8 July 2014, neumanph@in.tum.de 4

Technische Universitat Munchen

Application: Hierarchical Particle TransportCoarse-scale model: NS (fluid) + Faxen2 (particle)

NS Level 0

NS Level 1

NS Level 2LB Level 3

Fine-scale model: LB + Two-way coupledparticle solver3

2 H. Faxen. Appelberg, 1921.3 A.J.C. Ladd. J. Fluid Mech., 271:285–339, 1994.

P. Neumann et al.: Parallel Particle Simulation in Multiscale Fluid Representations

SuperMUC Review Workshop, 8 July 2014, neumanph@in.tum.de 5

Technische Universitat Munchen

Particle Displacement in Channel Flow (1)

NS Level 0

NS Level 1

NS Level 2LB Level 3

• Start with LB, Level 3Switch to

• NS-Faxen, Level 2 at t = 0.01• NS-Faxen, Level 1 at t = 0.60• NS-Faxen, Level 0 at t = 0.80

• Compare to dynamically adaptive LB solution4

LB LB–NSdxF = 3.1e-3 dxF = 3.1e-3dxC = 2.8e-2 dxC = 5.0e-1dtLB = 4.8e-7 dtLB = 3.2e-7

dtNS = 3.0e-5

4 P. Neumann and T. Neckel. Computational Mechanics, 51(2):237–253, 2013.

P. Neumann et al.: Parallel Particle Simulation in Multiscale Fluid Representations

SuperMUC Review Workshop, 8 July 2014, neumanph@in.tum.de 6

Technische Universitat Munchen

Particle Displacement in Channel Flow (2)

0.0 0.2 0.4 0.6 0.8 1.0 1.2 1.4 1.61.500

2.000

2.500

3.000

3.500

4.000

4.500Pos_X (LB-NS,0.75x0.35)

Pos_X (LB-NS,1.5x0.7)

Pos_X (LB-NS,3.0x1.4)

Pos_X (LB-NS,4.0x1.4)

Pos_X (LB)

posi

tion

(x-d

irect

ion)

time

• Start with LB, Level 3Switch to

• NS-Faxen, Level 2 at t = 0.01• NS-Faxen, Level 1 at t = 0.60• NS-Faxen, Level 0 at t = 0.80

• Compare to dynamically adaptive LB solution4

LB LB–NSdxF = 3.1e-3 dxF = 3.1e-3dxC = 2.8e-2 dxC = 5.0e-1dtLB = 4.8e-7 dtLB = 3.2e-7

dtNS = 3.0e-5

4 P. Neumann and T. Neckel. Computational Mechanics, 51(2):237–253, 2013.

P. Neumann et al.: Parallel Particle Simulation in Multiscale Fluid Representations

SuperMUC Review Workshop, 8 July 2014, neumanph@in.tum.de 7

Technische Universitat Munchen

Particle Displacement in Channel Flow (2)

0.0 0.2 0.4 0.6 0.8 1.0 1.2 1.4 1.61.500

2.000

2.500

3.000

3.500

4.000

4.500Pos_X (LB-NS,0.75x0.35)

Pos_X (LB-NS,1.5x0.7)

Pos_X (LB-NS,3.0x1.4)

Pos_X (LB-NS,4.0x1.4)

Pos_X (LB)

posi

tion

(x-d

irect

ion)

time

• Start with LB, Level 3Switch to

• NS-Faxen, Level 2 at t = 0.01• NS-Faxen, Level 1 at t = 0.60• NS-Faxen, Level 0 at t = 0.80

• Compare to dynamically adaptive LB solution4

LB LB–NSdxF = 3.1e-3 dxF = 3.1e-3dxC = 2.8e-2 dxC = 5.0e-1dtLB = 4.8e-7 dtLB = 3.2e-7

dtNS = 3.0e-5

4 P. Neumann and T. Neckel. Computational Mechanics, 51(2):237–253, 2013.

P. Neumann et al.: Parallel Particle Simulation in Multiscale Fluid Representations

SuperMUC Review Workshop, 8 July 2014, neumanph@in.tum.de 7

Technische Universitat Munchen

Particle Displacement in Channel Flow (2)

0.0064 0.0084 0.0104 0.0124 0.01441.500

1.502

1.504

1.506

1.508

1.510

1.512

1.514Pos_X (LB-NS,0.75x0.35)

Pos_X (LB-NS,1.5x0.7)

Pos_X (LB-NS,3.0x1.4)

Pos_X (LB-NS,4.0x1.4)

Pos_X (LB)

posi

tion

(x-d

irect

ion)

time

• Start with LB, Level 3Switch to

• NS-Faxen, Level 2 at t = 0.01• NS-Faxen, Level 1 at t = 0.60• NS-Faxen, Level 0 at t = 0.80

• Compare to dynamically adaptive LB solution4

LB LB–NSdxF = 3.1e-3 dxF = 3.1e-3dxC = 2.8e-2 dxC = 5.0e-1dtLB = 4.8e-7 dtLB = 3.2e-7

dtNS = 3.0e-5

4 P. Neumann and T. Neckel. Computational Mechanics, 51(2):237–253, 2013.

P. Neumann et al.: Parallel Particle Simulation in Multiscale Fluid Representations

SuperMUC Review Workshop, 8 July 2014, neumanph@in.tum.de 7

Technische Universitat Munchen

Algorithmic Requirement Analysis

• (Transient) Mapping from particles to grid and vice versa• Linked lists/ cells: inappropriate→ particles may move more than one cell

• For few particles: global scatter/gather of particle position→ each rank holds all particle information

• For many particles: localised approach→ only domains which hold particle maintain particle data

P. Neumann et al.: Parallel Particle Simulation in Multiscale Fluid Representations

SuperMUC Review Workshop, 8 July 2014, neumanph@in.tum.de 8

Technische Universitat Munchen

Particle-in-Tree: Algorithm

• Make each cell of spacetree holdarray of particles (dynamic, most oftime empty)

• Make physics move particle,i.e. update particle positions (localcomputation within cell)

• Before cell is left: check whether(moved) particle still is within cell.Otherwise lift it to next coarser level

• Apply lift checks recursively• Before algorithm descends in spacetree: Check whether refined cell holds particles.

If so, sieve them into the children (drop)• Apply drops recursively

Remarks:• Particles belonging to level ` are never dropped deeper than level ` (drop constraint)• Benchmarking difficulties: Density, speed, mesh width...

P. Neumann et al.: Parallel Particle Simulation in Multiscale Fluid Representations

SuperMUC Review Workshop, 8 July 2014, neumanph@in.tum.de 9

Technische Universitat Munchen

Particle-in-Tree: Scaling

P. Neumann et al.: Parallel Particle Simulation in Multiscale Fluid Representations

SuperMUC Review Workshop, 8 July 2014, neumanph@in.tum.de 10

Technische Universitat Munchen

Outlook: Lattice Boltzmann–Molecular Dynamics

ParticleInsertion

MomentumInsertion

MacroscopicCellServiceMacroscopicCellService

MomentumController

KineticEnergyController

MoleculeWrapper

MoleculeIterator

MDSolverInterfaceMacroscopicSolverInterface

MDSimulation ContinuumSimulation

implements

utilises

utilises

sendsdata to

receivesdata from/

sendsdata to

implements

utilises utilises

MaMiCo

User/Programmer

uses/provides newimplementation of

TransferStrategy

Goal:

• Resolve regions of interest by molecular dynamics• Resolve other regions by Lattice Boltzmann• Partitioned approach: Macro-Micro-Coupling Tool5

→ Separation: molecular dynamics↔ Mesh-based solver

5 P. Neumann and N. Tchipev. Proceedings of ISPDC, 2012.

P. Neumann et al.: Parallel Particle Simulation in Multiscale Fluid Representations

SuperMUC Review Workshop, 8 July 2014, neumanph@in.tum.de 11

Technische Universitat Munchen

LB–MD: Channel FlowLB codes:• Peano framework6

• waLBerla7

• OpenLB (in preparation)

MD codes:• Simple MD• LAMMPS8

• ESPResSo9 (inpreparation)

• MarDyn (planned)

→ new SuperMUC project

Massively Parallel MultiscaleSimulation of Nanoflows

0 13 26 39 52 65 78 91 104 117 1300

0.2

0.4

0.6

0.8

1

Velocity (LBMD)Velocity (LB)

y

Vel

ocity

MD LBLB

• 3D steady-state coupling→ velocity relaxation

• LB: 54× 54× 54 cells, dx = 2.5

• MD: 105 LJ atoms

• MD: periodic boundaries+ buffer regions

6 www.peano-framework.org 7 www.walberla.net 8 lammps.sandia.gov 9 espressomd.org

P. Neumann et al.: Parallel Particle Simulation in Multiscale Fluid Representations

SuperMUC Review Workshop, 8 July 2014, neumanph@in.tum.de 12

Technische Universitat Munchen

Thank You...

... for your attention!

P. Neumann et al.: Parallel Particle Simulation in Multiscale Fluid Representations

SuperMUC Review Workshop, 8 July 2014, neumanph@in.tum.de 13