Large-Scale Granular and Fluid (DEM/SPH) …on-demand.gputechconf.com/.../SC429-large-scale-granular-fluid... · Title: Microsoft PowerPoint - SC14_NVIDIA_booth_talk.pptx Author:

Copyright © Global Scientific Information and Computing Center, Tokyo Institute of Technology

GP GPUGP GPU

Large-Scale Granular and Fluid (DEM/SPH) Simulations using Particles

Large-Scale Granular and Fluid (DEM/SPH) Simulations using Particles

1

Takayuki Aoki

Global Scientific Information and Computing CenterTokyo Institute of Technology

SC14 NVIDIA booth talk, November 19, 2014, New Orleans


GP GPUGP GPU

Compute Node(3 Tesla K20X GPUs)

Performance: 4.08 TFLOPSMemory: 58.0GB(CPU)

+18GB(GPU)

Rack (30 nodes)

Performance: 122 TFLOPSMemory: 2.28 TB

System (58 racks)1442 nodes: 2952 CPU sockets,

4264 GPUs

Performance: 224.7 TFLOPS (CPU) ※ Turbo boost5.562 PFLOPS (GPU)

Total: 17.1 PFLOPS

TSUBAME 2.5TSUBAME 2.5


GP GPUGP GPU

3

TSUBAME SupercomputerTSUBAME Supercomputer

Graph 500No. 3 (2011)

wire

文部科学大臣表彰

(2012)

Gordon Bell Prize (2011)

Tesla S1070X170(680GPU)

Tesla K20XTesla M2050

CUDACOE


GP GPUGP GPU

4


GP GPUGP GPU

Weak Scalability: 2.0000 PFLOPS on 4,000 TSUBAME2.0, 330 billion cells44.5 % the peak performance


GP GPUGP GPU

Granular Material Simulationsusing Discrete Element Method

Granular Material Simulationsusing Discrete Element Method

7


GP GPUGP GPU

8

Golf Bunker ShotsGolf Bunker Shots


GP GPUGP GPU

Contact interaction

Normal direction

Tangential direction

SpringViscosity

Viscosity

Friction

Spring

ijijij xkxF

Simulation for Granular MaterialsSimulation for Granular MaterialsDEM (Discrete Element Method)DEM (Discrete Element Method)


GP GPUGP GPU

10

In 2005In 2005

■ kn = 5×108 dyn/cm■ Time Integration:

2-stage Ruge-Kutta

■ = 8×104 dyn・sec/cm■ t = 4×10-7 sec

DEM (Discrete Element Method) 76,000 Particles: 48 hours

Future work:

CPU 0 CPU 1 CPU 2


GP GPUGP GPU

2 dimensional slice-grid method

Dynamic Load Balance Dynamic Load Balance

Many particles

no particle

2. Move boundary

1. Move boundary


GP GPUGP GPU

2 dimensional slice-grid method

Dynamic Load Balance Dynamic Load Balance

Many particles

no particle

2. Move boundary

1. Move boundary


GP GPUGP GPU

Computational domain is dynamically decomposed into 64 sub-domains.

Slice grid

Dynamic Domain DecompositionDynamic Domain Decomposition

KD-tree Octree


GP GPUGP GPU

• Particle Collision detection of particles with complex shapes described by CAD data is efficiently carried out by using Level Set Function.

Collision Detection using Level Set FunctionCollision Detection using Level Set Function

Particle

Polygon ofCAD data Φ > 0 Φ < 0Negative areaPositive area


GP GPUGP GPULevel Set Function describing CAD surfaceLevel Set Function describing CAD surface

Surface patches of CAD data Level Set Function

negative distance area far from the surface

positive distance area far from the surface

• Generation from 3D CAD data on the uniform mesh• Fast generation algorithm and inside/outside judgment


GP GPUGP GPU

Neighbor Particle ListNeighbor Particle List

Local domain0 6

3

0

0 6 3

NULL

87 percent of memory usage is reduced compared to regular neighbor list.

Linked-list methodLinked-list method


GP GPUGP GPU

AOKI Lab.螺旋すべり台

AOKI Lab.バンカーショット計算16.7 millions particles

with 64 GPUs


GP GPUGP GPU


GP GPUGP GPUDEM using non-spherical particles

Considering more realistic shapes of rocks, non-spherical particles are used in DEM.

Many spherical particles with rigid body connections


GP GPUGP GPU

Using spherical particles,


GP GPUGP GPU

Using non‐spherical tetrapod particles,


GP GPUGP GPUMultiple GPU ScalabilityMultiple GPU Scalability

• Conditions Particles : 2 × 106, 1.6 × 107, 1.29 × 108

Domain Decomposition: Dynamic load Balance using Slice Grid Method Time-Integration : 2-stage

Runge-Kutta


GP GPUGP GPUSPH for Fluid DynamicsSPH for Fluid Dynamics

: Kernel function

First derivatives

h

h : Kernel radius

Particle interaction within a kernel radis


GP GPUGP GPU

A list of Particle Difference Operators

Improved SPHImproved SPH

Interpolation

Gradient

Divergence

Laplacian

2nd polynomial function (Spiky shaped):

r

Generalization of Finite Difference Operators (Imoto, Tagami 2014)


GP GPUGP GPU

• Explicit Time-integration using Predictor-corrector Method

Improved SPHImproved SPH

Predicator

Collector

Temporary pressures are calculated from Birch-Murnaghan’s equations:

Positions are computed as follows:

Pressures are computed as follows:

(1)

(2)

(3)

(4)

(5)


GP GPUGP GPUA Dam Break SimulationA Dam Break Simulation

• Initial setting and Parameters

12 m

2.2 m

0.8 m

Water

Object 6 m10 m

4.8 m


GP GPUGP GPUDescription of the Object ShapeDescription of the Object Shape

• A object is represented by particles arrangement generated from CAD data


GP GPUGP GPUA Dam Break SimulationA Dam Break Simulation

72 M particles with 80 GPUs


GP GPUGP GPU

Fluid-Structure Interaction


GP GPUGP GPU

33

SUMMARYSUMMARY

Particle Method (DEM/SPH) based on short-range interaction are also suitable for GPU computing as well as stencil computation.

Successful many granular simulations GPU-based supercomputer TSUBAME 2.0/2.5 have been shown.

Fluid simulations using SPH is suitable to describe free-surface flows.

Particle methods can be applied to Fluid-Structure Interaction easily.

Documents

Large-Scale Granular and Fluid (DEM/SPH) …on-demand.gputechconf.com/.../SC429-large-scale-granular-fluid... · Title: Microsoft PowerPoint - SC14_NVIDIA_booth_talk.pptx Author: