Local Adaptive Mesh Refinement on the GPUmartinsa.at.ifi.uio.no/files/siam_gs13_amr_on_gpu.pdf · Local Adaptive Mesh Refinement on the GPU. Technology for a better society Outline

Technology for a better society 1

Martin Lilleeng Sætra

University of Oslo

18-06-2013

Local Adaptive Mesh Refinement

on the GPU

Technology for a better society

Outline

2

• The starting point: Our GPU-based shallow water simulator + Original Local Adaptive Mesh Refinement-paper

• Novel work: Adaptive mesh refinement fully implemented on the GPU

• Performance results

• Video: Malpasset dam break case with adaptive mesh refinement


The Shallow Water Equations (SWE)

3

Vector ofConserved variables Flux Functions

Bed slopesource term

Bed frictionsource term

Numerical Simulation of the SWE:• Hyperbolic partial differential equation

– Enables explicit schemes.• Solutions form discontinuities / shocks

– Require high accuracy in smooth parts without oscillations near discontinuities.

• Solutions include dry areas– Avoid negative water depths in

simulations.• Accuracy

– 2nd order spatial/temporal discretization.

Scheme of choice: A. Kurganov and G. Petrova, A Second-Order Well-Balanced Positivity Preserving Central-Upwind Scheme for the Saint-Venant System Communications in Mathematical Sciences, 5 (2007), 133-160


Spatial Discretization

4

Write on vector form:

Impose finite-volume grid with discrete fluxes:

Continuous equation

Discrete spatial grid

Discrete flux calculation


Calculate Fluxes

5

Continuous variables Discrete variables

Dry states fix

Slope reconstruction

Evaluate integration pointsFlux calculation

Vector ofConservedvariables

Flux FunctionsBed slope

source termBed frictionsource term


Evolve in Time


One Full Time Step Using the Kurganov-Petrova-scheme on the GPU

7

3. Halfstep

1. Calculate fluxes

4. Calculate fluxes5. Evolve in time

6. Apply boundaryconditions

2. Calculate Δt


Domain Decomposition

8

• “Traditional” CUDA block decomposition.• Each Streaming Multiprocessor of the GPU computes on a small 2D block.• Neighboring blocks use overlap to exchange information.• Global ghost cells for boundary conditions (wall, open, fixed depth etc.).


Local Adaptive Mesh Refinement (AMR)

9

• Goal: Increase accuracy, minimize cost.

Cost

Accuracy

Grid re

finement


Capture Local Features in the Solution

10


“AMR Grid” Hierarchy

11

• “Traditional” CUDA block decomposition within each grid.• Initialize new subgrids at level ℓ +1, with twice the resolution, using

reconstructed cell values from current grids at level ℓ. • The boundary interfaces of grid ℓ +1 must be aligned with grid cell

boundaries in grid ℓ.• Each new subgrid can be viewed as a stand-alone simulator.

(ℓ=0)

(ℓ=1)

(ℓ=1)

(ℓ=2)

Grid 0

Grid 1 Grid 2

Grid 3


Keep Data on the GPU

12


AMR: Time Integration

13

AMR: Refine


Extending the Simulation Cycle

BC

Compute boundaries

Reset AMR data(Not a CUDA kernel)

Flux correction

Coarsen / Average

• Pre-step • “Regular” time stepping

• Post-step


Boundary Values – Space

15

• Save the solution at the beginning and at the end of each time step from the level ℓ grid at all boundaries to level ℓ+1 subgrids.

• Each subgrid has data structures for saving these values.• The solution is reconstructed in space at both the beginning and the

end of the time step.

ℓ

ℓ+1


Boundary Values – Time

16

• The saved boundary values are then linearly interpolated in time, by a special boundary condition kernel used on all grids except the root grid.

t+Δt0t

Grid 0 (ℓ)

Grid 1 (ℓ+1)

Δt0

Δt1 Δt2


Time Step Size

17

• Since the time step size, Δt, is dependent on the eigenvalues of the solution, each grid has a different Δt.

• After each time step on the level ℓ grid, the time step size is given to all level ℓ+1 grids.

• The last time step in Grid 1 and 2 (Δt2 and Δt4) is reduced so that all grids reach t+Δt0 after one full simulation cycle.

t+Δt0t

Grid 0 (ℓ)

Grid 1 (ℓ+1)

Grid 2 (ℓ+1)

Δt0

Δt1 Δt2

Δt3 Δt4


Averaging From Level ℓ +1 to ℓ

18

• After all level ℓ+1 subgrids are advanced to the same time as the level ℓ grid, the distribution of the solution will be more accurate in the level ℓ+1 subgrids.

• The solution is “moved up” in the AMR hierarchy by replacing the values in level ℓ by the values from a overlaying subgrid at level ℓ+1.

• A simple average is used.

ℓ+1 ℓ


Flux Correction Step – Fine-coarse

19

• Fluxes across all fine-coarse (ℓ+1 to ℓ) interfaces are accumulated for each time step in the level ℓ+1 subgrids.

• After all the level ℓ+1 subgrids are at the same advanced time as the level ℓ grid, a flux correction step is performed on the level ℓ grid.

• The correction step uses the difference between the flux from the level ℓ grid and the accumulated fluxes from the level ℓ+1 subgrids, and corrects the values in the level ℓ grid.

t0

t0

t1

t2

t1

t2

ℓ+1 ℓ

(Δt1 + Δt2 = Δt0)


Flux Correction Step – Fine-fine

20

• Fine grid cells adjacent to other fine grids also needs to be corrected.• Again, we use the difference between the flux from the level ℓ grid and

the accumulated fluxes from the level ℓ+1 subgrids, but now we write the correction to the adjacent ℓ+1 grid, instead of the level ℓ grid.

• This, together with the coarse-fine flux correction, maintains conservation of mass.

t2

t2

t0

t1

t0

t1

ℓ+1

(Δt0 + Δt1 = Δt2 + Δt3)

ℓ+1

t3

t3


Extending the Simulation Cycle

BC

Compute boundaries

Reset AMR data(Not a CUDA kernel)

Flux correction

Coarsen / Average

• Pre-step • “Regular” time stepping

• Accumulate fine fluxes

• Reduce last time step

• Special BC

• Post-step


AMR: Refine

22

AMR: Time Integration


“Correct” Refinement Criteria

23


Refinement Test

24

• After a given number of time steps, each cell in every grid is checked against a given refinement criterion, and potentially flagged for refinement. This test is performed by a dedicated kernel.

• The same kernel does a reduction to tiles, which is a collection of cells, and writes the number of cells marked for refinement per tile to a map.

• This map is used to generate new proposed grids.

• The new proposed grids are checked for overlaps with existing grids. In case of overlap, the proposed grids are broken down into smaller grids.

• The two last points are the only work done by the CPU.


Initializing New Subgrids

25

• The bathymetry of new subgrids are bilinearly interpolated from the parent grid using effective dedicated GPU hardware (texture memory).

• The initialization of the variables is done by reconstructing the solution from the last time step on the parent grid, and evaluating it in the cell centers of the new grid.


Performance Results – N Subgrids (2D)

26


Performance Results – 1 Subgrid

27



Video: Malpasset Dam Break Case w/AMR

29

./malpasset_amr.avi


Thank you for your attention

Contact:Martin Lilleeng Sæ[email protected]: http://martinsa.at.ifi.uio.no/

SINTEF homepage: http://www.sintef.no/heterocomp

Acknowledgements: Mustafa Altinakar, André R. Brodtkorb, Christopher Dyken, Trond R. Hagen, Knut-Andreas Lie, and Jostein R. Natvig.


• A. R. Brodtkorb, M. L. Sætra, and M. Altinakar, Efficient Shallow Water Simulations on GPUs: Implementation, Visualization, Verification, and Validation, Computers & Fluids 55(0):1–12, 2012.

33

References

Documents

Local Adaptive Mesh Refinement on the GPUmartinsa.at.ifi.uio.no/files/siam_gs13_amr_on_gpu.pdf · Local Adaptive Mesh Refinement on the GPU. Technology for a better society Outline