Efficient High-Order Methods for Geophysical Fluid Dynamics Models*

Frank Giraldo

Department of Applied Mathematics

Naval Postgraduate School, Monterey CA USA

http://faculty.nps.edu/fxgirald

Collaborators: Matthias Läuter (AWI, Potsdam), Marco Restelli (Max-Planck, Hamburg), Emil Constantinescu (Argonne NL, Chicago), Jim Kelly (NPS)

*Funded by the Office of Naval Research: (1) Computational Mathematics and (2) Meteorology

Motivation for this Work

The goal of this research is to construct the best possible climate and numerical weather prediction models for the atmosphere and ocean that:

1. Produce high-order accuracy solutions (or good quality);2. Can use unstructured (adaptive) grids to better handle the physical geometry of the

problem;3. Are efficient to run on all types of computers; and4. Scale well on parallel computers.

1. Producing good quality solutions efficiently requires different methods for different types of flow problems (e.g., continuous versus discontinuous methods).

2. Element-based Galerkin (EBG) methods (e.g., finite elements/volumes, SE, DG) allow for easy construction of the discrete derivatives on unstructured grids.

3. To construct efficient models on any type of computer requires the use of implicit, semi-implicit, or Lagrangian time-integrators (TI).

4. To scale well on parallel computers requires very good domain decomposition techniques. EBG methods are such methods but the matrix problem required in the TI must use better preconditioners to achieve linear scalability.

Talk Summary

I. Equation Sets being explored in this work • Hydrostatic Primitive Equations• Euler/Navier-Stokes Equations• Shallow Water Equations

II. Spatial Discretization • Continuous and Discontinuous Element-based Galerkin Methods

III. Time-Integrators• Explicit SSP Methods• Semi-implicit Methods • Fully-Implicit Methods

IV. Results of the three models under development• Global Hydrostatic Atmospheric Model (NSEAM)• Mesoscale Atmospheric Models• Coastal Ocean Model

I. Equation Sets

Hydrostatic Primitive Equations(Euler Equations with no Vertical Acceleration)

(Mass)

(Momentum)

(Energy/Entropy)

(State)

∂π∂t

+∇ ⋅ π u( ) +∂

∂σπ &σ( ) = 0

∂u

∂t+ u ⋅∇u + &σ

∂u

∂σ+

2Ωz

a2(k × u) = −∇ϕ − cpθ

∂PE

∂π∇π + μx + Su

∂θ∂t

+ u ⋅∇θ + &σ∂θ

∂σ= Sθ

∂ϕ∂PE

= −cpθu =(u,v,w)T ,

x =(x,y,z)T ,

∇=∂∂x

,∂

∂y,

∂

∂z

⎛⎝⎜

⎞⎠⎟

T

π =pS (x, t) − pT PE =p(x,σ,t)

pA

⎛

⎝⎜⎞

⎠⎟

R/cp

Mesoscale Equations(Navier-Stokes Equations)

(Mass)

(Momentum)

(Energy)

(Equation of State )

(Viscous Fluxes)

∂ρ∂t

+∇ • U = 0

∂U

∂t+∇ •

U⊗U

ρ+ PI2

⎛

⎝⎜⎞

⎠⎟= − f k×U( ) − ρgk +∇ • Fu

visc + SU

∂E

∂t+∇ •

E + P

ρU

⎛

⎝⎜⎞

⎠⎟= ∇ • Fe

visc + SE

P =γ −1( ) E−U • U2ρ

−ρϕ⎛

⎝⎜⎞

⎠⎟

Fuvisc =μ ∇u+ ∇u( )T + λ ∇• u( )I 2( )

Fevisc =u• Fu

visc + μcp

Pr∇T

U =ρu,

E =ρe,

u =(u,w)T ,

x =(x,z)T ,

∇=∂∂x

,∂

∂z⎛⎝⎜

⎞⎠⎟

T

Coastal Ocean Equations(Shallow Water Equations)

(Mass)

(Momentum)

(Geopotential)

∂ϕ∂t

+∇ • U = 0

∂U

∂t+∇ •

U⊗U

ϕ+

1

2ϕ 2 − ϕ B

2( )I2 − ν∇U⎛⎝⎜

⎞⎠⎟

= −ϕ S∇ϕ B − f (k × U) + gt

ρ− γ U

ϕ =g hS + hB( )u =(u,v)T ,

x =(x,y)T ,

∇=∂∂x

,∂

∂y

⎛⎝⎜

⎞⎠⎟

T

U =ϕu,

II. Spatial Discretization

• Primitive Equations:

• Approximate the solution as:

– Interpolation O(N)

• Write Primitive Equations as:

• Weak Problem Statement: Find

– such that • Integration O(2N)

• Choice of determines the method (e.g., global function defines spectral method, local function defines EBG)

Galerkin Methods

∂q

∂t+∇ ⋅F = S(q)

qN = ii=1

MN

∑ qi FN =F qN( )

R(qN )≡∂qN

∂t+∇⋅FN −SN =ε

qN ∈Σ(Ω) ∀ ∈Σ

Ω /Ωe

∫ R qN( )dΩ = 0

Σ = ∈L2 (Ω) :ψ ∈PN (Ωe )∀Ωe{ }

Σ = ∈H 1(Ω) :ψ ∈PN (Ωe )∀Ωe{ }

(DG)

(CG)

SN =S qN( )

• Order of the method can be changed automatically (via Basis Functions)

– For many problems, high-order is the most efficient way to reach a certain level of accuracy

• Unstructured adaptive (conforming and non-conforming) grids can be used quite naturally (since the method is based on constructing discrete operators on elements/volumes

• Excellent performance on MPP (especially in high-order mode) since the amount of on-processor work is huge while the communication footprint (stencil) is very small. Numerous Gorden Bell prizes at SC have been awarded to Spectral Element codes (at least 5 to my knowledge).

• Easy to maintain and improve codes since basis functions can be whatever you wish. All one needs to do is to swap out the basis functions routines.

Advantages of Element-based Galerkin Methods

Comparison of EBG Methods

Continuous Galerkin Methods

• High order accurate yet local construction (via DSS)

– globally conservative = good for hydrostatic primitive equations

• Theoretically optimal for self-adjoint operators (elliptic equations)

– Excellent for incompressible Navier-Stokes

– also extremely good for non-self-adjoint operators (hyperbolic equations)

• Simple to construct efficient semi-implicit time-integrators

– Semi-implicit = nonlinear terms are explicit and linear terms are implicit

Discontinuous Galerkin Methods

• High order like Continuous Galerkin

• Completely local in nature

– no global assembly/DSS required as in CG (truly local)

• High order generalization of the FV

– locally and globally conservative = excellent choice for non-hydrostatic equations (mesoscale models)

– upwinding and BCs implemented naturally (via Riemann solvers)

• Designed for hyperbolic equations (i.e., shock waves)

– Simple provision for elliptic equations (e.g., LDG)

• Not so straightforward to construct efficient semi-implicit time-integrators, due to the inherent nonlinear nature of the numerical flux, until recently (Restelli-Giraldo, SIAM J. Sci. Comp. and Giraldo-Restelli, Int. J. Num. Meths Fluids)

Icosahedral TelescopingHexahedral

Icosahedral Adaptive Adaptive

Geometric Flexibility of EBG Methods

Adaptive

Quadrilaterals

Triangles

Banded

Convergence Rate for EBG Methods(DG Spatial Operators)

Note that the method achieves the expected convergence rate; that is, error =O(hN+1)

Scalability of EBG Methods(Surface Values for T185 L26 during 0-30 days)

Pressure Temperature

30 day simulation of a Baroclinic Instability for a fully 3D atmospheric model using 8th order polynomials

Scalability of EBG Methods IBM SP4 (1.7 GHz)

Note that the problem size from T249 L30 to T498 L60 has increased by a factor of 16 (22 hor x 2 vert x 2 time-steps). However, the Wallclock Time has only decreased by a factor of 8. Furthermore, NSEAM T498 L60 scales linearly with processor count! At T498 L60 NSEAM can theoretically accommodate 70,000 processors!

T249 (54 km) L30 DT=300 secs T498 (20 km) L60 DT=150 secs

III. Time-Integrators

Explicit Time-Integrators

• Let’s rewrite the governing equations as

• SSP-RK(2,3,4) For k=1,…,K

• SSP-BDF2Which we write as

or in general, more compactly, as

∂q

∂t= S(q)

qk =m=0

k−1

∑ αk,mqm+ Δtβk,mS qm( )( ), q0 =qn, qK =qn+1

qn+1 =α0qn +α1q

n−1 +γΔt β0 S(qn) + β1 S(qn−1)( )

qn+1 = αmm=0

K−1

∑ qn−m+γΔt βmm=0

K−1

∑ S(qn−m)

Semi-Implicit Time-Integrator(Building an implicit method on top of an explicit one)

• Let’s rewrite the governing equations as

• Now, if we knew the linear operator L, then we could write

• Discretizing by Kth order time-integrator yields

Where

and

∂q

∂t= S(q)

∂q

∂t= S(q) − δ L(q){ } + δ L(q)[ ]

qtt =q̂+ λL qtt( )

λ =δγΔt, qtt = ρmqn−m

m=−1

K−1

∑ , q̂ =ρ−1qexplicit + ρmqn−m

m=0

K−1

∑

qexplicit = αmm=0

K−1

∑ qn−m+γΔt βmm=0

K−1

∑ S(qn−m)

Semi-Implicit (IMEX) Time-Integrators (BDFK)(SWE: Linear Kelvin Wave)

Accuracy Performance

DG with N=10

BDFK Time-Integrators(Stability Regions)

Explicit Implicit

dq

dt=λq

Additive Runge-Kutta Time-Integrators(Stability Regions)

Explicit Implicit

dq

dt=λq

ARK3

RK3

RK2BDF3

ARK3

Semi-Implicit (IMEX) Time-Integrators(Constructing the Schur Complement)

• The implicit problem that we have is:

• Where the dims are:– For 2D Euler d=4, 3D d=5, etc.– For 2D Euler we solve a 16N2 system– For 2D SWE d=3, and solve a 9N2

• A better approach is to write out the system as follows (for 2D SWE):

• Applying block LU decomposition:

– Where the Schur Complement is:

– And the dimensions are:

I −λL( )qtt =q̂

A ∈RdNxdN , qtt ∈RdN , b∈RdN

A11 A12

A21 A22

⎛⎝⎜

⎞⎠⎟

utt

ϕ tt

⎛⎝⎜

⎞⎠⎟=

b1b2

⎛⎝⎜

⎞⎠⎟

Aqtt =b

A11 A12

0 A22 −A21A−111A12

⎛⎝⎜

⎞⎠⎟

utt

ϕ tt

⎛⎝⎜

⎞⎠⎟=

b1b2 −A−1

11A12b1

⎛⎝⎜

⎞⎠⎟

A22 −A21A−111A12( )ϕ tt =b2 −A−1

11A12b1 Aϕ tt =b

A ∈RNxN , ϕ tt ∈RN , b∈RN

Semi-Implicit (IMEX) Time-Integrators(Important Properties of Schur Complement)

• The original system is:

• The Schur Complement system is:

• This system is clearly smaller than the original system (NxN instead of 3Nx3N for 2D SWE).

• Equally important is that the Schur System is better conditioned than the original system. This means that fewer iterations are required by an iterative solver to reach convergence.

• Key Point: A semi-implicit method should be more efficient than the most efficient explicit methods, but with the addition of the Schur Complement system, the gains are much bigger. Thus, whenever possible we must strive to find such a system. This is not always very easy.

A22 −A21A−111A12( )ϕ tt =b2 −A−1

11A12b1

A11 A12

A21 A22

⎛⎝⎜

⎞⎠⎟

utt

ϕ tt

⎛⎝⎜

⎞⎠⎟=

b1b2

⎛⎝⎜

⎞⎠⎟

Semi-Implicit (IMEX) Time-Integrators(Which Methods offer a Schur Complement?)

• It turns out that it can be done as long as one can write the method in the IMEX form:

• Additive Runge-Kutta Methods can be written in the Schur Complement form. This now extends the order of accuracy in time as well as the maximum time-step allowed for efficiency (big gains in the explicit part for increasing K).

I −λL( )qtt =q̂

4. Fully-Implicit Time-Integrators

• Writing the governing equations as

• We then discretize this implicitly using implicit methods (eg., BDFK, IRK)

• Where the matrix problem is now– Which clearly requires a nonlinear

implicit solution(eg., Newton-Krylov methods)

• We write the problem as the functional:

• And then solve the nonlinear problem:

• With the resulting Linear System:

∂q

∂t= S(q)

F =qn+1 −γΔtS(qn+1)− αmm=0

K−1

∑ qn−m ≡0

F q( )(k+1) =F q( )(k) +∂F q( )(k)

∂qq(k+1) −q(k)( ) ≡0

∂F q( )(k )

∂qq(k +1) − q(k )( ) = −F q( )

(k )

Comparison of Various Time-Integrators(Rising Thermal Bubble)

x

z

IV. Results of the Three Models

IV. Results of the Three ModelsSE Global Hydrostatic Model

Collaborators: Y.J. Kim, Maria Flatau, Chi-Sann Liou, and Melinda Peng (NRL-Monterey)

ControlControl ControlControl

Day 0~180Day 0~180

Surface Temperature & Convective PrecipitationSurface Temperature & Convective PrecipitationAqua-Planet Experiments

ControlControl3.75° x 2.5° L193.75° x 2.5° L19

Neale & Hoskins (2001)Neale & Hoskins (2001)

180180 0W0W0E0E00 180180180180

ControlControl ControlControl

Convective Precipitation (-5°~5°)Convective Precipitation (-5°~5°)

NSEAM E6P8L20 (~2.2°~T54)NSEAM E6P8L20 (~2.2°~T54) UKMOUKMO

TomitaTomita

NICAM Cloud Resolving ModelNICAM Cloud Resolving Model

180180 0W0W0E0E

w/ reducedHyperviscosit

y

(μ =5e5)

IV. Results of the Three ModelsSE/DG NonHydrostatic Atmospheric Model

Collaborators: Matthias Läuter (AWI, Potsdam), Emil Constantinescu (Argonne NL, Chicago), Marco Restelli (MPI,Hamburg), Saša Gaberšek

(NRL, Monterey), Jim Doyle (NRL, Monterey), Jim Kelly (NPS)

Rising Thermal Bubble(nelx=nely=20 N=10, 5 m res, Time=700 seconds)

x

z

Potential Temperature Profile along x=500 meters

Inertia-Gravity Wave(nelx=120, nely=4 N=10, 250 m res, Time=3000 seconds)

x

z

Potential Temperature Profile along z=5000 meters

Density Current(nelx=128, nely=32 N=8, 250 m res, Time=900 seconds)

Potential Temperature

x

z

Profile along z=1200 meters

Linear Hydrostatic Mountain(nelx=20, nely=10 N=10, 250 m res, Time=10 hours)

x

z

Vertical VelocityMomentum FluxVertical Velocity

Linear Nonhydrostatic Mountain Waves(360 x 310 meter resolution with 10th order polynomials)

Vertical Velocity

Convective Storm Simulation(SE2NC-SIBDF2: 500 x 200 m res, N=10, Time=10 hours)

Vapor Rain

Clouds

IV. Results of the Three ModelsTriangular DG Coastal Ocean Model

Collaborators: Tim Warburton (Rice), Marco Restelli (MPI, Hamburg), Dimitrios Alevras (Hellenic Navy Hydrographic Service, Athens), Jörn

Behrens (Univ. of Hamburg)

Propagation of the 2004 Indian Ocean Tsunami(Grid Dimensions: Np=66715, Ne=130444, N=1, K=2, J=1)

Time evolution of the water surface height (grid data provided by J. Behrens, AWI-Bremerhaven, and data formatted by D. Alevras, NPS)

x

y

Propagation of the 2004 Indian Ocean Tsunami(Grid Dimensions: Np=66715, Ne=130444, N=1, K=2, J=1)

y

Riemann Problem on Sphere(Discontinuous Galerkin Shallow Water)

Height V-Velocity

U-Velocity

Current Work and Future Directions

• Continuous and Discontinuous EBG methods are a good choice for the development of next-generation GFD models

• To make this new class of models ready for operational use requires the introduction of good Time-Integrators such as semi-implicit, fully-implicit and Lagrangian methods

• Current Areas of Research are:– Time-Integration

• Semi-Implicit Runge-Kutta Methods (and Rosenbrock Methods)• Fully-Implicit Methods• Multi-rate methods based on extrapolation• Preconditioners

– Adaptivity • Conforming and non-conforming for triangles• Non-conforming for quadrilateral

– Physical Parameterizations• Continue testing simple Kessler microphysics• Positivity-preserving schemes for moisture• Extension to more complicated physics

Semi-Implicit (IMEX) Time-Integrators(Additive RK Methods)

• Starting with:

• We can then write this in the IMEX form:

• Discretizing by Kth order time-integrator yields

For k=1,…,K

Where

and

∂q

∂t= S(q)

∂q

∂t= S(q) − δ L(q){ } + δ L(q)[ ]

I −λL( )q(k)tt =q̂(k)

λ =a(k ,m )δΔt, q(k )

tt =q(k) +%a(k,m) −a(k,m)

%a(k,m) q(m)

m=1

k−1

∑ ,

q̂(k ) =qexplicit +

%a(k,m) −a(k,m)

%a(k,m) q(m)

m=1

k−1

∑