High-order Finite Elements for Computational Physics

High Order Finite Elements For Computational Physics:An LLNL Perspective

FEM 2012 Workshop

Estes Park, Colorado, June 4th, 2012

Robert N. Rieben

LLNL-PRES-559274

This work was performed under the auspices of theU.S. Department of Energy by Lawrence LivermoreNational Laboratory under contractDE-AC52-07NA27344. Lawrence Livermore NationalSecurity, LLC

2/24

Introduction Spatial Discretization Temporal Discretization Software Implementation Applications Conclusions

Acknowledgments

This is joint work with:

• P. Castillo

• V. Dobrev

• A. Fisher

• T. Kolev

• J. Koning

• G. Rodrigue

• M. Stowell

• D. White

LLNL-PRES-559274

3/24


Motivation

Our goal is to model, and ultimately predict, the behavior of complex physical systems bysolving a set of coupled partial differential equations

Our simulation codes must:

• work on general unstructured 2D and 3D meshes

• run on massively parallel computing architectures

• be adaptable and extendable

We like high order algorithms because they:

• minimize numerical dispersion

• maximize FLOPS/byte (well suited for exascale platforms)

• often lead to more robust algorithms

We have adopted a general approach for solving PDEs using high order finite elements that:

• is implemented in modular software libraries

• has been integrated into production multi-physics codes

• forms the basis for new research codes founded on high order methods

LLNL-PRES-559274

4/24


Weak Variational Formulation of Continuum PDE1. We begin with a set of continuum PDEs

2. We multiply each equation by a test function from a particular space

3. We integrate over the problem domain

4. We perform integration by parts

As an example, consider the coupled Ampere-Faraday equations:

ε∂~E

∂t= ∇× (µ−1~B)

∂~B

∂t= −∇× ~E

Each test function belongs to a particular function space. We use the concept of differentialforms as a convenient way to categorize and label these spaces:

Func. Space Diff. Form Interface Continuity Example Units

H(Grad) 0-form Total Scalar Potential, φ V /m0

H(Curl) 1-form Tangential Electric Field, ~E V /m1

H(Div) 2-form Normal Magnetic Field, ~B W /m2

L2 3-form None Material Density, ρ kg/m3

LLNL-PRES-559274P. Castillo, J. Koning, R. Rieben, D. White, “A discrete differential forms framework for computational electromagnetism,” CMES, 2004

4/24







ε∂~E

∂t· ~W 1 = ∇× (µ−1~B) · ~W 1, ~W 1 ∈ H(Curl)

∂~B

∂t· ~W 2 = −∇× ~E · ~W 2, ~W 2 ∈ H(Div)








4/24







RΩ(ε

∂~E

∂t· ~W 1) =

RΩ(∇× (µ−1~B) · ~W 1)

RΩ(∂~B

∂t· ~W 2) =

RΩ(−∇× ~E · ~W 2)








4/24







RΩ(ε

∂~E

∂t· ~W 1) =

RΩ(µ−1~B · ∇ × ~W 1)−

H∂Ω(µ−1~B × ~W 1 · n)

RΩ(∂~B

∂t· ~W 2) =

RΩ(−∇× ~E · ~W 2)








4/24







RΩ(ε

∂~E

∂t· ~W 1) =

RΩ(µ−1~B · ∇ × ~W 1)−

H∂Ω(µ−1~B × ~W 1 · n)

RΩ(∂~B

∂t· ~W 2) =

RΩ(−∇× ~E · ~W 2)








5/24


Finite Element ApproximationWe use a Galerkin finite element method to convert the variational form into a semi-discreteset of linear ODEs. We define a finite element by the 4-tuple (Ω,P,A,Q) where:

1. Ω is a reference element of a certain topology (e.g. unit cube or tetrahedron)

2. P is a finite element space defined on Ω

3. A is the set of degrees of freedom (linear functionals) dual to P4. Q is a quadrature rule defined on Ω

The element mapping defines the transformationfrom reference to physical space

Φ : x ∈ Ω 7→ x ∈ Ω

0 0.2 0.4 0.6 0.8 1

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 0.5 1 1.5

!0.2

0

0.2

0.4

0.6

0.8

1

1.2

x ∈ Ω x ∈ Ω

• Element topology is conceptuallyseparated from element geometryto support general curved elements

• We approximate all integrals bytransforming from physical spaceto reference space and applyingquadrature

• Well defined degrees of freedomare essential for implementingsource and boundary terms andfor normed error analysis

LLNL-PRES-559274

5/24


Finite Element ApproximationWe use a Galerkin finite element method to convert the variational form into a semi-discreteset of linear ODEs. We define a finite element by the 4-tuple (Ω,P,A,Q) where:

1. Ω is a reference element of a certain topology (e.g. unit cube or tetrahedron)

2. P is a finite element space defined on Ω

3. A is the set of degrees of freedom (linear functionals) dual to P4. Q is a quadrature rule defined on Ω

The element mapping defines the transformationfrom reference to physical space

Φ : x ∈ Ω 7→ x ∈ Ω

0 0.2 0.4 0.6 0.8 1

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 0.5 1 1.5

!0.2

0

0.2

0.4

0.6

0.8

1

1.2

x ∈ Ω x ∈ Ω

• Element topology is conceptuallyseparated from element geometryto support general curved elements

• We approximate all integrals bytransforming from physical spaceto reference space and applyingquadrature

• Well defined degrees of freedomare essential for implementingsource and boundary terms andfor normed error analysis

LLNL-PRES-559274

6/24


Basis Functions and Discrete Differential Forms

We define a discrete differential form as a basis function expansion using a basis that spans asubpsace of the continuum function space:

f ≈ Πl (f ) ≡PDim(P l )

i Ali (f ) W l

i , l ∈ 0, 1, 2, 3

Example Transformation: We define a basis of arbitrary polynomial degree on the referenceelement Ω and use the element mapping Φ to transform W l andits derivative dW l to real space:

Basis: W l Derivative of Basis: dW l

0-forms W 0 Φ ∂Φ−1(∇W 0 Φ)

1-forms ∂Φ−1(W 1 Φ) 1|∂Φ|∂ΦT (∇× W 1 Φ)

2-forms 1|∂Φ|∂ΦT (W 2 Φ) 1

|∂Φ| (∇ · W2 Φ)

3-forms 1|∂Φ| (W 3 Φ) —

The order of the basis p and element mapping s are chosenindependently

LLNL-PRES-559274P. Castillo, R. Rieben, D. White, “FEMSTER: An object oriented class library of high-order discrete differential forms,” ACM-TOMS, 2005.

6/24





i Ali (f ) W l

i , l ∈ 0, 1, 2, 3

Example Transformation:

l = 1, p = 2, s = 1

We define a basis of arbitrary polynomial degree on the referenceelement Ω and use the element mapping Φ to transform W l andits derivative dW l to real space:


0-forms W 0 Φ ∂Φ−1(∇W 0 Φ)

1-forms ∂Φ−1(W 1 Φ) 1|∂Φ|∂ΦT (∇× W 1 Φ)

2-forms 1|∂Φ|∂ΦT (W 2 Φ) 1

|∂Φ| (∇ · W2 Φ)

3-forms 1|∂Φ| (W 3 Φ) —



6/24





i Ali (f ) W l

i , l ∈ 0, 1, 2, 3

Example Transformation:

l = 2, p = 1, s = 2

We define a basis of arbitrary polynomial degree on the referenceelement Ω and use the element mapping Φ to transform W l andits derivative dW l to real space:


0-forms W 0 Φ ∂Φ−1(∇W 0 Φ)

1-forms ∂Φ−1(W 1 Φ) 1|∂Φ|∂ΦT (∇× W 1 Φ)

2-forms 1|∂Φ|∂ΦT (W 2 Φ) 1

|∂Φ| (∇ · W2 Φ)

3-forms 1|∂Φ| (W 3 Φ) —



7/24


Matrix OperatorsWe define symmetric matrix operators using bilinear forms:

Mass Matrix: (Mlα)ij ≡

RΩ αW l

i ∧W lj

Stiffness Matrix: (Slα)ij ≡

RΩ αdW l

i ∧ dW lj

We also define rectangular matrix operators using mixed bilinear forms:

Hodge “Star” Matrix: (Hl,m)ij ≡R

Ω W li ∧W m

j

Derivative Matrix: (Dl,mα )ij ≡

RΩ αdW l

i ∧W mj

Topological Derivative Matrix: (Kl,m)ij ≡ Ami (dW l

j )

Derivativematrices are mesh

dependent firstorder differential

operators

Dl,mα ≡ Mm

α Kl,m

D0,1 ≈ GradD1,2 ≈ CurlD2,3 ≈ Div

Stiffness matricesare mesh

dependent secondorder differential

operators

Slα ≡ (Kl,m)T Mm

α Kl,m

S0 ≈ Div-GradS1 ≈ Curl-CurlS2 ≈ Grad-Div

LLNL-PRES-559274

7/24




RΩ αW l

i ∧W lj


RΩ αdW l

i ∧ dW lj



Ω W li ∧W m

j


RΩ αdW l

i ∧W mj


j )



operators

Dl,mα ≡ Mm

α Kl,m




operators

Slα ≡ (Kl,m)T Mm

α Kl,m


LLNL-PRES-559274

7/24




RΩ αW l

i ∧W lj


RΩ αdW l

i ∧ dW lj



Ω W li ∧W m

j


RΩ αdW l

i ∧W mj


j )



operators

Dl,mα ≡ Mm

α Kl,m




operators

Slα ≡ (Kl,m)T Mm

α Kl,m


LLNL-PRES-559274

8/24


Discrete DeRahm Complex: Operator Range and Null Spaces

The topological matrix operators and finite element spaces satisfy a discrete DeRahmcomplex:

H(Grad)∇−→ H(Curl)

∇×−→ H(Div)∇·−→ L2

↓ Π0 ↓ Π1 ↓ Π2 ↓ Π3

P0 K0,1

−→ P1 K1,2

−→ P2 K2,3

−→ P3

This ensures that our discrete operators have the correct range and null spaces, which iscritical for preventing spurious modes

• 2 elements

• 12 nodes

• 20 edges

• 11 faces

Polynomial Basis Degree p = 1

K1,2 K0,1 = 0LLNL-PRES-559274

8/24





∇×−→ H(Div)∇·−→ L2

↓ Π0 ↓ Π1 ↓ Π2 ↓ Π3

P0 K0,1

−→ P1 K1,2

−→ P2 K2,3

−→ P3


• 2 elements

• 12 nodes

• 20 edges

• 11 faces


K2,3 K1,2 = 0LLNL-PRES-559274

8/24





∇×−→ H(Div)∇·−→ L2

↓ Π0 ↓ Π1 ↓ Π2 ↓ Π3

P0 K0,1

−→ P1 K1,2

−→ P2 K2,3

−→ P3


• 2 elements

• 12 nodes

• 20 edges

• 11 faces


K1,2 K0,1 ≈ 10−12

LLNL-PRES-559274

8/24





∇×−→ H(Div)∇·−→ L2

↓ Π0 ↓ Π1 ↓ Π2 ↓ Π3

P0 K0,1

−→ P1 K1,2

−→ P2 K2,3

−→ P3


• 2 elements

• 12 nodes

• 20 edges

• 11 faces


K2,3 K1,2 ≈ 10−12

LLNL-PRES-559274

9/24


High Order Time Integration

We apply high order, explicit and implicit time stepping methods to the semi-discreteequations, resulting in a fully-discrete method:

ε∂~E

∂t= ∇× (µ−1~B)

∂~B

∂t= −∇× ~E

−→ M1ε

∂e

∂t= (K1,2)T M2

µb

∂b

∂t= −K1,2e

Example: Explicit Symplectic Maxwell Algorithm

for n = 1 to nstep doeold = en

bold = bn

for j = 1 to order doenew = eold + βj ∆t (M1

ε)−1(K1,2)T M2

µbold

eold = enew

bnew = bold − αj ∆t K1,2eold

bold = bnew

end foren+1 = enew

bn+1 = bnew

end for

Order k = 1

α1 = 1 β1 = 1

Order k = 2

α1 = 1/2 β1 = 0

α2 = 1/2 β2 = 1

Order k = 3

α1 = 2/3 β1 = 7/24

α2 = −2/3 β2 = 3/4

α3 = 1 β3 = −1/24

Order k = 4

α1 = (2 + 21/3 + 2−1/3)/6 β1 = 0

α2 = (1 − 21/3 − 2−1/3)/6 β2 = 1/(2 − 21/3)

α3 = (1 − 21/3 − 2−1/3)/6 β3 = 1/(1 − 22/3)

α4 = (2 + 21/3 + 2−1/3)/6 β4 = 1/(2 − 21/3)

LLNL-PRES-559274R. Rieben, D. White, G. Rodrigue, “High order symplectic integration methods for finite element solutions to time dependent Maxwell

equations,” IEEE-TAP,2004

9/24


High Order Time Integration

We apply high order, explicit and implicit time stepping methods to the semi-discreteequations, resulting in a fully-discrete method:

ε∂~E

∂t= ∇× (µ−1~B)

∂~B

∂t= −∇× ~E

−→ M1ε

∂e

∂t= (K1,2)T M2

µb

∂b

∂t= −K1,2e

Example: Explicit Symplectic Maxwell Algorithm

for n = 1 to nstep doeold = en

bold = bn

for j = 1 to order doenew = eold + βj ∆t (M1

ε)−1(K1,2)T M2

µbold

eold = enew

bnew = bold − αj ∆t K1,2eold

bold = bnew

end foren+1 = enew

bn+1 = bnew

end for

Order k = 1

α1 = 1 β1 = 1

Order k = 2

α1 = 1/2 β1 = 0

α2 = 1/2 β2 = 1

Order k = 3

α1 = 2/3 β1 = 7/24

α2 = −2/3 β2 = 3/4

α3 = 1 β3 = −1/24

Order k = 4

α1 = (2 + 21/3 + 2−1/3)/6 β1 = 0

α2 = (1 − 21/3 − 2−1/3)/6 β2 = 1/(2 − 21/3)

α3 = (1 − 21/3 − 2−1/3)/6 β3 = 1/(1 − 22/3)

α4 = (2 + 21/3 + 2−1/3)/6 β4 = 1/(2 − 21/3)

LLNL-PRES-559274R. Rieben, D. White, G. Rodrigue, “High order symplectic integration methods for finite element solutions to time dependent Maxwell

equations,” IEEE-TAP,2004

10/24


Software Implementation

We use object oriented methods to implement finite element concepts in modular libraries

• FEMSTER (Castillo, Rieben, White)• Developed under LLNL LDRD program, no longer in public domain• Arbitrary order element geometry and basis functions• Operates at element level, client code responsible for global (parallel) assembly

• mfem (Dobrev, Kolev)• LLNL’s state of the art, open source, finite element library: mfem.googlecode.com• Originated by Tzanio Kolev and Veselin Dobrev at Texas A&M University• Arbitrary order element geometry and basis functions, including NURBS• Supports global, parallel matrix assembly on domain decomposed meshes via HYPRE

We have production and research simulation codes which utilize the FEM libraries

• EMSolve: parallel, high order electromagnetic wave propagation and diffusion

• ALE3D: parallel, multi-material ALE hydrodynamics + resistive MHD

• ARES: parallel, multi-material ALE radiation-hydrodynamics + resistive MHD

• BLAST: parallel, high order multi-material Lagrangian hydrodynamics

Each code uses LLNL’s HYPRE library for linear solver operations

LLNL-PRES-559274

http://mfem.googlecode.com

11/24


Application: Electromagnetic Wave Propagation

• Physics Code: EMSolve

• FEM Library: FEMSTER

• PDEs: coupled Ampere-Faraday laws or dissipative electric field wave equation

• Spatial Discretization: high order, curvilinear, unstructured

• Temporal Discretization: high order symplectic or simple forward Euler

Continuum PDEs

Coupled Ampere-Faraday Laws:

ε∂~E

∂t= ∇× (µ−1~B)

∂~B

∂t= −∇× ~E

Dissipative Electric Field Wave Equation:

ε∂2~E

∂t2= −∇× (µ−1∇× ~E)− σ

∂~E

∂t

Semi-Discrete Finite Element Method

Coupled Ampere-Faraday Laws:

M1ε

∂e

∂t= (K1,2)T M2

µb

∂b

∂t= −K1,2e

Dissipative Electric Field Wave Equation:

M1ε

∂2e

∂t2= −S1

µe−M1σ

∂e

∂t

LLNL-PRES-559274

12/24


Example: Coaxial Waveguide

p = 1, s = 1, fine p = 2, s = 2, coarse

This example pre-dates our ability to plot highorder field and mesh data!

High-order methods excel at minimizing phaseerror / numerical dispersion

0 20 40 60 80 1000

0.05

0.1

0.15

0.2

0.25

Time

Max

( ||E

−Eh|| 2 )

p=1, s=1, fine meshp=2, s=2, coarse meshlinear fitlinear fit

0 10 20 30 40 50 60 70 80 90 100 110

−2.5

−2

−1.5

−1

−0.5

0

0.5

Propagation Distance

log

10(

||E−E

h|| 2 )

p=1, s=1, k=1p=2, s=2, k=1p=3, s=2, k=1p=3, s=2, k=3linear fitlinear fitlinear fitlinear fit

LLNL-PRES-559274R. Rieben, G. Rodrigue, D. White, “A high order mixed vector finite element method for solving the time dependent Maxwell equations on

unstructured grids,” JCP 2005.

13/24


Example: Photonic Crystal Waveguides

High-order space discretization (p = 3) enablessingle element PML

LLNL-PRES-559274R. Rieben, D. White, G. Rodrigue, “Application of novel high order time domain vector finite element method to photonic band-gap

waveguides,”. Proc. IEEE TAP Sym. 2004.

13/24


Example: Photonic Crystal Waveguides

High-order time discretization (k = 3) reducesenergy error amplitude

k = 1 k = 3

0.07 0.08 0.09 0.1 0.11 0.12 0.13Time -ps-

1.4537

1.4538

1.4539

Ene

rgy

0.07 0.08 0.09 0.1 0.11 0.12 0.13Time -ps-

1.45384

1.45385

1.45386

Ene

rgy

LLNL-PRES-559274R. Rieben, D. White, G. Rodrigue, “Application of novel high order time domain vector finite element method to photonic band-gap

waveguides,”. Proc. IEEE TAP Sym. 2004.

14/24


Example: RF Signal Attenuation in a Cave200 MHz loop antenna in a smooth cave

Cartesian mesh Cylindrical mesh200 MHz loop antenna in a random rough cave

LLNL-PRES-559274J. Pingenot, R. Rieben, D. White, D. Dudley, “Full wave analysis of RF signal attenuation in a lossy rough surface cave using a high order time

domain vector finite element method,” JEMWA 2006.

15/24


Application: Electromagnetic Diffusion• Physics Code: EMSolve

• FEM Library: FEMSTER

• PDEs: H-field or scalar-vector potential diffusion formulations


• Temporal Discretization: generalized Crank-Nicholson

Continuum PDEs

H-Field Diffusion Formulation:

µ∂~H

∂t= −∇× (σ−1∇× ~H)

~B = µ~H~J = ∇× ~H

Vector Potential Diffusion Formulation:

∇ · σ∇φ = 0

σ∂~A

∂t= −∇× (µ−1∇× ~A)− σ∇φ

~B = ∇× ~A~J = −σ(∇φ+ ∂~A

∂t)


H-Field Diffusion Formulation:

M1µ

∂h

∂t= −S1

σh

M2µb = H1,2h

j = K1,2h

Vector Potential Diffusion Formulation:

S0σv = f

M1σ

∂a

∂t= −S1

µa−D0,1σ v

b = K1,2a

M2σ j = −H1,2(K0,1v + ∂a

∂t)

LLNL-PRES-559274

16/24


Example: Conducting Cylinder at Rest

Analytic solutions check error convergence in space

Orthogonal mesh Skewed mesh

LLNL-PRES-559274R. Rieben, D. White, “Verification of high-order mixed finite-element solution of transient magnetic diffusion problems,”IEEE TMag, 2006.

17/24


Example: Conducting Sphere at Rest

Analytic solutions check error convergence in time

Snapshot of transient ~A field in conducting sphere

LLNL-PRES-559274R. Rieben, D. White, “Verification of high-order mixed finite-element solution of transient magnetic diffusion problems,”IEEE TMag, 2006.

18/24


Application: ALE Magnetohydrodynamics

• Physics Codes: ALE3D and ARES

• FEM Library: FEMSTER (or FEMSTER-like routines)

• PDEs: Operator split Lagrangian magnetic diffusion and Eulerian magnetic advectioncoupled to Euler equations

• Spatial Discretization: low order (p = 1), tri-linear hexahedral (Q1), unstructured

• Temporal Discretization: implicit backward Euler

Continuum PDEs

Lagrangian Magnetic Diffusion:

∇ · σ∇φ = 0

σd~E

dt= ∇× (µ−1~B) + σ∇φ

d~B

dt= −∇× ~E

Eulerian Magnetic Advection:

∂~B

∂t= −∇× ~vm × ~B


Lagrangian Magnetic Diffusion:

S0σv = f

M1σ

de

dt= (K1,2)T M2

µb + D0,1σ v

db

dt= −K1,2e

Eulerian Magnetic Advection:

∂b

∂t= −K1,2euw

LLNL-PRES-559274R. Rieben, D. White, B. Wallin, J. Solberg, “An arbitrary Lagrangian-Eulerian discretization of MHD on 3D unstructured grids,” JCP 2007.

19/24


Example: Rotating Conducting Cylinder

We consider a rotating, conducting cylinder immersed in an external magnetic field:

computed FEM solution

0 2 4 6 8 10 12 14Time

0.5

0.0

0.5

1.0

1.5

Magneti

c Fi

eld

Exact Solution3DMHD code

Comparison to exact solution

A time and space dependent analytic solution to this problem has been derived

LLNL-PRES-559274D. Miller, J. Rovny, “Two-Dimensional Time Dependent MHD Rotor Verification Problem” LLNL-TR, 2011

20/24


Example: Explosively Driven Magnetic Flux Compression Generator

We consider a flat plate magnetic flux compression generator, an inherently 3D problem thatrequires a 3D MHD simulation code

Materials Pressure contours andmagnetic field magnitude

This is a complex multi-material, multi-physics, ALE calculation

LLNL-PRES-559274J. Shearer et. al., “Explosive-Driven Magnetic-Field Compression Generators,” J. App. Phys, 1968

21/24


Application: Multi-Material Lagrangian Hydrodynamics

• Physics Code: BLAST

• FEM Library: mfem

• PDEs: Euler equations in Lagrangian (moving) frame


• Temporal Discretization: energy conserving, explicit, high order RK

Continuum PDEs

Momentum: ρd~v

dt= ∇ · σ

Mass: 1ρ

dρ

dt= −∇ · ~v

Energy: ρde

dt= σ : ∇~v

Motion:dx

dt= ~v

Eq. of State: σ = −EOS(ρ, e) I


Fij ≡R

Ω(t)(σ : ∇ ~W 0i )W 3

j

M0ρ

dv

dt= −F · 1

M3ρ

de

dt= FT · v

dx

dt= v

LLNL-PRES-559274V. Dobrev, T. Kolev, R. Rieben, “High-order curvilinear finite element methods for Lagrangian hydrodynamics,” SISC, in review, 2011.

22/24


Example: Rayleigh-Taylor InstabilityQ4-Q3 FEM solution

High order Lagrangian methodsbetter resolve complex flow

t = 3.0 t = 4.0 t = 4.5 t = 5.0

Q1-Q0

Q2-Q1

Q8-Q7

LLNL-PRES-559274

23/24


Example: Multi-Material Shock Triple Point

We consider a 3 materialRiemann problem in both2D axisymmetric and full

3D geometries

High order mesh and fieldvisualization is essential

GLVis (Dobrev, Kolev) isused for high order data

visualization

glvis.googlecode.com

LLNL-PRES-559274

http://glvis.googlecode.com

24/24


Conclusions

High order finite elements offer many benefits for computational pysics,including:

• better convergence (smooth problems)

• lower dispersion errors

• greater FLOPS/byte ratio for advanced architectures

• more robust algorithms for Lagrangian methods

Our general, high order discretization approach has several practicaladvantages, including:

• flexibility with respect to choice of continuum PDEs

• generality with respect to space and time discretization order

• ability to use curvilinear elements

LLNL-PRES-559274

Documents

High-order Finite Elements for Computational Physics