Development and applications of HiFi â€“ adapti e implicit high

Development and applications of HiFi –adapti e implicit high order finite elementadaptive, implicit, high order finite element

code for general multi-fluid applications.

Vyacheslav (Slava) Lukin

C ll b t Al H Gl W t L i E i M iCollaborators: Alan H. Glasser, Weston Lowrie, Eric Meier

Plasma Science and Innovation Center (PSI-Center)University of Washington, Seattle, WA, USAUniversity of Washington, Seattle, WA, USA

* This research is supported, in part, by the U.S. DOE Fusion Energy Postdoctoral Fellowship.

Recent Advances in Parallel Implicit Solution of Fluid Plasma SystemsSIAM Conference on Computational Science and Engineering

Miami, Florida, March 2-6, 2009

Outline

Chronology, Motivation, Code Overview

Formulation of the PDE System to be Discretized

Spatial Discretization of an Arbitrary Physical Domainp y y

Temporal Discretization and Advance

Available Boundary Condition OptionsAvailable Boundary Condition Options

I/O, Post-Processing and Visualization

V ifi i d O i P d i Si l iVerification and Ongoing Production Simulations

V. S. Lukin, et. al., “Development and applications of HiFi – ....”SIAM Conference on Computational Science and Engineering, Miami, Florida, March 2-6, 2009

History of the Project

Originally envisioned and the core organization structure formulated by Alan H. Glasser, with contributions from X.Z. Tang at LANL in 2002-03;H. Glasser, with contributions from X.Z. Tang at LANL in 2002 03;

HiFi precursor -- 2D SEL code brought to maturity, tested and utilized for strongly non-linear Extended MHD simulations by Lukin, in collaboration with

l ( ) d d ( ) d ib i f dGlasser (LANL) and S.C. Jardin (PPPL), and contributions from Andrei N. Simakov (LANL) in 2003-07;

Development of HiFi as well as continued improvements to 2D SEL haveDevelopment of HiFi, as well as continued improvements to 2D SEL, have been ongoing at the PSI-Center, University of Washington since December 2007 with a small team consisting of Lukin, Glasser, Weston Lowrie and Eric Meier, with contributions from Masahiko Sato (NIFS, Japan), George J. Marklin and U Sh l kUri Shumlak.


Motivation: Technical

(Predictive) modeling of many real-life plasma experiments requires allowing for their generally non-axisymmetric geometry and self-consistent g g y y g ytreatment of boundary conditions;

For many, it is necessary to model not only two-fluid ion-electron plasma dynamics but also their interaction with neutral gas and hot particle kineticdynamics, but also their interaction with neutral gas and hot particle kinetic effects;

Accurate and well resolved simulations are necessary to capture dissipation-sensitive processes such as dynamic self organization;sensitive processes, such as dynamic self-organization;

Large separation between dynamically relevant and CFL imposed time-scales naturally leads to implicit numerical methods;

Of course, the code has to be maximally robust, efficient and scalable;


Motivation: OrganizationalThe code has to:

Be able to take advantage of new algorithms and libraries that are continuously being developed in the applied mathematics community;

Allow for flexible and straight-forward specification of the equations, initial and boundary conditions of interest to the user;

Be usable by experimenters and graduate students with time to learn toBe usable by experimenters and graduate students, with time to learn to independently use the code much shorter than time to complete a PhD degree;

Be usable for:) Ed i la) Educational purposes;

b) Development and testing of semi-empirical and numerical models of physical processes, where exact analytical description is not known or accessible;accessible;

c) Massively parallel production simulations that lead to new physical understanding of complex non-linear systems and, ultimately, to predictive modeling useful for experimental design.p g p g


HiFi Overview

Algorithm:

Fortran 90/95; Modular; Parallelized with MPI and PETSc;

PDE Formulation: generic flux-source form of the equations;

Spatial discretization: high order C0 spectral elements – very low numerical dispersion + geometric flexibility + adaptable grid + domain decomposition preconditioning;

Time step: fully implicit, 2nd-order accurate, Newton iteration, direct or iterative solvers + multiple algebraic preconditioner options available in PETSc + physics based preconditioning (under development);

Adaptivity: robust and automated in time – based on the rate of Newton convergence, and in space – based on the spatial convergence error and use of harmonic map generation (needs extension to full 3D);

I/O: parallelized with the portable HDF5 data format;


HiFi Overview

User Interface:

Main algorithm compiled into a library that is transparently used by theMain algorithm compiled into a library that is transparently used by the user-specified physics file constructed according to a provided generic template;

i l id i d hi i h CPre-Processing: external grid generation and smoothing with CUBIT package + advanced grid quality evaluation (under development) + check for some of the common mistakes in user-constructed physics file;

Post-Processing: parallelized conversion of check-point datafiles into data on a physical grid + user-specified integral data diagnostics;

Visualization: external VisIt package for up-to 3D data and XDRAW for up-Visualization: external VisIt package for up to 3D data and XDRAW for upto 2D data;


Flux-Source Form: Definition

Fluxes and sources are arbitrary nonlinear functions of dependent variables and their gradients; A(x) and B(x) are arbitrary functions of space;

Allows to specify varying sets of PDEs on a single computational domain, using time and/or location and/or values of dependent variables to decide which PDEs to evolve at each location at each moment in time;

Allows for dynamically evolving computational grid with moving domain boundary;

Wide range of applicability far beyond magnetized plasma physicsWide range of applicability, far beyond magnetized plasma physics.


Flux-Source Form: Examples

Helical Incompressible Hall MHD( ) 2Dtρ ρ ρ∂

+ ∇ = ∇∂

v

( )r jt φ

ψ η φ∂ ⎡ ⎤= − ×⎣ ⎦∂v B

1 p γ⎛ ⎞∂

Visco-resistive Compressible MHD

( )

( )

// //

2

11 1

:

p p T Tt

p jφ

γ κ κ κγ γ

η µ

⊥ ⊥

⎛ ⎞∂+ ∇ − − ∇ + ∇⎡ ⎤⎜ ⎟⎣ ⎦− ∂ −⎝ ⎠

⎡ ⎤= ∇ + + ∇ ∇ + ∇⎣ ⎦

v

v v v v T

( ) ( )2

02

Bpt

ρρ µ

⎛ ⎞∂ ⎛ ⎞ ⎡ ⎤+ ∇ + + − − ∇ + ∇ =⎜ ⎟⎜ ⎟ ⎣ ⎦∂ ⎝ ⎠⎝ ⎠

vvv I BB v v

t T

ˆ( ) ˆjφ φ= ∇ × B where

1 1ˆ ˆ, r zr z r r

ψ ψ∂ ∂= −

∂ ∂B , ˆ ˆ, r zv r v z=v ,

and axisymmetry is assumed ( 0φ∇ → ).

Dozens of different sets of PDEalready implemented

and countingEuler Fluid Eqns.

Euler Lagrange Eqn Anisotropic Heat Conduction Eqn.

and counting…

Euler-Lagrange Eqn.( ) 0T T S

t∂

+ ∇ • •∇ = =∂

Dt

where ( ) ( ) ( ) ( ) ( )

( ) ( )

2 2// //

2 2//

cos sin sin cos. sin cos

D D D Dsymm D D

⊥ ⊥

⊥

⎛ ⎞Φ + Φ − Φ Φ= ⎜ ⎟⎜ ⎟Φ + Φ⎝ ⎠

Dt

.

p q


Spatial Representation: Basics

High order spectral element representation of dependent variables in all spatial dimensions on a block-structured quadrilateral/hexahedral grid in logical space;

Mapping from logical space to physical space is provided using the same spectral element set of basis functions;p ;


Spatial Representation: Grid Generation

Initial logical-to-physical domain mapping is specified either analytically, or using the CUBIT automatic grid generator;


Spatial Representation: Adaptation

Two-dimensional static r-refinement, i.e. adaptation of the logical-to-physical domain mapping whenever deemed necessary and according to a well defined directional spatial convergence error estimator, is already available;

The same core code used to solve the primary set of PDEs is also used to solveThe same core code used to solve the primary set of PDEs is also used to solve for the new grid mapping – make use of the flux-source form!

Extension to fully three-dimensional adaptation is straightforward pending an elegant formulation of a 3D quasi-smooth error estimator to be fully functionalelegant formulation of a 3D quasi smooth error estimator to be fully functional.


Spatial Representation: Plans & Goals

Presently, the codes use C0 spectral element representation by explicitly enforcing continuity of the solution, but not its derivatives, across neighboring grid cells;

HiFi data structure also allows for non-trivial but straightforward extension of the codes’ capabilities to compute solutions requiring discontinuous spatial representation by coupling the existing global implicit solve with an appropriate Ri lRiemann solver;

Next development goal – allow for arbitrary connectivity of multiple hexahedral structured blocks.


Time Advance: Implicit Formulation

Rewrite the flux-source set of PDEs in the weak form and end up with a nonlinear time-dependent matrix-vector equation;

Discretize in time and Taylor expand the solution for each next time-step aroundDiscretize in time and Taylor expand the solution for each next time step around the solution for the previous time-step(s);

Two adaptive implicit time-advance schemes presently available:1 Θ-scheme:1. Θ scheme:

θ = 0.5 – Crank-Nicholson, 2nd order accurate, non-dissipative;θ = 1. – backward Euler, 1st order accurate, dissipative;

2. BDF2 – 2nd order accurate, dissipative for high frequency perturbations;, p g q y p ;

Jacobian matrix J, which is generally necessary to complete a Newton iteration, is computed using an analytic calculation of derivatives of the residual R with respect to all dependent variables;p p

Each Newton iteration requires solving a large linear matrix-vector equation:


Time Advance: Newton Iteration I

The linear system can be reduced/preconditioned by taking advantage of cell-wise locality of the high order (higher than linear) basis functions:

• static condensation preconditioning is available in both 2D and 3D codes, works very robustly in all cases, reduces the size of the system by a factor of np – the polynomial order of the basis set;

• FETI-DP preconditioning is available in 2D and works well for symmetric parabolic problems; further development of the method within HiFi is conditional on its generalization for asymmetric systems;

Additionally, following Luis Chacon’s work, an option to use physics-based preconditioning with user specified Schur complement matrix has been recently incorporated into the code. This is an ongoing effort showing promising results on li t t bl ith if b k d It i t b l t d i thlinear test problems with uniform background. It remains to be evaluated in the context of HiFi by simulating realistic non-linear and highly non-uniform experimentally relevant plasmas.


Time Advance: Newton Iteration II

Parallel solve of the reduced linear system(s) is accomplished by making use of the Portable Extensible Toolkit for Scientific computation (PETSc) library;

PETSc allows for nearly effortless access to a number of additional problem independent preconditioners (e.g. global Block Jacobi or Additive Schwartz with overlap and local LU/ILU sub-preconditioning), both iterative and direct linear solvers (e.g. flexible GMRES or SuperLU_dist), and other linear algebra libraries such as Hypre;

Newton iterations are determined to have converged when some criteria on theNewton iterations are determined to have converged when some criteria on the smallness of the residual R has been met;

Advanced non-linear Newton solver available through PETSc, SNES Solve, has demonstrated superior performance with fast and robust non linear convergence anddemonstrated superior performance with fast and robust non-linear convergence and is now used exclusively.


Boundary Condition Options

three general categories of boundary conditions (BC) + periodic BC:

• li it l l BC h th l ti t ti f l l l li• explicit local BC, where the solution must satisfy a local general non-linear time-dependent equation of the form:



• Flux BC – specify the normal flux Fn through the boundary of the domain:

where is the user specified boundary flux. Whenever it is equal to the flux given by the interior PDE, “natural” BC result, which implies that boundary g y p yvalues for all dependent variables on which such flux depends are already specified by other boundary conditions;



• integral BC – which impose a restriction of some type on the integral of the solution along a given face or edge of the logical domain and therefore globally couple the degrees of freedom along that boundary. An example of such BC which is presently implemented in the code is the “polar” r = 0 boundary condition;condition;

l i l t ti f i t l BC• general implementation of integral BC may be desired and pursued in the future.


I/O and Visualization

Read, write and post-process data in parallel using the portable HDF5 format;

Use both XDRAW and VisIt packages for data visualization purposes:Use both XDRAW and VisIt packages for data visualization purposes:


Verification: 2D SEL

2D SEL code has been thoroughly tested in linear and non-linear regimes g y gagainst analytics, multiple published simulation results, and other macroscopic modeling codes in both compressible and incompressible, single and two-fluid regimes [V.S. Lukin, PhD Dissertation, Princeton University, (2007)];

Recent (spring of 2008) verification study against NIMROD code for an identical FRC translation problem with identical resolution on the same machine has demonstrated good quantitative agreement of results; comparison of run-times g q g pshowed SEL to have comparable yet favorable performance (note that no problem-specific code tuning was done before running either code, as in both cases the test was performed by users with no knowledge of the codes’ inner algorithm layouts);


Verification: HiFi

Since HiFi is a straightforward extension of SEL and the 3rd dimension is exactly the same as the other two, there is no reason to expect any deterioration in i l i Sits accuracy relative to SEL;

Linear anisotropic heat equation calculations in highly distorted domains demonstrate the accuracy of the spatial discretization and the logical-to-physical

imapping;

Linear visco-resistive MHD wave tests have been performed and show high accuracy of the time-advance for both wave propagation and dissipation;

Non-linear visco-resistive Hall MHD simulations of a spheromak tilt mode have been performed and show that the final relaxed state of the spheromak is in precise agreement with that predicted by an independent Taylor-state calculation with an i l / i PSI TET d b G M klieigenvalue/eigenvector PSI-TET code by George Marklin.


Verification: anisotropic heat equation

Anisotropic linear heat equation with a time-dependent source at the bottom of the domain and insulator boundary conditions elsewhere:


Verification & Production: 3D Hall MHD


Verification & Production: 3D Hall MHDSpheromak tilt simulations in a short can

Initial condition in a cylinder of height y gL = 2, radius r0=1, with initially uniform density ρ0=1, pressure p0 and

J/B ≡ λ = 4 138J/B ≡ λ = 4.138

Ab initio calculations shown below were performed with α = 0, η=0, µ=5.e-2

d 1 1and κ=1.e-1;

Perfect conductor, perfect slip, heat insulator solid wall boundary conditions and initial rotation perturbation around the y-axis are applied to the problem.


Verification & Production: 3D Hall MHDSpheromak tilt simulations in a short can

Result of an ab initio spheromak tiltTaylor-state with zero pressure generated by PSI-TET code

λ = 3.978

Result of an ab initio spheromak tilt calculation with p0=0 (no density/pressure

evolution), di=0, di2ν=1.e-5,

λav = 3.978av


A Verification & Production: 3D Hall MHD


Validation & Predictive Simulations

That is the ultimate goal of the project!

Cannot be done without proper physical model, geometry, and boundary conditions

being applied to the problem.


Documents

Development and applications of HiFi â€“ adapti e implicit high