default

Crustal Deformation Modeling TutorialReview of PyLith Capabilities and Features

Brad AagaardCharles WilliamsMatthew Knepley

June 24, 2013

default

Crustal Deformation ModelingElasticity problems where geometry does not change significantly

Quasi-static modeling associated with earthquakes

Strain accumulation associated with interseismic deformationWhat is the stressing rate on faults X and Y?Where is strain accumulating in the crust?

Coseismic stress changes and fault slipWhat was the slip distribution in earthquake A?How did earthquake A change the stresses on faults X and Y?

Postseismic relaxation of the crustWhat rheology is consistent with observed postseismicdeformation?Can aseismic creep or afterslip explain the deformation?

PyLith Overview

default

Crustal Deformation ModelingElasticity problems where geometry does not change significantly

Dynamic modeling associated with earthquakes

Modeling of strong ground motionsForecasting the amplitude and spatial variation in groundmotion for scenario earthquakes

Coseismic stress changes and fault slipHow did earthquake A change the stresses on faults X and Y?

Earthquake rupture behaviorWhat fault constitutive models/parameters are consistent withthe observed rupture propagation in earthquake A?

PyLith Overview

default

Crustal Deformation ModelingElasticity problems where geometry does not change significantly

Volcanic deformation associated with magma chambers and/ordikes

InflationWhat is the geometry of the magma chamber?What is the potential for an eruption?

EruptionWhere is the deformation occurring?What is the ongoing potential for an eruption?

Dike intrusionsWhat the geometry of the intrusion?

PyLith Overview

default

PyLith

DevelopersBrad Aagaard (USGS, lead developer))Charles Williams (GNS Science, formerly at RPI)Matthew Knepley (Univ. of Chicago, formerly at ANL)

Combined dynamic modeling capabilities of EqSim (Aagaard)with the quasi-static modeling capabilities of Tecton (Williams)Use modern software engineering (modular design, testing,documentation, distribution) to develop an open-source,community code

PyLith Overview

default

Crustal Deformation ModelingOverview of workflow for typical research problem

GeologicStructure

MeshGeneration

PhysicsCode

Visualization

Gocad

Earth Vision

CUBIT/Trelis

LaGriT

TetGen

Gmsh

PyLith

Relax

GeoFEST

Abaqus

ParaView

Visit

Matlab

Matplotlib

GMT

CIG

Open Source

Free

Commercial

Available

Planned

PyLith Overview

default

Governing Equations

Elasticity equation

σij,j + fi = ρu in V , (1)σijnj = Ti on ST , (2)

ui = u0i on Su, and (3)

Rki(u+i − u−

i ) = dk on Sf . (4)

Multiply by weighting function and integrate over the volume,

−∫

V(σij,j + fi − ρui)φi dV = 0 (5)

After some algebra,

−∫

Vσijφi,j dV +

∫ST

Tiφi dS +

∫V

fiφi dV −∫

Vρuiφi dV = 0 (6)

PyLith Governing Equations

default

Governing Equations

Writing the trial and weighting functions in terms of basis (shape)functions,

ui(xi , t) =∑

m

ami (t)N

m(xi), (7)

φi(xi , t) =∑

n

cni (t)N

n(xi). (8)

After some algebra, the equation for degree of freedom i of vertexn is

−∫

VσijNn

,j dV+

∫ST

TiNn dS+

∫V

fiNn dV−∫

Vρ∑

m

ami NmNn dV = 0

(9)

PyLith Governing Equations

default

Discretize Domain Using Finite Elements

(a) Interpolated triangular mesh

7

8

9

10

2

3

4

5

6

0 1

Vertices

Edges

Cells

7 8 9 10

2 3 4 5 6

0 1

(b) Interpolated quadrilateral mesh

9

10

11

12

13

14

2

3

4

5

6

7

80 1

9 10 11 12 13 14

2 3 4 5 6 7 8

0 1

(c) Optimized triangular mesh

2

3

4

50 1

Vertices

Cells

2 3 4 5

0 1

(d) Optimized quadrilateral mesh

2

3

4

5

6

7

0 1

2 3 4 5 6 7

0 1

PyLith Governing Equations

default

Governing Equations

Using numerical quadrature we convert the integrals to sums overthe cells and quadrature points

−∑

vol cells

∑quad pts

σijNn,j wq|Jcell|+

∑surf cells

∑quad pts

TiNnwq|Jcell|

+∑

vol cells

∑quad pts

fiNnwq|Jcell|

−∑

vol cells

∑quad pts

ρ∑

m

ami NmNnwq|Jcell| = ~0 (10)

PyLith Governing Equations

default

Quasi-static SolutionNeglect inertial terms

Form system of algebraic equations

A(t)~u(t) = ~b(t) (11)

where

Anmij (t) =

∑vol cells

∑quad pts

14

Cijkl(t)(Nm,l + Nm

,k )(Nn,j + Nn

,i )wq|Jcell| (12)

bi(t) =∑

surf cells

∑quad pts

Ti(t)Nnwq|Jcell|+∑

vol cells

∑quad pts

fi(t)Nnwq|Jcell|

(13)

and solve for ~u(t).

PyLith Governing Equations

default

Fault InterfaceFault tractions couple deformation across interface

Sf+Sf-

n

u+,T+

u-,T-

PyLith Fault Implementation

default

Fault Implementation: Governing EquationsTerms in governing equation associated with fault

Tractions on fault surface are analogous to boundary tractions

. . . +

∫ST

~φ · ~T dS −∫

Sf+

~φ ·~l dS +

∫Sf−

~φ ·~l dS . . . = 0

Neumann BC Fault + Fault -

Constraint equation relates slip to relative displacement∫Sf

~φ · ( ~d } − (~u+ − ~u−) )dS = 0

Slip Relative Disp.

PyLith Fault Implementation

default

Governing Equations (cont.)

Express weighting function ~φ, displacement field ~u, Lagrangemultipliers (fault tractions)~l , and fault slip ~d as linear combinationsof basis functions,

~φ = Nm · ~am (14)

~u = Nn · ~un (15)~l = Np ·~lp (16)~d = Np · ~dp (17)

PyLith Fault Implementation

default

Governing Equations (cont.)

Lagrange multiplier (fault traction) terms:

. . .−∫

Sf+

NTm · Np ·~lp dS +

∫Sf−

NTm · Np ·~lp dS = ~0 (18)

Constraint equation∫Sf

NTp ·(

Np · ~dp − Nn+ · ~un+ + Nn− · ~un−

)dS = ~0 (19)

PyLith Fault Implementation

default

Fault Slip ImplementationUse Lagrange multipliers to specify slip

System without cohesive cellsConventional finite-element elasticity formulation

A~u = ~b

Fault slip associated with relative displacements across fault

C~u = ~d

System with Lagrange multiplier constraints for fault slip(A CT

C 0

)(~u~l

)=

(~b~d

)Prescribed (kinematic) slipSpecify fault slip (~d) and solve for Lagrange multipliers (~l)Spontaneous (dynamic) slipAdjust fault slip to be compatible with fault constitutive model

PyLith Fault Implementation

default

Implementation: Fault InterfacesUse cohesive cells to control fault behavior

PyLith Fault Implementation

default

Implementing Fault Slip with Lagrange multipliers

AdvantagesFault implementation is local to cohesive cellSolution includes tractions generating slip (Lagrangemultipliers)Retains block structure of matrix, including symmetryOffsets in mesh mimic slip on natural faults

DisadvantagesCohesive cells require adjusting topology of finite-elementmeshScalable preconditioner/solver is more complex

PyLith Fault Implementation

default

Ingredients for Running PyLith

Simulation parametersFinite-element mesh

Mesh exported from LaGriTMesh exported from CUBITMesh constructed by hand (PyLith mesh ASCII format)

Spatial databases for physical properties, boundaryconditions, and rupture parameters

SCEC CVM-H or USGS Bay Area Velocity modelSimple ASCII files

PyLith Running PyLith

default

Spatial DatabasesUser-specified field/value in space

ExamplesUniform value for Dirichlet (0-D)Piecewise linear variation in tractions for Neumann BC (1-D)SCEC CVM-H seismic velocity model (3-D)

Generally independent of discretization for problemAvailable spatial databasesUniformDB Optimized for uniform valueSimpleDB Simple ASCII files (0-D, 1-D, 2-D, or 3-D)

SCECCVMH SCEC CVM-H seismic velocity model v5.3ZeroDispDB Special case of UniformDB

PyLith Running PyLith

default

Features in PyLith 1.9A few small, under-the-hood changes and several parallel processing bug fixes

Time integration schemes and elasticity formulationsImplicit for quasistatic problems (neglect inertial terms)

Infinitesimal strainsSmall strains

Explicit for dynamic problemsInfinitesimal strainsSmall strainsNumerical damping via viscosity

Bulk constitutive modelsElastic model (1-D, 2-D, and 3-D)Linear Maxwell viscoelastic models (2-D and 3-D)Generalized Maxwell viscoelastic models (2-D and 3-D)Power-law viscoelastic model (2-D and 3-D)Drucker-Prager elastoplastic model (2-D and 3-D)

PyLith Features

default

Features in PyLith 1.9 (cont.)

Boundary and interface conditionsTime-dependent Dirichlet boundary conditionsTime-dependent Neumann (traction) boundary conditionsAbsorbing boundary conditionsKinematic (prescribed slip) fault interfaces w/multiple rupturesDynamic (friction) fault interfacesTime-dependent point forcesGravitational body forces

Fault constitutive modelsStatic frictionLinear slip-weakeningLinear time-weakeningDieterich-Ruina rate and state friction w/ageing law

PyLith Features

default

Features in PyLith 1.9 (cont.)

Automatic and user-controlled time steppingAbility to specify initial stress/strain stateImporting meshes

LaGriT: GMV/PsetCUBIT: Exodus IIASCII: PyLith mesh ASCII format (intended for toy problemsonly)

Output: VTK and HDF5 filesSolution over volumeSolution over surface boundaryState variables (e.g., stress and strain) for each materialFault information (e.g., slip and tractions)

Automatic conversion of units for all parametersParallel uniform global refinementPETSc linear and nonlinear solvers

Custom preconditioner with algebraic multigrid solver

PyLith Features

default

PyLith Development

Short-term prioritiesUnder-the-hood improvements

New finite-element data structures [done]Support higher order basis functions [in progress]Provides much higher resolution for a given meshPrepare for multi-physics [done]

Multi-cycle earthquake modelingResolve interseismic, coseismic, and postseismic deformationElastic/viscoelastic/plastic rheologiesCoseismic slip, afterslip, and creep

Long-term prioritiesMultiphysics: Elasticity + Fluid flow + Heat flowScaling to 1000 processors

PyLith Features

default

PyLith DevelopmentPlanned Releases

v2.0 (Summer 2013)New finite-element data structuresSupport for higher order basis functions

v2.1 (Spring 2014)Coupling of quasi-static and dynamic simulationsMoment tensor point sources

v2.2 (Fall 2014)Support for incompressible elasticityHeat and fluid flow coupled to elastic deformation

v2.xSupport for finite-element integrations on GPUs

PyLith Features

default

Design PhilosophyModular, extensible, and smart

Code should be flexible and modularUsers should be able to add new features without modifyingcode, for example:

Boundary conditionsBulk constitutive modelsFault constitutive models

Input/output should be user-friendlyTop-level code written in Python (expressive, dynamic typing)Low-level code written in C++ (modular, fast)

PyLith Architecture

default

PyLith Design: Focus on GeodynamicsLeverage packages developed by computational scientists

PyLith

SievePyre Proj.4 FIAT

PETSc numpy

BLAS/LAPACKMPI boost

PyLith Architecture

default

PyLith as a Hierarchy of ComponentsComponents are the basic building blocks

���������

������

����

���

����������

����������

������������������

����������

��������

PyLith Architecture

default

PyLith as a Hierarchy of ComponentsPyLith Application and Time-Dependent Problem

���������

����������

����������

����

��������������

�������

�����

�������������

����������

����������

���������

����������

���������

��

����������

�������������

�����������

PyLith Architecture

default

PyLith as a Hierarchy of ComponentsFault with kinematic (prescribed slip) earthquake rupture

FaultCohesiveKin

properties

facilities

id

name

up_dir

normal_dir

quadrature

eq_srcs

output

EqKinSrc

properties

facilities

origin_time

slip_function

PyLith Architecture

default

PyLith as a Hierarchy of ComponentsDiagram of simple toy problem

PyLith Architecture

default

PyLith as a Hierarchy of Components

PyLith Architecture

default

PyLith Application Flow

PyLithAppmain()

mesher.create()

problem.initialize()

problem.run()

TimeDependent (Problem)initialize()

formulation.initialize()

run()

while (t < tEnd)

dt = formulation.dt()

formulation.prestep(dt)

formulation.step(dt)

formulation.poststep(dt)

Implicit (Formulation)initialize()

prestep()

set values of constraints

step()

compute residual

solve for disp. incr.

poststep()

update disp. field

write output

PyLith Architecture

default

Unit and Regression TestingAutomatically run more than 1800 tests on multiple platforms whenever code ischecked into the source repository.

Create tests for nearly every function in code duringdevelopment

Remove most bugs during initial implementationIsolate and expose bugs at origin

Create new tests to expose reported bugsPrevent bugs from reoccurring

Rerun tests whenever code is changedCode continually improves (permits optimization with qualitycontrol)

Binary packages generated automatically upon successfulcompletion of testsAdditional full-scale tests are run before releases

PyLith Testing

default

General Numerical Modeling TipsStart simple and progressively add complexity and increase resolution

Start in 2-D, if possible, and then go to 3-DMuch smaller problems⇒ much faster turnaroundExperiment with meshing, boundary conditions, solvers, etcKeep in mind how physics differs from 3-D

Start with coarse resolution and then increase resolutionMuch smaller problems⇒ much faster turnaroundExperiment with meshing, boundary conditions, solvers, etc.Increase resolution until solution resolves features of interest

Resolution will depend on spatial scales in BC, initial conditions,deformation, and geologic structureIs geometry of domain important? At what resolution?Displacement field is integral of strains/stressesResolving stresses/strains requires fine resolution simulations

Use your intuition and analogous solutions to check yourresults!

Troubleshooting General

default

Mesh Generation TipsThere is no silver bullet in finite-element mesh generation

Hex/Quad versus Tet/TriHex/Quad are slightly more accurate and fasterTet/Tri easily handle complex geometryEasy to vary discretization size with Tet, Tri, and Quad cellsThere is no easy answerFor a given accuracy, a finer resolution Tet mesh that variesthe discretization size in a more optimal way might run fasterthan a Hex mesh

Check and double-check your meshWere there any errors when running the mesher?Do all of the nodesets and blocks look correct?Check mesh quality (aspect ratio should be close to 1)

CUBITName objects and use APREPRO or Python for robust scriptsNumber of points in spline curves/surfaces has huge affect onmesh generation runtime

Troubleshooting Meshing

default

PyLith Tips

Read the PyLith User ManualDo not ignore error messages and warnings!Use an example/benchmark as a starting pointQuasi-static simulations

Start with a static simulation and then add time dependenceCheck that the solution converges at every time step

Dynamic simulationsStart with a static simulationShortest wavelength seismic waves control cell size

CIG Short-Term Crustal Dynamics mailing listcig-short@geodynamics.org

PyLith User Resourceshttp://www.geodynamics.org/cig/software/pylith/user resources

Troubleshooting PyLith

default

PyLith Debugging Tools

pylithinfo [--verbose] [PyLith args]

Dumps all parameters with their current values to text fileCommand line arguments

--help

--help-components

--help-properties

--petsc.start in debugger (run in xterm)--nodes=N (to run on N processors on local machine)

Journal info flags turn on writing progress[pylithapp.journal.info]

timedependent = 1

Turns on/off info for each type of component independentlyExamples turn on writing lots of info to stdout using journalflags

Troubleshooting PyLith

default

Getting Started

Read the PyLith User ManualWork through the examples

Chapter 7 of the PyLith manualInput files are provided with the PyLith binarysrc/pylith/examples

Input files are provided with the PyLith source tarballsrc/examples

Modify an example to look like a problem of interest

Troubleshooting PyLith