NUMERICAL SIMULATION OF SUBSURFACE ENERGY SYSTEMS

SPARSE JACOBIAN MATRIX CALCULATIONS

Rami M Younis McDougall School of Petroleum Engineering

The University of Tulsa

13th European AD Workshop, INRIA Sophia-Antipolis 6/10/2013

FUture Reservoir Simulation Systems & Technology FURSST Industrial Research Consortium

PETROLEUM RESERVOIR SIMULATION

COMPLEXITY

Example: Underground Coal Gasification (UCG)

COMPLEXITY

SOFTWARE

NONLINEAR ANALYSIS

USAGE PARADIGMS

The business-as-usual trend

+ Single problem

+ Disposable

The business-as-usual trend

+ Single problem

+ Disposable + Multi-application

+ Maintainable (5 years)

The business-as-usual trend

+ Single problem

+ Disposable + Multi-application

+ Maintainable (5 years)

+ Multi-physics

+ Extendable ?

Business-as-usual and a brick wall

AUTOMATIC DIFFERENTIATION

Automating vector calculus within discrete problem specification

Automating problem specification

Fully-Implicit Forward Simulation

ℛ 𝒖 𝑥, 𝑡 =𝜕

𝜕𝑡𝒜 𝒖 +ℱ 𝒖, 𝛻𝒖, ∆𝒖

𝑹ℎ 𝑈𝑛+1; 𝑈𝑛, ∆𝑡 = 𝑨ℎ 𝑈

𝑛+1 − 𝑨ℎ 𝑈𝑛 +∆𝑡𝑭ℎ 𝑈

𝑛+1; 𝑈𝑛, ∆𝑡

1. Fix time-step, ∆𝒕, given old state, 𝑼𝒏

2. Obtain new state, 𝑼𝒏+𝟏 , by solving;

𝑹ℎ 𝑈𝑛+1; 𝑈𝑛, ∆𝑡 =0

For each 𝒏 = 𝟎, 𝟏,…

Newton’s Method

𝑈𝑛+1 𝜈+1 ≔ 𝑈𝑛+1 𝜈 − 𝑱−𝟏𝑹 𝑈𝑛+1 𝜈;𝑈𝑛, ∆𝑡

CHALLENGES

Challenge 1: Derivatives + placement

𝜕

𝜕𝑝

𝑒−𝑝/𝑅𝑇𝑝

𝑎 + 𝑝 + 𝑝2

1. Differentiate

2. Place appropriately

3. Expensive to modify

4. Impossible to debug

Picard

iterations

Physical Properties

and Thermodynamics

Stenciling

Challenge 2: Varying intermediate sparsity

- Number of degrees of freedom

- Independent unknowns

Challenge 3: Phase Changes…

All cells 3-phase

Call 1:

phase disappearance phase (dis)appearance

1 ROUTINE and 3 CALLS

Call 2: Call 3:

Objective: Automate Formulation of Equations

+ Algorithmic differentiation and placement

+ Ubiquitous (generic) representations

+ Natural semantics of variable switching

SPARSE AD

Approach 1: Compression and a Seed Matrix

1. Infer (or provide) sparsity pattern

2. Graph coloring on pattern matrix

3. Pick a seed matrix

4. Compute compressed dense AD

5. Back-solve to get sparse Jacobian

Approach 2: Runtime direct sparse AD

a b a b

f a 'f a a

.a b b a a b

JUST use SPARSE

vector gradients

1. Sparse operations versus dense

2. EXAMPLE: SparsLinC

Provide means of activation

𝑥 = 𝑥, 1 e.g. Univariate

𝑥3 = 𝑥, 0,0,1,0 Multivariate

Example:

ADvector< ADscalar< double >, fastl::freelist_pool > a;

Automatically Differentiable Expression

Templates Library ADETL

In-house Generic C++ Library

Forward Mode, Operator Overloading, C++ templates

Select specific internal data-type

Memory allocators and manipulators

Concepts that your wish to subscribe to

Dense indexable, scalar differentiable, … , linearly combinable

Top layer abstracts the notion of a variable set

RUNTIME SPARSITY:

Compute kernel is K-way Sparse Linear Combination

What is the fastest way to compute it ?

Lazy tactics like Sacado does for dense

Algorithms to reduce memory demands

1

k

i i

i

W V

Evaluate

, where , N

iW V R are sparse vectors

LAZY EVALUATION STRATEGIES

In an expression with 20 binary operators,

should we

Evaluate 1 operator at a time into a temporary ?

“Build an expression” and evaluate it in one go ?

1

Lazy Tactic 1: Classic Expression Templates

Lazy Tactic 2: Runtime SPLC lists

REDUCING LOADS/STORES by caching ! 2

1

k

i i

i

W V

Evaluate , where , N

iW V R are sparse vectors

Algorithm 1: the obvious

Algorithm 2: the less obvious

Sparsity Metrics + Auto-tuning

N = 25

Na = 9

NNZ = 7

S = 1 – 7 / 25 = 0.72

Sa = 1 – 7 / 9 = 0.22

NNZ Number of logical non-zero entries

N Nominal dimension Na Apparent dimension

S Sparsity ( 1 – NNZ / N ) Sa Apparent sparsity ( 1 – NNZ / Na )

EXAMPLE:

ADETL Choices

Dense

Block

Sparse

Sparse x x

SPLC

Expressions

Classic

Expressions

B

ADETL selects best sparse linear combination algorithms on the fly

Direct

Evaluation

Univariate x

Nine-component two-phase

Pressure

Water Saturation

AD Overhead

GPRS

(sec)

AD-GPRS (sec)

Overhead

Discretization per iteration

0.074 0.057 -23 %

Properties per iteration

0.321 0.256 -20 %

Total 139 123 -12 %

Compositional three-phase heterogeneous

Pressure Gas Saturation

Permeability

AD Overhead

GPRS

(sec)

AD-GPRS (sec)

Overhead

Discretization per iteration

0.147 0.152 3 %

Properties per iteration

0.192 0.334 73 %

Total 426 494 16 %

NUMERICAL SIMULATION OF SUBSURFACE ENERGY SYSTEMS

SPARSE JACOBIAN MATRIX CALCULATIONS

Rami M Younis McDougall School of Petroleum Engineering

The University of Tulsa

ryounis@utulsa.edu

13th European AD Workshop, INRIA Sophia-Antipolis 6/10/2013