NUMERICAL SIMULATION OF SUBSURFACE ENERGY SYSTEMS
SPARSE JACOBIAN MATRIX CALCULATIONS
Rami M Younis McDougall School of Petroleum Engineering
The University of Tulsa
13th European AD Workshop, INRIA Sophia-Antipolis 6/10/2013
FUture Reservoir Simulation Systems & Technology FURSST Industrial Research Consortium
PETROLEUM RESERVOIR SIMULATION
COMPLEXITY
Example: Underground Coal Gasification (UCG)
COMPLEXITY
SOFTWARE
NONLINEAR ANALYSIS
USAGE PARADIGMS
The business-as-usual trend
+ Single problem
+ Disposable
The business-as-usual trend
+ Single problem
+ Disposable + Multi-application
+ Maintainable (5 years)
The business-as-usual trend
+ Single problem
+ Disposable + Multi-application
+ Maintainable (5 years)
+ Multi-physics
+ Extendable ?
Business-as-usual and a brick wall
AUTOMATIC DIFFERENTIATION
Automating vector calculus within discrete problem specification
Automating problem specification
Fully-Implicit Forward Simulation
ℛ 𝒖 𝑥, 𝑡 =𝜕
𝜕𝑡𝒜 𝒖 +ℱ 𝒖, 𝛻𝒖, ∆𝒖
𝑹ℎ 𝑈𝑛+1; 𝑈𝑛, ∆𝑡 = 𝑨ℎ 𝑈
𝑛+1 − 𝑨ℎ 𝑈𝑛 +∆𝑡𝑭ℎ 𝑈
𝑛+1; 𝑈𝑛, ∆𝑡
1. Fix time-step, ∆𝒕, given old state, 𝑼𝒏
2. Obtain new state, 𝑼𝒏+𝟏 , by solving;
𝑹ℎ 𝑈𝑛+1; 𝑈𝑛, ∆𝑡 =0
For each 𝒏 = 𝟎, 𝟏,…
Newton’s Method
𝑈𝑛+1 𝜈+1 ≔ 𝑈𝑛+1 𝜈 − 𝑱−𝟏𝑹 𝑈𝑛+1 𝜈;𝑈𝑛, ∆𝑡
CHALLENGES
Challenge 1: Derivatives + placement
𝜕
𝜕𝑝
𝑒−𝑝/𝑅𝑇𝑝
𝑎 + 𝑝 + 𝑝2
1. Differentiate
2. Place appropriately
3. Expensive to modify
4. Impossible to debug
Picard
iterations
Physical Properties
and Thermodynamics
Stenciling
Challenge 2: Varying intermediate sparsity
- Number of degrees of freedom
- Independent unknowns
Challenge 3: Phase Changes…
All cells 3-phase
Call 1:
phase disappearance phase (dis)appearance
1 ROUTINE and 3 CALLS
Call 2: Call 3:
Objective: Automate Formulation of Equations
+ Algorithmic differentiation and placement
+ Ubiquitous (generic) representations
+ Natural semantics of variable switching
SPARSE AD
Approach 1: Compression and a Seed Matrix
1. Infer (or provide) sparsity pattern
2. Graph coloring on pattern matrix
3. Pick a seed matrix
4. Compute compressed dense AD
5. Back-solve to get sparse Jacobian
Approach 2: Runtime direct sparse AD
a b a b
f a 'f a a
.a b b a a b
JUST use SPARSE
vector gradients
1. Sparse operations versus dense
2. EXAMPLE: SparsLinC
Provide means of activation
𝑥 = 𝑥, 1 e.g. Univariate
𝑥3 = 𝑥, 0,0,1,0 Multivariate
Example:
ADvector< ADscalar< double >, fastl::freelist_pool > a;
Automatically Differentiable Expression
Templates Library ADETL
In-house Generic C++ Library
Forward Mode, Operator Overloading, C++ templates
Select specific internal data-type
Memory allocators and manipulators
Concepts that your wish to subscribe to
Dense indexable, scalar differentiable, … , linearly combinable
Top layer abstracts the notion of a variable set
RUNTIME SPARSITY:
Compute kernel is K-way Sparse Linear Combination
What is the fastest way to compute it ?
Lazy tactics like Sacado does for dense
Algorithms to reduce memory demands
1
k
i i
i
W V
Evaluate
, where , N
iW V R are sparse vectors
LAZY EVALUATION STRATEGIES
In an expression with 20 binary operators,
should we
Evaluate 1 operator at a time into a temporary ?
“Build an expression” and evaluate it in one go ?
1
Lazy Tactic 1: Classic Expression Templates
Lazy Tactic 2: Runtime SPLC lists
REDUCING LOADS/STORES by caching ! 2
1
k
i i
i
W V
Evaluate , where , N
iW V R are sparse vectors
Algorithm 1: the obvious
Algorithm 2: the less obvious
Sparsity Metrics + Auto-tuning
N = 25
Na = 9
NNZ = 7
S = 1 – 7 / 25 = 0.72
Sa = 1 – 7 / 9 = 0.22
NNZ Number of logical non-zero entries
N Nominal dimension Na Apparent dimension
S Sparsity ( 1 – NNZ / N ) Sa Apparent sparsity ( 1 – NNZ / Na )
EXAMPLE:
ADETL Choices
Dense
Block
Sparse
Sparse x x
SPLC
Expressions
Classic
Expressions
B
ADETL selects best sparse linear combination algorithms on the fly
Direct
Evaluation
Univariate x
Nine-component two-phase
Pressure
Water Saturation
AD Overhead
GPRS
(sec)
AD-GPRS (sec)
Overhead
Discretization per iteration
0.074 0.057 -23 %
Properties per iteration
0.321 0.256 -20 %
Total 139 123 -12 %
Compositional three-phase heterogeneous
Pressure Gas Saturation
Permeability
AD Overhead
GPRS
(sec)
AD-GPRS (sec)
Overhead
Discretization per iteration
0.147 0.152 3 %
Properties per iteration
0.192 0.334 73 %
Total 426 494 16 %
NUMERICAL SIMULATION OF SUBSURFACE ENERGY SYSTEMS
SPARSE JACOBIAN MATRIX CALCULATIONS
Rami M Younis McDougall School of Petroleum Engineering
The University of Tulsa
13th European AD Workshop, INRIA Sophia-Antipolis 6/10/2013