A parallel scientific software for heterogeneous hydrogeoloy

Conference on Parallel CFD

Antalya, Turkey May 2007

Jocelyne Erhel INRIA Rennes

Jean-Raynald de Dreuzy Geosciences Rennes

Anthony Beaudoin

LMPG, Le Havre

Etienne Bresciani

INRIA Rennes

Damien Tromeur-Dervout

CDCSP, Lyon

Partly funded by Grid’5000

french project

From Barlebo et al. (2004)

Dispersion

Flow

Injection of tracer

Tracer evolution during one year (Made, Mississippi)

Heterogeneous permeability

Physical context: groundwater flow

Flow governed by the heterogeneous permeability

Solute transport by advection and dispersion

Physical context: groundwater flow

Spatial heterogeneity

Stochastic models of flow and solute transport

-random velocity field-random solute transfer time and dispersivity

Lack of observationsPorous geological mediafractured geological media

Flow in highly heterogeneous porous medium

3D Discrete Fracture Network

Head

Numerical modelling strategy

NumericalStochasticmodels

Simulationresults

Physical model

natural system

Simulation of flowand solute transport

Characteriz

ation of

heterogeneity

Model validation

Hydrolab scientific software

Object-oriented and modular with C++Parallel algorithms with MPIEfficient numerical librariesfree software

PHYSICAL MODELS

PorousMedia

FractureNetworks

FracturedPorous

Hydrolab platform: physical models

Physical equations

Permeability field in porous media

Simple 2D or 3D geometrySimple 2D or 3D geometrystochastic permeability fieldstochastic permeability field

finitely or infinitely correlatedfinitely or infinitely correlated

MultifractalD2=1.7

finitely correlated medium

MultifractalD2=1.4

D=1

dw=2

10 h

100 h

D=2

dw=2

Well test Interpretation

Generalized flux equation

r

hr

rr

T

t

hS wdD

D21

1

Cristalline aquifer ofPloemeur (Brittany, France)

D=1.5dw=2.8

0 250 500 750 100010-1

100

101

2=9

2=6,25

2=4

2=2.25

2=1

2=0.25

DL(t

)

tN

0 500 1000-2

-1

0

1

2 2=9

2=6.25

2=4

2=2.25

2=1

2=0.25

DT(t

N)

tN

Longitudinal dispersion Transversal dispersion

Macro-dispersion analysis

Natural Fractured Media

Fractures exist at any scale with no correlation

Fracture length is a parameter of heterogeneity

0.1 1 10 10010

-7

10-6

10-5

10-4

10-3

10-2

10-1

100

101

102

n(l)~l-2.7

prob

abili

ty d

ensi

ty

n(l)

Fracture length l

Site of Hornelen, Norway

Discrete Fracture Networks with impervious matrix

Stochastic computational domainlength distribution has a great impact : power law n(l)=l-a

3 types of networks based on the moments of length distribution

Existing meanNo variation2 < a < 3

Existing meanExisting variationNo third moment3 < a < 4

Existing meanExisting variationExisting third momenta > 4

Output of simulations in 2D fracture networks : upscaling

Pas de réseaux

p , param ètre de percolation (échelle, densité de fractures)

Théorie de la percolation >3a

: longueur de corré lation

M odèle à deux échelles propres 2< <3a

M odèle de superposition de fractures

infinies <2a

Saturated medium: one water phaseSaturated medium: one water phase Constant density: no saltwaterConstant density: no saltwater Constant porosity and constant viscosityConstant porosity and constant viscosity Linear equationsLinear equations Steady-state flow or transient flowSteady-state flow or transient flow Inert transport: no coupling with chemistryInert transport: no coupling with chemistry No coupling between flow and transportNo coupling between flow and transport No coupling with heat equationsNo coupling with heat equations No coupling with mechanical equationsNo coupling with mechanical equations Classical boundary conditionsClassical boundary conditions Classical initial conditionsClassical initial conditions

Physical equations

Flow equations: Darcy law and mass conservationFlow equations: Darcy law and mass conservationTransport equations: advection and dispersionTransport equations: advection and dispersion

NUMERICAL METHODS

PDE solversEDO/DAE solversLinear solversParticle tracker Multilevel methods

Monte-Carlo methodUQ methodsParametric simulations

Hydrolab platform: numerical methods

Darcy law and mass conservationDarcy law and mass conservation

BoundaryBoundary conditions conditions

Given head

Nul flux

3D fracture network3D fracture network

Giv

en

Head

Giv

en

H

ead

Nul flux

Nul flux

2D porous medium2D porous medium

Steady-state flow equations

Uncertainty Quantification methods

Probabilistic framework

Given statistics of the input data,

compute statistics of the random solution

stochastic permeability field K stochastic network Ω

stochastic flow equations

stochastic velocity field

n(l)=l-a

Monte-Carlo simulations

For j=1,…,M

sample network Ωj

compute vj

sample permeability field Kj

Spatial discretization

2D heterogeneous porous mediumFinite volume and regular grid

3D Discrete Fracture NetworkMixed Finite Elements and non structured grid

Meshing a 3D fracture network

• Direct mesh : poor quality or unfeasible

• Projection of the fracture network: feasible and good quality

Mesh and flux computation in 3D fracture networks

Parallel algorithms

Domain decomposition

Parallel computing facilities

16,8 millions of unknowns in 100 seconds

with 16 processors

©INRIA/Photo Jim Wallace

Numerical model

Clusters at Irisa

Grid’5000

project

Funded by

French

Government

and Brittany

council

Discrete flow numerical model

Linear system Ax=b

b: boundary conditions and source termA is a sparse matrix : NZ coefficientsMatrix-Vector product : O(NZ) opérationsDirect linear solvers: fill-in in Cholesy factor

Regular 2D mesh : N=n2 and NZ=5NRegular 3D mesh : N= n3 and NZ=7N

Fracture Network : N and NZ depend on the geometry

N = 8181

Intersections and 7 fractures Parallel sparse linear solvers

2D heterogeneous porous medium2D heterogeneous porous medium

memory size and CPU time with memory size and CPU time with PSPASESPSPASES

Theory : NZ(L) = O(N logN) Theory : Time = O(N1.5)

variance = 1, number of processors = 2

Sparse direct linear solvers

2D heterogeneous porous medium2D heterogeneous porous medium

CPU time with HYPRE/AMGCPU time with HYPRE/AMG

Linear complexity of BoomerAMG

Sparse iterative linear solvers

variance = 1, number of processors = 4residual=10-8

Flow computation in 2D porous medium

Finitely correlated permeability fieldFinitely correlated permeability field

Impact of permeability varianceImpact of permeability variance

matrix order N = 106

PSPASES and BoomerAMG independent of varianceBoomerAMG faster than PSPASES with 4 processors

matrix order N = 16 106

parallel sparse linear solvers

2D heterogeneous porous medium2D heterogeneous porous medium

Direct and multigrid solversDirect and multigrid solvers

Parallel CPU timeParallel CPU time

variance = 9

matrix order N = 106 matrix order N = 4 106

Solute transport

Fix

ed

head

an

d C

=0

Fix

ed

head

an

d

C/

n=

0

Nul flux and C/ n = 0

Nul flux and C/ n=0

inje

ctio

n

Advection-dispersion equationBoundary conditionsInitial condition

injection

Solute transport : particle tracker

Many independent particlesBilinear interpolation for V

Homogeneous molecular diffusion Dm

Stochastic differential equationFirst-order explicit scheme

Parallel particle trackerBunch of independent particles

Subdomain decomposition

1 and Pe=1 3 and Pe=100

Solute transport in 2D heterogeneous porous media

Permeability Particles

Horizontal velocity Vertical velocity

particle tracker : convergence analysis

Np = 1000 is a good trade-off between efficiency and convergence

Longitudinal dispersion

Transversal dispersion

Pure advection Advection-diffusion with Pe=100

Grid size 8192*8192 with 32 processors

More efficient in pure advection caseSlightly more efficient with moderately heterogeneous media

particle tracker : impact of diffusion and heterogeneity

particle tracker : parallel performances

Grid size 4096*4096

Good speed-up in any configuration

Current work and perspectives

Current work

• Iterative linear solvers for 3D fracture networks

• 3D heterogeneous porous media

• Subdomain method with Aitken-Schwarz acceleration

• Transient flow in 2D and 3D porous media

• Grid computing and parametric simulations

Future work

• Porous fractured media with rock

• Site modeling

• UQ methods