Upload
dokhue
View
214
Download
0
Embed Size (px)
Citation preview
Some applications of MUMPs in Some applications of MUMPs in computational fluid dynamics and computational fluid dynamics and
electromagneticselectromagnetics
Al dAl d G lf tG lf tAlexander Alexander GelfgatGelfgatSchool of Mechanical School of Mechanical Engineering, Faculty Engineering, Faculty of Engineeringof Engineering
TelTel--Aviv UniversityAviv University
Area: Area: CComputational omputational FFluid luid DDynamicsynamicseaea CCo pu a o ao pu a o a u du d y a csy a csCFD is relevant for:
Basic science:AstrophysicsClassical mechanicsClassical mechanicsGeophysicsOceanography
Applications:Applications:AerospaceAutomotiveBiomedicalBiomedicalChemical ProcessingMarineOil & GasOil & GasPower GenerationSports
Our current project:Our current project:spiraling growth of rarespiraling growth of rare earthearth scandatesscandatesspiraling growth of rarespiraling growth of rare--earth earth scandatesscandates
caused by melt flow instabilitiescaused by melt flow instabilities
GdScO3DyScO3 SmScO3 NdScO3
2
0.5
1
1.5
Z
0
-1
-0.5
0
0.5
1
X
-1
-0.5
0
0.5
1
Y
Some historySome history"Observe the motion of the surface of the waterObserve the motion of the surface of the water,which resembles that of hair, which has twomotions, of which one is caused by the weight ofthe hair, the other by the direction of the curls;thus the water has eddying motions one part ofthus the water has eddying motions, one part ofwhich is due to the principal current, the other tothe random and reverse motion.“ Leonardo daVinci, translated by Ugo Piomelli, University ofMaryland)Maryland),
1452-1519
, , ' ,t t t v r v r v r
1, ,t T
t t dt
v r v r , ,t
t t dtT v r v r
' , , ,t t t v r v r v r
Osborne Reynolds,1842-1921
, , ,t t tv v v
Nowadays understandingNowadays understandingAll turbulent features and properties are described by classical All turbulent features and properties are described by classical
equations of fluid motionequations of fluid motion
Navier-Stokes equation: fvvvv
Rep
t1
+ b d & i iti l diticontinuity equation:
Ret
0vdiv+ boundary & initial conditionscontinuity equation:
With t h i l l ti d t b l t flWith an accurate enough numerical solution we can reproduce turbulent flow properties of Reynolds experiment up to Re<10,000
However: we need larger However: we need larger ReRe and faster solvers !!!and faster solvers !!!gg
ObjectiveObjectiveLowerLower--order CFD solver for order CFD solver for modellingmodelling of of fluid dynamics, heat and mass transfer fluid dynamics, heat and mass transfer
Computational modelling in fluid dynamics : - Steady flows
in nature and technologyin nature and technologyComputational modelling in fluid dynamics : y
- Flow instabilities
- Supercritical flows p
- Turbulent flows
- Flow control
Numerically challenging tasks: - Solution of nonlinear algebraic equations
- Generalized eigenvalue problem
- Projection on central manifold
Di i l i l i- Direct numerical simulation
- Fast linear system solvers
Steady solver: Steady solver: yyfull Jacobian exact Newton iteration full Jacobian exact Newton iteration
, 0 F X
System of algebraic equations:
101055 –– 101077
equations equations forfor 22D problemsD problemsX - vector of all nodal values (uij, vij, wij, Tij, pij) for for 22D problemsD problems
Jacobian matrix: iij
j
FJX
X
calculated exactly from the numerical scheme
Newton iterations: ,n JδX F X
Newton iterations:
1n n
X X δX
Bottleneck:each Newton iteration
needs solution of f li isystem of linear equations
of very large order
Solution of linear equations for Newton iterationSolution of linear equations for Newton iterationSolution of linear equations for Newton iterationSolution of linear equations for Newton iteration
An iterative solver:
- traditional choice: BiCGstab or GMRES (Krylov subspace)t lti id ith i l ti l th- we try: multigrid with a semi-analytical smoother
A direct solver for sparse matrices:A direct solver for sparse matrices:
- we use: MUMPS – Multifrontal Massively Parallel Solver
3D problems: out-of-core tool, parallel implementationto be done in near future
Eigenproblem for analysis of instabilitiesEigenproblem for analysis of instabilities
u uv v
Generalized eigenvalue problemwhere J is the Jacobian matrix and detB=0
w wp p
B J
Solution: Arnoldi iteration in the shift-and-invert mode
1 1, , , , , , ,T T
u v w p u v w p
J B B
Solution: Arnoldi iteration in the shift and invert mode
Choose as close as possible to the leading eigenvalue and calculate several with the largest ||.
Bottleneck: calculation of the Krylov basis 1 2, , ,..., nx x x x J B J B J B
The matrix (J−B) is iteration independent !
Calculation of LU-decomposition of (J−B)resolves the bottleneck ! Instead of an iterative solver use resolves the bottleneck ! a direct solver (we use MUMPS in
a complex mode)
Performance on a single CPUPerformance on a single CPU
non-isothermal rotating
102
N2.6
gflow in a cylinder
2e
(sec
)10
N2
N2.75
N2
PUtim
e
101
N
CP
200 400
100 real Jacobian matrix for Newton iterationcomplex Jacobian matrix for linear stabilityArnoldi eigensolver
N=Nr=Nz
200 400
Computer memory demandsComputer memory demands4104
non-isothermal rotating
flow in a cylinder
ytes
)103
3
flow in a cylinder
N2.3
ry(M
by
N2.3
Mem
or
102
real Jacobian matrix for Newton iterationcomplex Jacobian matrix for linear stability
N=Nr=Nz
100 200 300 400 500101complex Jacobian matrix for linear stability
Test problems in rectangular geometryTest problems in rectangular geometryInt. J.Int. J. NumerNumer. Meth. Fluids,. Meth. Fluids, 20072007, vol., vol. 5353, pp., pp. 485485--506506Int. J. Int. J. NumerNumer. Meth. Fluids, . Meth. Fluids, 20072007, vol. , vol. 5353, pp. , pp. 485485 506506
B tiBuoyancy convection:Marangoni convection:
- Different boundary conditions- Different Prandtl numbers- Different aspect ratios
Test problems in cylindrical geometryTest problems in cylindrical geometryInt. J. Int. J. NumerNumer. Meth. Fluids, . Meth. Fluids, 20072007, vol. , vol. 5454, pp. , pp. 269269--294294pppp
Buoyancy convection in non-
Thermocapillary convection in non
Isothermal and non-isothermal
Thermocapillary convection inconvection in non
uniformly heated cylinders
convection in non-uniformly heated
cylinders
non-isothermal flow in a cylinder with rotating disk
convection in annular pools
z
Thot
(T)
H
(
T) Tcold Thot
rR Tcold
Effect of stretching Effect of stretching
y
j
y
jyj
x
i
x
ixi A
ysinb
Ay
Ay,Axsina
AxAx 22
yy
a=b=0.06 a=b=0.12a=b=0
Convergence and effect Convergence and effect of stretching of stretching
Test case: convection in square cavity, A=1, Pr=0, G 9 472×106 8242Grcr=9.472×106, cr=8242
1
r R
0.98
1
R
0 98
1
Gr cr
/Gr
0.94
0.96uniform grida=b=0.08
cr/
0 96
0.98
uniform grida=b=0.08
0.92a=b=0.12
0 94
0.96a=b=0.12
N=Nx=Ny
100 200 300 4000.9N=Nx=Ny
100 200 300 4000.94
TimeTime--dependent solver using MUMPSdependent solver using MUMPS
Assembling Assembling of of the the LU F t i tiLU F t i ti U d tiU d ti RHPRHP
STARTSTART
ggStokes Operator Stokes Operator A A
for Momentumfor Momentum--Continuity System Continuity System
LU Factorization LU Factorization of Aof A
MUMPSMUMPS
Updating Updating RHPRHPof the of the EEnergy Equationnergy Equation
Solution of the Energy Solution of the Energy Equation toEquation toObtain Obtain n+n+11
NoNo
Updating RHP of the Momentum
Equations
Backward SubstitutionBackward Substitutionto to ObtainObtainvvn+n+11, , ppn+n+11
Is the Steady Is the Steady State or Time State or Time
Limit Reached?Limit Reached?
YesYesFINISHFINISH
MUMPSMUMPSLimit Reached?Limit Reached?
The Full Pressure Coupled Direct (FPCD) Solution
An efficient 3D time marching solver The FPCD CharacteristicsThe FPCD Characteristics(Cont.1)
1
10
ec)
1 E 03
1.E+04
y (M
gb)
10-6 ×N 1.09480.0015×N 1.0501
0.1
1
Cpu
Tim
e (s
e
1.E+02
1.E+03
ompu
ter M
emor
y
0.011.0E+04 1.0E+05 1.0E+06
The Total Nodes Number
1.E+011.E+04 1.E+05 1.E+06
The Total Nodes Number
C
Direct method treats whole computational domain
Constant numerical convergence rate
B d lti f t l l (MUMPS k )Based on multi-frontal sparse solver (MUMPS package)
Highly effective only for linearized problems (Stokes operator implementation)
Extremely memory demanding for 3D calculations (no more then 803 resolution up to now)
Non-parallelized back-substitution prevents multi-processor computations
18
Making a steady state solver from a timeMaking a steady state solver from a time--stepper stepper (together with L. Tuckerman)(together with L. Tuckerman)( g )( g )
Navier-Stokes equation: tULNU Ut Navier-Stokes equation: Ut
Semi-implicit time step: ttULNttUttU U p p tUtNItLI U
U
1
An idea: instead of classical Newton iteration uUU,UAuLNU
Consider an equivalent equation:
)U(AuItNItLItuttu 1 tUttUUItNItLI
)U(AuItNItLItuttu U
1
Each time step produced by the MUMPS back substitution produces a new Krylov basis vector for BiCGstab algorithm
Making an eigenvalue solver from a timeMaking an eigenvalue solver from a time--stepper stepper ((together with Ltogether with L. Tuckerman). Tuckerman)(( gg ))
Stability problem: 0 BdetvLNBvStability problem: 0 Bdet,vLNBv U
To build Krylov basis for shift-and-inverse Arnoldi iteration process we need to solve:y p n
Un BvBLNv 11
vBNLtLBtvttv nU
nn 1111 An idea: consider an equivalent equation:
tvttvvtLB nnn 1one linearized time step
one linear time step
MUMPS back substitution
and combine the time-stepper with BiCGstab algorithm
Examples of results:Examples of results: multiplicitymultiplicity
0.2
Convection of water in partially heated cavity (Gelfgat’s group)
0 1
u
0.1
RN
u-N
u
0L
-0.1
Gr0 500 1000 1500 2000
Examples of results:Examples of results: stability diagramsstability diagramsConvection in laterally heated cavity
1.2E+06
9.0E+05
6.0E+05Gr c
r
3.0E+05
0.0E+002 3 4 5 6 7 8 9 10
A
Examples of results:Examples of results: melting of a square blockmelting of a square block
T=06000
a)
2000
4000
T=0T=Tmelting
eal(L
ambd
a
0
Re
-4000
-2000
Each point ≈ 2000 MUMPS runs
T=0time
0.02 0.04 0.06 0.08-6000
-4000
time
ElectromagneticsElectromagnetics: Maxwell vs. Helmholtz equations: Maxwell vs. Helmholtz equations
Maxwell equations: Helmholtz equations:ddivdiv
DEB 0
:);tiexp(~ 0HE,
rott
rot
BE
DjH
00
HHEE
ii
Solve all the t
rot
E
HB Solve a scalar elliptic problem
equations together using MUMPS
EjDEHB
for each component
...Ej divhB ≠ 0, divhE ≠ 0
ElectromagneticsElectromagnetics: results for rotating magnetic field: results for rotating magnetic field
ConcludingConcluding remarksremarksConcluding Concluding remarksremarks
• Where MUMPS is successfully applies it yields acece MUMPS • Where MUMPS is successfully applies it yields a qualitative improvement of results
perf
orm
anc
perf
orm
anc
Iterative solvers
• We have more ideas for future studiesar
sol
vers
par
sol
vers
p
Pre-MUMPS direct solvers
ss o
f lin
eass
of l
inea
Hap
pine
Hap
pine
Thanks for the nice tool !!!Thanks for the nice tool !!!