COMPUTATIONAL FLUID DYNAMICS:

MODULAR ELEMENTS FOR CACTUS?Sumanta Acharya

L. R. Daniel Professor & Director

Turbine Innovation & Energy Research Center

College of Engineering

AcknowledgementsFrank Muldoon, Mayank Tyagi

NASA, ONR, CCT, BOR

Motivation• Fluid flow controls all aspects of our lives!!

– Human body– Drug manufacture & delivery, DNA Analysis– Transportation– Entertainment & Sports– Energy– Chemical, petrochemical, refining– Defense– Many others!!

• Research in fluid dynamics is driven by a need to survive on EARTH!

Physics, Mathematics and Simulations of Fluid Turbulence

Energy containingscales

Dissipative Scales(Kolmogorov Scales)

Inertial Range (k -5/3scaling)

Lo

g(E

(k))

Log(k)

SP

EC

TR

AL

PIC

TU

RE

PH

YS

ICA

LS

PA

CE

PIC

TU

RE

Small Scales(Probably more universal)

Large scales(Boundary condition dependent)

Vis

cous

Dis

sipa

tion

Forward Transfer

Backscatter

Phenomenology of Turbulence

•Stochastic (Randomness)

•Non-linear

•Dissipative

•Enhanced Diffusion

• Fluid flow is governed by the Navier-Stokes Equations. So we can compute?

2

2

Re

1

0

j

i

ij

jii

i

i

x

u

x

p

x

uu

t

u

x

u

SIMULATIONS OF FLUID TURBULENCE

Simulation of turbulent flows

Direct Numerical Simulation (DNS)•Resolves all the scales of turbulent flows

•Extremely computational intensive

Large eddy simulation (LES)•Resolves large energy containing scales and models small scales

•Less computationally intensive as compared to DNS

Reynolds-averaged Navier-stokes (RANS) models •Solves ensemble averaged equations and models all the scales•Most widely used in industries

kc kd

log(k)

E(k

)

DNSCoarse DNSLES

kr

Scales in spectrum of turbulent flow field

DNS: Resolved up to kd

LES: Resolved up to kr and filters the scales larger than kc

RANS: Models the entire spectrum

Direct Numerical Simulations of Turbulent Flows

•No modeling needed (Needs better schemes though)

•Computationally prohibitively expensive (Degrees of freedom active in a turbulent flow at Reynolds number Re are approximately Re9/4 (upper bound))

•Motivation for Grid-computing & more efficient algorithms. Fastest machine too slow; Largest cluster too small.

Direct numerical simulation of this flow

9×10¹¹ years of computing time & 2×10¹¹Gbytes of storage

For this reason most flows of industrial interest cannot be solved using Navier-Stokes equations

For flow around a car the smallest eddies are 10-5 meters in size

Turbulence Modeling using RANS Closure•Mathematically sound but lacks physics (Hierarchy of moment equations is NOT closed and one must invoke closure hypothesis always at some agreeable level)

•Not-universal (All kinds of approximations are made that may not be valid in different scenarios)

•Will be widely used in industry for foreseeable future (Computationally affordable for really complex problems of industrial interests)

DNS or LES Best RANS?Muldoon & Acharya, 2004

Navier-Stokes Equations

j

SGSij

j

i

ij

jii

xx

u

x

p

x

uu

t

u

2

2

j

Rij

j

i

ij

jii

xx

U

x

P

x

UU

t

U

2

2

dyyfyxGxfb

a

2

2

0

j

i

ij

jii

i

i

x

u

x

p

x

uu

t

u

x

u

RANS (hrs)

LES (days)

iii uuu 0

i

iii

u

uUu

DNS

(months)

Need faster machines, more numbers of them, and better algorithms to move froma world of “blur” to a world of “clear vision”.

Elements of a CFD Algorithm• Models: Differential Equation L(u)=f• Grid Generation: Cartesian, block body-fitted, adaptive mesh refinement,

Unstructured; Immersed Boundary Methods; Adaptive time-stepping • Discretization: Finite differences, finite elements, spectral schemes;

AU=b; Higher order schemes, Compact schemes, Upwind, TVD, Adaptive discretization

• Boundary Conditions: Dirichelet, Neumann, Periodic, Convective • Solvers: Linearized system of coupled algebraic equations-sparse

matrices AU=b. Explicit schemes, Implicit Elliptic solvers: Direct, Iterative (GMRES, CG, Line-relaxation); Multi-Grid

• Pressure-velocity Coupling: Poisson Equations, (Pseudo)-Compressible Formulations, Matrix pre-conditioning

• Parallelization Issues: Domain Decomposition, Ghost Cells, Message Passing

• I/O• Data Processing & Visualization

ij

ij

j

i

ij

jii

j

j

fxx

U

x

p

x

UU

t

U

x

U

2

2

Re

1

0

U

V

W

A Staggered Cell

ii-1i-2 i+1 i+2 ii-1 i+1 i+2 i+3

Ui > 0 Ui < 0

Upwinding on the interface location Xi

Five point stencil to get staggered velocity components at the interfacelocations for the respective momentum equation

)0( 632

2

)0( 236

22

,,,,2,,1,,,,1

,,

,,,,1,,,,1,,2

,,

,,,,

,,kji

kjikjikjikjikji

kjikjikjikjikji

kji

kjikji

kji ux

uuuuu

ux

uuuuu

x

uu

x

uu

x

vuvuvuvu

y

uv kjikjikjikjikjikjikjikji

kji

24

2727 *,2,

*,2,

*,1,

*,1,

*,,

*,,

*,1,

*,1,

,,

2

,,2,,1,,,,1,,2

,,2

2

12

163016

x

uuuuu

x

u kjikjikjikjikji

kji

Mass Conservation

Momentum Conservation

Convection Term

Diffusion Term

DISCRETIZATION

Staggered Grid vs. Collocated GridNote for Cactus: Array indexing

Note for Cactus: Felxiblity in choosing order and stencil

Note for Cactus: Interpolation stencils & order

Note for Cactus: Felxiblity in choosing order and stencilDesign code for higher order stencilsImplementation-specify stencil locations and order

•Recursive relation for the finite-difference weights on arbitrary grids

•The Fornberg Module can be used for interpolation as well as nth- derivatives for desired order of accuracy (provided that sufficient data is given)

•Simple and Ingenious

Remarks on Fornberg Algorithm-Module

Large Eddy Simulations (LES)

•Subgrid Scale Modeling

•Filtering

j

SGSij

j

i

ij

jii

xx

u

x

p

x

uu

t

u

2

2

jijiSGSij uuuu

dyyfyxGxfb

a

Homogeneously filtered incompressible Navier-Stokes equations

TOP HAT

GAUSSIAN

Examples of some filters

)(

)

(2

)(48

64

1ˆ

1,1,11,1,11,1,11,1,11,1,11,1,11,1,11,1,1

,1,1,1,11,,11,,11,1,1,1,

1,1,1,1,1,,11,,1,1,1,1,1

1,,1,,,1,,1,,,1,,1,,

,,

kjikjikjikjikjikjikjikji

kjikjikjikjikjikji

kjikjikjikjikjikji

kjikjikjikjikjikjikji

kji

•Compact or Pade filters

•Fourier cut-off (or use some transfer function to do convolution in fourier space)

•Averaging (cell-volume)

•Quadrature rules (Tensorial product in multi-dimensions for Simpson’s or Trapezoidal Rule)

A volume-averaging 2Δ width filter

Typical filters used in finite-difference calculations

dyyfyxGxfb

a

Cactus: Filtering leads to weighted averaging or interpolation. Use interpolation thorns

•Need explicit filters for finite-difference calculations

•Need filters for Large-eddy simulations methodology

•Filters are subjected to several physical and mathematical constraints

Remarks on Filter-module

1

1

11

~2

1

2

3~

nn

nnnnn

pt

uu

DCDCt

uu

uDuuC 2

Re

1,

t

up

u

t

u

t

up

n

n

~0

~

2

1

12

Pressure-Poisson Equation: Basic Details of Fractional-Step Scheme

Fractional Step Scheme

Convection, Diffusion Terms

Continuity Constraint

Pressure-Poisson Equation

ij

ij

j

i

ij

jii

j

j

fxx

U

x

p

x

UU

t

U

x

U

2

2

Re

1

0

kjikji

kjikji

kji

kjikjikji

gpkfy

p

x

p

gz

p

y

p

x

p

,,,,

,,2

2

,,2

2

,,

,,2

2

,,2

2

,,2

2

ˆˆˆˆ

kGkYkPkPX

kGkPIkfYkPkPX

ˆˆˆ

ˆˆ)(ˆˆ

kPkDkPkYPDPX yyyxxx11 )()(,

kPkGPkG

kPkPPkP

kGkDkPkPD

yx

yx

yx

ˆˆ

ˆˆ

ˆˆˆ

1

1

kPkPPkP

k

kGkP

yx

yjxi

jiji

1

,,

ˆˆ

ˆˆ

Expand Poisson’s Equation

Take Fourier Transform in z-direction

Write differential operators in Matrix form

Absorb wavenumbers in one of the operators (say Y)

Introduce Eigenvalue Decomposition

Simple matrix manipulations by pre/post multiplying the appropriate operators

Take Inverse Fourier transform of the last step to get the solution

Pressure Poisson Solver: Assembly of Thorns

•Parallel direct sparse solver (PSPASES)

•Parallel (fast and efficient) elliptic solvers of any kind

•Parallelization using domain decomposition (Influence matrix approach)

Remarks on Pressure-Poisson equation-Module

Remarks on Pressure-Velocity Coupling Strategies

•Poisson Equation-Direct Solver

•Poisson Equation-Iterative Solver

•Pseudo-Compressibility Approach

•SCGS, SCGS-PP, SCGS-PPV

• SIMPLE, SIMPLER, SIMPLEC, SIMPLEM (Moukalled)

• Note to Cactus: Thorns for different solution strategies; specify which during compilation

• Straightfoward Domain decomposition is to simply break up the Cartesian grid into blocks of approximately the same size. Need thorn for domain decomposition.

• Give each process ghost cells which are filled using information communicated from the neighbors.

• The size of these ghost cells (and therefore the size of the data to be communicated) depends on the order of the schemes used. Need thorn for automatically assigning ghost cells based on the order of scheme.

• Use calls to the MPI library to explicitly send and receive data.

Parallelization

Domain decomposition

Adaptive Mesh Strategies

• Concept of Springs with stiffness parameters (Harvey and Acharya)

• Concept of truncation error estimate defining region to be flagged for further refinement. Flagged region is then re-gridded leading to embedded grids with potential multi-grid strategies (Moukalled and Acharya)

• Concept of Adaptive Differencing schemes (Rhodes and Acharya)

Harvey & Acharya, 1991

Moukalled & Acharya, 1980’s

1. Error estimator2. Surface Mesh3. Metrics4. Poisson Solver

Block Body Fitted Grids

Immersed Boundary Method

11

111

11

~

]2

1

2

3[

2

1

2

3~

nn

nnnnnn

nnnnn

pt

uu

xxDCDCt

uuf

fDCDCt

uu

f

t

up

ut

uf

t

up

n

n

~0

~

2

1

1

2

• Poisson Equation Thorn• Interpolation Thorn

pi-2,j

vi+2,j-1

pi,j

vi-2,j-3

ui,j ui+2,j

vi,j-1

vi,j

ui-1,j-2

ui-1,j

ui+2,j-2ui,j-2

ui-1,j+1i-3,j+1

vi+2,j

vi-2,j-1

pi,j-2 pi+2,j-2

pi,j+1

pi-2,j-2

pi+2,j

pi+2,j+1pi-2,j+1 ui,j+1

i-3,j-2

i-3,j

vi-2,j

vi+2,j+1vi,j+1vi-2,j+1

vi+2,j-3vi,j-3

ui+2,j+1

ui-2,j

ui-2,j-2

ui-2,j+1

pi-1,j

pi-1,j+1

pi-1,j-2

vi-1,j-1

vi-1,j

vi-1,j+1

vi-1,j-2

vi-1,j-3

vi-2,j-2 vi-1,j-2 vi,j-2 vi+2,j-2

pi,j-1 pi+2,j-1pi-1,j-1pi-2,j-1i-3,j-1 ui-2,j-1 ui-1,j-1 ui,j-1 ui+2,j-1

ui+1,j+1

ui+1,j

ui+1,j-1

ui+1,j-2

vi+1,j-2

vi+1,j-1

vi+1,j

vi+1,j-3

pi+1,j-2

pi+1,j-1

pi+1,j

pi+1,j+1

vi+1,j+1

+j

+i

immersed boundary point

rejected stencil

immersed boundary

chosen stencil

Vext

Vint

Vd

Vc

VimV2

V1

h1

h2

Vim = (V1/h1+V2/h2)/(1/h1+1/h2)Vm : Computed Velocity influencing the immersed pointVim : Immersed Velocityhm : Distance between immersed point and calculated point

Vint = Vd [2]/[ 2 – ] – Vc []/[ 2 – ] Vext = Vd []/[ 2 – ] + Vc [ – ]]/[ 2 – ]

Different Interpolation Strategies for Immersed Boundary forcing

Flow past heated circular cylinder in a channel at ReD = 100

Validation of Immersed Boundary Method

Temperature Vorticity

Table 1: Comparison of experimental and computed values of drag coefficients, Nusselt number and Strouhal number (a: White (1990), b: Incropera and Dewitt (1990)).

ComputedTheoretical / Experimental

CDp Pressure Drag

CDf Friction Drag

0.6200.593

1.2 (Total drag)a

Nusselt Number 5.45 5.21 (±20%)b

Strouhal Number 0.283 0.281-0.287a

Film Cooling

Internal Cooling

Stator-Rotor InteractionApplications of LES in Turbomachinery

95 Cylinders array (Staggered Arrangement)

Handling Complex Geometries With IBM

Pressure Contours

Velocity Vectors

Biological Application: Suspension Feeding by a Mussel

1

2

3 4

5

6

7891011

Trajectory for the frontal and lateralciliary strutures

Forward Stroke(1-7):Rotation of Line about the pivoting point

Backward Stroke(8-11-1):Specified trajectory for Ciliary tip andprofile for the cilia between pivot and tip(assume a parabolic profile)

1. Cilia is described by collection of points oneach strand

2. Motion of each strand will be pre-defined3. Grid resolution for flow is much larger than

ciliary diameter4. Algorithm to influence fluid around cilia is

based on identification of cell containing thesestrand points and evaluation of velocity andaccelaration of the configurational center ofsuch strands

Frontal/Lateral Cilia

Trajectory of Latero-Frontal Cirri is defined on planeswhich have simpler motion i.e. rotation along oneaxis and translation

Points on cirri are allowed to move on each of these planes

Motion is defined by asymmetric ellipse for each pointon the strand

This yields wider cross-section (projected) to sweepfluid in forward stroke and smaller cross-sectionin backward stroke

Symmetry line for CirriFinal motion of the points is obtained bytransformation of the in-plane trajectorieson the rotated planes

Latero-Frontal Cirri

Algorithm to spread influence of body force (Lagrangian) on grid nodes (Eulerian)

1. Define a curve through the lagrangian points

2. Find intersections with grid lines and interpolate the flow fields at these intersection points

3. Calculate the weights for spreading operation between the grid nodes and the intersection points

Simpler Problem: Flow Field around Cilia RowContours of Streamwise Component of velocity

Free Slip

Free Slip

Free Slip

Free Slip

No Slip

Chemical Industry Application: Mixing in Stirred-Tank Reactors

Turbomachinery Application: Unsteady Stator-Rotor Interactions

Vorticity

Pressure

•Need Immersed boundary forcing scheme to solve moving geometries

•Need systematic way to introduce immersed boundary points

•Applications: Biological -> Chemical Industry -> Turbomachinery

Remarks on Immersed Boundary Method-Module