Openpipeflow 1.02b Doc

8/12/2019 Openpipeflow 1.02b Doc

1/22

A Pipe-Flow Code

Primitive variable version

Ashley P. Willis

May 29, 2014


2/22

CONTENTS

1 Getting started 3

1.1 Overview of files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31.2 Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

1.3 Input files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41.4 Output . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

1.4.1 Snapshot data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41.4.2 Time-series data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

1.5 Compiling libraries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51.6 Compiling and running . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61.7 Monitoring a run . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61.8 Making utils . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

2 Spatial representation 8

2.1 Finite differences r . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

2.1.1 Differentiation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82.1.2 Integration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82.1.3 Matrix representation of linear operators . . . . . . . . . . . . . . . 9

2.2 Fourier representaion , z . . . . . . . . . . . . . . . . . . . . . . . . . . 92.2.1 Expansion of variables . . . . . . . . . . . . . . . . . . . . . . . . . . 92.2.2 Orthogonality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92.2.3 Energy integral . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92.2.4 Volume integral . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

2.3 Enforced conditions at the axis . . . . . . . . . . . . . . . . . . . . . . . . . 10

3 Model 11

3.1 Governing equations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113.1.1 Non-dimensionalisation / scales . . . . . . . . . . . . . . . . . . . . . 113.1.2 Dimensionless parameters . . . . . . . . . . . . . . . . . . . . . . . . 113.1.3 Evolution equations . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

3.2 Decoupling the equations . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113.3 Simple projection method . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123.4 PPE-formulation with correct boundary conditions . . . . . . . . . . . . . . 123.5 Predictor-corrector . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133.6 Timestep control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

4 The code 15

4.1 Naming conventions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15


3/22

CONTENTS 2

4.1.1 Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154.1.2 Public variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

4.2 Ordering the Fourier modes . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

4.3 Data types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164.3.1 coll, spec storage of coefficients . . . . . . . . . . . . . . . . . . . 164.3.2 phys real space data . . . . . . . . . . . . . . . . . . . . . . . . . 174.3.3 mesh, lumesh storage of banded matrices . . . . . . . . . . . . . . 174.3.4 rdom the radial domain . . . . . . . . . . . . . . . . . . . . . . . 17

4.4 Modifications for the parallel implementation . . . . . . . . . . . . . . . . . 17

A Differential op erators in cylindricalpolar coordinates 20


4/22

1GETTING STARTED

1.1 Overview of files

Makefilewill require modification for your compiler and libraries (see 1.5). Samplecommands for other compilers can be found near the top of the file.

parallel.hEnsure _Npis set to 1if you do not have MPI. This file contains macrosfor parallelisation that are only invoked if the number of processes Np is greater than1.

program/parameters.f90, where youll find parameters. See 1.2

utils/ contains a number of utilities. Almost anything can be done in a util, both

post-processing and analysing data at runtime. There should be no need to alter thecore code. See 1.8

Matlab/, a few scripts. See Matlab/Readme.txt.

doc/, this document!


5/22

Chapter 1. Getting started 4

1.2 Parameters

./parallel.h:

_Np Number of processors (set 1 for serial)./program/parameters.f90:

i_N Number of radial pointsn [1, N]i_K Maximum k (axial), k (K, K)i_M Maximum m (azimuthal),m [0, M)i_Mp Azimuthal periodicity, i.e. m = 0, Mp, 2Mp, . . . , (M 1)Mp

(set =1 for no symmetry assumption)d_Re Reynolds numberReor Rmmd_alpha Axial wavenumber = 2/Lzb_const_flux Enforce constant fluxUb =

12 .

i_save_rate1 Save frequency for snapshot data files

i_save_rate2 Save frequency for time-series datai_maxtstep Maximum number of timesteps (no limit if set< 0)d_cpuhours Maximum number of cpu hoursd_time Start time (taken fromstate.cdf.inif set < 0)d_timestep Fixed timestep (typically 0.01 or dynamically controlled if set< 0)d_dterr Maximum corrector norm,fcorrd_courant Courant number Cd_implicit Implicitness c

1.3 Input files

State files are stored in the NetCDF data format and can be transferred across differ-ent architectures safely. The program main.out runs with the compiled parameters (seemain.info) but will load states of other truncations. Copy any state file e.g. state0018.cdf.datto state.cdf.in It will interpolate if necessary.

state.cdf.in:

t Start time. Overridden by d_timeifd_time0t Timestep. Ignored, see parameter d_timestep.N, Mp, rn Number of radial points, azimuthal periodicity,

radial values of the input state. If the radial points differ the fields are

interpolated onto the new points.

ur, u, uz Field vectors

. IfK= K or M= M, harmonics are truncated orzeros appended.

1.4 Output

1.4.1 Snapshot data

Data saved every i_save_rate1timesteps:

state????.cdf.dat

vel_spectrum.dat

All output is sent to the current directory, and state files are numbered 0000,0001,0002, . . . .Each file can be renamed state.cdf.in should a restart be necessary. To list times t for


6/22


each saved state file,

grep state OUT

The spectrum files are overwritten each save as they are retrievable from the state data.To verify sufficient truncation, a quick profile of the energy spectrum can be plotted with

gnuplot> set log

gnuplot> plot vel_spectrum.dat w lp

1.4.2 Time-series data

Data saved every i_save_rate2timesteps:

tim_step.dat t, t, tf, tCF L current and limiting step sizes

vel_energy.dat t, E energies and viscous dissipation

vel_friction.dat t, Ub or , uz(r= 0)z , u bulk velocites, pressure, friction vel.

1.5 Compiling libraries

Take a look at Makefile. The compiler and flags will probably need editing. There aresuggested flags for many compilers at the top of this file.

Building the main code is successful if running makeproduces no errors, but this requiresthe necessary libraries to be present - LAPACK, netCDF and FFTW. Often it is necessaryto build LAPACK and NetCDF with the compiler and compile flags that will be used forthe main simulation code.

The default procedure for building a package (FFTW3, netCDF) is

tar -xvvzf package.tar.gzcd package/

[set environment variables if necessary]

./configure --prefix=

make

make install

FFTW3. This usually requires no special treatment. Install with your package man-ager or build with the default settings.

LAPACK. If using gfortran you might be able to use the binary version supplied foryour linux distribution. Otherwise, edit the file make.inc that comes with LAPACK, setting

the fortran compiler and flags to those you plan to use. Type make. Once finished, copythe following binaries into your library path (see Makefile LIBS -L/lib/)cp lapack.a /lib/liblapack.a

cp blas.a /lib/libblas.a

netCDF. If using gfortran you might be able to use the supplied binary version foryour linux distribution. Otherwise, typical environment variables required to build netCDFareCXX=""

FC=/opt/intel/fc/10.1.018/bin/ifort

FFLAGS="-O3 -mcmodel=medium"

export CXX FC FFLAGS

After the make install, ensure that files netcdf.mod and typesizes.mod appear in your


7/22


include path (see Makefile COMPFLAGS -I/include/). can be found at the top ofthe pagehttp://www.unidata.ucar.edu/downloads/netcdf/current/index.jspVersion 4.1.3 is rela-tively straight forward to install, the above flags should be sufficient. It is trickier for morerecent versions [2013-12-11 currently netcdf-4.3.0.tar.gz]. First./configure --disable-netcdf-4 --prefix-

which disables HDF5 support (not currently required in code). Fortran is no longer bun-dled, so get netcdf-fortran-4.2.tar.gz or a more recent version from herehttp://www.unidata.ucar.edu/downloads/netcdf/index.jsp

Build with the above flags, and in additionCPPFLAGS=-I/include

export CPPFLAGS

Add the link flag -lnetcdffbefore -lnetcdf in the Makefile.

1.6 Compiling and running

For serial use set _Npto 1 in parallel.h. Otherwise MPI is required and the compile mustbe updated in the Makefile. First set parameters in program/parameters.f90. Next type

> make

> make install

The second command creates the directoryinstall/and a text file main.info, which is arecord of settings at compile time. Next an initial condition state.cdf.inis needed,

> mv install ~/runs/job0001

> cd ~/runs/job0001/> cp .../state0100.cdf.dat state.cdf.in

> nohup ./main.out > OUT 2> OUT.err &

Press enter again to see if main.out stopped. If it has stopped, checkOUT.err or OUT forclues why. If it appears to be running, wait a minute or two then

> rm RUNNING

This will terminate the job cleanly.NOTE: Any output state, e.g. state0012.cdf.dat can be copied to state.cdf.in to be

used as an initial condition. If resolutions do not match, they are automatically interpolated

or truncated. This is how I generate almost all initial conditions, by taking a state from arun with similar parameters. If there is a mismatch in Mp, use utils/changeMp.f90.

1.7 Monitoring a run

Immediately after starting a job, its a good idea to check for any warnings

> less OUT

To find out number of timesteps completed, or for possible diagnosis of an early exit,

> tail OUT


8/22


The code outputs timeseries data and snapshot data, the latter has a 4-digit number e.g.state0012.cdf.dat.

To see when the in the run each state was saved,

> head -n 1 vel_spec* | less

I often monitor progress with tail vel_energy.dator

> gnuplot

> plot vel_energy.dat w l

Use rm RUNNINGto end the job.

1.8 Making utils

The core of the code in program/ rarely needs to be changed. Almost anything can b edone by creating a utility; there are many examples in utils/

InMakefile, setUTIL = utilname (omitting the.f90extension), then typemake utilwhich builds utilname.out.


9/22

2SPATIAL REPRESENTATION

2.1 Finite differences r

A function p(r) is represented by the values pn = p(rn), where p(r) is evaluated on theNradial points rn.

2.1.1 Differentiation

Taylor expansions about central point x0, k neighbouring points each side

f(xk) = f(x0) + (xk x0) f(x0) +

(xkx0)

2

2! f(x0) + . . .

...

f(xk) = f(x0) + (xk x0) f(x0) + (xkx0)22! f

(x0) + . . .

(2.1)

The expansions (2.1) can be written

f= A df, df = A1 f, f=

f(xk)...

f(xk)

, df=

f(x0)f(x0)f(x0)

...

(2.2)

Derivatives are calculated using weights from the appropriate row of A1.

2.1.2 Integration

Integrating (2.1) the indefinite integral may be approximated about x0 f(x) dx = (x x0) f(x0) +

(xx0)2

2! f(x0) +

(xx0)3

3! f(x0) + . . . (2.3)

= [ (x x0) (xx0)2

2! . . . ] A1f (2.4)

The total integral is approximated by

1

2 n xn+1

xn1

f(x) dx (2.5)


10/22

Chapter 2. Spatial representation 9

2.1.3 Matrix representation of linear operators

Linear equations can be written in the matrix form

Lp= Mq+ s. (2.6)

Land Mare N Nmatrices representing linear operators. Ifq and s are known then theright-hand side is simple to evaluate. Consider the equation

Lp= q, (2.7)

where p is unknown. If the matrix operators are formed by linear combinations of theweights in2.1.1, using onlyk neighbouring points to each side, they are banded. Boundaryconditions are formed in the same way and are placed in the end rows. The equation issolved by forward-backward substitution, using the banded LU-factorisation of L.

BC

p q

=

LU

2.2 Fourier representaion , z

2.2.1 Expansion of variables

A(, z) =km

Akmei(kz+m0m) (2.8)

Areal impliesAkm = Ak,m (coefficients are conjugate-symmetric). It is sufficient to keep

only m 0, and for m = 0 keep k 0.

2.2.2 Orthogonality

Exponentials

20 e

i(m+n)

d= 2 m,n ,

2

0 ei(k+j)

dz =

2

k,j (2.9)

2.2.3 Energy integral

E = 12

A A dV =

22

km

|Akm|

2 r dr (2.10)

E=

m=0

Em, Em =

22

A200 r dr+

42

k>0

|A2k0| r dr, m= 0,

42

k |Akm |2 r dr, m >0.

(2.11)


11/22

Chapter 2. Spatial representation 10

2.2.4 Volume integral

E = 12

A dV = 42

A00 r dr (2.12)

2.3 Enforced conditions at the axis

The geometry enforces conditions on each variable at the axis when expanded over Fouriermodes in . Given a vector A, each mode for Az is even in r ifm is even, and is odd ifmis odd. Each mode for Ar and A is even in r ifm is odd, and is odd ifm is even.


12/22

3MODEL

3.1 Governing equations

3.1.1 Non-dimensionalisation / scales

length R,velocity, fixed flux 2 Ub,

velocity, fixed pressure Ucl of HPF.For HPF Ucl= 2 Ub.

3.1.2 Dimensionless parameters

Reynolds number, fixed flux, Rem = 2UbR/

Reynolds number, fixed pressure, Re = UclR/1 + = Re/Rem

Re =uR/= (2 Rem(1 + ))1

2 = (2 Re)1

2

3.1.3 Evolution equations

Fixed flux,

(t+ u )u= p + 4

Rem(1 + )z+

1

Rem

2u (3.1)

Fixed pressure

(t+ u )u= p + 4

Rez+

1

Re

2u (3.2)

Let u= W(r)z+ u. Using the scaling above, W(r) = 1 r2. Then

(t 1

Rem

2)u =u ( u)dW

dr uzz W zu

+ 4

Remz p . (3.3)

3.2 Decoupling the equations

The equations for ur and u are coupled in the Laplacian. They can be separated in aFourier decompositon by considering

u= uri u, (3.4)


13/22

Chapter 3. Model 12

for which the are considered respectively. Original variables are easily recovered

ur

=1

2(u

++ u

), u

=

i

2(u

+ u

). (3.5)

Governing equations are then

(t 2) u = N(p), (3.6)

(t 2) uz = Nz(p)z , (3.7)

where

2= 2

1

r2

i

r2 (3.8)

3.3 Simple projection method

The basic projection method offers simplicity and flexibility. The error measured by thepredictor-corrector is second order (on the nonlinear terms), but the method is first orderoverall. See following section for a better method.

The following steps are performed, where qrefers to time tq:

u uq

t = Nq+

1

2 , (3.9)

uq+1 = u , (3.10)

where

2 = u. (3.11)

3.4 PPE-formulation with correct boundary conditions

Write the time-discretised NavierStokes equations in the form

Xuq+1 = Yuq + Nq+

1

2 p ,

2p = (Yuq + Nq+1

2 ),(3.12)

whereqdenotes timetq, which is sixth order in r for uq+1 and second order forp, where the

solenoidal condition is not explicitly imposed. Symmetry conditions provide the conditionsat the axis. The difficulty is in imposing the remaining four this system should b e

inverted, in principle, simultaneously for p and uq+1

with boundary conditions uq+1

= 0and uq+1 = 0 on r = R (Rempfer 2006). In practice it would be preferable to invert forp first then for uq+1, but the boundary conditions to not involve p directly.

Note that the Yuq term has been included in the right-hand side of the pressure-Poissonequation, the divergence of which should be small. Assume that pressure boundary condi-tion is known: the right-hand side of the NavierStokes equation is then projected onto thespace of solenoidal functions though p and hence after inversion, uq+1 will be solenoidal.

Consider the bulk solution, {u,p}, obtained from solution of the following:

Xu = Yuq + Nq+

1

2 p ,

2p = (Yuq + Nq+1

2 ),(3.13)


14/22

Chapter 3. Model 13

with boundary conditionsu= 0 and r p= 0. Introduce the following systems:

Xu = p ,2p = 0, (3.14)

with boundary conditionsu= 0 and rp= 1 on r= R, and Xu = 0, (3.15)

with boundary conditions u+ = 1, u = 1, uz = i on r = R. The system (3.14) providesa linearly independent function uj that may be added to u without affecting the right-hand side in (3.13), but altering (to correct) the boundary condition applied. Similarly thesystem (3.15) provides a further three functions. The superposition

uq+1

=u+

4j=1

ajuj (3.16)

may be formed in order to satisfy the four boundary conditions, uq+1 =0 and uq+1 = 0on r = R. Substituting (3.16) into the boundary conditions, they may be written

Aa= g(u), (3.17)

where A = A(g(u)) is a 44 matrix. The appropriate coefficients required to satisfy theboundary conditions are thereby recovered from solution of this small system for a. Theerror in the boundary conditions gj(u

q+1) using the influence-matrix technique is at thelevel of the machine epsilon, typically order 1e-18.

The functionsuj(r), the matrix A and its inverse may all be precomputed. The boundary

conditions for u have been chosen so that that u are pure real, uz is pure imaginary, and

A is real. For each timestep, this application of the influence matrix technique requiresonly evaluation of the deviation from the boundary condition, multiplication by a 44 realmatrix, and the addition of only two functions to each component ofu, each either purereal or pure imaginary. Compared to the evaluation of nonlinear terms, the computationaloverhead is negligible.

3.5 Predictor-corrector

The model equation for each Fourier mode is

(t 2)f=N, (3.18)

where nonlinear terms have been evaluated on each radial point by the transform method,and is solved as in2.1.3. The predictor at time tq, with Euler nonlinear terms and implic-itnessc

fq+11 fq

t

c2fq+11 + (1 c)

2fq

= Nq, (3.19) 1

t c2

fq+11 =

1

t+ (1 c)2

fq + Nq. (3.20)

Corrector iterations are

1t c

2

fq+1j+1 =

1t + (1 c)2

fq + c Nq+1j + (1 c) Nq, (3.21)


15/22

Chapter 3. Model 14

or equivalently, for the correction fcorr = fq+1j+1 f

q+1j

1t c

2fcorr = c N

q+1

j c N

q+1

j1 , (3.22)

where j = 1, 2, . . . and Nq+10 =Nq. The size of the correction fcorr must reduce at each

iteration. For c = 12 the scheme is second order such that fcorr t2.

3.6 Timestep control

t= C min( / |v|), 0< C < 1, (3.23)

where C is the Courant number.The timestep should also be small enough such that the corrector norm fcorr is satis-

factorily small. This may be important for integrating initial transients, but ought not tobe the limiting factor in general.


16/22

4THE CODE

4.1 Naming conventions

4.1.1 Parameters

Parameters defined within parameters.f90are given a prefix indicating the data type.For example:

N i_N integert d_timestep doublefixed flux? b_fixed_flux boolean

Note that the naming convention frees the variable name for dummy variables, e.g.:

integer :: ndo n = 1, i_N

...

4.1.2 Public variables

By default variables declared within a module are accessible to any other module orsubroutine that uses the module. To ensure that it is clear where a public variable isdeclared, variables are given a prefix. Examples:

t tim_dt timestep.f90ur vel_r velocity.f90

4.2 Ordering the Fourier modes

A=km

Akmei(kz+m0m), (4.1)

Coefficients are stored as in the following example for m0=Mp= 2, M= 4 andK= 4:


17/22

Chapter 4. The code 16

m m_ nh

3 6 18 19 20 21 22 23 24

2 4 11 12 13 14 15 16 171 2 4 5 6 7 8 9 100 0 0 1 2 3

-3 -2 -1 0 1 2 3 k

Rather than two indices k and m, the single index nh labels the modes. In the code thisordering corresponds to the typical loop

nh = -1

do m = 0, M-1

m_ = m*Mp

do k = -(K-1), (K-1)

if(m==0 .and. k


18/22


4.3.2 phys real space data

The type (phys) is used for data in real space.

type phys

double precision :: Re(0:i_Z-1, 0:i_Th-1, i_N)

end type phys

The element Re(k,m,n)refers to the value at zk, m and rn.

4.3.3 mesh, lumesh storage of banded matrices

The finite-difference stencil has finite width (2.1) and so matrix operations involve ma-trices that are banded. The type (mesh), defined in meshs.f90, is used for matrices onthe radial mesh involving kl points to either side of each node.

type mesh

double precision :: M(2*i_KL+1, i_N)

end type mesh

For a matrix A, the element M(KL+1+n-j, j) = A(n,j). See the man page for the lapackroutine dgbtrf. The LU-factorisation of a banded matrix is also banded:

type lumesh

integer :: ipiv(i_N)

double precision :: M(3*i_KL+1, i_N)

end type lumesh

The element M(2*KL+1+n-j, j) = A(n,j). Pivoting information is stored along with thematrix.

4.3.4 rdom the radial domain

The type (rdom) contains information the radial domain. The public variable mes_Dis declared in meshs.f90. Data includes the number of radial points N; powers of r,rpn=r(n,p); weights for integration rdr; matrices for taking the p

th derivative dr(p); . . .

type rdom

integer :: N

double precision :: r(i_N,i_rpowmin:i_rpowmax)double precision :: intrdr(i_N)

type (mesh) :: dr(i_KL)

end type rdom

4.4 Modifications for the parallel implementation

The code has been parallelised using the Message Passing Interface (MPI). Sections of thecode with special MPI function calls are separated from the serial sections by preprocessordirectives. The number of processors is given by the parameter _Npset inparallel.h. Therank of the local processor is given by mpi_rnkdeclared in mpi.f90, and mpi_sze=_Np.

Data type changes for the model equation (see equations 2.7, 3.18) are as follows:


19/22


q * s=pL

mesh

= g

lumesh

on radial subdomainall real space values

for radial subdomainall spectral coefficients

selection of modes

all radial points forcoll

spec

transpose

transform

g

coll

spec

phys

multiply

invert

p

coll

coll

spec

phys

actorise

transform

for radial subdomainall spectral coefficients

selection of modesall radial points for

transpose

phys

Data is split over the spherical harmonics for the linear parts of the code evaluatingcurls, gradients and matrix inversions for the timestepping. These linear operations donot couple modes. Here all radial points for a particular mode are located on the sameprocessor, separate modes may be located on separate processors. In the definition oftype (coll)the parameteri_H1is replaced byi_pH1 = (_Np+i_H1)/_Np-1. The variablevar_H has elements to determine which harmonics are on which processor. The elementspH0 and pH1 are the zeroth index and one-minus the number of harmonics on the localprocessor. The corresponding values for all processors are stored in the arrays pH0_(rank)and pH1_(rank). For a variable of type (coll), valid values for the harmonic index inRe(n,nh)are nh=0..pH1, which refer to the indices (4.2) nh=pH0..pH0+pH1.

Data is split radially when calculating Fourier transforms and when evaluating productsin real space. In the definitions of type (spec) and type (phys) the parameter i_N isreplaced by i_pN = (_Np+i_N-1)/_Np. The location of the radial subdomain on the localprocessor is given by additional elements of the variable mes_D. The elements pNi and pNare the location of the first (inner) radial point and the number of points. The valuesfor all processors are stored in the arrays pNi_(rank) and pN_(rank). For a variable oftype (spec), valid values for the radial index in Re(nh,n)are n=1..pNwhich refer to theindices ofrn, where n=pNi..pNi+pN-1.

The bulk of communication between processors occurs during the data transposes inthe functions var_coll2spec() and var_spec2coll(). For _Np processors the data istransfered in _Np1 steps. At each step each processor sends and receives a block of data,the source and destination are selected systematically as set in the following loop:

do step = 1, mpi_sze-1

dest = modulo(mpi_rnk+step, mpi_sze)

src = modulo(mpi_rnk-step+mpi_sze, mpi_sze)

...


20/22


0 1 2 3

2 3step=1 src / dest

data

sections

nr / nh

The sketch is for the case _Np= 4. The size of each block is approximatelyi_pN(i_pH1+1),and the data within each block must also be transposed.


21/22

ADIFFERENTIAL OPERATORS IN CYLINDRICALPOLAR

COORDINATES

Gradient

(f)r = rf, (f) = 1

rf, (f)z = zf. (A.1)

Laplacian

2f= (1

rr+ rr)f+

1

r2f+ zzf. (A.2)

Laplacian of vector

(2A)r =2Ar

2

r2A

Arr2

,

(2A) =2A+ 2r2

Ar A

r2, (A.3)

(2A)z =2Az.

Divergence

A= (1

r+ r)Ar+

1

rA+ zAz. (A.4)

Curl

( A)r =1

rAz zA,

( A) = zAr rAz, (A.5)

( A)z = ( 1r

+ r)A 1r

Ar.

Directional derivative

(A B)r = ArrBr+A

r Br+ AzzBr

ABr

,

(A B) = ArrB+A

r B+ AzzB+

ABrr

, (A.6)

(A B)z =ArrBz+A

r Bz+ AzzBz.


22/22

Appendix A. Differential operators in cylindricalpolar coordinates 21

Toroidalpoloidal decompositionsAxial

A= (z) + (z), (A.7)

= (r,,z), = (r,,z).

2hf= (1

rr+ rr)f+

1

r2f. (A.8)

Ar =1

r+ rz,

A =r+1

rz, (A.9)

Az = 2h.

Radial

A= 0 + 0 z+ (r) + (r), (A.10)

0= 0(r), 0= 0(r), = (r,,z), = (r,,z).

2cf= 1

r2f+ zzf. (A.11)

Ar = r2c,

A =0+ rz+ r, (A.12)

Az =0 + (2 + rr)z.

Documents

Openpipeflow 1.02b Doc