Numerical Methods Used in Fusion Science Numerical Modeling

Masatoshi Yagi

Japan Atomic Energy Agency, Rokkasho Fusion Institute

7th ITER International School, Aix-en-Provence, 25-28, Aug. 2014

Quick Tour in BA Rokkasho Site

3!

Broader  Approach  (BA)  Ac.vi.es  In  parallel  to  the  ITER  program,  BA  ac.vi.es  are  being  implemented  by  the  EU  and  Japan,    aiming  at  early  realiza.on  of  the  fusion  energy

Naka

Cadarache

Rokkasho

4!

Rokkasho  BA  site  

JAPAN

Tokyo!

Fukushima!

Aomori Pref.!

http://map.google.co.jp

Rokkasho!

Aomori City!

BA  Site  

Pacific Ocean

Rokkasho BA site!

Naka site

5!5

IFMIF/EVEDA  Accelerator  Bldg

Computer  Simula.on  &    Remote  Experimental  Bldg  DEMO  R&D  Bldg

Administra.on  &    Research  Bldg  

Rokkasho BA site u 　Constructions of all buildings were completed by March 2010.!

Security

Power Station(30

MVA)

ITER

Material Behavior �

Two-phase Heat Flow �

Contribu.on  to  Comprehensive  Promo.on    of  Demo  Reactor  Development�

Demo Reactor Design �

Interna.onal  Fusion  Energy  Research  Center  Computer  Simula.on  Center

Simula'on  Research  on  Burning  and  Steady  State  Advanced  Plasma  Behaviors    for  ITER/Satellite  Tokamak,  Demo  Reactor  Design,  Advanced  Fusion  Material  Development,  etc   Effec've  and  Efficient  Promo'on  of  ITER  Project  and  early  Realiza'on  of    Fusion  Power  Plant

Satellite Tokamak �

~1.5PF    Super  Computer

Plasma Behavior

Burning Plasma Prediction

Advanced Plasma Data

7

HELIOS  in  IFERC-­‐CSC

2011 2012 2013 2014 2015 2016 2017

IFERC-­‐CSC Introduc.on Opera.on Shutdown

TOP500  Super  Computer  List  on  2012/11

Ini.al  Opera.on Normal  Opra.on

7

Rank  Site  Rmax  (Tflops) Rpeak  (Tflops)

1 DOE/SC/ORNL,  United  States

17590 27112.5

2 DOE/NNSA/LLNL    United  States

16324.75 20132.66

3 RIKEN  KEI,  Japan 10510.00 11280.38

・  ・  

・  ・  

・  ・  

・  ・  

15 IFERC-­‐CSC　HELIOS,  Japan

1237.0 1524.1

Interna.onal   1st  cycle   2nd  cycle   3rd  cycle   4th  cycle   5th  cycle  

Bull  computers  provided  by  F4E/CEA

Ø  TOP500  Super  Compu.ng  Ranking  on  2013/11  World  24th,  Domes.c  3rd

Ø  2014/1  System  Enhancement            Intel  Xeon  Phi,  Many  Integrated  Core  (MIC)    architecture)  

•  Theore.cal  performance　427Tflops  •  Linpack　　 225.1Tflops  

flops  (Floa.ng-­‐point  Opera.ons  Per  Second)  

Giga  (G):  109,  Tera  (T):  1012,  Peta  (P):  1015,  Exa  (E):  1018  

ü  Explicit Space Discretization

ü  Explicit Time Discretization

n  Boundary Value Problem

ü  Tomas Algorithm

n  Initial Value Problem

ü  Von Neumann Analysis

ü  Implicit Time Discretization

n  Eigen Value Problem

ü  Power Method

ü  Inverse Shifted Power Method

n  High Performance Computing

ü  Vectorization

ü  Open MP Programming

ü  Message Passing Interface (MPI) Programming

ü  Hybrid Programming

Example: One-Dimensional Boundary Value Problem

θ ''(x)=q(x), θ(0)=θ(1)=0

We will solve this problem, numerically.

Explicit Space Discretization

θi±1≡θ(x±Δx)=θ(x)±Δxθx(x)+Δx22 θxx(x)±

Assuming an equidistant grid,

Here the x subscript denotes differentiation, and the i subscript refers to the index of data points.

Three types of differences:

Forward:

Backward:

Central:

θx( )i =θi+1−θiΔx +Ο(Δx)

θx( )i =θi−θi−1Δx +Ο(Δx)

θx( )i =θi+1−θi−12Δx +Ο(Δx2)

n  Boundary Value Problem (BVP)

θxx( )i =θi+1−2θi+θi−1

Δx2+Ο(Δx2)

Second derivative

Formula for high-order approximation for interior points and for boundary points

θx( )i =aθi+bθi−1+cθi−2

Δx +Ο(Δx2)

One-sided, second-order finite difference

θi−2 =θi−2Δx(θx)i+2Δx2 θxx( )i−

2Δx( )3

6θxxx( )i+

θi−1=θi−Δx(θx)i+Δx22 θxx( )i−

Δx( )3

6θxxx( )i+

θi =θi

Δx θx( )i =(a+b+c)θi−Δx(2c+b) θx( )i+Δx22 (4c+b) θxx( )i+Ο(Δx

3)

a+b+c=02c+b=−14c+b=0

"

#$

%$

⇒a= 32

, b=−2, c= 12

θx( )i =3θi−4θi−1+θi−2

2Δx+Ο(Δx2)

Similarly, θx( )i =−3θi+4θi+1−θi+2

2Δx+Ο(Δx2)

1Δx2 (θi+1−2θi+θi−1)=qi, i=1,,N -1

θ0 =θN =0

−2 11 −2 1

. . .1 −2 1

. . .1 −2 1

1 −2

"

#

$$$$$$$$

%

&

''''''''

θ1θ2θ3.

θN−3θN−2θN−1

"

#

$$$$$$$$$$

%

&

''''''''''

=

Δx2q1−u0Δx2q2Δx2q3.

Δx2qN−3Δx2qN−2

Δx2qN−1−uN

"

#

$$$$$$$$$$$

%

&

'''''''''''

Dirichlet boundary condition

Tomas Algorithm for Tridiagonal Systems

Aθ =qSolution of BVP reduces to solving the linear system

where the matrix A is tridiagonal if the boundary condition are Dirichlet

1.  LU decomposition of matrix A=LU, L: lower trianglar matrix, U: upper triangular matrix

2.  Forward substitution Ly=q

3.  Backward substitution Ux=y

a1 c1b2 a2 c2

b3 a3 c3. . .

. . .bN−1 aN−1 cN−1

bN aN

"

#

$$$$$$$$$$

%

&

''''''''''

=

1l2 1

l3 1. .. .

lN−1 1lN 1

"

#

$$$$$$$$$

%

&

'''''''''

d1 u1d2 u2

d3 u3. .

. .dN−1 uN−1

dN

"

#

$$$$$$$$$$

%

&

''''''''''

d1=a1, u1=c1

Step 1: A=LU

lidi−1=biliui−1+di =aiui =ci

"

#

$$

%

$$

⇒ li=bi /di−1

di=ai−liui−1

ui=ci

"

#$

%$

, (i=2,N −1) lN =bN /dN−1, dN =aN −lNuN−1

Step 2: Ly=q

1l2 1

l3 1. .. .

lN−1 1lN 1

"

#

$$$$$$$$$

%

&

'''''''''

y1y2y3..

yN−1yN

"

#

$$$$$$$$$$

%

&

''''''''''

=

q1q2q3..

qN−1qN

"

#

$$$$$$$$$$

%

&

''''''''''

y1=q1

liyi−1+yi =qi⇒ yi =qi−liyi−1, (i=2,,N)

d1 u1d2 u2

d3 u3. .

. .dN−1 uN−1

dN

"

#

$$$$$$$$$$

%

&

''''''''''

θ1θ2θ3..

θN−1θN

"

#

$$$$$$$$$$

%

&

''''''''''

=

y1y2y3..

yN−1yN

"

#

$$$$$$$$$$

%

&

''''''''''

Step 3: Ux=y

θN = yN /dN,

diθi+uiθi+1= yi⇒θi =(yi−uiθi+1)/di, (i=N -1,,1)

Tomas algorithm will always converge if

ak ≥ bk + ck , k=2,,N −1

a1 > c1 , & aN > bN

n  Advection Equation

∂Θ∂t+U∂Θ

∂x=0

Θ(x,t=0)=sin2π x

with periodic boundary condition Θ(x=0,t)=Θ(x=1,t)

and initial condition

We will solve this equation, numerically

Euler-Forward/Central-Difference Scheme (EF/CD) [Explicit Scheme]

Θ jn+1−Θ j

n

Δt +UΘ j+1n −Θ j−1

n

2Δx =0, j=1,,N −1 Θ0n =ΘN

n and Θ j0 =sin2π xj

Von Neumann Stability Analysis

Θ jn = ak

n

k=−∞

∞∑ e2πikx j

akn+1=ak

n(1−iCsin2πkΔx) where C=UΔtΔx

CFL number (Courant, Friedrichs and Lewy)

akn+1

akn= 1+C2 sin2 2πkΔx

1/2>1

EF/CD scheme is absolutely unstable

Euler-Forward/Upwind-Differencing Scheme (EF/UD) [Explicit Scheme]

Θ jn+1−Θ j

n

Δt +UΘ jn−Θ j−1

n

Δx =0, j=1,,N −1

which can be rewritten as

Θ jn+1=Θ j

n−C2(Θ j+1

n −Θ j−1n )+C

2(Θ j+1

n −2Θ jn+Θ j−1

n )numerical diffusion

Von Neumann Stability Analysis

akn+1=ak

n[1−C(1−e−2πikΔx)]

akn+1

akn=[1+2(C2−C)(1−cos2πkΔx)]1/2

Stability Condition C≤1

Stability Region in Complex Plane an∝eλtn, an+1∝eλtn+1 =eλ(tn+Δt),Assuming

akn+1−ak

n

Δt=λak

n =−1−e−2πikΔx

Δxakn λΔt=−C(1−cos2πkΔx)−iCsin2πkΔx

Crank-Nicolson/Center-Differencing Scheme (CN/CD) [Implicit Scheme]

Θ jn+1−Θ j

n

Δt +U2Θ j+1n+1−Θ j−1

n+1

2Δx +U2Θ j+1n −Θ j−1

n

2Δx =0, j=1,,N −1

Von Neumann Stability Analysis

akn+1=

1−iC2sin2πkΔx

1+iC2sin2πkΔx

#

$

%%%

&

'

(((akn,

akn+1

akn=1 CN/CD scheme is neutrally stable

Eigen Values

λΔt=−C2

4sin2 2πkΔx−iC

2sin2πkΔx

1−C2

4sin2 2πkΔx

which are on the left half-plane for any positive value of C

1 C/4−C/4 1 C/4

. . .−C/4 1 C/4

. . .−C/4 1 C/4

−C/4 1

"

#

$$$$$$$$

%

&

''''''''

Θ1n+1

Θ2n+1

Θ3n+1

.ΘN−3n+1

ΘN−2n+1

ΘN−1n+1

"

#

$$$$$$$$$$$

%

&

'''''''''''

=

CΘ0n /4

+Θ1

n−CΘ2n /4+CΘ0

n+1/4

CΘ1n /4+Θ2

n−CΘ3n /4

CΘ2n /4+Θ3

n−CΘ4n /4

.CΘN−4

n /4+ΘN−3n −CΘN−2

n /4

CΘN−3n /4+ΘN−2

n −CΘN−1n /4

CΘN−2n /4+ΘN−1

n −CΘNn /4+CΘN

n+1/4

"

#

$$$$$$$$$$$$$

%

&

'''''''''''''

Tomas Algorithm is available to solve CN/CD scheme

Exercise #1 Consider full implicit scheme for advection equation and perform von Neumann stability analysis and calculate eigenvalue

Effects of Boundary Conditions

Example ∂Θ∂t+U∂Θ

∂x=0, Θ(x,t=0)sin2π x, 0< x<1, Θ(x=0,t)=−sin2πt

CN/CD Scheme

Θ jn+1−Θ j

n

Δt+1

2Θ j+1n+1−Θ j−1

n+1

2Δx+Θ j+1n −Θ j−1

n

2Δx

$

%&&

'

())=0, j=1,,N −1

Θ0n+1=−sin2πtn+1, Θ j

0 =sin2π xj

Fictitious Node j=N+1

Linear extrapolation ΘN+1=ΘN +ΔxΘN−ΘN−1

Δx=2ΘN −ΘN−1

⇒ΘNn+1−ΘN

n

Δt+12ΘNn+1−ΘN−1

n+1

Δx+ΘNn −ΘN−1

n

Δx

%

&''

(

)**=0

Addition of upwind derivative does not influence the stability of CN scheme

Eigenvalue Problem

Ax=λx

Local Eigensolvers

Power Method: simple method to obtain the maximum eigenvalue

xk+1=cAxk C: normalization constant

Assume there exists an eigenvalue λ1 that dominates

λ1 > λ2 ≥ λ3 ... ≥ λn

Initial Guess x0 =c1v1+c2v2++cnvn

xk =Axk−1==Akx0 =c1λ1kv1++cnλnkvn

xkc1λ1k

=v1+c2c1

λ2λ1

!

"#

$

%&k

v2 ++cnc1

λnλ1

!

"#

$

%&k

vn⇒v1 for k→∞

Convergence Rate λ2λ1

Pseudo-code

Initialize: x0

Begin Loop: for k=1,2,…

x̂k =Axk−1

xk = x̂kmax(x̂k)

endfor

End Loop:

max(y) returns the entry of y with maximum modulus

y=(4.32,−9.88,2.9)T, max(y)=−9.88Ex.

max(x̂k )→λ1, xk→v1

Rayleigh Quotient

R(A,x)≡ xTAxxTx

= (c1v1++cnvn)T A(c1v1++cnvn)

c12+c2

2++cn2

= λ1c12+λ2c2

2++λncn2

c12+c2

2++cn2 =λ1

1+ λ2λ1

"

#

$$

%

&

''

c2c1

"

#$$

%

&''

2++ λn

λ1

"

#

$$

%

&

''cnc1

"

#$$

%

&''

2

1+ c2c1

"

#$$

%

&''

2++ cn

c1"

#$$

%

&''

2 →λ1

Inverse Shifted Power Method: to selectively compute minimum eigenvalue

Ax=λx⇔A−1x=λ−1x

Axk+1=cxk This method is most effective with a proper shift

(A−σ I)xk+1=cxk Convergence Rate λn−σ

λn−1−σ

Pseudo-code

Initialize: x0

Begin Loop: for k=1,2,…

x̂k =U−1L−1xk−1

xk = x̂kmax(x̂k)

endfor

End Loop:

Choose Choose σ

Factorize A−σ I=LU

if R(A,xk )−R(A,xk−1) <ε return

LU decomposition

a11x1+a12x2++a1nxn =b1a21x1+a22x2++a2nxn =b2

an1x1+an2x2++annxn =bn

First stage of Gaussian elimination

a11x1+a12x2++a1nxn =b1

a22(1)x2++a2n

(1)xn =b2(1)

an2(1)x2++ann

(1)xn =bn(1)

aij(1)=aij−a1 jli1

(1), li1(1)= ai1

a11, bi

(1)=bi−b1ai1a11

(n-1)th stage gives upper triangular system U

L is constructed from lij(k) where k corresponds to kth column and i < j

Modified Inverse Shifted Power Method with Rayleigh Quotient

Pseudo-code

Initialize: x0

Begin Loop: for k=0,1,…

x̂k+1=(Uk1)−1(Lk)−1xk

xk+1 = x̂k+1max(x̂k+1)

endfor

End Loop:

Choose Compute σ0 =

(x0)TAx0

(x0)T x0

LkUk =A−σ kI

if σ k+1−σ k <ε return

σ k+1=(xk+1)TAxk+1

(xk+1)T xk+1

Exercise #2 Develop the code for modified inverse shifted power method,

then investigate eigenvalue of the matrix changing initial guess

A=1 1/2 1/31/2 1/3 1/41/3 1/4 1/5

!

"

###

$

%

&&&

x0 =1.01.01.0

!

"

###

$

%

&&&,

0.4−0.4−0.4

!

"

###

$

%

&&&

You may calculate eigenvalue analytically and compare it to numerical result for verification

High Performance Computing

Code Tuning

ü  Intel AVX/AVX2 (Advanced Vector Extension)

ü  Open MP Parallel Programming

ü  MPI Parallel Programming

256bit SIMD (Single Instruction Multiple Data)

Intel AVX-512, Xeon Phi Knights Landing (~2015)

CPU CPU

memory memory

Cores

Node

CPU CPU

memory memory

Cores

Node

Interconnect

Open MP Open MP

MPI

Open MP/MPI Hybrid Programming

cc/ifort –xAVX

Alignment of 32 bye boundary

float A[1000] __attribute__((aligned(32)));

REAL*4 A(1000) !DIR$ATTRIBUTES ALIGN: 32:: A

You should use MKL library, which is already optimized for AVX

ü  Vectorization

DO K=1, KMAX

DO J=1, JMAX

DO I=1,IMAX

END DO

END DO

END DO

Numerical Operations

Vectorization

Open MP

MPI

Loop-level Parallelization

ü  Open MP

!$OMP parallel do istep =1, nstep !$OMP do do j=2, n – 1 do i = 2, n – 1 g(i,j)=0.25d0 * (f(i-1,j)+f(i+1, j) & + f(i, j-1) + f(i, j+1) end do end do !$OMP end do nowait !$OMP single er=0.0d0 !$OMP end single !$OMP do reduction(+:er) . end do !$OMP end parallel

$ifort –openmp ….

$export OMP_NUM_THREADS=4

$./a.out

ü  MPI (Message Passing Interface)

call MPI_INIT(ierr) call MPI_COMM_RANK(MPI_COMM_WORLD,myid,ierr) call MPI_COMM_SIZE(MPI_COMM_WORLD,nprocs,ierr) s=1+myid*(n/nprocs) e=s+(n/nprocs)-1 do j=s,e ! Domain Decomposition do i=1, n a(i,j)=& 0.25*(b(i-1,j)+b(i,j+1)+b(i,j-1)+b(i+1,j)) - & h*h*f(i,j) end do end do

call MPI_SENDRECV( & a(1,e), nx, MPI_DOUBLE_PRECISION, nbrtop, 0, & a(1,s-1), nx, MPI_DOUBLE_PRECISION, nbrbottom, 0, & comm1d, status, ierr )

call MPI_SENDRECV( & a(1,s), nx, MPI_DOUBLE_PRECISION, nbrbottom, 1, & a(1,e+1), nx, MPI_DOUBLE_PRECISION, nbrtop, 1, & comm1d, status, ierr ) . call MPI_FINALIZE(ierr)

$mpiifort …..

$mpirun –np 4 ./a.out

Hybrid Programming

call MPI_INT(ierr) call MPI_COMM_RANK(MPI_COMM_WORLD, myid, ierr) call MPI_COMM_SIZE(MPI_COMM_WORLD, numprocs, ierr) … s=1+myid*(n/nprocs) e=s+(n/nprocs)-1 if(myid == numprocs-1) e=n !$omp parallel do private(i) do j=s,e y(j)=0.0d0 do i=1,n y(j)=y(j)+A(j,i)*x(i) end do end do !$omp end parallel do

$mpiifort -openmp…..

$export OMP_NUM_THREADS=4 $mpirun –np 2 ./a.out

2MPI x 4Threads