10
Some HPC challenges for multi-physics extended CFD computations F. Vuillot, A. Refloch ONERA/DSNA 2 "workshop on HPC-CFD in Energy/Transport Domains", HPCC 2014, August 20-22, 2014, Ecole des Mines, Paris Outline of presentation Context (multi-physics extended CFD for research and industry) ONERA's CEDRE code 1) Some HPC challenges Memory I/O library Lagrangian vs Eulerian Coupling and CWIPI library Scalability 2) Large scale LES examples Aeroacoustics Reactive flows Two-phase flows Conclusions

d industry) Some HPC challenges for multi-physics ......Some HPC challenges for multi-physics extended CFD computations F. Vuillot, A. Refloch ONERA/DSNA 2 "workshop on HPC-CFD in

  • Upload
    others

  • View
    5

  • Download
    0

Embed Size (px)

Citation preview

Page 1: d industry) Some HPC challenges for multi-physics ......Some HPC challenges for multi-physics extended CFD computations F. Vuillot, A. Refloch ONERA/DSNA 2 "workshop on HPC-CFD in

So

me

HP

C c

ha

lle

ng

es

fo

r

mu

lti-

ph

ys

ics

ex

ten

de

d C

FD

co

mp

uta

tio

ns

F.

Vuill

ot,

A. R

efloch

ON

ER

A/D

SN

A

2"w

ork

sh

op

on

HP

C-C

FD

in

En

erg

y/T

ran

sp

ort

Do

ma

ins", H

PC

C 2

014,

Au

gu

st

20-2

2, 2014,

Eco

led

es M

ines, P

ari

s

Ou

tlin

e o

f p

res

en

tati

on

•C

onte

xt

(multi-physic

s e

xte

nded C

FD

for

researc

h a

nd industr

y)

•O

NE

RA

'sC

ED

RE

code

1)

Som

e H

PC

challe

nges

–M

em

ory

–I/

O lib

rary

–L

ag

ran

gia

nvs

Eu

leri

an

–C

ou

plin

g a

nd

CW

IPI

libra

ry

–S

ca

lab

ility

2)

Larg

e s

cale

LE

S e

xam

ple

s

–A

ero

aco

ustics

–R

ea

ctive

flo

ws

–T

wo

-ph

ase

flo

ws

•C

onclu

sio

ns

Page 2: d industry) Some HPC challenges for multi-physics ......Some HPC challenges for multi-physics extended CFD computations F. Vuillot, A. Refloch ONERA/DSNA 2 "workshop on HPC-CFD in

3"w

ork

sh

op

on

HP

C-C

FD

in

En

erg

y/T

ran

sp

ort

Do

ma

ins", H

PC

C 2

014,

Au

gu

st

20-2

2, 2014,

Eco

led

es M

ines, P

ari

s

HP

C c

ha

lle

ng

es

: 1 t

arg

et

–3 a

xes

���������

� �

����������������

����� ��������

�����

��� �

�����

���������������

������������

��� �����������������

���������

����

� ��������

������ ���

�������� �� ���

�������

����������������

����

���

4"w

ork

sh

op

on

HP

C-C

FD

in

En

erg

y/T

ran

sp

ort

Do

ma

ins", H

PC

C 2

014,

Au

gu

st

20-2

2, 2014,

Eco

led

es M

ines, P

ari

s

Co

nte

xt

•H

ard

ware

evolu

tion

–F

rom

ve

cto

r m

ain

fra

me

s (

Cra

y,

NE

C,

Fu

jitsu

), la

rge

me

mo

ry p

er

co

re

–T

o s

ca

lar

pa

ralle

l clu

ste

rs,

red

uce

d m

em

ory

pe

r co

re

–T

ow

ard

s H

PC

clu

ste

rs a

nd

exa

sca

lem

ach

ine

s,

limite

d m

em

ory

pe

r

co

re

•M

ulti-physic

s e

xte

nded C

FD

–F

rom

pu

re a

ero

dyn

am

ic c

on

ditio

ns (

1 s

pe

cie

s,

pe

rfe

ct/

ide

al g

as)

–T

o r

ea

ctive

co

nd

itio

ns (

mu

lti-

sp

ecie

s,

ch

em

ica

l kin

etics,

rea

l g

ase

s)

–T

o m

ulti-

ph

ase

s c

on

ditio

ns (

dis

pe

rse

d,

se

pa

rate

d,

mix

ed

)

–T

o m

ulti-

ph

ysic

s c

on

ditio

ns (

ae

roa

co

ustics,

flu

id-s

olid

in

tera

ctio

ns

(he

at

tra

nsfe

rs,

me

ch

an

ics),

ele

ctr

o-m

ag

ne

tism

, m

ovin

g m

esh

es,

...)

•C

ode c

halle

nges

–M

em

ory

an

d I

/O m

an

ag

em

en

ts

–A

uto

ma

tic p

art

itio

nin

g a

nd

ba

lan

cin

g

–C

ou

plin

g p

roce

du

res

–H

PC

re

qu

ire

me

nts

at

eve

ry le

ve

l

Page 3: d industry) Some HPC challenges for multi-physics ......Some HPC challenges for multi-physics extended CFD computations F. Vuillot, A. Refloch ONERA/DSNA 2 "workshop on HPC-CFD in

5"w

ork

sh

op

on

HP

C-C

FD

in

En

erg

y/T

ran

sp

ort

Do

ma

ins", H

PC

C 2

014,

Au

gu

st

20-2

2, 2014,

Eco

led

es M

ines, P

ari

s

ON

ER

A's

CE

DR

E c

od

e

•O

NE

RA

'sre

fere

nce

co

de

fo

r e

ne

rge

tics

an

dp

rop

uls

ion

•U

nstr

uctu

red

me

sh

es

–C

ell

cente

red

(fin

ite

volu

me),

expl/im

plic

it, M

US

CL u

pto

hig

hord

er

–G

enera

lized, fu

ll poly

hedra

l, m

eshes

(face

base

d)

•P

ara

llel

–M

PI based

–A

uto

matic

part

itio

nnin

gand

bala

ncin

g

•M

ulti-

ph

ysic

s

–In

tern

al(d

iffe

rentC

ED

RE

solv

ers

for

diffe

rent

ph

ysic

alsyste

ms)

•C

ha

rme

: N

avie

r-S

toke

s, tu

rbu

lent, r

ea

ctive

, re

alg

ases, m

ulti-p

hase

s

•S

pa

rte: d

ispe

rse

dp

ha

ses (

La

gra

ng

ian

)

•S

pir

ée

: d

isp

ers

ed

ph

ases (

Eu

leri

an)

•A

str

e: ra

dia

tio

n (

Mo

nte

-Ca

rlo

)

•R

éa

: ra

dia

tio

n (

dis

cre

teo

rdin

ate

)

•A

ca

cia

: th

erm

al so

lutio

ns in s

olid

s

•F

ilm:

liqu

idfilm

s

–E

xte

rnal(c

ouplin

glib

rari

es: M

pC

CI, C

WIP

I, O

penP

alm

)

–M

ovin

g/d

efo

rma

ble

mesh

(conserv

ative A

LE

ap

pro

ach)

•Z

on

al a

pp

roa

ch

(mo

de

lsp

er

use

r's

do

ma

in)

6"w

ork

sh

op

on

HP

C-C

FD

in

En

erg

y/T

ran

sp

ort

Do

ma

ins", H

PC

C 2

014,

Au

gu

st

20-2

2, 2014,

Eco

led

es M

ines, P

ari

s

HP

C c

ha

lle

ng

es

fo

r s

pe

cif

icin

du

str

ialn

eed

s

•U

nstr

uctu

red

mesh

–no

alig

ned

data

•M

em

ory

•I/

O

•M

ulti-physic

s-

couplin

gappro

aches

•S

cala

bili

ty

•M

ovin

gm

esh

(not

pre

sente

d)

Page 4: d industry) Some HPC challenges for multi-physics ......Some HPC challenges for multi-physics extended CFD computations F. Vuillot, A. Refloch ONERA/DSNA 2 "workshop on HPC-CFD in

7"w

ork

sh

op

on

HP

C-C

FD

in

En

erg

y/T

ran

sp

ort

Do

ma

ins", H

PC

C 2

014,

Au

gu

st

20-2

2, 2014,

Eco

led

es M

ines, P

ari

s

ME

MO

RY

•Im

port

ance o

fm

em

ory

–m

easure

insid

ecode (

house k

eepin

g)

•C

urr

ent

trend o

f decre

asin

g a

vaila

ble

mem

ory

per

core

HO

WE

VE

R:

–R

esili

en

ce

pro

ble

m:

MT

TI

ch

eckp

oin

t-re

sta

rt

–Im

pa

ct

of

Big

Da

ta (

als

on

ee

ds

of

big

me

mo

ry)

–A

dva

nce

sin

te

ch

no

lgy

•P

RA

M (

para

mete

rra

ndom

access

mem

ory

) in

pla

ce o

fR

AM

•R

eR

am

(HP

–"T

he

Machin

e"

pro

ject)

–M

ore

me

mo

ryca

na

lso

ha

ve

an

in

flu

en

ce

on

mu

lti-

ph

ysic

s c

ou

plin

g

Mem

ory

red

ucti

on

tren

d m

ay

be

co

un

ter

bala

nced

(fo

rtu

nate

ly)

8"w

ork

sh

op

on

HP

C-C

FD

in

En

erg

y/T

ran

sp

ort

Do

ma

ins", H

PC

C 2

014,

Au

gu

st

20-2

2, 2014,

Eco

led

es M

ines, P

ari

s

I/O

is

su

e (

1/2

)

Unstr

uctu

red

data

=>

non c

ontinu

ous

/ in

tele

ave

ddata

Larg

e n

um

ber

of

sm

all

file

s (

per

core

):

Pro

ble

mfo

r ba

ckup

and

for

para

llelfile

syste

m(L

ustr

e, �����

Sin

gle

file

:

in-h

ouse

libra

ry,

bin

ary

,

com

patib

ility

little-e

ndia

n/b

ig-e

ndia

n

Resta

rt o

fcom

puta

tions o

n d

iffe

rent

num

ber

of

core

s(s

trong

users

dem

an

ds)

Page 5: d industry) Some HPC challenges for multi-physics ......Some HPC challenges for multi-physics extended CFD computations F. Vuillot, A. Refloch ONERA/DSNA 2 "workshop on HPC-CFD in

9"w

ork

sh

op

on

HP

C-C

FD

in

En

erg

y/T

ran

sp

ort

Do

ma

ins", H

PC

C 2

014,

Au

gu

st

20-2

2, 2014,

Eco

led

es M

ines, P

ari

s

I/O

is

su

e (

2/2

)

Actu

albott

leneck:

ways

to e

scape

–T

est

thro

ug

hP

roje

t P

RA

CE

: C

ed

reP

rep

ara

tory

pro

ject

PR

A0

17

,

20

10

, co

mp

ute

r Ju

ge

ne

fro

mJu

elic

h,

1M

h

–I/

O lib

rary

tests

on

32

00

0 c

ore

s,

GP

FS

is

be

tte

rth

an

Lu

str

e

–"I

nte

rle

ave

d"

co

st

islim

ite

dfo

r re

sta

rt f

iles

�������� ���!���"��#

����� $ %����������

����� & %��� ����'()*�+(,'-�+.+��

�'��������� ���!

����� / %��� ����'()*�+(�012332*�

����� 4 %��� ����'()*�+(,'-�+.+��

10

"w

ork

sh

op

on

HP

C-C

FD

in

En

erg

y/T

ran

sp

ort

Do

ma

ins", H

PC

C 2

014,

Au

gu

st

20-2

2, 2014,

Eco

led

es M

ines, P

ari

s

Mu

ltip

hys

ics

: E

ule

ria

nv

s L

ag

ran

gia

n

•M

ulti-

ph

ysic

s:

mo

re c

ha

llen

ge

s t

o b

e f

ace

d

•E

ule

ria

na

pp

roa

ch

: p

ara

llel b

y d

om

ain

•L

ag

ran

gia

na

pp

roa

ch

: p

ara

llel b

y t

raje

cto

ry (

e.g

. p

ho

ton

s)

•B

oth

ap

pro

ach

es in

CE

DR

E

•P

rob

lem

of

loa

d b

ala

ncin

g f

or

the

co

mp

lete

sim

ula

tio

n

•S

om

e L

ag

ran

gia

na

pp

roa

ch

ne

ed

s a

ll th

e m

esh

on

ea

ch

co

re (

e.g

.

so

lve

r A

str

e)

–Lim

it o

f m

em

ory

: h

ybri

d s

olu

tion thro

ugh

Ope

nM

P

–T

oday t

his

solu

tion p

erm

its to

push th

e lim

it fro

m 3

M e

lem

ents

to 2

0 M

ele

ments

(48G

o/n

ode)

•W

ork

s a

re s

till

ne

ed

ed

on

th

at

su

bje

ct

Page 6: d industry) Some HPC challenges for multi-physics ......Some HPC challenges for multi-physics extended CFD computations F. Vuillot, A. Refloch ONERA/DSNA 2 "workshop on HPC-CFD in

11

"w

ork

sh

op

on

HP

C-C

FD

in

En

erg

y/T

ran

sp

ort

Do

ma

ins", H

PC

C 2

014,

Au

gu

st

20-2

2, 2014,

Eco

led

es M

ines, P

ari

s

CW

IPI:

Co

up

lin

g W

ith

Pa

ralle

l In

terf

ac

e

•D

evelo

ped

at

ON

ER

A

•O

pen s

ourc

e (

cedre

.onera

.fr)

•T

est

on C

urie 3

2k c

ore

s–

1 B

illio

n n

odes

–S

imu

late

s t

he

co

up

ling

be

twe

en

tw

o s

oft

wa

res

thro

ug

h a

n

exch

an

ge

acro

ss a

n in

terf

ace

co

nsis

tin

g o

f a

pa

rtitio

ne

d s

urf

ace

.

Th

e d

iscre

tiza

tio

no

f th

e s

urf

ace

is u

niq

ue

to

ea

ch

co

de

.

–S

yn

ch

ron

ou

svs a

syn

ch

ron

ou

s:

sp

ee

d r

atio

10

-15

12

"w

ork

sh

op

on

HP

C-C

FD

in

En

erg

y/T

ran

sp

ort

Do

ma

ins", H

PC

C 2

014,

Au

gu

st

20-2

2, 2014,

Eco

led

es M

ines, P

ari

s

CE

DR

E:

Sc

ala

bilit

yte

sts

We a

ckn

ow

led

ge

PR

AC

E f

or

aw

ard

ing

us a

cce

ss to r

eso

urc

e C

UR

IE b

ase

d in

FR

AN

CE

at

TG

CC

, P

roje

ct

PA

13

89

, 20

13,

20

0 k

h

����� �� ���"����� ����"��� � � ����"�5��!�� ���"����� "�

6��!� +��� ������� �� /78 ����

������������������ ����������� ����

Page 7: d industry) Some HPC challenges for multi-physics ......Some HPC challenges for multi-physics extended CFD computations F. Vuillot, A. Refloch ONERA/DSNA 2 "workshop on HPC-CFD in

13

"w

ork

sh

op

on

HP

C-C

FD

in

En

erg

y/T

ran

sp

ort

Do

ma

ins", H

PC

C 2

014,

Au

gu

st

20-2

2, 2014,

Eco

led

es M

ines, P

ari

s

Ex

am

ple

so

fla

rge

sc

ale

LE

S "

rea

llife

" a

pp

lica

tio

ns

•A

ero

acoustics

•R

eactive

flow

s

•T

wo-p

hase

reactive

flow

s

14

"w

ork

sh

op

on

HP

C-C

FD

in

En

erg

y/T

ran

sp

ort

Do

ma

ins", H

PC

C 2

014,

Au

gu

st

20-2

2, 2014,

Eco

led

es M

ines, P

ari

s

La

rge

sc

ale

LE

S:

Ae

roa

co

usti

cs

(1/2

)

•Jet

nois

e

–A

ircra

ft n

ois

e r

ed

uctio

n s

tud

ies

–H

yb

rid

ap

pro

ach

: C

ED

RE

LE

S-C

FD

+ K

IM a

co

ustic c

od

e

–2

50

Mce

lls+

re

so

lve

d n

ozzle

bo

un

da

ry la

ye

r tr

ipp

ing

, 9

60

co

res

M.

Lort

eau, F

. C

léro

, F

. V

uil

lot,

"R

ecen

tpro

gre

ssin

LE

S c

om

puta

tion f

or

aero

acoust

ics

of

turb

ule

nt

hot

jet.

Com

par

ison

to e

xper

imen

ts

and

nea

rfi

eld

anal

ysi

s",

AIA

A 2

014-3

057, 20th

AIA

A/C

EA

S A

ero

acoust

ics

Confe

ren

ce, 16-2

0 J

une

2014, A

tlan

ta, G

a, U

SA

Page 8: d industry) Some HPC challenges for multi-physics ......Some HPC challenges for multi-physics extended CFD computations F. Vuillot, A. Refloch ONERA/DSNA 2 "workshop on HPC-CFD in

15

"w

ork

sh

op

on

HP

C-C

FD

in

En

erg

y/T

ran

sp

ort

Do

ma

ins", H

PC

C 2

014,

Au

gu

st

20-2

2, 2014,

Eco

led

es M

ines, P

ari

s

La

rge

sc

ale

LE

S:

Ae

roa

co

usti

cs

(2/2

)

•Landin

g g

ear

nois

e–

Air

cra

ftn

ois

e r

ed

uctio

nstu

die

s

–C

om

ple

xg

eo

me

trie

s:

un

str

uctu

red

me

sh

es

are

re

qu

ire

d

–H

yb

rid

ap

pro

ach

: C

ED

RE

LE

S-C

FD

+ K

IM a

co

ustic

co

de

–7

0 M

ce

lls+

ZD

ES

tu

rbu

len

ce

mo

de

ling

L. S

and

ers,

N.

Lupo

gla

zoff

, F

. V

uil

lot,

E. M

anoh

a, D

. L

uqu

et, F

. d

e la

Puen

te, "F

urt

her

flow

and

nois

e pre

dic

tions

of

the

Gulf

stre

am

PD

CC

nose

landin

g g

ear

bas

edon t

he

CE

DR

E u

nst

ruct

ure

dso

lver

", 1

9th

AIA

A/C

EA

S A

ero

acoust

ics

Confe

ren

ce, 27-2

9 M

ay 2

013,

Ber

lin, G

erm

any

F. de

la P

uen

te,

L. S

and

ers,

F. V

uil

lot,

"O

n L

AG

OO

N n

ose

landin

g g

ear

CF

D/C

AA

com

puta

tion o

ver

unst

ruct

ure

dm

esh

usi

ng

a Z

DE

S

appro

ach",

AIA

A 2

014

-, 2

0th

AIA

A/C

EA

S A

ero

acoust

ics

Confe

ren

ce, 16

-20 J

une

2014, A

tlan

ta, G

a, U

SA

Gulf

stre

am

/Nasa

PD

CC

nose

landin

g g

ear

(BA

NC

work

shop)

Air

bus

Lagoon N

LG

(B

AN

C w

ork

shop)

16

"w

ork

sh

op

on

HP

C-C

FD

in

En

erg

y/T

ran

sp

ort

Do

ma

ins", H

PC

C 2

014,

Au

gu

st

20-2

2, 2014,

Eco

led

es M

ines, P

ari

s

La

rge

sc

ale

LE

S:

Re

ac

tiv

efl

ow

s

•S

upers

onic

reactive

flow

in o

ver-

expan

ded

nozzle

+ f

ilm c

oolin

g

–L

au

nch

er

stu

die

s(C

NE

S,

AT

AC

pro

ject)

–6

sp

ecie

s +

se

ve

n s

tep

s r

ea

ctive

sch

em

e

–7

.5 M

ce

lls+

ZD

ES

tu

rbu

len

ce

mo

de

ling

B. S

ainte

-Rose

, N

. B

erti

er, S

. D

eck, F

. D

upoir

ieux

, "N

um

eric

al s

imula

tions

and p

hysi

cal

anal

ysi

s of

an o

ver

-ex

pan

ded

rea

ctiv

e gas

flo

w

in a

pla

nar

nozz

le",

Com

bust

ion a

nd F

lam

e 159 (

2012),

2856-2

871

Iso-Q

surf

ace

colo

ure

db

y m

od(U

x)

Page 9: d industry) Some HPC challenges for multi-physics ......Some HPC challenges for multi-physics extended CFD computations F. Vuillot, A. Refloch ONERA/DSNA 2 "workshop on HPC-CFD in

17

"w

ork

sh

op

on

HP

C-C

FD

in

En

erg

y/T

ran

sp

ort

Do

ma

ins", H

PC

C 2

014,

Au

gu

st

20-2

2, 2014,

Eco

led

es M

ines, P

ari

s

La

rge

sc

ale

LE

S:

Tw

o-p

ha

se

rea

cti

ve

flo

ws

•Liq

uid

pro

pella

nt

rocket

moto

rcom

bustion c

ham

ber

–M

AS

CO

TT

E c

ryo

ge

nic

test

be

nch

(LO

X-G

H2

typ

e)

–C

ou

plin

g o

f tw

o C

ED

RE

Eu

leri

an

so

lve

rs

•C

HA

RM

E for

separa

ted

phase

flo

w (

gas a

nd

liq

uid

jet)

•S

PIR

EE

for

the d

ispers

ed d

rople

t spra

y f

low

–1

0 M

ce

llsg

rid

, 2

x4

80

co

res

A. M

urr

one,

N. F

did

a, C

. L

e T

ouze

, L

. V

inger

t(2

014),

"A

tom

izat

ion o

f cr

yo

gen

ic r

ock

et e

ngin

es c

oax

ial

inje

ctors

. M

odel

ing a

spec

ts a

nd

exper

imen

tal

inv

esti

gat

ions"

, S

pac

e P

ropuls

ion, 2014, C

olo

gn

e, G

erm

any,

19-

22 m

ai, 2014

18

"w

ork

sh

op

on

HP

C-C

FD

in

En

erg

y/T

ran

sp

ort

Do

ma

ins", H

PC

C 2

014,

Au

gu

st

20-2

2, 2014,

Eco

led

es M

ines, P

ari

s

Wra

p-u

p

•H

PC

challe

nges f

or

researc

hand

industr

ialflow

s

–U

nstr

uctu

red

da

ta s

tru

ctu

res

–I/

O a

nd

me

mo

rym

an

ag

em

en

t

–E

xch

an

ge

an

dco

up

ling

pro

ce

du

res

•O

NE

RA

'sC

ED

RE

code for

energ

etics

and

pro

puls

ion

•Larg

e s

cale

LE

S:

recentexam

ple

s

•C

lear

HP

C n

eeds

–U

nste

ad

yflo

wso

lutio

ns

–M

ulti-

ph

ysic

s

–D

ata

ma

na

ge

me

nt

•W

ork

s a

re in p

rogre

ss

in these

are

as

Page 10: d industry) Some HPC challenges for multi-physics ......Some HPC challenges for multi-physics extended CFD computations F. Vuillot, A. Refloch ONERA/DSNA 2 "workshop on HPC-CFD in

19

"w

ork

sh

op

on

HP

C-C

FD

in

En

erg

y/T

ran

sp

ort

Do

ma

ins", H

PC

C 2

014,

Au

gu

st

20-2

2, 2014,

Eco

led

es M

ines, P

ari

s

Th

an

kyo

ufo

r yo

ur

att

en

tio

n

•Q

UE

ST

ION

S ?