High-Fidelity Terascale Simulations of Turbulent Combustion

High-Fidelity Terascale Simulations of Turbulent CombustionHigh-Fidelity Terascale Simulations of Turbulent Combustion

Jacqueline H. Chen, Evatt R. Hawkes, Ramanan Sankaran, James C. Sutherland, and Joseph C. Oefelein

Combustion Research FacilitySandia National Laboratories

Livermore CA

Supported by Division of Chemical Sciences, Geosciences, and Biosciences,

Office of Basic Energy Sciences, and DOE SciDAC

Computing: LBNL NERSC, ORNL CCS/NLCF, PNNL MSCF,SNL HPCNP Infiniband test-bed,

SNL CRF BES Opteron cluster

Visualization: Kwan-Liu Ma and Hiroshi Akiba UC Davis

OutlineOutline

1. DNS of turbulent combustion – challenges and opportunities

2. Sandia S3D DNS capability – terascale simulations on Office of Science platforms

3. New combustion science to advance predictive models:

• 3D simulations of turbulent jet flames with detailed chemistry.

• Layering Large Eddy Simulation and DNS approaches. Vorticity fields in DNS of a turbulent jet flame

(volume rendering by Kwan-Liu Ma and Hiroshi Akiba)

Turbulent combustion is a grand challenge!Turbulent combustion is a grand challenge!

Diesel Engine Autoignition, Soot IncandescenceChuck Mueller, Sandia National Laboratories

• Stiffness : wide range of length and time scales

– turbulence

– flame reaction zone

• Chemical complexity– large number of species and

reactions (100’s of species, thousands of reactions)

• Multi-Physics complexity – multiphase (liquid spray, gas

phase, soot, surface)

– thermal radiation

– acoustics ...

• All these are tightly coupled

Several decades of relevant scalesSeveral decades of relevant scales

• Typical range of spatial scales– Scale of combustor: 10 – 100 cm– Energy containing eddies: 1 – 10 cm– Small-scale mixing of eddies: 0.1 – 10 mm– Diffusive-scales, flame thickness: 10 – 100 m – Molecular interactions, chemical reactions: 1 – 10 nm

• Spatial and temporal dynamics inherently coupled

• All scales are relevant and must be resolved or modeled

O(4)Range

O(4)

Co

nti

nu

um

Terascale computing:~3 decades in scales

(cold flow)

Combustion CFD Approaches to tackledifferent length scale rangesCombustion CFD Approaches to tackledifferent length scale ranges

• Reynolds–Averaged Navier–Stokes (RANS)– Coarse meshes, full range of dynamic scales modeled– Empirical closure, bulk approximation, current engineering CFD

• Large Eddy Simulation (LES)– Energetic scales resolved, sub-grid scale dynamics modeled– combustion closure required, captures large-scale unsteadiness

• Direct Numerical Simulation (DNS)– No sub-grid modeling required but limited on range of scales– Building-block configurations, research tool

Incr

ea

sin

g c

os

t a

nd

re

so

luti

on

fo

r fi

xed

ph

ys

ica

l p

rob

lem

uL

Re

Role of Direct Numerical Simulation (DNS)Role of Direct Numerical Simulation (DNS)

• A tool for fundamental studies of the micro-physics of turbulent reacting flows

• A tool for the development and validation of reduced model descriptions used in macro-scale simulations of engineering-level systems

DNS PhysicalModels

Engineering-levelCFD codes(RANS andfuture LES)

DNS

• Physical insight into chemistry turbulence interactions

• Full access to time resolved fields

CH4

Oxidizer

Fuel

CH3O

HO2

O

Piston Engines

S3D MPP DNS capability at SandiaS3D MPP DNS capability at Sandia

• S3D code characteristics:– Solves compressible reacting Navier-Stokes– F90/F77, MPI, domain decomposition. – Highly scalable and portable– 8th order finite-difference spatial– 4th order explicit RK integrator– hierarchy of molecular transport models– detailed chemistry– multi-physics (sprays, radiation and soot) from

SciDAC “Terascale Simulation of Turbulent Combustion”

• 70% parallel efficiency on 4096 processors on NERSC; 90% parallel efficiency on 5120 CrayXT3 processors; 75% parallel efficiency on 1000 CrayX1E processors

• Terascale computations => need scalar optimization customized to architecture

– CCS, NERSC consultants

S3D is a state-of-the-art DNS code developed with 13 years of BES sponsorship.

S3D scales up to 1000s of processors… and beyond?

S3D performance improvements on SeaborgS3D performance improvements on Seaborg

Application Software Case Studies in FY05 for MICS (K. Roche)

• 8% improvement in performance by minimizing conversion between dimensional and nondimensional units for interfacing with Chemkin

• interface to Mathematical Acceleration Subsystem (MASS) library for exponential and logarithmic functions resulted in 10% improvement. Another 5% due to using vectorized exp function in VMASS

•Tabulating Gibbs energy as a function of temperature resulted in 8% savings

•Remove excessive dynamic allocation calls resulted in 12% savings.

•Eliminate MPI derived data types for communicating noncontiguous boundary data with neighbors

Using Xprofiler, IPM and working with David Skinner:

45% efficiency improvement

OutlineOutline




• 3D simulations of turbulent jet flames with detailed chemistry

• Layering Large Eddy Simulation and DNS approaches

Vorticity fields in DNS of a turbulent jet flame(volume rendering by

Kwan-Liu Ma and Hiroshi Akiba)

FY05 INCITE award: Direct simulation of a 3D turbulent nonpremixed CO/H2/air jet flame with detailed chemistry (350 million grids, 12 species)FY05 INCITE award: Direct simulation of a 3D turbulent nonpremixed CO/H2/air jet flame with detailed chemistry (350 million grids, 12 species)

Objectives:

•Study fine-grained coupling between turbulent mixing and finite-rate chemical effects

• Extinction-reignition dynamics

• Mixing time scales

• Reynolds number effect

• Preferential diffusion

• Flow unsteadiness

•Test assumptions in subgrid mixing and reaction models

Isoscalar surface of scalar dissipation rate (local mixing rate)

INCITE run test case (1/2 resolved) 350 million grid points, Re=5000

x

x

y

z

The effect of chemistry-turbulence interactions in turbulent nonpremixed jet flames: mixing of passive and reacting scalars

The effect of chemistry-turbulence interactions in turbulent nonpremixed jet flames: mixing of passive and reacting scalars

Evatt Hawkes, Ramanan Sankaran,

James Sutherland, and Jacqueline Chen

Computational Platforms: NERSC Seaborg, ORNL CCS Phoenix Cray X1E, XT3 and PNNL MPP2

Description of runs- Temporally evolving non-premixed plane jet flames Description of runs- Temporally evolving non-premixed plane jet flames

• A canonical flow with shear-driven turbulence

Heat release iso-contours

Fuel

Air

Air

Mixing,Reaction

Mixing,Reaction

x

y

• Large computing allocations are enabling new science runs– INCITE at NERSC, capability computing at NLCF ORNL, ERCAP at PNNL, NERSC

• Detailed H2/CO chemistry (17 d.o.f., Li et al. 2005)

• Parameters selected to maximize Re

• Case A: – Re 6000– 40 million grid points– 480 processors on PNNL MPP2– ~ 1 TB data

• Case B:– Re 8000– 100 million grid points– 1728 processors on Seaborg– 192/240 processors on ORNL CCS Cray X1/E– ~ 2.5 TB data

DNS data-sets of turbulent nonpremixed H2/CO flames (mixing, extinction/reignition)DNS data-sets of turbulent nonpremixed H2/CO flames (mixing, extinction/reignition)

• INCITE calculation:– Currently running on Seaborg

– Re 5000

– 350 million grid points

– ~4096 Seaborg processors

– 3 million hours total

– ~ 9TB raw data

• Two additional calculations, parametric study in Re

– Re 2500, 150 million grid points

– Re 10,000, 500 million grid points

Community data setsCommunity data sets

• New opportunities for numerical benchmarks – highly resolved LES and DNS

• TNF workshop (1996-2004): International Collaboration of Experimental and Computational Researchers

– Framework for detailed comparison of measured and modeled results (http://www.ca.sandia.gov/TNF/abstract.html)

Non-premixed combustion conceptsNon-premixed combustion concepts

• Mixture fraction Z: the amount of fluid from the fuel stream in the mixture

• Z is a conserved (passive) scalar – (no reactive source term)

• Scalar dissipation, a measure of local molecular mixing rate:

ZZD 2

Example development of the jet:40 million grid PNNL runExample development of the jet:40 million grid PNNL run

• Left: Vorticity. Right: Simultaneous volume rendering of mixture fraction, scalar dissipation and OH radical.

(Rendering by Hiroshi Akiba and Kwan-Liu Ma, UC Davis)

100 million grid run on NERSC Seaborg and ORNL Cray X1: Re = 8000100 million grid run on NERSC Seaborg and ORNL Cray X1: Re = 8000

Scalar Dissipation Vorticity

uZZD 2


HO2 dissipation OH dissipation

22222 HOHOHOHO YYD OHOHOHOH YYD 2

Mixing timescalesMixing timescales

• Models for molecular mixing are required in the PDF approach to turbulent combustion (Pope 1985), a sub-grid model used in RANS and LES approaches.

• TNF workshop – CFD predictions are dependent on mixing timescale choice.

• Models assume that scalar mixing timescales are identical for all scalars and determined by the turbulence timescale.– scalars with different diffusivities?

– reactive scalars?

DefinitionsDefinitions

• Mechanical time-scale:

• Scalar time-scale:

• Time-scale ratio:

k

u

D2

'2

ur

is assumed to be order unity in most modelsris assumed to be the same for all scalarsr

Mixture fraction to mechanical timescale ratioMixture fraction to mechanical timescale ratio

• Confirmation that mixture fraction to mechanical time scale ratio is order unity.

• Average value about 1.5, similar to values reported by experiments, simple chemistry DNS, and used successfully in models.

Tim

esca

le R

atio

rZ

Effect of diffusivityEffect of diffusivity

• Smaller, more highly diffusive species do have faster mixing timescales

Increasing diffusivityr φ

Finite-rate chemistry effects on mixingFinite-rate chemistry effects on mixing

• HO2 and H2O2 have faster mixing times in the middle of the simulation, while OH and O are lower

• Diffusivity trend does not appear to hold for HO2 and H2O2 versus O and OH.

• What is going on?

r φ

Radical production and destruction in high dissipation regionsRadical production and destruction in high dissipation regions

• OH is consumed while HO2 is produced in high dissipation regions

HO2OH

Color scale: mass fraction (blue is low; red is high)White iso-contours: scalar dissipation rate

Dissipation of passive and reactive scalarsDissipation of passive and reactive scalars

• Blue: Z, Green: OH, Red: HO2

• Dissipation fields of Z and HO2 are co-incident and aligned with principal strain directions

• OH dissipation occurs elsewhere, more in the centre of the jet

• OH is reduced in the high dissipation regions, leading to longer mixing times, opposite for HO2

• At later times, mixing rates relax, OH returns and HO2 decreases

100 million grid run on Seaborg / Cray X1100 million grid run on Seaborg / Cray X1

• We need to establish any Re dependence

• Need parametrics at larger Re

• Run in progress… but seems to be showing same effects

Conclusions - mixing timescalesConclusions - mixing timescales

• New finding: detailed transport and chemistry effects can alter the observed mixing timescales

• Models may need to incorporate these effects– a poor mixing model could lead to incorrectly predicting a stable

flame when actually extinction occurs

• This type of information cannot be determined any other way at present– ambiguities in a-posteriori model tests

– too difficult to measure

– need 3D and detailed chemistry to see this

Knowledge Discovery From Terascale Datasets Knowledge Discovery From Terascale Datasets

Challenge: Need interactive knowledge discovery software for terascale high-fidelity scientific data sets where computing pipeline is distributed geographically.

Solution: Intelligent visualization pipeline

•Simultaneous cognitive visualization

•Intelligent feature extraction/tracking

•Scalable transparent data sharing and parallel I/O across platforms

INCITE Award: DNS of a 3D turbulent flame (9 TB raw data - 350 million grids, 17 variables, 100,000 steps)

Advanced post-processing for extinction/re-ignitionAdvanced post-processing for extinction/re-ignition

• Extinction viewed as radical or reaction-rate “hole” in a mixture-fraction isosurface

• Stems from fast chemistry flame-sheet “flamelet” models of turbulent combustion

• Holes can be expanding or contracting

• Speed of the edge can be monitored – need algorithm for edge detection

Advanced post-processing for extinction/re-ignitionAdvanced post-processing for extinction/re-ignition

• Holes could be tracked to determine any history effects– need parallel feature tracking

OutlineOutline




• 2D DNS of HCCI combustion

• 3D simulations of turbulent jet flames with detailed chemistry

• Layering Large Eddy Simulation and DNS approaches

Vorticity fields in DNS of a turbulent jet flame(volume rendering by

Kwan-Liu Ma and Hiroshi Akiba)

Layering of LES and DNS approaches(the future...)Layering of LES and DNS approaches(the future...)

Joseph C. Oefelein, Evatt R. Hawkes, Jacqueline H. Chen

RANS-LES-DNS at High ReRANS-LES-DNS at High Re

• LES gets lots more than RANS

• High Re is too expensive for DNS (at present)– can afford 1000s now

useful but…

– would like 100000s and more

Swirl-Stabilized Premixed Combustion

Azimuthal Velocity Component

RANS LES DNS

Why LES?Why LES?

• LES treats the large scales directly

• Large scales are geometry dependent

• LES can couple directly with high Re experiments

• But – LES needs sub-grid models

Layering LES and DNSLayering LES and DNS

High Re Experiments

LES

•Moderate Re:–Canonical problems–Cleverly designed experiments

DNS

•Sub-grid models

•Better BCs, ICs•Identify modelling questions•Identify relevant parameter space

Applications

Multiphysics capabilities and algorithmic framework – Oefelein Multiphysics capabilities and algorithmic framework – Oefelein

• Theoretical framework (LES DNS)– Fully-coupled conservation equations– Compressible, chemically reacting system– Real-fluid EOS, thermodynamics, transport– Detailed chemistry, multiphase flow, sprays– Dynamic subgrid-scale modeling

• Numerical framework– Dual-time, all-Mach-number formulation– Generalized preconditioning methodology– Complex geometry, generalized coordinates– Massively-parallel (MPI), highly-scalable– Ported to all major platforms

• Over 10 years development • Fine-grain scalability (NERSC IBM SP – Seaborg)• Grid size fixed, number of processors increased

Magnitude of vorticity (normalized)Magnitude of vorticity (normalized)

DNS(all scales)

40 million grid, 120,000 hours

LES(large scales)

0.6 million grid, 600 hours

SummarySummary

• S3D is a state-of-the-art DNS capability for turbulent combustion simulations that scales to thousands of processors and is ported to all Office of Science platforms.

• DOE supercomputing facilities are enabling new combustion science.

• 3D DNS of detailed finite-rate chemistry effects in turbulent jets provides new insights and data for combustion modeling:– mixing of reactive scalars can be very different from conserved

scalars

• Layering LES and DNS approaches will open new avenues of exploration with great potential for the future


HO2 dissipation OH dissipation

S3D performance on different architecturesS3D performance on different architectures

Documents

High-Fidelity Terascale Simulations of Turbulent Combustion