SDMCenter

End-to-end data management capabilities in the GPSC & CPES

SciDAC’s: Achievements and Plans

End-to-end data management capabilities in the GPSC & CPES

SciDAC’s: Achievements and Plans

SDM AHM

December 11, 2006

Scott A. KlaskyEnd-to-End Task Lead

Scientific Computing Group

ORNL

SDM AHM

December 11, 2006

Scott A. KlaskyEnd-to-End Task Lead

Scientific Computing Group

ORNL

SDMCenterOutlineOutline

• Overview of GPSC activities.• The GTC and GEM codes.• On the path to petascale computing.• Data Management Challenges for GTC.

• Overview of CPES activities.• The XGC and M3D codes.• Code Coupling.• Workflow Solutions.

• ORNL’s end-to-end activities.• Asynchronous I/O.• Dashboard Efforts.

SDMCenterIt’s all about the enabling technologies…It’s all about the enabling technologies…

CS CS

Math Math

ApplicationsApplications

Enabling technologies respond

Applications drive

D. Keyes

It’s all about the data

It’s all about the features

which lead us to Scientific discovery!

SDMCenter

GPSC gyrokinetic PIC codes used for studying microturbulence in plasma core

GPSC gyrokinetic PIC codes used for studying microturbulence in plasma core

• GTC (Z. Lin et al., Science 281, p.1835, 1998) • Intrinsically global 3D nonlinear gyrokinetic PIC code• All calculations done in real space• Scales to > 30,000 processors• Delta-f method • Recently upgraded to fully electromagnetic

• GEM (Y. Chen & S. Parker, JCP, in press 2006) • Fully electromagnetic nonlinear delta-f code• Split-weight scheme implementation of kinetic electrons• Multi-species• Uses Fourier decomposition of the fields in toroidal and poloidal directions

(wedge code)

• What about PIC noise.• “It is now generally agreed that these ITG simulations are not being

influenced by particle noise. Noise effects on ETG turbulence remain under study but are beginning to seem of diminishing relevance.” PSACI-PAC.

SDMCenter

GTC Code performance. GTC Code performance.

0.01

0.1

1

10

100

1000

10000

1996 1998 2000 2002 2004 2006 2008 2010

TflopsTB

0.01

0.1

1

10

100

1000

10000

1996 1998 2000 2002 2004 2006 2008 2010

TflopsTB

Increase output because of•Asynchronous metadata rich I/O.•Workflow automation.•More analysis services in the workflow.

Historical Prediction of GTC Data Production

Cray XT3

Cray T3E IBM SP3 Cray X1E Cray Baker

SDMCenter

GTC: Towards a Predictive Capability for ITER Plasmas

GTC: Towards a Predictive Capability for ITER Plasmas

• Petascale Science• Investigate important physics problems for ITER

plasmas, namely, the effect of size and isotope scaling on core turbulence and transport (heat, particle, and momentum).

• These studies will focus on the principal causes of turbulent transport in tokamaks, for example, electron and ion temperature gradient (ETG and ITG) drift instabilities, collisonless and collisional (dissipative) trapped electron mode (CTEM and DTEM) and ways to mitigate these phenomena.

QuickTime™ and aMPEG-4 Video decompressor

are needed to see this picture.

SDMCenter

Impact: How does turbulence cause heat, particles and momentum to escape from plasmas?

Impact: How does turbulence cause heat, particles and momentum to escape from plasmas?

• Investigation of the ITER confinement properties is required• a dramatic step from 10 MW for 1 second to the projected 500 MW

for 400 seconds.

• The race is on to improve predictive capability before ITER comes on line (projected 2015).

• More realistic assessment of ignition margins requires more accurate calculations of steady-state temperature and density profiles for ions, electrons and helium ash.

• The success of ITER depends in part on its ability to operate in a gyroBohm scaling regime which must be demonstrated computationally.

• Key for ITER is the fundamental understanding of the effect of deuterium-tritium isotope presence (isotope scaling) on turbulence.

SDMCenterCalculation DetailsCalculation Details

• Turbulent transport studies will be carried out using the present GTC code, which uses a grid of the size of ion gyroradius.

• The electron particle transport physics requires the incorporation of the size of the electron skin depth in the code for the TEM physics, which can be an order of magnitude smaller than the size of ion gyroradius.

• A 10,000x10,000x100 grid and 1 trillion particles (100 particles/cell) are estimated to be needed. (700 TB/scalar field, 25TB particles(1 time step).

• For the 250TF machine a 2D domain decomposition (DD) for electrostatic simulation of ITER size machine (a/rho>1000) with kinetic electron is necessary.

W. Lee

SDMCenterGTC Data Management IssuesGTC Data Management Issues

• Problem: Move data from NERSC to ORNL then to PPPL as the data was being generated.• Transfer from NERSC to ORNL, 3000 timesteps, 800GB within the

simulation run (34 hours).

• Convert each file to HDF5 file• Archive files to 4GB chunks to HPSS at ORNL.• Move portion of hdf5 files to PPPL.

• Solution: Norbert Podhorszki

Transfer

Convert

Archive

Watch

SDMCenter

GTC Data Management Achievements

GTC Data Management Achievements

• In the process to remove• Ascii output.• Hdf5 output.• Netcdf output.

• Replace with• Binary (parallel) I/O with metadata tags.

• Conversion to HDF5 during the simulation on a ‘cheaper’ resource.

• 1 XML file to describe all files output in GTC.• Only work with 1 file from the entire simulation.

• Large buffer writes.• Asynchronous I/O when it becomes available.

SDMCenterThe data-in-transit problemThe data-in-transit problem

• Particle data needs to be examined occasionally.• 1 trillion particles = 25TB/hour. (Demand <2% I/O overhead).

• Need 356GB/sec to handle burst! (7GB/sec aggregate).

• We can’t store all of this data! (2.3 PB/simulation) x 12 simulations/year = 25 PB.

• Need to analyze on-the-fly and not save all of the data for permanent storage. [Analyze on another system].

• Scalar data needs to be analyzed during the simulation.• Computational Experiments too costly to let simulation run and

ignore it. [Estimated cost = $500K/simulation on Pflop machine].• GTC already = 0.5M CPU hours/simulation; approaching 3M CPU

hours on 250Tflop system.

• Need to compare new simulations with older simulations and experimental data.• Metadata needs to be stored in databases.

SDMCenter

Workflow Simulation monitoring.

Workflow Simulation monitoring.

• Images generated from the workflow.

• User needs to set angles, min/max and then the workflow produces the images.

• Still need to put this in our everyday use.

• Really need to identify the features as it’s running.• Trace back features once

they are known to earlier timesteps (where are they born?)

QuickTime™ and aH.264 decompressor

are needed to see this picture.

SDMCenter5D Data Analysis -15D Data Analysis -1

• Common in fusion to look at puncture plots. (2D).• To gleam insight, we need to be able to detect ‘features’• Need temporal perspective, involving the grouping of similar

items to possibly identify interesting new plasma structures (within this 5D-phase space) at different stages of the simulations.

2D Phase Space

SDMCenter5D Data Analysis -25D Data Analysis -2

Our turbulence covers the global volume as opposed to some isolated (local) regions The spectral representation of the turbulence, evolves in time by

moving to longer wavelengths. Understanding key nonlinear dynamics here involves extracting

relevant information from the data sets for the particle behavior. The trajectories of these particles are followed self-consistently in

phase space Tracking of spatial coordinates and the velocities.

The self- consistent interaction between the fields and the particles is most important when viewed in the velocity space because particles of specific velocities will resonate with waves in the plasma to transfer energy.

Structures in velocity space could potentially be used in the future development of multi- resolution compression methods.

W. Tang

SDMCenterData Management ChallengeData Management Challenge

QuickTime™ and aCinepak decompressor

are needed to see this picture.

Decomposition shows transient wave components in time

• A new discovery was made by Z. Lin in large ETG calculations.

• We were able to see radial flow across individual eddies.

• The Challenge:• Track the flow across the

individual eddies, give statistical measurements on the velocity of the flow

• Using Local Eddy Motion Density (PCA) to examine data.• Hard problem for lots of

reasons! Ostrouchov ORNL

SDMCenterPhysics in tokamak plasma edgePhysics in tokamak plasma edge

• Plasma turbulence• Turbulence suppression (H-mode)• Edge localized mode and ELM

cycle• Density and temperature pedestal• Diverter and separatrix geometry• Plasma rotation• Neutral collision

Edge turbulence in NSTX (@ 100,000 frames/s)Diverted magnetic field

SDMCenterXGC codeXGC code

• XGC-0 self-consistently includes• 5D ion neoclassical dynamics, realistic magnetic geometry

and wall shape• Conserving plasma collisions (Monte Carlo)• 4D Monte Carlo neutral atoms with recycling coefficient• Conserving MC collisions, ion orbit loss, self-consistent Er • Neutral beam source, magnetic ripple, heat flux from core.

• XGC-1 includes• Particle source from neutral ionization • Full-f ions, electrons, and neutrals• Gyrokinetic Poisson equation for neoclassical and turbulent

electric field• Full-f electron kinetics for neoclassical physics• Adiabatic electrons for electrostatic turbulence• General 2d field solver in a dynamically evolving 3D B field

QuickTime™ and aYUV420 codec decompressor

are needed to see this picture.

SDMCenter

Neoclassical potential and flow of edge plasma from XGC1

Neoclassical potential and flow of edge plasma from XGC1

Electric potential Parallel flow and particle positions

SDMCenter

Phs-0: Simple coupling:

with M3D and NIMROD

XGC-0 grows pedestal along neoclassical root

MHD checks instability and crashes the pedestal

The same with XGC-1 and 2

Phs-1: Kinetic coupling:

MHD performs the crash

XGC supplies closure information to MHD during crash

Phs-2: Advanced coupling:

XGC performs the crash

M3D supplies the B crash information to XGC during the crash

XGC-MHD coupling planXGC-MHD coupling plan

Blue: Developed • Red: To be developed

•Need real-time visualization to help monitor/debug these simulations. •Need better integration with interactive debugging sessions. •Need to be able to look at derived quantities from raw data.

SDMCenter

Data replication

XGC-M3D code couplingCode coupling framework with KeplerXGC-M3D code couplingCode coupling framework with Kepler

XGC on Cray XT3

End-to-end system 160p, M3D runs on 64PMonitoring routines here

Ubiquitous and transparent data access via logistical networking

User monitoring Data replication

Post-processing

40 Gb/s

Data

arch

iving

 

SDMCenterCode Coupling FrameworkCode Coupling Framework

XGC1XGC1 R2D0R2D0 M3DOMPM3DOMP

M3DMPPM3DMPPlustre

Bbcp first then portals with sockets.

lustreNecessary steps for initial completionR2D0, M3DOMP becomes a serviceM3DMPP is launched from Kepler once M3DOMP returns a failure condition.XGC1 stops when M3DMPP is launched.Get incorporated into Kepler

SDMCenter

Kepler workflow frameworkKepler workflow framework

Kepler: developed by the SDM Center

• Kepler is an adaptation of the UC Berkeley tool, Ptolemy

• Can be composed of sub-workflows• Uses event-based “director” and

“actors” methodology• Features in Kepler relevant to CPES

• Launching components (ssh, command line)

• Execution logger – keep track of runs

• Data movement – Sabul, Gridftp, Logistical Networks (future), data streaming (future).

SDMCenter

Original View of CPES workflow(a typical scenario)

Original View of CPES workflow(a typical scenario)

What’s wrong with this picture?

RunSimulation

RunSimulation

Move filesIn time step

Move filesIn time step

AnalyzeTime step

AnalyzeTime step

VisualizeAnalyzed data

VisualizeAnalyzed data

SimulationProgram

(MPI)

SimulationProgram

(MPI)

TS

IterateOn TS

DiskCache

SRMData Mover

SRMData Mover

SeaborgNERSC

HPSSORNL

TS TS

DiskCache

Disk cackeEwok-ORNL

AnalysisProgram

AnalysisProgram

CPESVIS tool

CPESVIS tool

KeplerWorkflowEngine

Softwarecomponents

Hardware+ OS

KEPLER

SDMCenterWhat’s wrong with this picture?What’s wrong with this picture?

• Scientists running simulations will NOT use Kepler to schedule jobs on super-computers• Concern about dependency on another system• But need to track when files are generated so Kepler can move them• Need a “FileWatcher” actor in kepler

• ORNL permit only One-Time-Password (OTP)• Need a OTP login actor in Kepler

• Only SSH can be used to invoke jobs including data copying• Cannot use GridFTP (requires GSI security support at all sites)• Need an ssh-based DataMover actor in Kepler: scp, bbcp, …

• HPSS does not like a large number of small files• Need an actor in Kepler to TAR files before archiving

SDMCenter

New actors in CPES workflowto overcome problems

New actors in CPES workflowto overcome problems

Detect whenFiles are

Generated

Detect whenFiles are

Generated Movefiles

Movefiles

Tar files

Tar files

OTPLoginactor

OTPLoginactor

DiskCache

FileWatcher

actor

FileWatcher

actor

SeaborgNERSC

HPSSORNL

DiskCache

Disk cackeEwok-ORNL

ScpFile copier

actor

ScpFile copier

actorTar’ing

actor

Tar’ingactor

KeplerWorkflowEngine

Softwarecomponents

Hardware+ OS

LoginAt ORNL

(OTP)

LoginAt ORNL

(OTP)Archive

files

Archivefiles

Localarchiving

actor

Localarchiving

actor

SimulationProgram

(MPI)

SimulationProgram

(MPI)

StartTwo

Independentprocesses

KEPLER

1

2

SDMCenterFuture SDM work in CPESFuture SDM work in CPES

• Workflow Automation of the coupling problem.• Critical for for code debugging.• Necessary to track provenance to ‘replay’ coupling

experiments.• Q: Do we stream data or write files?

• Dashboard for monitoring simulation.• Fast SRM movement of data NERSC<--> ORNL.

SDMCenter

Asynchronous petascale I/O for data in transit

Asynchronous petascale I/O for data in transit

• High-performance I/O• Asynchronous• Managed buffers• Respect firewall

constraints

• Enable dynamic control with flexible MxN operations• Transform using

shared-space framework (Seine)

User applications

Seine coupling framework interface

Other program

paradigms

Shared space management

Load balancing

Directory layer Storage layer

Communication layer (buffer management)

Operating system

SDMCenterCurrent Status Asynchronous I/OCurrent Status Asynchronous I/O

• Currently working on XT3 development machine (rizzo.ccs.ornl.gov).

• Current implementation based on RDMA approach.• Current benchmarks indicate 0.1% overhead

writing 14TB/hour on jaguar.ccs.ornl.gov.• Looking at changes in ORNL infrastructure to deal

with these issues.• Roughly 10% of machine will be carved off for real-time

analysis. (100 Tflop for real-time analysis with TBs/sec bandwidth).

SDMCenter

SDM/ORNL Dashboard: Current Status

SDM/ORNL Dashboard: Current Status

• Step 1:• Monitor ORNL and

NERSC machines.

• Log in• https://ewok-web.ccs.ornl.

gov/dev/rbarreto/SDMP/WebContent/SdmpApp/rosehome.php

• Uses OTP.

• Working to pull out users jobs.

• Workflow will need to move data to ewok web disk.• Jpeg, xml files

(metadata).

SDMCenterDashboard- futureDashboard- future

•Current and old simulations will be accessible on webpage.•Schema from simulation will be determined by XML file the simulation produces.•Pictures and simple metadata (min/max…) are displayed on the webpage.•Later we will allow users to ‘control’ their simulations.

SDMCenterThe End-to-End FrameworkThe End-to-End Framework

SRMSRM LNLN Async. NXM streamingAsync. NXM streaming

Workflow AutomationWorkflow Automation

Applied MathApplied Math

ApplicationsApplications

Data MonitoringData Monitoring

CCACCA

VIZ/DashboardVIZ/Dashboard

Me

tada

ta rich o

utput from

com

po

nen

ts.

SDMCenterPlansPlans

• Incorporate workflow automation into everyday work.• Incorporate visualization services into the workflow.• Incorporate asynchronous I/O (data streaming) techniques.• Unify Schema in fusion SciDAC PIC codes.

• Further Develop workflow automation for code coupling.• Will need dual-channel Kepler actors to understand data streams.• Will need to get certificates to deal with OTP with workflow systems.• Autonomics in workflow automation.• Easy to use for non-developers!

• Dashboard.• Simulation monitoring (via push method) available end Q2: 2004.• Simulation control!