TeraScale Supernova Initiative

04/22/23 Anthony MezzacappaSciDAC PI Meeting, Charleston, SC

1


2

www.tsi-scidac.orgTeraScale Supernova InitiativeTeraScale Supernova Initiative

11 Institution, 21 Investigator, 34 Person, Interdisciplinary Effort ascertain the core collapse supernova mechanism(s) understand supernova phenomenology

e.g.: (1) element synthesis, (2) neutrino, gravitational wave, and gamma ray signatures provide theoretical foundation in support of OS experimental facilities (RHIC, RIA, NUSEL, …) develop enabling technologies of relevance to many applications

e.g. 3D, multifrequency, precision radiation transport serve as computational science testbed

drive development of technologies in simulation “pipeline” (data management, networking, data analysis, and visualization)

Explosions ofExplosions ofMassive StarsMassive Stars Relevance:Relevance:

Element ProductionElement ProductionCosmic LaboratoriesCosmic LaboratoriesDriving ApplicationDriving Application

With ISIC and other collaborators:With ISIC and other collaborators:89 people from 28 institutions involved.89 people from 28 institutions involved.


3

Components of a Supernova Model1. Accurate (Boltzmann) Neutrino Transport2. Turbulent Stellar Core Flow3. Stellar Core Magnetic Fields4. Gravity (Einsteinian)5. Nuclear (Stellar Core) and Weak Interaction (Neutrino) Physics

Core collapse supernovae are radiatively (neutrino) driven. Fluid instabilities, rotation, and magnetic fields will play a role. Strong (Einsteinian) rather than weak (Newtonian) gravity.

Bruenn, DeNisco, and Mezzacappa (2001)


4

TSI’s 3 Paths to 2D/3D Supernova Models

1) 2D/3D Hydrodynamics/MHD with Radial Ray TransportUse existing 1D codes along each ray.Transport parallelized trivially (one ray per processor).

2) 2D/3D Hydrodynamics/MHD with 2D/3D Multigroup Flux-Limited Diffusion (MGFLD)Fully 2D/3D.Sophisticated approximation to Boltzmann transport.Less computationally intensive than Boltzmann transport.Access to 3D science on TeraScale (versus PetaScale) platforms.

3) 2D/3D Hydrodynamics/MHD with 2D/3D Boltzmann TransportFinal word.Explosion mechanism extremely sensitive to transport treatment.MGFLD compares well with Boltzmann in 1D. What about 2D? 3D?


5

• TSI members first to perform 1D models with Boltzmann transport.Mezzacappa and Bruenn (1993)Mezzacappa et al. (2001)

• Progress on 2D (and 3D) Boltzmann transport progressing rapidly.

Development of formalism for conservative (energy and lepton number) general relativistic

neutrino transport (analytical tour de force).Cardall and Mezzacappa (2003)

Development of finite differencing. Construction of GenASiS. Completion of test problems. Initiation of realistic 2D supernova studies.

1. Difficult to develop number- and energy- conservative differencing for these “observer corrections (aberration, frequency shift).”2. Difficult to handle “advection” terms when neutrinos and matter are in equilibrium.3. Memory and CPU requirements.

What makes neutrino transport difficult?

Without this, supernova simulationsdifficult to interpret.


6

Development of radiation field stationary state in nonspherical fixed medium:

Density Distribution Radiation Field Energy Density and Flux

2D Boltzmann Neutrino Transport Test Problem


7

Solve for first moment of neutrino distribution(truncation of 2N-1 moments obtained with Boltzmann solution).

2D MGFLD Equations

Advection Terms

Observer Corrections


8

First 2D simulations with multifrequency neutrino transport, advection terms, and observer corrections:

Scientific Target: Development of Proto-Neutron Star Instabilities Close coupling of matter and neutrinos requires fully 2D transport for an accurate assessment. What impact do the neutrinos have on the development of these instabilities?

Swesty and Myra (2004)

Running on 1024 processors at NERSC.

Scaling now to 2048.

Fully implicit solve.QuickTime™ and aPlanar RGB decompressor

are needed to see this picture.


9

Initial shock location/strength depend on knowledge of nuclearstates and their occupationduring core collapse.

This is a challenge in nuclear computation being addressed by TSI’s nuclear theorists.

This challenge is exacerbated by the fact that nuclei increase in size (neutron and proton number)/complexity (population of states, collective excitations) during collapse.


10

Significant change in initial shock location and strength and stellar core profiles when state of stellar core nuclei computed with more realistic nuclear models and when this new nuclear physics is included in the supernova models.

Hix et al. 2003, Physical Review Letters, 91, 201102.Langanke et al. 2003, Physical Review Letters, 90, 241102.

Merger of two fields at their respective states of the art. (SciDAC enabled.)


11

In addition to physics of nuclei during collapse, supernova mechanism depends on high-density physics of “nuclear matter.”

Complex nuclear many-body problem. How sensitive is the supernova mechanism to changes in this physics?

TSI using several “equations of state” (EOSs).1. Lattimer-Swesty (LS, Industry Standard)2. Wilson3. Stone-Newton (SN, New)

LS and SN EOSs are both phenomenological at some level, but fundamentally different.

Will allow us to explore sensitivities to high-densityEOSs despite uncertainties.

Nuclear Matter

Nuclei


12

2 fundamentally new instabilities discovered by TSI (computationally):

Stationary Accretion Shock Instability (SASI)

Lepto-Entropy Fingers

Supernova shock wave may become unstable.Instability will

help drive explosion, define explosion’s shape.

Operates between the proto-neutron star andsupernova shock wave.

Blondin, Mezzacappa, and DeMarino (2003)

Operates in the proto-neutron star.Instability may help boost neutrino luminosities,which power the explosion.

Bruenn, Raley, and Mezzacappa (2004)


13

QuickTime™ and aAnimation decompressor



14

r-process Element Synthesis• Responsible for half the elements heavier than iron.

• Believed to occur above proto-neutron star after explosion in a neutron-rich, neutrino-driven wind.• In past, difficult to obtain “right” conditions in supernova simulations.

Neutrinos driving wind also drive it towardproton richness.

TSI Discovery: An r-process can occur under such conditions…

Near “Symmetric” (equalnumbers of protons andneutrons)

…for rapid wind velocitiesMeyer 2002, Physical Review Letters, 89, 231101


15

Implicit Time Differencing…• Extremely Short Neutrino-Matter Coupling Time Scales• Neutrino-Matter Equilibration• Neutrino Transport Time Scales

…Leads to Nonlinear Algebraic Equations Linearize Solve via Multi-D Newton-Raphson Method

Solve Large Sparse Linear Systems

Tera- to Peta-Scale Systems

Correspondence between structure of integro-PDE and underlying linear systems...


16

Progress (in conjunction with TOPS):

2D/3D MGFLD

Sparse Approximate Inverse PreconditionerSaylor, Smolarski, and Swesty (2004)

Successfully implemented in 2D MGFLD code (V2D).

2D/3D Boltzmann Transport

“ADI” Preconditioner D’Azevedo et al. (2004)

Successfully implemented in 1D Boltzmann code (AGILE-BOLTZTRAN).• Dense LU factorization was used for dense blocks (D’Azevedo).

Being implemented in 2D/3D Boltzmann code (GenASiS).• Sparse incomplete LU factorization for dense blocks (D’Azevedo, Eijkhout).


17

Hydrodynamics VH-1 (PPM) ZEPHYR (Second Order Finite Difference) Zeus-MP (Second Order Finite Difference)

Magnetohydrodynamics Zeus-MP (Second Order Finite Difference)

Neutrino Transport MGFLD_TRAN: 1D General Relativistic Hydrodynamics

with 1D General Relativistic Multifrequency Flux-Limited Diffusion AGILE-BOLTZTRAN: 1D General Relativistic Adaptive Mesh Hydrodynamics

with 1D General Relativistic Boltzmann Transport V2D/V3D: 2D/3D MGFLD Transport Code (Under Development) GenASiS: 2D/3D Boltzmann Code (Under Development)

TSI Radiation Magnetohydrodynamics Code Suite

Extensively analyzed by PERC.Analysis by PERC recently begun.

Zeus-MP Single-CPU performance boosted by factor of 2. Highest stat ever seen at NERSC (30% of peak).

AGILE-BOLTZTRAN Parallel port, scalable linear solve, PERC analysis reduced run times from weeks to days.


18

InputData

HighlyParallelCompute

Output~500x500files

Aggregate to ~500 files (< 2.2GB each)

HPSSarchive

Data Depot

Logistic NetworkL-Bone

Local MassStorage 14+TB)

Aggregateto one file (~1 TB each)

VizWall

Viz Client

Local 44 Proc.Data Cluster- data sits on local nodes for weeks

Viz Software

As TSI enters “production mode” managing its Workflows has become a paramount issue.

Ideally, we would like to automate these workflows.

Data ManagementNetworkingVisualization

These must be viewed together.

Collaboration between:

SDM Arie Shoshani Nagiza Samatova Guru Kora Ian Watkins Mladen Vouk

Networking Beck Atchley Moore Rao

Visualization (TSI) Blondin Toedte


19

Addressing Bulk Data Transfer Needs:Addressing Bulk Data Transfer Needs:

Logistical Networking Light Weight Low Level Deployable … Solution

New Paradigm Integrate storage and networking. Multi-source, multi-stream. Easy for TSI members to share data.

QuickTime™ and aAnimation decompressor


Atchley, Beck, and Moore (2003)

Data transfer rates 200-300 Mbps using TCP/IP!Limit set by ORNL firewall.Greater rates expected

outside firewall,other protocols (e.g., Sabul).

Direct impact on TSI’s workflow!

Current Data Generation Rate:500 Mbps (10 TB in 2 days).


20

CERN

Chicago

Sunnyvale

Atlanta

ANLFNAL

ORNL

CalTech

SLAC

LBL

NERSC

PNNL

10 Gbps

10 Gbps

DOE Science UltraNet + NSF CHEETAH

Seattle

BNL

JLab

University

DOE National Lab

Future Connections

UltraNetCHEETAH

UVa

NCSU

CUNY

TSI serving as a testbed for an NSF-funded network (Cheetah) designed to develop provisioning technologies for dedicatedchannels.

PI: Rao

TSI serving as a testbed for a proposed effortto bring together UltraNet and Cheetah.

PI: Rao

TSI is testbed for a proposed effort to allow a broad research community to access and use UltraNet.

PI: Beck


21

QuickTime™ and aYUV420 codec decompressor


New Rendering Techniques/Quantitative, Comparative Visualization [Kwan-Liu Ma (UC Davis), Pat McCormick, Jim Ahrens (LANL)]

Rapid rendering. Rendering in hardware. Interactivity.

High-resolution, high-quality rendering. Quantitative visualization.

• Multiple variables, gradients and functions of variables.• Computation done in hardware.• Interactivity. Kwan-Liu Ma (UC Davis)

TSI members played a major role in development of Exploratorium/Powerwall at ORNL. Successfully deployed EnSight for

production, remote, and collaborative visualization.

• Development of representations for higher dimensional data for 2D supernova models. Close coupling of

data management, networking, and visualization.


22

Kwan-Liu Ma (UC Davis)

QuickTime™ and aYUV420 codec decompressor



23

A number of scientific discoveries have been made in TSI’s first two years of operation.A number of technical and technological breakthroughs have enabled our science.

Execution of the next major stage in our science has begun: Delivery of 2D supernova models.Continue to serve as a testbed for enabling technologies of relevance to other SciDAC applications.

Documents

TeraScale Supernova Initiative