Kares cei viz08

Slide 1

Operated by Los Alamos National Security, LLC for NNSA

U N C L A S S I F I E D


EnSight and Computing at the EnSight and Computing at the Petascale Petascale

LA-UR-08-05876LA-UR-08-05876Robert J. Kares and Robert K. GreeneRobert J. Kares and Robert K. Greene

Applied Physics (X) DivisionApplied Physics (X) Division

Los Alamos National LaboratoryLos Alamos National Laboratory

Presented to:Presented to:

CEI VIZ’08 ConferenceCEI VIZ’08 Conference

Baltimore, MDBaltimore, MD

September 23-25, 2008September 23-25, 2008

2Operated by Los Alamos National Security, LLC for NNSA



Acknowledgements

Thanks to my LANL colleague and friend:

Robert Greene

for creating most of the visualizations in this presentation.

Thanks also to my LANL colleagues:

Brian Albright, Will Dearholt, Chris Fryer, Fernando Grinstein, Paul Henning, Jeremy Margulies, Paul Woodward (UMN) and Lin Yin

for their kind permission to use results from their work.




Celebrating a True Milestone in Supercomputing !

On May 26, 2008 a joint team of physicists and computer scientists from Los Alamos National Laboratory and IBM achieved a long deamed of goal in supercomputing.

At 3:00 AM EST at the IBM assemble plant in Poughkeepsie, NY the new RoadRunner machine under construction for LANL by IBM demonstrated a sustained performance of:

1,026 teraflops/s or 1.026 petaflops/s

on the standard Top500 LINPACK benchmark.

In the weeks that followed several real applications were run on the full machine that demonstrated some truly amazing performance numbers.




SPaSM - Molecular Dynamics• Full 17 CU run achieved 361 TF • 26% of theoretical peak (double precision 1.376 PF) • 37 GF per Cell (36% of SPE peak) • Kernel operation achieves 45% of Cell theoretical peak

VPIC - Laser-Plasma Interactions• 1.00 Trillion particle run• Aggressive test of full 17 CU system• Achieved sustained performance of >374.25 TF

— 11% of theoretical max performance (single precision 3.0 PF)• Cell processes used 42.8 TB RAM (93.8% of available Cell memory)

— Opteron processes used 7.3 TB RAM

Sustained Petascale Performance from Real Applications

Image courtesy of Lin Yin, LANL




EnSight Image from VPIC Laser-Plasma Interaction Problem

Image courtesy of Lin Yin, LANL




Paul Woodward’s PPM Hydro• A single CU test problem with two fluid Rayleigh-Taylor mixing• Ran 20,320 time steps on a 768 X 2176 X 896 grid

in about 2 hours of wall clock time• On a single CU (1/17th of the full machine) achieved

a sustained performance of 15.1 TF• Paul estimates that he can run a 60 billion

cell problem, 115,000 time steps on the

full 17 CU machine in about 34 hours of

wall clock time at a sustained performance

of 200 TF

Sustained Petascale Performance from Real Applications

Image courtesy of Paul Woodward, University of Minnesota




Roadrunner at a glance

Cluster of 17 Connected Units• 6,120 AMD dual-core Opterons• 12,240 IBM PowerXCell 8i processors• 44.1 Teraflops peak (Opteron)• 1.26 Petaflops peak (Cell eDP)• 1.026 PF sustained Linpack, #1 Top500

InfiniBand 4x DDR fabric• 2-stage fat-tree; all-optical cables• Full bisection BW within each CU

— 384 GB/s (bi-directional)• Half bisection BW among CUs

— 3.3 TB/s (bi-directional)• Non-disruptive expansion to 24 CUs

98 TB aggregate memory• 49 TB Opteron• 49 TB Cell

204 GB/s sustained File System I/O• 204x2 10G Ethernets to Panasas

Fedora Linux (RHEL possible)

SDK for Multicore Acceleration• Cell compilers, libraries and tools

xCAT Cluster Management• System-wide GigEnet network

2.35 MW Power• 437 MFlop/Watt, #3 on Green500

Size:• 278 racks• 5200 ft2

• 55 miles of IB cable• 250 tons




PowerXCell 8i4 GB

PowerXCell 8i

4 GB

Roadrunner is a cluster-of-clusters of “Triblade” nodes

PowerXCell 8i4 GB

PowerXCell 8i

4 GB

Opteron x2

8 GB

Opteron x2

8 GB

InfiniBand 4x DDR

2 GB/s, 2µs

2 PCI-E x8 links, 2 GB/s, 2µs each

Triblade schematic

A “connected unit” is 180 Triblades on a first level InfiniBand

switch




PowerXCell 8i4 GB

PowerXCell 8i

4 GB

The PowerXCell 8i is a heterogeneous network on a chip, and the floating point workhorse of a Triblade

PowerXCell 8i4 GB

PowerXCell 8i

4 GB

Opteron x2

8 GB

Opteron x2

8 GB

2 GB/s, 2µs

2 PCI-E x8 links, 2 GB/s, 2µs each

SPE

LS

SPE

LS

SPE

LS

SPE

LS

SPE

LS

SPE

LS

SPE

LS

SPE

LS

EIB

PPE

4 GB

InfiniBand 4x DDR




Interactive TeraScale Visualization and Analysis with the EnSight Server-of-Servers

EnSight SOS

ServerServerServer

ServerServerServerServer

ServerServerServerServerServerServerServerServerServerServerServerServerServerServerServerServerServerServerServerServerServerServerServerServerServerServerServerServerServerServerServerServerServer

Local Network orASC DISCOM WAN

LANL Los Alamos, NMCompute Platform

Local/Remote

Roadrunner or LLNL Purple NodesViewMasterViz Cluster

EnSightMaster Client

264 HP xw8200 nodes nVidia 3400/4500 cards 3.2 TB memory 35 TB scratch disk

DR clientDR client

DR clientDR client

DR client

LANL Data Visualization Corridor delivers high

resolution client images to desktops, PowerWalls and the CAVE at 60 fps with stereo

GPFS• Leave the data on the compute platform.

• Run the SOS in distributed mode.

• If you have to move the data…you loose !




Today LANL relies primarily on ViewMaster, our 264 node HP Linux rendering cluster for its large data production visualization needs:– Each of the 264 nodes is an HP xw8200 with dual 64 bit 3.4 GHz Xeons.– 132 of the nodes are display nodes connected to either user desktops or

shared facilities, have 16 GB of memory each and an nVidia 4500 or 4500G graphics card.

– The remaining 134 rendering pool nodes have 8 GB each and an nVidia 3400 card for parallel rendering.

– ViewMaster has its own 35 TB NetApp filer which is hung off its local GigE switch and is cross-mounted on all the nodes to provide global disk space dedicated entirely to viz.

– All nodes have both a GigE and InfiniBand 4X interconnect: – GigE carries all external and file I/O traffic. – IB carries data for parallel rendering with the EnSight DR client.

Petascale Data Requires Big Dedicated Viz Hardware




ViewMaster drives 50 user offices, 3 stereo powerwalls and the five-sided CAVE in the Metropolis Center for Modeling and Simulation.

Each office user has their own dedicated ViewMaster node with a 1920 X 1080 stereo display.

34 ViewMaster nodes are dedicated to driving the Metropolis Center’s five-sided CAVE using nVidia 4500G’s for genlocked stereo.

Images are delivered via a high bandwidth, point-to-point, digital fiber optic distribution system to offices and theaters.

And of course EnSight, EnVideo and Reveal run all of these shared facilities and provide all the visualization and analysis needs for office users.

Petascale Data Requires Big Dedicated Viz Hardware




LANL Metropolis Center’s Stereo Collaboratory PowerWalls

Two 3840 X 2048 pixel stereo powerwalls.

Designed for interactive data exploration with groups of about a dozen users.

Designed to be used by end-users on a walk-in basis.

EnSight GUI runs on a separate console panel with the stereo display on the powerwall.

So if you can run EnSight your desktop, you can run it on one of these stereo walls.

Teach users to run EnSight and they can use all of our big shared display facilities with no extra effort.




LANL Metropolis Center’s PowerWall Theater

• 7680 X 4096 pixel stereo wall• Full multimedia• Seating for 85• For shared, interactive data exploration and presentations• EnVideo plays 60 fps in stereo




World’s largest and most complex VR environment:• 33 Christie Digital Mirage 2000 projectors:

— 3 X 3 projectors on the front panel.— 3 X 2 projectors on floor and ceiling.— 2 X 3 projectors on left and right side walls.

• Rear projection on all five surfaces including both the floor and ceiling.• VICON Motion Systems tracking system with 8 IR cameras viewing the tracked volume.• Uses a unique configuration of IR reflecting balls attached to the user’s stereo glasses

and input wand to track the position and orientation of the head and 6D input device.

EnSight DR client running on 34 nodes of ViewMaster drives the five-sided CAVE environment.

User controls the environment by selecting items from the EnSight heads up macro (HUM) display with the wand or by pressing buttons on the wand that are bound to different transformations or user-defined macros.

Metropolis Center’s Five-Sided CAVE












EnSight from Desktops to Petaflops

EnSight is used by LANL users for visualization and analysis of everything from our smaller 2D simulations to our largest 3D hero calculations.

EnSight is run on everything from desktop PC’s and Mac’s to our 3 TF ViewMaster rendering cluster attached to the five-sided CAVE:• LANL users have only one tool to learn for their visualization and analysis needs.• If you can use EnSight on your desktop, you can use it anywhere.

EnSight is interfaced to all of our codes:• This makes code-to-code comparisons and V&V activities easy.• EnSight also reads Silo and Exodus II files so we can compare results from our

codes with those of other laboratories.

EnSight enables highly effective, scalable distance computing:• Leave the data on the compute platform and run EnSight distributed.• Can compare different datasets no matter how large they are or

where they are located.




Interactive TeraScale Visualization and Analysis with the EnSight Server-of-Servers

EnSight SOS

ServerServerServer

ServerServerServerServer

ServerServerServerServerServerServerServerServerServerServerServerServerServerServerServerServerServerServerServerServerServerServerServerServerServerServerServerServerServerServerServerServerServer

Local Network orASC DISCOM WAN

LANL Los Alamos, NMCompute Platform

Local/Remote

Roadrunner or LLNL Purple NodesViewMasterViz Cluster

EnSightMaster Client

264 HP xw8200 nodes nVidia 3400/4500 cards 3.2 TB memory 35 TB scratch disk

DR clientDR client

DR clientDR client

DR client

LANL Data Visualization Corridor delivers high

resolution client images to desktops, PowerWalls and the CAVE at 60 fps with stereo

GPFS• Leave the data on the compute platform.

• Run the SOS in distributed mode.

• If you have to move the data…you loose !




The Same Viz Architecture is Used for Our Smaller 2D Problems

EnSightServer

Local GigE Network

LANL Los Alamos, NMLocal Compute Platform

Redtail or TLCC NodesDesktop PC

EnSightClient

GPFS

• Leave the data on the compute platform.

• Run EnSight in distributed mode.




EnSight and Other Visualization Efforts at LANL

EnSight is the production visualization and analysis tool used by all of the ASC users at LANL:• EnSight serves an extremely broad community of LANL users.• Rich set of powerful visualization and analysis features.• Fully scalable, parallel and distributed.• Supports all ASC platform architectures and is interfaced to all the codes.• EnSight is a very well supported, stable, commercial product.

Open source packages like VTK and ParaView have their place in the LANL community as research tools for exploring and prototyping new visualization and analysis techniques:• Exploring uses for the Cell processors on Roadrunner like ray tracing.• Prototyping uses for GPU’s in data analysis.

But open source packages are no substitute for a fully supported, stable, commercial software product in a demanding production environment like the one we have at LANL.




EnSight and Legacy Viz Tools at LANL

We have many legacy visualization tools at LANL.

Over time EnSight is replacing all of them with a single powerful solution.

The timescale to complete this transformation is not limited by any technical shortcomings with EnSight or by EnSight’s ease of use.

Rather it is limited by users’ reluctance to learn new tools.

So why are people changing at all?• Because they see other users doing things with EnSight that they cannot do with

legacy tools.• Using EnSight for their analysis results in a much better understanding of their

simulation results.




Studying Turbulence with EnSight: the Taylor-Green Vortex in 3D

The Taylor-Green vortex is one of the simplest systems in which to study the generation of small scales and turbulence by 3D vortex stretching:• Evolution from a simple initial vortex flow provides a dynamical model of decaying

turbulence.• Used as a test of our continuous adaptive mesh refinement code RAGE.

Here we wish to understand the dynamical onset and eventual decay of the turbulence by using EnSight to display many diverse quantities simultaneously:• Use EnSight to extract a brick of floats containing the 3 velocity components.• Perform 3D FFT’s on the 3 velocity components with a Fortran routine.• Take angular averages over direction in k space to form the power spectrum of the

kinetic energy vs |k|.• Convert the power spectrum vs |k| on every timestep to a 1D part. • Animate everything together on the same image to study the dynamical evolution

from the initial coherent vortex flow to fully turbulent Kolmogorov flow that decays away in late time.




Studying Turbulence with EnSight: the Taylor-Green Vortex in 3D

Simulation courtesy of F. Grinstein,

LANL




EnSight and Python: Comparing 3D Simulations of the Vetter-Sturtevant Shocktube Experiments

The Vetter-Sturtevant shocktube experiments:• Two gases, air and SF6, initially separated by a membrane inside a tube, are mixed

by the passage of a strong shock through the interface between them.• Used as a test of our continuous adaptive mesh refinement code RAGE.

How can you compare different 3D simulations of these experiments?

Use EnSight and Python to perform a dimensional reduction of the 3D simulation data:• Use Python script with EnSight to place an integration plane at positions along the

tube axis oriented perpendicular to the axis. • At each position EnSight computes the mix parameter by integration over the plane

and returns the result to Python. • In this way a 1D plot of the mix parameter vs position is generated at each time.• 3D mix is reduced to a plot of the1D mix parameter vs position that can be

compared between simulations over time.• EnSight is used to integrate the mix parameter along the axis and plot the result

over time.• Animate everything together on the same image to study the quantitative evolution

of the shock induced mixing.




EnSight and Python: Comparing 3D Simulations of the Vetter-Sturtevant Shocktube Experiments

Simulation courtesy of F. Grinstein, LANL




EnSight and Transparency: 3D SPH Simulations of Supernova

3D SPH Simulation of a Supernova detonation:• 15 Solar mass progenitor star• Symmetric explosion in 3D

EnSight uses multiple transparent isosurfaces of density colored by temperature to display the instabilities in the expanding layers of the exploding star.

An interesting alternative to traditional volume rendering methods.

The transparency in this animation uses the new depth peeling feature to greatly accelerate the rendering speed.

The full 3D structure of the expanding shells are much more readily apparent in the stereo version of the animation … check it out at the vendor show.




EnSight and Transparency: 3D SPH Simulations of Supernova

Simulation courtesy of C. Fryer, LANL




EnSight and Extrusion: 2D Simulations of Explosively Formed Cu Jet Impacting on a Plate

2D simulation of a shaped charge explosively forming a penetrating Cu jet and the impact of the jet on a steel plate:• Material strength parameters were used in both metals.• Used as a test of our continuous adaptive mesh refinement code Nobel.

Here we use the new EnSight extrude feature to create a 3D version of the 2D axisymmetric problem by rotation about the axis.

Different materials are rotated by different amounts to visually separate them in the animation:• HE is rotated by 180 degrees• CU and Steel by 270 degrees

Materials are colored by speed to show the jet formation and its impact on the steel.




EnSight and Extrusion: 2D Simulations of Explosively Formed Cu Jet Impacting on a Plate

Simulation courtesy of W. Dearholt, LANL




EnSight and Measured Data: Geologic Mapping of the Pajarito Plateau

Data courtesy of EES Division, LANL




EnSight and Textures: Geologic Mapping of the Pajarito Plateau

Data courtesy of EES Division, LANL




The DOE ASC Program’s Commitment to EnSight Remains Strong

The US Department of Energy’s Advanced Simulation and Computing (ASC) program was established by Congress in 1995 to provide the modeling and simulation capabilities required to maintain the US nuclear stockpile in the absence of nuclear testing.

Since the beginning of ASC EnSight has played a central role in both the LANL and national ASC programs.

Since June of 2000, ASC contracts with CEI from both LANL and the ASC Tri-Lab have helped to accelerate the development of many EnSight features designed to address the problems of big data:• EnSight Gold and the EnSight Server-Of-Servers• EnSight DR Distributed Rendering• EnSight support for VR with distributed memory clusters• EnSight support for many advanced analysis features like Python scripting• CEI tools like EnVideo, EnVe and Reveal




The DOE ASC Program’s Commitment to EnSight Remains Strong

In FY08 LANL ASC contracts with CEI have resulted in several additional new features that are driving the release of EnSight 9: • Full MPI support for the EnSight SOS• Restructuring of the communication between SOS and the DR client to improve

performance

All the benefits of this enormous multi-year development effort are available to everyone in the HPC community via the standard commercial release of EnSight Gold.

In the future ASC faces many challenges:• How to utilize the power of multi-core CPU’s for analysis.• How to track the changing landscape of ASC platform architectures and data sizes.

ASC will continue to work with CEI to insure that EnSight meets these challenges as we move into the petascale future.

Technology

Kares cei viz08