Diving deep: data-management and visualization strategies for adaptive mesh refinement simulations

A daptive mesh refinement, developedby Marsha Berger and her colleaguesin the 1980s for gas dynamical simu-lations,1 is a type of multiscale algo-

rithm that achieves high spatial resolution in lo-calized regions of dynamic, multidimensionalnumerical simulations. Greg Bryan’s excellent ar-ticle2 in the March/April 1999 issue of Computingin Science & Engineering describes our cosmolog-ical AMR algorithm and how we have applied it tostar, galaxy, and galaxy cluster formation. Basi-cally, the algorithm allows us to place very highresolution grids precisely where we need them—where stars and galaxies condense out of diffusegas. In our applications, AMR allows us to achievea local mesh refinement, relative to the globalcoarse grid, of more than a factor of 106. Such res-olution would be totally impossible to achievewith a global, uniform fine grid. Thus, AMR al-lows us to simulate multiscale phenomena that areout of reach with fixed grid methods.

The AMR algorithm accomplishes this byproducing a deep, dynamic hierarchy of increas-

ingly refined grid patches. The data structuresfor storing AMR data are complex, hierarchical,dynamic, and in general quite large. For exam-ple, the simulation of an X-ray galaxy clustershown in Figure 1 used a grid hierarchy sevenlevels deep containing over 300 grid patches.

Existing visualization, animation, and data-management tools developed for simple meshdata structures cannot handle AMR data sets.Consequently, in the past several years we havebeen working at the National Center for Super-computing Applications to overcome this deficit.Here we describe our progress in four main ar-eas: portable file formats, desktop visualizationtools, virtual-reality navigation and animationtechniques, and Web-based workbenches for han-dling and exploring AMR data. Although we haveapplied our work specifically to cosmology, we be-lieve our solutions have broader applicability.

AMR data structures

Within an evolving AMR simulation, the data isorganized in a grid hierarchy data structure—think of a tree of arbitrary structure and depth(see Figure 2). The AMR algorithm generatesthis tree recursively and adaptively. Every nodeand leaf of the tree is associated with a 3D gridpatch (hereafter, simply a grid); these have vari-ous sizes, shapes, and spatial resolutions.

DIVING DEEP: DATA-MANAGEMENTAND VISUALIZATION STRATEGIESFOR ADAPTIVE MESH REFINEMENTSIMULATIONS

The authors’ cosmological applications illustrate problems and solutions in storing,handling, visualizing, virtually navigating, and remote-serving data produced by large-scale adaptive mesh refinement simulations.

MICHAEL L. NORMAN, JOHN SHALF, STUART LEVY, AND GREG DAUES

National Center for Supercomputing Applications

1521-9615/99/$10.00 © 1999 IEEE

M A S S I V E D A T AV I S U A L I Z A T I O N

36 COMPUTING IN SCIENCE & ENGINEERING

JULY/AUGUST 1999 37

In our cosmological simulations, each gridstores arrays of data describing the physical stateof the cosmic fluid (density, temperature, and soon), the gravitational potential, and a list of par-ticle positions and velocities for stars and darkmatter. The cell size ∆x of the grid decreases withdepth in the hierarchy as 1/Rlevel, where R is aninteger refinement factor (2 in our application)and level is the level of grid in the hierarchy.

A given grid in the hierarchy overlies andmore finely resolves a region in its parent grid,and it can also be the parent grid for a yetsmaller, more refined child grid. Every grid, re-gardless of its level, contains a complete repre-sentation of the solution in the region it covers.That is, there are no holes in the parent gridwhere the child grids exist. Consequently, we canobtain an approximate representation of theglobal solution at any level of the hierarchy bycompositing the solutions at or above that level.

The solutions on the grid hierarchy advanceforward in time with individual time steps ∆t thatdecrease proportionally to ∆x, starting at thecoarsest grid (the root grid), and proceeding tofiner and finer grids. When the algorithm reachesthe finest level, each grid at that level will advanceseveral times until its time “catches up” with thetime of its parent grid. The schedule of opera-tions is similar to the W cycle of classic ellipticmultigrid.

Although a tree of grids is a useful way tothink about the grid hierarchy at an instant intime, over time the number of levels and thenumber and location of grids at a given levelchange. Generally, the deeper into the hierar-chy we go, the more grids there are and the

more rapidly they change. This becomes im-portant when devising file structures for writ-ing AMR data to disk. The solution we have de-vised abandons any notion of the hierarchicalrelationship between the grids and instead justconsiders a collection of grids that changes overtime. Each grid writes its own disk file contain-ing attributes such as spatial domain, level of re-finement, and persistence, as well as the fieldand particle data (for more detail, see the side-bar on FlexIO). In addition, the AMR applica-tion writes a single ASCII file to disk with all

Figure 1. A deep adaptive mesh refinement (AMR) grid hierarchy.Contours of cosmic density in a simulation of an X-ray galaxy clus-ter are plotted on a grid hierarchy seven levels deep with severalhundred individual grids.

CompositeAMR grid

Level 0

Level 1

Level 2

(a) (b)

Figure 2. Sch-ematic of anAMR grid hier-archy in 2D:(a) explodedview, showingthe level struc-ture, and (b)logical view oftree datastructure.


the information required to assemble the com-plete grid hierarchy from the grid files.

AMR visualization strategies

AMR data structures have required a reevalua-tion of visualization system architecture. Tradi-tionally, researchers have divided simulation datasets into three primary categories: structuredgrids, unstructured grids, and particles. AMRdata doesn’t really fit into any of these categories,leaving us with a gaping hole in our ability to un-

derstand the results of this emerging class ofsimulations. Consequently, we have investedconsiderable effort in making visualization ofAMR data as sophisticated as that available forthe traditional structured and unstructured grids.Of course, our first attempts at building the nec-essary infrastructure involved recasting the AMRdata into a form that our traditional visualizationsystems could handle.

The most naïve way of approaching AMR vi-sualization is to sample the entire hierarchy into asingle structured grid. The primary advantages of

The sharing of data and visualization tools across researchgroups and applications requires common file formats. AnAMR file format must be both platform- and application-independent. Platform independence means the file formatcannot depend on any particular operating system or proces-sor architecture. Application independence means that thefile format must not depend on a particular set of simulationdata structures and must be amenable to a wide variety ofsimulation implementations. Given the great diversity incode implementations and grid hierarchy structuresdeveloped so far by the AMR community, application inde-pendence has been much more difficult than platform inde-pendence. Above all, the file format must be simple enoughthat it can be easily modified to accommodate new file for-mats not considered in the initial implementation.

The HDF and NetCDF standards (see the Glossary of abbre-viations in the main text) are both popular platform-indepen-dent file formats for storing binary simulation data sets. (Seehttp://hdf.ncsa. uiuc.edu/ for NCSA’s HDF home page andwww.unidata.ucar. edu/packages/netcdf for information on Uni-data’s NetCDF.) Because HDF can interoperate with NetCDF,HDF forms the basis for a platform-neutral file representationfor AMR data. This is all wrapped in a more abstract API—FlexIO,developed by John Shalf—that is capable of more ad-vanced file operations such as redirecting the I/O through anetwork socket with little effort on the part of the programmer(see http:// bach.ncsa.uiuc.edu/FlexIO/). FlexIO is available withbindings for Fortran, C, and C++. It also allows translation toan ancillary file format called IEEEIO, which can be read andwritten using 100% pure Java or IDL (www.rsinc.com) ratherthan requiring Java be linked to native methods.

Both NetCDF and HDF store data in a self-describing man-ner. Each array of data stored in the file intrinsically includesits data type and dimensions. The user can extend the des-cription of the data through attributes—additional namedarrays of information that can store information like coordi-nate systems, time step, and calibration information. Table Aand Table B list the standard attributes used in the FlexIOAMRwriter API. We add these NetCDF-style attributes to each

grid to fully describe its position in the hierarchy and thusuniquely identify it.

As you can see, there is a considerable amount of redun-dancy in these attributes. However, this ensures that themetadata will be in a form that is as convenient as possiblefor application designers, who tend to make different choicesin expressing the same set of concepts. Also, all parametersrelated to refinement and grid placement are expressed bothas integers and in floating-point form. The floating-point rep-resentations are necessary because they are most convenientfor visualization systems, and they are what most simulationsuse internally to represent grid placement. However, the nu-merical precision of floating-point numbers becomes inade-quate after about 10 levels of refinement. The only truly ac-curate representation of placement is integer coordinates. Sointeger and double-precision floating-point descriptions ofrefinement and grid placement are necessary to provideboth maximum fidelity and convenience of representation.

Typical AMR simulation codes simply write one file per gridso that by the end of the simulation run you have a directoryfilled with thousands of files. The AMRwriter API enables youto combine all of the grids in the AMR hierarchy into a singlefile, simplifying data management. In fact, a sequence oftime steps can be stored in a single file that takes into ac-count the fact that grids on different levels are updated atdifferent frequencies.

Finally, the FlexIO AMR API suite includes several sampleimplementations of readers for this file format. The simplestof the readers simply fills out a data structure with metadatainformation that it reads from the file and presents you with alist of information about every grid in the file. A more sophis-ticated reader allows you to select grids that are active at aparticular time step so you can animate changes in the hierar-chy through time. It minimizes data movement from disk au-tomatically by loading only grids that have changed whenthe time step changes. It also allows you to select particularlevels in the hierarchy (or remove intermediate levels thatmay clutter the scene). Finally, an enhanced version of thissecond reader automatically converts the AMR hierarchy into

FlexIO: A portable file format for AMR data

JULY/AUGUST 1999 39

this approach are that resampling is very easy toimplement, and that the visualization-processingalgorithms on structured grids tend to give muchbetter performance than their unstructured-grid(finite-element) counterparts. Resampling cantake place in full 3D (sampling into a uniformmesh) or by projection of all the data onto a 2Dslice. This methodology is clearly not scalable,however, because in many cases an AMR simula-tion has such a huge range of length scales thatthere is insufficient memory in the graphics work-station and indeed in the supercomputer to rep-

resent the entire hierarchy. For example, samplingthe results shown in Figure 1 to a uniform gridwould require a 8,1923 grid—far larger thanwould fit into the memory of the largest super-computer in the world. This method also resultsin a lot of redundant work on grid points thathave been projected from the hierarchy’s coarsergrids into the visualization grid’s finer mesh. Wecan compensate for the ratio of scale by specifyingsmall regions of interest, but this leaves us withan incomplete view of the problem domain. Someearly tools we implemented using AVS and IDL

a mesh of hexahedral finite-element cells (removing all gridoverlaps in the process). After selecting the desired simulationtime step and levels of the hierarchy to show, this third readergives you an array of points and a connectivity list for an arrayof hexahedral cells that connect those points. This operateswithin an AVS module as an example of how to integrateAMR data into existing commercial visualization systems.

In the near future, the HDF group, led by Mike Folk ofNCSA, will update the file format to use HDF5, which pro-vides a much richer and more efficient means of storingmetadata and intergrid relationships (implied or otherwise).HDF5 also has full support for parallel I/O under HDF andMPI; this makes it much more suitable for wide-area distrib-

uted simulation codes. We are also closely watching the ASCIDMF efforts in enhancing HDF5 with a file storage API basedon vector bundles.1 This is a far more ambitious project thanthe standard described above, so we expect it will take sometime to reach production. However, when it does arrive, itwill provide a much more robust data model than we havebeen accustomed to under NetCDF and HDF4.x.

Reference1. D.M. Butler and S. Bryson, “Vector-Bundle Classes Form Powerful Tool for

Scientific Visualization,” Computers in Physics, Vol. 6, No. 6, 1992, pp.

576–584.

Table A. Attribute specification: Real-valued attributes.

Attribute Description

origin Floating-point origin of the data setdelta Physical spacing between grid points in the data setmin_ext Minimum coordinate of the data set in physical spacemax_ext Maximum coordinate of the data set in physical spacetime Current real-valued time that this grid represents

Table B. Attribute specification: Integer-valued attributes.

Attribute Description

level Level of this grid in the AMR hierarchy; levels are numbered from 0 to (n levels – 1)timestep Integer time step in the file with respect to the evolution of the finest-resolution grid

in the hierarchy; in effect, the global simulation time steplevel_timestep Integer time step for the grid with respect to the evolution on its particular levelpersistence Number of time steps the grid persists on this particular level; in terms of the other

parameters, the number of timesteps per level_timestepspatial_refinement Refinement in each Cartesian direction; a divisor that defines the refinement with

respect to the top-level (0th level) gridtime_refinement Refinement for the time step with respect to the top-level (0th level) grid step sizegrid_placement_refinement Attribute created independently from the spatial_refinement for staggered gridsiorigin Integer origin of the grid placed in the coordinates of the spatial_refinement defined

above


were very valuable in the development of theAMR codes (see Figure 3 and the Glossary of ab-breviations). But these are reaching their limits ofscalability as our AMR data sets ramp up in sizeand dynamic range.

To capitalize on AMR’s multiresolution naturewhile still fitting it into existing visualizationframeworks, an obvious choice is to convert theAMR hierarchy into an unstructured grid com-posed of hexahedral cells (cubes). This requiresremoval of overlapping cells in the hierarchy, anoperation that can be implemented with O(n)complexity (where n is the number of vertices inthe entire hierarchy). This technique gives usgraphics output that fully integrates the range ofscale from the simulations. However, finite-element or unstructured grids are much lessmemory-efficient than the structured grids in theoriginal AMR hierarchy because all connectivityin unstructured meshes must be explicit, whereasstructured-mesh vertex connectivity is com-

pletely implicit. Furthermore, visualization algo-rithms on unstructured grids tend to be muchslower than their structured-grid counterpartsand much more difficult to parallelize efficiently.So, while translating AMR hierarchies into un-structured grids has greatly improved the visual-ization fidelity, we are still left with the same sortof performance and scalability problems we hadwhen sampling into a uniform grid.

Unfortunately, these previous endeavors intoAMR visualization treated AMR grid structuresas second-class data types. Visualization systemsand libraries are designed to deal only with datathat they consider first-class data types such asparticles, structured grids, and unstructuredgrids. All other data structures must be trans-lated to fit into this narrow conception of thecomputational domain.

The treatment of AMR grids as first-classdata structures for visualization computationsleads to a variety of performance optimizations.For instance, visualization algorithms can cap-italize on each grid’s spatial location and datacontent to remove it from the computation us-ing lightweight tests. A naïve isosurfacing algo-rithm must search every point in a structuredmesh to determine where the surface should beplaced. But an algorithm can cull a set of AMRgrids based on their precomputed minimumand maximum data values to remove gridswhose data content falls outside the range thatwould intersect with the level surface. An AMRray tracer can trivially evaluate whether a gridin the hierarchy would resolve to subpixel res-olution and thereby choose whether to descendany farther down the hierarchy as it makesprogress through the data. A plane slicer needonly consider grids that it would intersect.These are just examples of the most obvious setof applicable optimizations. We have barely be-gun to explore the rich set of possibilities of agrid structure that expresses data locality andlevel of detail natively as AMR does.

Opportunities for parallelismAs mentioned earlier, when we scale up the sizeof AMR simulations, we still face scalabilityproblems. Most visualization algorithms, li-braries, and systems are inherently serial be-cause that was sufficient back in the days of vec-tor computing. However, to create visualizationtools that can interactively explore the terascaledata sets produced on today’s massively parallelsystems, we must parallelize the visualizationcomputations. That is our only means of fully

Glossary of abbreviationsAMR Adaptive mesh refinementASCI Accelerated Strategic Computing InitiativeAPI Application programming interfaceAVS Advanced Visualization System, a popular graphical visual-

ization tool from Advanced Visual Systems, Maynard, Mass.CAVE Computer-Assisted Virtual Environment. Developed at the

Electronic Visualization Laboratory of the University ofIllinois, Chicago, CAVE is a room-sized virtual-reality en-vironment where the floor, ceiling, and walls project a 3Dstereoscopic view of a scene.

CGI Common gateway interfaceDMF Data management facilityEVL Electronic Visualization Laboratory at the University of

Illinois, ChicagoHDF Hierarchical data format, a platform-independent binary

file format for storing simulation and image data developedby the NCSA HDF group.

HTTP Hypertext transfer protocolIDL Interactive Data Language, a popular interpreted-langu-

age-based visualization tool created by Research SystemsInc. of Boulder, Colo.

MPI Message-passing interfaceNCSA National Center for Supercomputing ApplicationsNetCDF Network common data format, a platform-independent bi-

nary file format for storing simulation data sets developedby the Unidata group at the University Corporation for At-mospheric Research.

RMI Remote method invocationSPMD Single-program, multiple-dataVE Virtual environment

JULY/AUGUST 1999 41

exploiting the performance of this new crop ofsupercomputers. Therefore, from the very start,we have considered opportunities for paralleliz-ing AMR visualization so that it becomes an in-tegral standard as these new libraries develop.

Opportunities for visualization algorithm par-allelism abound. There is plenty of research intoparallelizing visualization computations on struc-tured and unstructured grids, and we can applythat research to AMR visualization algorithms.For structured grids (often called unigrids), a typ-ical parallelization scheme is a regular domaindecomposition of the grid into smaller subcubesof data that processors can operate on in astraightforward data-parallel (SPMD) manner.For unstructured grids, domain decompositionis much more difficult; researchers have proposeda variety of heuristic algorithms to address theproblem.3

AMR data offers many more opportunities forparallel execution in the visualization pipeline.We can, for example, partition individual gridsfor data-parallel computations (treating each as aunigrid computation); however, such domain de-composition incurs a large cost because it re-quires a sizable communication volume betweenprocessors to distribute the grid data. Becausethe cost of distributing the data often exceeds thebenefits of parallelism, this technique has lim-ited utility. On shared-memory architectures likethe Origin 2000, we can use an even lighter-weight partitioning, such as handing out grids todifferent processors in a worker-slaves arrange-ment. When each slave thread finishes comput-ing on a particular grid, it can request anothergrid to compute from the master processor. Thismethod raises the same issues as domain decom-position, but it eliminates the costs of copyingthe data across processors.

Level of detailPerhaps the ultimate way to improve the user’sability to interact with visualization applicationsis to reduce or stagger the amount of data thatmust be computed at any given time. Normally,the stages of the visualization pipeline executesequentially, forcing the user to wait until the en-tire pipeline has executed before the image onthe screen updates. Even if each stage executesin parallel, the wait can be considerable. WithAMR data, we can extend the parallelism in thepipeline direction. Each stage can operate si-multaneously to pass the grids through the pipe-line in a streaming manner. So the root grid ofthe hierarchy can pass through to the viewer

Figure 3. VisualizingAMR data using com-mercial software: (a)screen shot of AVSpipeline, which auto-matically converts anAMR grid data struc-ture into an unstruc-tured mesh of hexa-hedral cells; (b–e)zoom sequenceshowing the depth ofthe grid hierarchy.

(e)

(d)

(b)

(a)

(c)


very rapidly, and over time the higher-detail sub-grids make their way through the pipeline togradually improve the quality of the scene. Us-ing this streaming-AMR-grids paradigm givesthe visualization system a natural continuum be-tween interactivity and fidelity.

We can modify the order in which these levelsof detail fill in simply by sorting the list of gridsat the beginning of the pipeline to fit a particularsorting criteria. For instance, the simplest sort isto fill in the grids by their level in ascending order:coarsest to finest grids on a level-by-level basis.But we can change the sorting algorithm to fill indetail near a point of interest. Alternatively, wecan sort the grids so that they fill in preferentiallyalong the line of sight of the current camera viewof the scene. Grid streaming opens up a robustset of methods for improving interactive perfor-mance and optimizing the method of filling in de-tail to fit a wide range of requirements.

Desktop visualization tools andworkbenches

Back in the 1980s, the supercomputing commu-

nity began to realize that 3D visualization toolswere a basic necessity for understanding the out-put of supercomputer simulation codes. TheLaboratory for Computational Astrophysics4 re-leased a tool called 4D2 to provide a free andpublicly accessible means of visualizing unigriddata coming from its ZEUS-3D code. Thisproved a great success among LCA’s communityof users. LCA is now developing a portable re-placement for 4D2 that will provide a simple-to-use visualization package that handles AMR datain addition to the original complement of uni-grid data visualization features.

Our current implementations for native AMRcomputations are built around the VisualizationToolkit, an extensible, freely available, open-source set of visualization libraries. (See www.kitware.com/vtk.html.) VTK gives us a set of toolsthat we can modify at a very low level to treatAMR data as a first-class (native) data type. Thislets us cast algorithms in a form that takes max-imal advantage of AMR’s regularity and multi-scale nature. We anticipate that this will greatlyimprove the performance and flexibility of visu-alization systems for AMR data. By buildingonto VTK, we’re making these techniques wide-ly available; other research groups can incorpo-rate them into their own visualization tools, thusextending them far beyond our initial efforts.Figure 4 is a screen shot of the 4D2-VTK toolwith an AMR slicer in operation.

The NCSA Computational CosmologyObservatoryDesktop tools have their limits, however. One ofthe biggest problems with general-purpose vi-sualization tools is that they give users too manychoices. It may seem hard to believe that suchflexibility is not necessarily a good thing, butmany full-featured visualization tools bewilderscientists with a dizzying array of widgets andmenu choices, many of which are irrelevant tothe problems they want to solve.

The “workbench” paradigm restricts a tool’sflexibility without making it thoroughly useless.The tool’s designers can optimize the workbenchuser interface for a particular audience and classof problems, emphasizing visualization para-digms and statistical metrics specific to the tar-geted research community. This customizationcan even include details such as the specificnomenclature for units of measurement ratherthan the more generic terms you would find in a general-purpose tool. Furthermore, a goodworkbench operates as a gateway to accessing

Figure 4. Visualizing AMR data using custom software based on theVisualization Toolkit. Here, the gas density on a slice through thecenter of a galaxy cluster is colorized (red = low, blue = high). Forclarity, “slicelets” are offset from one another by a distance propor-tional to the level of the grid hierarchy from which the data is ob-tained. By treating the AMR hierarchical data structure as a first-class data type, VTK makes possible many optimizations that speedup visualization performance.

JULY/AUGUST 1999 43

data archives and running simulation codes. Thislets scientists take advantage of sophisticated re-search codes without themselves being expertcomputer programmers. Designers of these toolsusually implement workbench interfaces as Webtools to maximize flexibility.

The NCSA Cosmology Applications Team—in collaboration with cosmologists at the Univer-sity of Missouri, the Massachusetts Institute ofTechnology, and New Mexico State University—is performing roughly 100 extremely high-reso-lution AMR simulations of X-ray galaxy clustersand placing the results in an online archive calledthe NCSA Computational Cosmology Observa-tory. (See http://sca.ncsa. uiuc.edu.) Theorists willfind this archive useful because they can use thestatistical properties of the simulated cluster sam-ple to constrain allowable cosmological models.Observers will be able to conduct simulated ob-servations of the X-ray clusters and their environsto help plan their observing campaigns. Our goalis to make this archive available to the scientificcommunity while shielding users from the sizeand complexity of the raw AMR data. We alsowant to give users a variety of data manipulation,export, visualization, and analysis tools.

To accomplish these goals, we developed theObservatory as a workbench-style system that

lets users interact with archived simulation dataover the Web. This environment supports dataretrieval across several machines at the NCSAand between those machines and the local clientsystems. The Observatory further supports dy-namic and interactive multidimensional visual-ization and basic analysis tools to extract speci-fied physical attributes of the cluster systems.

Figure 5 shows a diagram of the Observatory.The user begins by selecting a cluster from a cat-alog. The server then retrieves the raw AMR datafrom NCSA’s mass storage system. Once the datais on the Observatory Web server, the user ex-tracts from the AMR files particle data and fielddata sampled to a uniform grid of user-specifiedsize and spatial resolution. At this point, the usermay export the extracted data as HDF files forlocal analysis or use the suite of analysis tools pro-vided as a part of the Observatory.

As we considered what type of visualizationsoftware to provide for clients’ local machines,our critical requirements were that the softwarebe platform-independent and thin (small andquick to download). These characteristics are es-pecially important for 3D visualizations. Wehave successfully addressed these concerns intwo ways. First, the Observatory renders 3D ob-jects on the server side. Client applets down-

Special Focuses

Intelligent Information Retrieval...Intelligent information retrieval—the problem of delivering truly relevant documentsmatching user needs—has become increasingly central in recent years. This special issuewill present creative approaches to solving intelligent information retrieval problems usingAI or ML techniques.

...and ConstraintsA wide variety of problems can be naturally modeled as constraint-satisfaction andoptimization problems, including scheduling, planning, resource allocation, routing, design,configuration, and diagnosis applications in the engineering, manufacturing, and servicesectors. The special issue will provide insight into the increasing popularity of constrainttechnology.

IEEE Intelligent Systems

MARCH/APRIL 1999

& the i r app l i cat ions

IEEE

ALSO:ALSO:

Room service, AI-styleGleaning the WebOnline maps: help or hindrance?DARPA’s High-Performance Knowledge Base program IEEEhttp://computer.org

July

/A

ugust

Iss

ue

AI News You Can Use


loaded from the site let the user specify whichcluster to visualize, which field quantities todownload, and the type of visualization, such asan isosurface. The applet then connects back toa CGI script via the HTTP protocol, which al-lows the user’s applet to post its input to the CGIscript. The script then takes the user’s prefer-ences and launches a server-side VTK-basedrendering program. Using the server for ren-dering achieves the goal of having a thin clienton the user’s local machine.

Second, although visualizations created by VTKare not platform-independent, VTK does provideclasses to convert the visualizations into a VRML2.0 format and then write them to a file. However,instead of writing to a file, our CGI scripts writethe contents to the standard output, and the serversends them back to the applet. The applet thensimply opens a new browser window and specifiesthe content type (VRML), and the browser loadsthe appropriate VRML-viewer plug-in. Becausebrowsers running under many different operatingsystems can display VRML-based graphical rep-resentations, this gives us a platform-independentway of displaying complex 3D visualizations. Inshort, the site leverages widely and freely availabletechnology to deliver high-quality 3D visualiza-tions to a variety of platforms without over-whelming the processing power and storage ca-pacity of a client’s local machine.

We also meet the requirement of lightweightclient visualization software with Observatorysite tools that implement lower-dimensional vi-sualizations through Java graphics. Specifically,the client applet allows the user to launch CGIscripts that sample subsets of large 3D HDF files.The server retrieves these large files from thearchive storage, and they remain on the serverside. Only the smaller subsets of the data are

transferred over the network for theclient software to process.

One capability of the client applet isthe construction of 2D color contourplots of the sampled data. Figure 6 is ascreen shot of a working session of theObservatory implementing these plots.In keeping with the consideration of pro-cessing speed and memory on the clientside, the client applet allows users to con-trol the size and number of images gen-erated. We are also developing archivetools for other forms of lower-dimen-sional visualizations—such as 2D projec-tions of 3D data and 1D line plots.

Future developments of the Observa-tory site will focus on implementing remotemethod invocation (RMI) to establish a more per-sistent and unified computing environment, al-lowing more finely controlled access of objects onthe server. We will also add links to other elec-tronic literature and develop software to interactwith existing digital libraries of observed data. Thiswill mean adding support for more data types andformats, as well as constructing additional analysistools to allow direct comparison with observeddata.

Multiscale virtual navigation andanimation

Virtual environments that provide dynamic viewscovering the full human visual field promise in-sights not otherwise easily achieved into the richvariety of spatial and temporal structures arisingfrom AMR calculations (see Figure 7). However,given the bulk of AMR data, creating graphicalrealizations at acceptable frame rates is a majorproblem even for high-end graphics systems. Vir-tual environments are more demanding in thisrespect than desktop systems: Although desktopusers might tolerate a few seconds’ delay whilean image is computed, VE users lose their senseof immersion and virtual user interfaces becomeunusable if frame rates fall below a few framesper second.

It becomes essential, in the exploration pro-cess, for the VE user to focus attention on sub-sets of the data. Fortunately, the AMR paradigmoffers natural ways to do this. The system cancheaply present data from the coarsest grids.Furthermore, the very distribution of the AMRgrids indicates active areas in the computation.In a cosmological simulation, viewing animatedgrid-bounding boxes alone can reveal locations

Java clientWeb serverMSS

Raw AMR data Raw AMR data Extracted data

FTPAMR grid

extract

AMR particleextract

Java applet Processed data

3DIsoViz

3DParticles

3DSlice

2DContour

1DLinePlot

X-rayImager

VRML

Javagraphics

Figure 5. NCSA Computational Cosmology Observatory.

JULY/AUGUST 1999 45

of filaments and forming galaxy clusters.From such cues, the user could indicate a vol-

ume of interest, perhaps by sweeping it out di-rectly with a gesture. Sizes of interesting regionsin AMR data might well span several orders ofmagnitude as exploration proceeds, but theycould still be naturally described by the virtualusers, who would be able to adjust their scale atwill relative to the data. The visualization systemcould then concentrate its effort on, say, the coars-est few AMR grids spanning that region, extend-ing outward (for context) and inward (for betterresolution) as computation and rendering timepermit. Applying the streaming-AMR-gridschemes described earlier will be crucial here.Much experimentation will be necessary to de-termine how the focusing process should workand how to design the balance between user spec-ification and automatic adaptation.

Focusing attention, of course, isn’t merely a wayto accommodate the limitations of available graph-ics systems; it will be important also in extricatingrelevant features from what can easily become avisually cluttered mass of detail. Maximal realism isnot always the best way to display scientific data!

Still, there will be a need for views of higherquality than those that can be generated at inter-active rates. It seems promising to build a hybridsystem that would decouple part of the renderingfrom the high-frame-rate virtual environment:The user would plant a virtual camera in space. Aseparate system (possibly a remote supercom-puter) would render clearer views of the camera’sscene at leisure and display the resulting imageswithin (or alongside) the virtual environment.Meanwhile, the user would continue to interactwith the virtual world.

Design of suitable user interfaces is anothermajor issue for virtual AMR. Describing justwhat should be shown, and when and how, willcall for plenty of knobs to tweak. While immer-sive VEs make direct spatial interaction easy,they lack good counterparts to the keyboard andpointer that make the familiar desktop widgetsso efficient to use.

Virtual AMR in the Virtual DirectorWe are using the NCSA Virtual Director as a hostfor developing virtual AMR visualization tools(http://virdir.ncsa.uiuc.edu/virdir/). The Virtual Di-rector is a CAVE-based software framework thatwraps around visualization applications, support-ing navigation through space and time, virtualcamera choreography and recording, and net-worked collaboration in shared virtual spaces.5 Its

purpose is to provide simple creation of anima-tions—either by automating a series of virtual-re-ality screen snapshots while driving the virtualcamera along user-designed animation paths, orby exporting the path for use by external render-ing software. Figure 8 illustrates the use of camerapath for multiscale exploration. We have pro-grammed the camera to adjust scale as it ap-proaches the forming protostar.

Figure 6. Screen shot of a working session on the Cluster Archivesite illustrating Java-based 2D visualization. The components of theJava applet allow the user to specify parameters for data retrieval,sampling of the data into lower-dimensional subsets, and 2D colorcontour visualization with multiple color maps and zoomingfeatures. Here, the user sees a plot of the gas density for a slicethrough the core of a typical cluster.

Figure 7. The authors explore AMR data from a cosmological simu-lation of primordial star formation using the NCSA Virtual Director.They are standing in front of an ImmersaDesk—a virtual environ-ment akin to a drafting table invented at the Electronic Visuali-zation Laboratory of the University of Illinois, Chicago.


Users invoke many of the Virtual Director’sfunctions through a combination of 3D gesturesand text-based commands. Using text makes itconvenient to express commands using voice,and indeed users often control the Virtual Di-rector using a commodity PC-based voice recog-nition system—a partial solution to the user-in-

terface conundrum. Likewise, it is easy to sendcommands as network messages, a facility thatproves useful in collaboration sessions.

The power and potential of AMR tosolve multiscale problems in compu-tational astrophysics and cosmologyexcite us. We view AMR as continu-

ing a trend toward simulations of realistic com-plexity begun in the 1980s with the emergence ofsupercomputers and accelerated in the 1990s withthe advent of massive parallelism. However, eventhe most powerful parallel computers cannot ac-commodate uniform grids larger than about1,0243, or a billion cells. With adaptive grids, wehave been able to achieve resolutions in excess of106 locally with far fewer cells. Thus, AMR canbe viewed as an algorithm for reducing the mem-ory and CPU requirements of simulations for aspecified range of length and time scales, or en-abling simulations not otherwise possible.

The physical sciences host an abundance ofmultiscale phenomena that can benefit from theapplication of AMR. Examples include atmos-pheric science, fluid dynamics, combustion, ma-terials science, and so forth. Nonetheless, AMRhas not been widely applied because of its com-plexity and the lack of available tools. The de-velopment and dissemination of AMR librariesand frameworks6 in recent years has begun to re-verse this situation. We hope that our work ondata formats and visualization strategies will beof use to AMR adopters in other disciplines.

AcknowledgmentsWe thank Donna Cox, Bob Patterson, and Matt Hall ofthe Virtual Director team for allowing us to show someof their unpublished work. We also thank GalinaPushkareva and Brad Miksa for permission to use Figure4. This work is partially supported by NASA grantsNAG5-7404, NAGW-3152, and NAG5-3923.

References1. M.J. Berger and P. Collela, “Local Adaptive Mesh Refinement for

Shock Hydrodynamics,” J. Computational Physics, Vol. 82, No. 1,May 1989, pp. 64–84.

2. G.L. Bryan, “Fluids in the Universe: Adaptive Mesh Refinementin Cosmology,” Computing in Science & Engineering, Vol. 1, No. 2,Mar./Apr. 1999, pp. 46–53.

3. V.E. Taylor et al., “A Decomposition Method for Efficient Use ofDistributed Supercomputers for Finite Element Applications,”Proc. Int’l Conf. Application-Specific Systems, Architectures, andProcessors, IEEE Computer Society Press, Los Alamitos, Calif.,1996, pp. 12–24.

4. D. Song and M.L. Norman, “4D2 User Guide,” DocumentTR016, National Center for Supercomputing Applications, Ur-bana, Ill., http://lca.ncsa.uiuc.edu/lca_intro_4d2.html.

(e)(d)

(b)

(a)

(c)

Figure 8. Diving deep using the NCSA Virtual Director. With wandand voice commands, the user interactively defines a camera paththat passes near a forming protostar. A range of scales in excess ofin excess 100,000 is resolved with a 13-level AMR grid hierarchy. (a)overview and (b–e) camera-eye view as it traverses the levels. Wire-frame isosurfaces and colorized points show the distribution of gasdensity in the neonatal cloud.

JULY/AUGUST 1999 47

5. D.J. Cox, “What Can an Artist Do for Science: ‘Cosmic Voyage’ IMAX Film,”Art @ Science, C. Sommerer and L. Mignonneau, eds., Springer-Verlag, NewYork, 1998, pp. 53–59.

6. S. Baden et al., “Workshop on Structured Adaptive Mesh Refinement GridMethods,” IMA Conference Series, Springer-Verlag, to be published 1999.

Michael L. Norman is a professor of astronomy at the University ofIllinois, Urbana-Champaign, and senior research scientist at theNational Center for Supercomputing Applications where he directsthe Laboratory for Computational Astrophysics, which is devotedto the development and dissemination of astrophysical modelingsoftware. His interests are in simulating astrophysical phenomena,cosmological structure formation, and scientific visualization. Hereceived his PhD from the University of California, Davis, and hasbeen a staff member at the Lawrence Livermore and Los AlamosNational Laboratories as well as the Max Planck Institute for Astro-physics. He has recently become a CiSE area editor for applications.Contact him at NCSA, Beckman Institute D-25, 405 N. Mathews St.,Urbana, IL 61801; norman @ncsa.uiuc.edu; http://lca.ncsa.uiuc.edu.

John Shalf works for the NCSA Visualization and Virtual Environ-ments group (www.ncsa.uiuc.edu/SCD/Vis) and the StarTap inter-national high-performance networking project (www.startap. net).His background is in computer engineering, but he tends to hangaround physicists and computer scientists these days. He has con-tributed to a variety of projects in networking, distributed com-puting, and application development frameworks. Contact himat [email protected]; http://bach.ncsa.uiuc.edu.

Stuart Levy develops software for the Virtual Director group atNCSA. His background includes work in mathematical computergraphics, data networking, and systems programming. He enjoysamateur astronomy, as well as peering over the shoulders of pro-fessionals. Contact him at [email protected].

Greg Daues is a postdoctoral research associate in the Laboratoryfor Computational Astrophysics of the NCSA. He has a PhD inphysics and has been working with Java and VRML for two years inthe development of Web-based scientific visualization tools. Con-tact him at daues @ncsa.uiuc.edu.

Get a copy of the next issue on

Software Certification for $10 at

[email protected]

Subscribe now! See http://computer.org for special pricing options or contact our customer service department at [email protected]

EDITORIALCALENDAR

1999JULY

Software Certification ✺Software Security

SEPTEMBEROrganizational Design ✺Architectural Design

NOVEMBERCross Pollinating Disciplines ✺Defining Software Engineeringas a Profession

2000JANUARY

Process DiversityMARCH

Cobol: New MilleniumAPRIL

Engineering in the Small

Keep track of advances in the software develop-ment industry! Access how-to’s, tutorials, and

experience reports direct from the trenches! Findcontroversy and disparate opinions!

Be part of the community with a subscription toIEEE Software. This widely respected magazinecovers crucial advances in software technology,

development, and professional issues.

As a subscriber, you’ll get coverage on:

• Leading-edge programming practices •• Internet development •

• Technical project management •• Component-based development •

• Object-oriented techniques •• Living with rapid technology change •

http://computer.org/software

CommunityOF

BUILD THE

Leading Software Developers!

Documents

Diving deep: data-management and visualization strategies for adaptive mesh refinement simulations