To appear in IEEE Transactions on Visualization and Computer Graphics

Scalable Topological Data Analysis and Visualization for EvaluatingData-Driven Models in Scientific Applications

Shusen Liu, Di Wang, Dan Maljovec, Rushil Anirudh, Jayaraman J. Thiagarajan, Sam Ade JacobsBrian C. Van Essen, David Hysom, Jae-Seung Yeom, Jim Gaffney, Luc Peterson, Peter B. Robinson

Harsh Bhatia, Valerio Pascucci, Brian K. Spears and Peer-Timo Bremer

Abstract— With the rapid adoption of machine learning techniques for large-scale applications in science and engineering comes theconvergence of two grand challenges in visualization. First, the utilization of black box models (e.g., deep neural networks) calls foradvanced techniques in exploring and interpreting model behaviors. Second, the rapid growth in computing has produced enormousdatasets that require techniques that can handle millions or more samples. Although some solutions to these interpretability challengeshave been proposed, they typically do not scale beyond thousands of samples, nor do they provide the high-level intuition scientistsare looking for. Here, we present the first scalable solution to explore and analyze high-dimensional functions often encountered inthe scientific data analysis pipeline. By combining a new streaming neighborhood graph construction, the corresponding topologycomputation, and a novel data aggregation scheme, namely topology aware datacubes, we enable interactive exploration of both thetopological and the geometric aspect of high-dimensional data. Following two use cases from high-energy-density (HED) physics andcomputational biology, we demonstrate how these capabilities have led to crucial new insights in both applications.

1 INTRODUCTION

Driven by growing computing power and ever more sophisticateddiagnostic instruments, many scientific applications have turned frombeing data poor to data rich. As a result, progress in these fieldsincreasingly depends on our ability to understand and learn from datathat are overwhelming in both size and richness. Scientists and analystshave turned to data-driven models, particularly those based on deepneural networks, to handle this increase in volume and complexity.These models exploit the complex nonlinear relationships in such large-scale data to learn effective predictive models. They allow us to dealwith multivariate and multimodal data (i.e., images, time-series, scalars,etc.) in ways inconceivable a decade ago. However, these flexiblemodels create new challenges, especially in scientific applications. Incommercial AI systems, sheer performance, measured by predictivepower or classification accuracy, is paramount. Science and engineeringpractitioners require satisfactory performance, but they also demandinsights and understanding. Yet, the very complexity that increasesmodel performance often obscures our understanding and hence failsto provide new insights. This scenario has produced a growing need fortechniques to interpret the results of data-driven models and to exploretheir training process and final error landscape to assess their reliability.

Despite a recent surge in techniques for model interpretation [28,31, 37, 38], most approaches focus on explaining a given predictionand often only for a specific problem domain, such as natural languageprocessing or computer vision. Although these techniques are useful tounderstand individual predictions, they provide few insights into theoverall behavior of a model or its internal representations.

In this paper, we are concerned with two large-scale applications: ahigh-energy-density physics ensemble exploring inertial confinementfusion and a multiscale simulation of a human cell membrane. In theformer application, some members of our team have built a sophisti-cated surrogate model to explore a large ensemble of simulations withup to 10 million members. The latter is driven by a sampling process ina high-dimensional latent space, and scientists are interested in under-standing and comparing sample distributions. In both cases the goal is

• Shusen Liu, Rushil Anirudh, Jayaraman J. Thiagarajan, Sam Ade Jacobs,Brian C. Van Essen, David Hysom, Jae-Seung Yeom, Jim Gaffney, LucPeterson, Peter B. Robinson, Harsh Bhatia, Brian K. Spears, Peer-TimoBremer are with Lawrence Livermore National Laboratory. E-mail: {liu42,anirudh1, jayaraman1, jacobs32, vanessen1, hysom1, yeom2, gaffney3,peterson76, robinson96, bhatia4, spears9, bremer5}@llnl.gov.

• Di Wang, Dan Maljovec, Valerio Pascucci are with SCI Institute, Universityof Utah. E-mail: {orpheus92, maljovec, pascucci}@sci.utah.edu.

not only to understand the gross simulation results, e.g., the dependenceof fusion yield on design parameters, but also, more importantly, toprovide insight and establish confidence in the model itself. Therefore,we need techniques to evaluate the errors and uncertainties in the modelwith a special emphasis on their variation within the model domain.For example, complex physics models are known to have strong non-linearities that often lead to a rapidly changing response in a very smallregion of the input domain of the simulation (also referred to as theparameter space). This sharp nonlinear behavior can lead to modelsthat exhibit low global errors and good model convergence, while atthe same time producing wildly incorrect results in localized regionsof parameter space that are often most interesting to the scientists.Moreover, scientists typically use these models and their predictions todesign new experiments. They need tools that can discriminate betweenregions of parameter space where the model has errors small enough tomake the model useful for design work. Likewise, they need to knowwhere errors have grown large enough to make the model unreliablefor predicting real-world conditions. Such requirement highlights theneed in the scientific community to augment global loss functions andresponse metrics with methods that can steer the scientific user to-ward regions of high value and confidence. Especially for the complexmodels considered in this paper, with multiple, interdependent lossfunctions and millions of training and evaluation samples, it is criticalto understand the impact of different loss functions and how the errorsmight be correlated to the underlying physics.

None of the existing techniques are particularly suited for the anal-ysis of these scientific models because they either do not scale to therequired sample sizes or do not provide the necessary details. Forexample, due to the curse of dimensionality, it is not uncommon for en-sembles, even with a large number of samples, to have a small numberof salient observations [21] that are crucial for interpretation, yet arenot easily detected automatically. Preserving such observations createsconflicting objectives, as the data size demands aggregating informationinto global trends, yet doing so obscures small-scale details.

As discussed in more detail in Section 3, many of the questionsraised above can be expressed as the analysis of a high-dimensionalfunction. For scientific analysis, this function might be a quantity ofinterest, such as the yield, whereas for model interpretation and vali-dation, it might be a loss function or another indicator of predictionquality. In general, we are given a high-dimensional domain, i.e., a setof input parameters or a latent space, as well as a scalar function onthat domain, to analyze. We propose to use topological techniques toaddress this challenge. In general, computing the topology of a given

1

arX

iv:1

907.

0832

5v1

[cs

.LG

] 1

9 Ju

l 201

9

function provides insights into both its global behavior and it localfeatures, since the measure of importance is variation in its functionvalue, i.e., persistence, rather than the size of a feature. Furthermore,topology provides a convenient abstraction for both analysis and vi-sualization, whose complexity depends entirely on the function itselfrather than the number of samples used to represent it. However, topo-logical information alone can provide only limited insight because, forexample, knowing that there exist two significant modes is of littleuse without knowing where in the domain these modes exist and howmuch volume they account for. We, therefore, propose to augment thetopological information with complementary geometric informationby introducing the notion of topological datacubes. These datacubesare sampled representations linked directly to the topological struc-ture, which provide capabilities for join exploration among scatterplots,parallel coordinates, and topological features. Finally, we introducenew streaming algorithms to compute both the topological informationand the corresponding datacubes for datasets ranging from hundreds ofthousands to eight million data points. More specifically, we have

• identified the shared analysis challenges posed by many data-driven modeling applications;

• developed a robust topological data analysis computation pipelinethat scales to millions of samples;

• developed an interactive visual analytics system that leveragesboth topological and geometric information for model analysis;

• addressed the model analysis challenges of a data-driven inertialconfinement fusion surrogate and solved the adaptive samplingevaluation problem for a large-scale biomedical simulation.

2 RELATED WORKS

Since we focus on addressing specific application challenges, thereare limited works aiming to solve the exact problem. Here we reviewseveral topics that relate to the individual components of the system orprovide relevant background for the overall application.Scientific Machine Learning. Data analysis has always been one ofthe driving forces for scientific discovery. Many statistical tools, i.e.,hypothesis testing [2], have co-developed with many scientific disci-plines. The application of modern machine learning tools in scientificresearch has long attracted scientists’ attention [27]. Machine learninghas been successfully utilized for solving problems in a variety of fields,such as bioinformatics [4], material science [8], and physics [29]. In arecent high-energy-density physics application [29], a random forest re-gressor has been adopted for identifying a previously unknown optionalconfiguration for the inertial confinement fusion experiments (morebackground on inertial confinement fusion is included in Section 6).With recent advances of powerful learners (i.e., deep neural network),coupled with the data analysis challenge, scientists are increasinglyinterested in leveraging these models for scientific discovery.Deep Learning Model Interpretation. The opaque nature of deepneural networks has prompted many efforts to interpret them. In themachine learning community, many approaches have been proposed,notably for common architecture such as the convolution neural network(CNN) [31,37,38], to probe into the mechanism of the model. Similarly,visual analytics systems that focus on interactive explorations of modelinternals have also been introduced in the visualization community,for vision [20], text [22], reinforcement learning [36], and more [26].However, these model-specific techniques are closely tied to particulararchitectures or setups, making them less flexible to a variety of ap-plication scenarios. Alternatively, we can approach the interpretationchallenge from a model agnostics perspective. Recently, a few stud-ies [17,24,30] have focused on building a general interpretation engine.For example, the LIME [30] explains a prediction by fitting a localizedlinear model that approximates the classification boundary around thegiven example. Considering the high-dimensional nature of the internalstate of the neural network models, we believe there is a unique oppor-tunity to build a general purpose neural-network-interpreting tool byexploiting high-dimensional visualization techniques. In this work, wedemonstrate a firm step toward this goal, where we provide valuablediagnostic and validation capabilities for designing a surrogate modelof a high-energy-density physics application.

Topological Data Analysis. Compared to traditional statistical analy-sis tools, topological data analysis employs quite different criteria thatallow it to pick up potentially important outliers that would otherwiseget ignored in standard statistical analysis, making topological dataanalysis tools uniquely suitable for many scientific applications. Thetopology of scalar valued function has been utilized for analyzing sci-entific data in previous works [5, 15, 16]. The scalability challenge forhandling functions defined on the large 3D volumetric domain has alsobeen addressed in several previous works, e.g., parallel merge tree com-putation [18]. However, in many scientific applications, scientists arealso interested in studying the properties defined in a high-dimensionaldomain. To address such a challenge, the HDVis work [13] was in-troduced for computing the topology of a sampled high-dimensionalscalar function. Unfortunately, the applications of the high-dimensionalscalar topology we see so far contain only a relatively small number ofsamples. An apparent mismatch exists between the scalability of theexisting high-dimensional scalar function topology implementation andthe large datasets for our target applications. This work aims to fill thegap by addressing both the computational and visualization challenges.For background on topological data analysis, please refer to Section 4.Data Aggregation Visualization. As the number of samples increasesin a dataset, a visual analytics system not only needs to cope with thecomputational/rendering strain but also to handle the visual encodingchallenges. For example, if we simply plot millions of points in aparallel coordinate or scatterplot, besides the speed consideration (i.e.,drawing each point as an SVG object using d3.js will not be ideal),the occlusion and crowding could potentially eliminate any possibilityof extracting meaningful information from the data. Many previousworks in visualization have been proposed to address these scalabilitychallenges, ranging from designing novel visual encoding (e.g., splatter-plot [25], stacking element plot [11]) to directly modeling and renderingthe data distribution (e.g., Nanocubes [19], imMens [23], density-basedparallel coordinate [3]). However, most existing approaches aggregateinformation globally or according to certain geological indexing forfaster queries. In this work, the proposed system combines visualizationcomponents for both topological and geometric features. Therefore,in order to scale the linked view visualization system beyond millionsof points, we adopt a topology-aware datacube design for aggregatinglarge data according to their topological partition.

3 APPLICATION TASKS ANALYSIS

This section provides the application background and identifies: (i)the analysis tasks shared by many data-driven applications; (ii) whythese tasks are important; and (iii) how we can solve these tasks bycombining topological data analysis with interactive data visualiza-tion. The system has been developed jointly through a continuingcollaboration with two application teams in high-energy physics andcomputational biology, respectively. In both cases we are tasked withsolving the analysis challenges encountered throughout the processingpipeline, including sample acquisition, sampling, modeling, and analy-sis. Despite the disparate application domains, we have encountereda number of recurring analysis tasks, which often lie at the very heartof the scientific interpretation. This has motivated the development ofdedicated visual analysis tool to streamline the analysis process andaccelerate discovery.

Data is at the center of any analysis task. Experimental data istypically considered the gold standard, but it is often too costly or timeconsuming to perform all desired experiments, and/or the phenomenonin question cannot be directly observed. In these cases, computersimulations at various fidelities have become indispensable to plan andinterpret observations. However, such simulations are rarely completelyaccurate and often rely on educated guesses of parameters or knownapproximations of physical processes. To deal with such uncertaintiesand calibrate the simulations, scientists turn to simulation ensembles inwhich the same (computational) experiment is simulated thousands ormillions of times with varying parameters, initial conditions, etc. Thecorresponding computational pipeline typically has three main stages:sampling, simulation, and modeling. We start by generating samplesin the input parameter space. We then run simulations on all input

2

To appear in IEEE Transactions on Visualization and Computer Graphics

parameter combinations and gather the outputs to create the ensemble.Finally, in the modeling stage, the simulation results are used to train acheaper surrogate for one or multiple of the output quantities to enablestatistical inference, uncertainty quantification, or parameter calibration.Both the sampling and the modeling stage can benefit significantly fromvisual analytics. In particular, we have identified four generic tasks wetypically encounter independent of the specific application:

• T1: Analyze the sampling pattern to ensure uniform coverage orverify a sampling objective

• T2: Explore quantities of interest with respect to the input param-eter space

• T3: Verify model convergence, explore residual errors, and eval-uate local and global reliability

• T4: Explore the sensitivities of the model with respect to theinput parameters

How we sample the high-dimensional input space has a significantimpact on the downstream analysis. Depending on the nature of theapplication, we may have different preferences in the sample patternand thus need to verify whether the sampling pattern satisfies the re-quired properties (T1). Given the simulation outputs, we then needto identify where we achieved or failed to achieve our objective, i.e.,induce nuclear fusion, how stable successful solutions are, etc. Typ-ically, such process requires us to explore some quantity of interest,i.e., energy yield of the physical simulation, in the high-dimensionalinput parameter space (T2). Once we obtain the simulation and builda surrogate model, we need to be able to evaluate the model behaviorand interpret the model’s internal representations (T3). The T2, T3focus on identifying regions in the high-dimensional space correspond-ing to certain meaningful measures. As discussed in the introduction,topological data analysis that identifies local peaks of a function canbe an effective tool for discovering and exploring these regions in amultiscale manner. However, due to the high-dimensional nature of thespace, a large number of samples are often required to provide adequatecoverage of the space. As a result, we need to make sure the topologicaldata analysis computation pipeline can reliably scale to large samplesizes (i.e., beyond tens of millions of points). We address the scalabilitychallenges by adopting a streaming computation pipeline (discussed inin Section 4).

Aside from studying the behavior of functions in certain high-dimensional space (T2, T3), we also want to understand the relationshipbetween specific input parameters and model output (T4), for example,judge sensitivities. A global regression will often not yield the desiredresult, because the relationships can be both complex and highly local-ized. An alternative is to leverage the topology-guided partition (T2,T3) and examine the localized and potentially simpler trend. However,the topological data analysis does not really capture any geometricstructure nor can it help reason about individual dimensions. There-fore, to fully utilize the revealed topological information, we need toprovide complementary geometric information. In the proposed sys-tem, we adopted scatterplots and parallel coordinates, which intuitivelyencode data dimensions and support flexible selection operations thatbenefit from the linked view visual exploration. In order to scale thecombined topological and geometrical explorations for large samplesize and support interactive query with respect to topological feature(i.e., different extrema), we devised a topology-aware datacube to en-able the interactively linked exploration between topological featuresand the corresponding samples’ geometric information (see details inSection 5).

4 STREAMING TOPOLOGICAL DATA ANALYSIS

As discussed above, the exploration and analysis introduced inthis work rely on high-dimensional topology and in particular high-dimensional extremum graphs [10, 33]. Topological information pro-vides insight into the overall structure of high-dimensional functionsand is a convenient handle for per-extremum localized analysis. Here,we first introduce the necessary background in Morse theory and reviewthe original algorithm to approximate extremum graphs for sampledhigh-dimensional functions. We then discuss a new streaming algo-rithm to compute extremum graphs and the corresponding streaming

neighborhood graph construction approach, which mainly aim to over-come the memory bottleneck that hinders the scalability of the existingimplementation.

4.1 Morse Complex and Extremum GraphsLet M be a smooth, d-dimensional manifold and f : M→ R be a

smooth scalar function with gradient ∇ f . A critical point is any pointx ∈M where ∇ f (x) = 0, and all other points are regular. Startingfrom a regular point p ∈M and following the gradient forward andbackward traces an integral line that begins at a local minimum andends at a local maximum. The set of points whose integral line ends at acritical point defines the stable/unstable manifold for that point, and theMorse complex is defined as the collection of all stable manifolds. Fordimensions beyond three, computing the entire Morse complex is in-feasible [14], so we follow [7,10] and concentrate on extremum graphs.An extremum graph contains all local maxima of the function togetherwith the saddles connecting them. To approximate the extremum graphfor sampled functions, we define an undirected neighborhood graph onall input vertices to approximate M. Given a graph, we approximate thegradient as the steepest ascending edge incident to a given vertex andconstruct the ascending integral line by successively following steepestedges to a local maximum. In practice, this traversal is implementedas a short-cut union-find at near linear complexity per vertex. In thisprocess, each vertex is labeled by the index of its corresponding stablemanifold, and saddles are identified as the highest vertex connectingtwo neighboring maxima [14]. Subsequently, we form the extremumgraph by considering arcs between saddles and neighboring maxima.

Fig. 1. Left: the performance comparison between the CPU baselinestreaming implementation and the CUDA accelerated version (5D samplewith varying sizes). Right: a plot showing the total edges extracted froma symmetric, relaxed Gabriel graph computed on the same dataset as afunction of the k used in the initial k-nearest neighbor graph.

4.2 Streaming Extremum GraphOne major road block for scaling the existing algorithm to large

point count is the size of the neighborhood graph. As the dimension ofM increases, more vertices are required to reliably approximate f . Toidentify the correct edges for topological computation, we often need tobuild neighborhood graphs that require several magnitude more storagespace than the vertex alone, which may quickly reach the memorylimitation of most desktop systems.

To address this challenge, we present a two-pass streaming algo-rithm for constructing extremum graphs that store only the verticesand an appropriate neighborhood look-up structure to avoid keepingthe massive neighborhood graph in memory. We can then obtain thenecessary neighboring information on the fly to construct edges foreach vertex. A vertex always maintains its currently steepest neighbor,which is initialized to be the vertex itself. Once all edges have beenseen, a near-linear union-find is used as a short-cut to the steepest neigh-bor link to point to the corresponding local maximum of each vertex,thereby constructing the sampled Morse complex. Even though thisstructure can be constructed in a single pass, note that the identificationof saddles and the subsequent cancellation of saddles and maxima alsorequire the neighborhood information. Consequently, we reiterate thesteaming graph in a second pass to assemble the complex.

To support streaming extremum graph computation, we introduce anew approach to compute neighborhood graphs in a streaming manner.For densely sampled space (i.e., sampling of simulation input param-eters), Correa et al. [9] have demonstrated that β -skeletons and theirrelaxed variants provide significantly more stable results for computingtopological structure [9], in which an approximated k-nearest neighbor

3

graph of sufficiently large k is computed first and then pruned usingthe empty region test defined by β . In this work, we build upon theirwork [9] and employ a streaming scheme to avoid storing the full graphin memory by doing neighborhood lookup and edge pruning for eachpoint individually. We then store the steepest direction for topologycomputation. In our implementation, we utilize GPU to exploit theparallelism in the neighborhood query and the edge-pruning steps. Acomparison between the baseline CPU implementation and the GPUaccelerated version is shown in the left plot of Fig. 1 on a test functionwith a 5D domain and varying sample sizes (note the log scale in they-axis, for the 1M case, the GPU version is approximately 70% fasterthan the CPU counterpart).

To determine a suitable k, as illustrated in the right plot of Fig. 1,we gradually increase k in the initial k-nearest neighborhood querystage and observe at which point the number of true edges of the emptyregion graph begins to stabilize. We can see, in 5D space, that thecurve starts to level off beyond k ∼ 500, indicating that adding morek neighbors will not result in many more edges being discovered inthe pruned graph. Note that the saturation point will vary based on thedistribution of the data in the domain; the results are shown for datadrawn from a uniform random distribution.

5 DATA AGGREGATION AND VISUALIZATION

We have described how to extend the scalability of the topologicaldata analysis to handle large datasets. Despite its strength, topolog-ical data analysis alone reveals little geometric information outsidethe location of critical points in parameter space. However, this infor-mation can be crucial to interpret data, and thus we propose a jointanalysis of topological features and their corresponding geometric dataas expressed through parallel coordinates and scatterplots. Here, weintroduce topology-aware datacubes, which not only aggregate data toenable interactive visual exploration but also maintain the topologicalfeature hierarchy that allows interactive linked view exploration. Asillustrated in Fig. 2, the overall visualization interface (built on top ofan existing high-dimensional data exploration framework ND2AV [6])consists of three views of the dataset: topological spines (a), scatterplots(b), and parallel coordinates plots (c). The topological spine showsthe relative distance and size of the peaks in the function of interests.During an exploration session, the user can first assess the global re-lationship presented in the topological spine and parallel coordinate,and then focus on individual peaks of the function in the topologicalspine by selecting the local extrema, which will trigger updates in theparallel coordinates plot and scatterplot that reveal the samples corre-sponding to that extrema. By examining the parallel coordinates plot /scatterplot patterns that correspond to different selected extrema, weare able to discern the locations of the peaks in the function that makethem different.

5.1 Topology-Aware Data CubesWhen dealing with large datasets, visualization systems often need

to address the scalability challenge from two aspects. On one hand, asthe number of sample increases, the standard visual encodings, such asscatterplots and parallel coordinates, become increasingly ineffectivedue to overplotting and thus overwhelm users’ perceptual capacity. Onthe other hand, the increasing data size induces a heavy computationalcost for processing and rendering, which may create latency that hin-ders the interactivity of the tool. One strategy to address this problem isdatacubes (OLAP-cubes) style aggregation techniques [19, 23], whichprovide summary information (i.e., density/histogram) to overcome theoverplotting while preserving joint information to enable linked selec-tion and interactive exploration. However, such aggregation techniquesare not directly applicable here as they summarize the dataset’s entiredomain without considering the topological segmentation. Instead, weaim to aggregate information with respect to the topological structure ofthe data because the ability to interactively explore the topological hier-archy at different granularities is central to our analysis task. Therefore,to achieve the design goal, the data aggregation structure should allowefficient query on different topological partitions of the data, whichmotivates us to introduce a topology-aware datacube design.

The extremum graph (Section 4) can be simplified by merging ex-

trema (each corresponding to a segment of the data) and removing lesssignificant saddles based on the persistence value (i.e., the functionvalue difference between the saddle and extremum pair: the higher thevalue, the more significant the topological feature). Since we often donot care about extrema with low persistence (i.e., corresponding to aless topologically significant structure), we can presimplify the topo-logical segmentation hierarchy and focus only on the top levels (i.e., upto a dozen segments) for our analysis. During the exploration, the usercan explore different levels of granularity by altering the persistencethreshold. In Fig. 2(a1), the number of peaks is shown (y-axis) for agiven persistence value (x-axis). The persistence values at the plateauregions (longer horizontal line segments in the diagram) signify morestable topological structures. A suitable persistence value is the onethat can produce significant and stable topological structures.

For the topology-aware datacubes, we precomputed datacubes forsamples in each leaf segment in the simplified topology hierarchy,where each datacube preserves the localized geometric features. Thesegments with lower persistence will be merged into a topologicallymore significant partition of the data. Since we already precomputed thedatacubes for each leaf segment, we can generate summary informationfor any higher level partition by aggregating the summary data in leafsegments on the fly. Despite the effectiveness of the datacube for rangequeries, storing a full joint distribution in the datacube form can beextremely expensive. For our intended exploration and interaction(displaying parallel coordinates plots and scatterplots), we do not needaccess to all the joint distribution information. Therefore, we computeand store lower dimensional data (up to three dimensions) cubes toreduce the storage while still supporting the interactive visualization.

(a) (b)

(c)

(a1)

Function Range

Extrem

a Cou

nt

Fig. 2. The proposed visualization interface consists of three views:topological spine (a), density scatterplot (b), density parallel coordinatesplot (c). These views provide complementary information, and the linkedselection enables a joint analysis of both geometric and topologicalfeatures. In (a1), we show the persistence plot, which controls thenumber of extrema displayed in the topological spine.

2D Terrain MetaphorHD Function

Saddle

Extrema

Fig. 3. Topological spine. A 2D terrain metaphor of high-dimensionaltopological information [10].

Topological Spine. The topological spine [10] view (see Fig. 2(a))visually encodes the extremum graph of a scalar field by showing theconnectivity of the extrema and saddles together with the size of localpeaks. As illustrated in Fig. 3, the spine utilizes a terrain metaphorto illustrate the peaks (local extrema) and their relationships in the

4

To appear in IEEE Transactions on Visualization and Computer Graphics

given function. Here, the contours around the extrema or across saddlesindicate the level sets of the function, similar to the contour line in atopographic map. In [10], the size of the contour C fi is the number ofsamples above the contour function value fi. In our implementation,we computed the size as d/2

√C fi to better reflect the relative “volume”

these points cover in the high-dimensional space (d is the dimensionof the function domain). The layout of the critical points (the extremaand saddles) is a 2D approximation of the location of critical pointsin the high-dimensional domain. Since the function may have manysmall local structures, we simplify the extremum graph to focus onimportant topological features. We consider a given local extremum lesssignificant when the function value difference between the extremumand the nearby saddle (i.e., the persistence of the local extremum) issmall. The extremum graph can be simplified by merging extrema withsmall persistence values. As illustrated in Fig. 2(a1), the number ofextrema in the topological structure is controlled by the persistence plot,where the x-axis is the normalized function range, and the y-axis showsthe number of local extrema at the current simplification level. Wecan identify the stable topological configuration by finding the widestplateau in the blue line along the x-axis. Since the complexity of thespine does not depend on the number of samples used to define thefunction, the topological spine is a perfectly scalable visual metaphorthat can easily be obtained from the precomputed datacubes instead ofthe raw samples.Density Scatterplots. By querying the datacube, we can directly ob-tain the 2D histogram and render the estimated joint distribution densityto avoid the overplotting issue in the standard scatterplot. Due to thetopology-aware datacube, the user can explore the density scatterplot(see Fig. 2(b)) for any topological segments by aggregating leaf dat-acubes on the fly. To better visualize the potentially large dynamicrange of the density value, we applied a gamma correction on the den-sity value: α = Adγ , where α denotes the opacity and d denotes thedensity of each bin.Density Parallel Coordinates. Similarly, the parallel coordinates plot(see Fig. 2(c)) can be drawn with selected 2D joint distributions fromthe datacube. To draw the density parallel coordinates plots, we firstdiscretize each axis according to the resolution (r) of the datacube; thus,there would be r bins on each axis. We then draw lines from each binto every other bin on the adjacent axis; thus, there would be r2 linesbetween every two adjacent axes. To draw each line, we query thecorresponding density from the 2D joint distribution of the neighboringaxes and map the value to the opacity (with gamma correction). Sinceevery line between two adjacent axes requires only the informationof corresponding dimensions, we need only the bivariate distributionof those two dimensions (or 3D if we want to support function rangeselection in PCP). Due to the discretization, the time to draw the parallelcoordinates plot depends only on the resolution and the total number ofdimensions.

Lasers

Capsule

Fuel

Fig. 4. Inertial confinement fusion (ICF). Lasers heat and compress thetarget capsule containing fuel to initiate controlled fusion.

6 APPLICATION: INERTIAL CONFINEMENT FUSION

In this section, we will discuss the application of the proposed analy-sis framework for analyzing simulations of inertial confinement fusion(ICF). Controlled fusion has often been considered to be a highly effec-tive energy source. At the National Ignition Facility (NIF) (at LawrenceLivermore National Laboratory), one of the key objectives is to providean experimental environment for controlled fusion research and thuslaying the groundwork for using fusion as a clean and safe energysource. Scientists utilize high-energy lasers to heat and compress a

millimeter-scale target filled with frozen thermonuclear fusion fuel (seeFig. 4). Under ideal conditions, the fusion of the fuel will produceenough neutrons and alpha particles (helium nuclei) for the target toself-heat, eventually leading to a runaway implosion process referred toas ignition. However, achieving an optimal condition by adjusting thetarget and laser parameters in response to the experimental output (e.g.,energy yield) is extremely challenging due to a variety of factors, suchas the prohibitive cost of the actual physical experiment and incompletediagnostic information. Consequently, ICF scientists often turn to nu-merical simulations to help them postulate the physical conditions thatproduced a given set of experimental observations.

In this application, we focus on a large simulation ensemble (10Msamples) produced by a recently proposed semianalytic simulationmodel [12, 32]. We capitalize on the abundance of simulations bybuilding a complex multivariate and multimodal surrogate based ondeep learning architectures. More specifically, the surrogate replicatesoutputs from the numerical physics model, including an array of sim-ulated X-ray images obtained from multiple lines of sight, as well asdiagnostic scalar output quantities (see Fig. 5). As discussed in moredetail below, this complex deep learning model is the first of its kind, atleast in this area of science. The model and the corresponding trainingdataset have been made publicly available [1].

ParameterSpace

LatentSpace

Fig. 5. System diagram of the deep learning-based surrogate modelingused in the ICF study. A 5D input parameter space X is mapped ontoa 20D latent space L, which is fed to a multimodal decoder D returningthe multiview X-ray images of the implosion as well as diagnostic scalarquantities.

6.1 Deep Learning for Inertial Confinement FusionAlthough describing all the architectural details of the model is

beyond the scope of the paper, Fig. 5 illustrates the key components.In this study, the input to the simulation code is a 5D parameter spaceX⊂R. For each combination of input parameters xi ∈X, the simulationcode produces 12 mulitchannel images I j = {I0

j , ..., I11j }, each of size

64×64×4 in the image space I, as well as 15 scalar quantities y j =

{y0j , ...,y

14j } ∈ S= R15. To jointly predict the images and scalars, the

model uses a bow-tie autoencoder [34] (I×S→ L→ I×S, not shownin the figure) to construct a joint latent space L⊂ R20 that captures allimage and scalar variations. The forward model X→ I×S is comprisedof two components: a multivariate regressor F : X→L and the decoderfrom the pretrained autoencoder: D : L→ I×S. In other words, theforward model does not directly predict the system output I×S, butinstead predicts the joint latent space L of the autoencoder. The decoderD : L→ I×S can then produce the actual outputs from the predicted L.Such a setup allows us to more effectively utilize relationships in theoutput domain I×S and improve the overall predictive performance.Furthermore, the model simultaneously considers the inverse modelG : L→ X and uses self-consistency to regularize the mapping, byminimizing ||x−G(F(x))||2 in addition to the prediction loss. Allsubmodels are implemented using deep neural networks (DNNs), andthe entire system is trained jointly to obtain the fitted models F andG, assuming that we have access to a pretrained autoencoder. In orderto enable neural network training at such large scales, and to supportthe system integration needs, we adopted the parallel neural networktraining toolkit, LBANN [35], for training the surrogate.

6.2 Surrogate Model AnalysisAlthough the ability of the model to produce accurate and self-

consistent predictions of multimodal outputs provides fundamentallynew capabilities for scientific applications, the complexity of the result-ing model naturally leads to challenges in its evaluation and exploration.Firstly, the application requirement for scientific deep learning is vastly

5

(a) (b1) (c1)(b2) (c2)

shapemode(4,3)

betti_u betti_v shapemode(2,1)

shapemode(1,0)

Fig. 6. Joint exploration of both topological and geometric characteristics of the surrogate’s errors as functions in the input parameter space. In (a1),the topological spine shows two local peaks of the output error (Fo(x)). The user can select the high error samples through parallel coordinates plotin (b2). These samples will also be highlighted in topological spine (b1). Also, the user can focus on an individual extremum (see c1, c2).

different from that of its commercial counterparts. Compared to tra-ditional applications such as image recognition and text processing(where a human user can easily provide the ground truth), in scientificproblems, even an expert has limited knowledge about the behavior ofan experiment, due to the inherent complexity of the physical system aswell as the exploratory nature of the domain. Secondly, these modelsare intended for precise and quantitative experimental designs that, wehope, can lead to scientific discovery. Consequently, physicists careabout not only the overall prediction accuracy, but also the localized be-haviors of a model, i.e., whether certain regions of the input parameterspace produce more error than others.

To address these challenges, we include the proposed visualizationtool as an integral part of the model design/training process. As dis-cussed in Section 3, the ability to interpret a high-dimensional functionis central for obtaining the localized understanding of a given system.In this application, the functions of interest are the high-dimensionalerror landscapes of the surrogate model. By adopting the proposedfunction visualization tool, we aim to provide the model designers andthe physicists with intuitive feedback on the prediction error distri-bution in the input parameter space. For over five months, we haveworked closely with the machine learning team to iteratively debugand fine-tune the training process. We have detected a crucial issuewith the normalization strategy applied to the x-ray images and iden-tified the problem with the batch scheduler based on the visualizationprovided by the proposed tool. The following analysis is carried outon a “well-behaved” model, where the aforementioned problems havealready been addressed, according to the traditional evaluation metrics(e.g., average prediction accuracy, loss convergence behavior). Here,we illustrate how the exploration of localized error behaviors can revealsome important yet unexpected issues of the surrogate model that arenot possible with the traditional summary metrics.Exploring surrogate error in the input parameter space. Tradition-ally, the machine learning community has relied on global summarystatistics to assess a model, i.e., global loss curves, that do not re-veal localized properties. By utilizing the proposed tool, we view thepredicative error as a function in the input domain, which can then beanalyzed as a high-dimensional scalar function (see T2-4 in Section 3).Such a line of inquiry is not only essential for scientific analysis but alsocan provide a critical feedback loop for validating and fine-tuning theactual machine learning model. In order to perform an elaborate as wellas unbiased study, we carry out our analysis on an 8 million validationset, hold-out during the training process. Despite the large sample size,due to the scalable design considerations, we are able to show that theproposed system allows an interactive linked-view exploration once thetopological structure and the corresponding datacube are obtained. Theprecomputation of the 8 million sample dataset took around 1.5-3.5hours to complete, depending on the initial neighborhood query size aswell as the hardware setup (i.e., utilizing GPU or not).

During the model training process, a global average loss functionis considered, which in this case consists of a weighted sum of thelosses in L (forward model), S (decoder), and X (inverse model). Eachterm addresses a different aspect of the overall training objectives.More specifically, we are given: (1) the autoencoder reconstructionerror, R(l) = ||y−D(l)||, for l ∈ L and l = E(y) denoting the encoderof the autoencoder; (2) the forward error in latent space Flat(x) =

||F(x)−E(y)||, where F(x) is the fitted forward model; (3) the forwarderror in output space Fo(x) = ||y−D(F(x))||; and (4) the cyclic self-consistency error G (x) = ||x− G(F(x))||.

To gauge the overall error of the surrogate, let us first look at the for-ward error in output space Fo(x) (i.e., the error in the predicted imagesand diagnostic scalars of the surrogate). We compute the topologicalstructure of the error in the input parameter space. As shown in Fig. 6(a), from the topological spine, we can see that there are two distinctlocal extrema of high errors in the 5D parameter space. We highlightthe relationships between those samples through an aggregated parallelcoordinate plot shown in (b2). We can also equivalently show wherethese samples are in the topological spine as shown in (b1). Note, adifferent shade of blue is used to indicate the fraction of the selectedsamples associated with each contour of the “topological terrain”. Adarker shade indicates a larger fraction. To understand the relationshipbetween the patterns in the PCP and the peaks in the topological spine,we can focus on one of the peaks in the topology spine (c1), whichwill also trigger an update of the PCP. As shown in (c2), the left peakappears to correspond to samples with low values in the first dimen-sion (shape mode (4, 3)), indicating that the two peaks are maximallyseparated in that direction.Interaction between different types of errors. The error in the outputspace Fo(x) is affected by both forward error in the latent space Flat(x)and the autoencoder error in R(l). Hence, we subsequently examineall three error components side by side. Let us first look at the error inthe autoencoder, which is trained separately from the forward/inversemodels. As shown in Fig. 7(a1), interestingly, the spine also containstwo peaks, and they are maximally separated by the first dimension(Fig. 7(a2)), which is similar to the topology of forward error in theoutput space (Fig. 6(b1, b2)). However, the error in latent space Flat(x),as illustrated in Fig. 7(b1), has a much more complex, yet stable,topological structure. When we focus on an individual topologicalsegment (in Fig. 7(b1, b2), we use orange and green to highlight two ofthe extrema (b1) and their corresponding lines in the parallel coordinate(b2)). We notice the extrema corresponds to outlying patterns in theparallel coordinate plot that will be often ignored by typical statisticalanalysis techniques. By viewing all three error patterns together, wefind that (1) despite similar overall similarity between the PCP plots, themajority of the local extrema of Flat(x) (such as the one highlightedin (b1,b2)) do not reappear in the Fo(x) (see Fig. 6); (2) the R(l) andFo(x) have a similar diverging pattern in the first dimension (see PCPplots), which is not found in Flat(x). Such an observation seems toindicate that the autoencoder error has a very strong influence on theoutput error of the surrogate. As a result, it is important to furtherexplore the potential cause of the high error in autoencoder. In addition,the forward error in the latent space Flat(x) has some very out-of-ordinary local extrema (b2) - it would also be interesting to understandthe cause (is the error due to faulty outputs in the simulation, or arethe image features more challenging to predict, e.g., high-frequencycontent) and potentially fix the errors.What contributes to the high error? Upon examination of the be-havior of the prediction error in the input domain, we are curious tounderstand the possible causes for the observed patterns (T3). Interest-ingly, from the PCP plot of the simulated energy yield (see Fig. 7(c2)),we find patterns (especially in the last three dimensions) similar to that

6

To appear in IEEE Transactions on Visualization and Computer Graphics

(a1) (a2)

(c1) (c2) (d1) (d2)

(b1) (b2)

Fig. 7. The autoencoder error (R(l)) and the latent space error (Flat(x)) is shown in (a),(b) respectively. The yield of the simulation is shown in (c). In(d), we illustrate the latent space error (Flat(x)) of the model that are trained for 80 epochs instead of 40.

of the previously explored error components. This interesting discoverycould have a significant impact on the application since the physicistsare interested in the transition regions between low and higher yield.Despite the similarity between the high yield and high error in theautoencoder (see Fig. 7(a2, c2), there is a clear discrepancy in thefirst dimension (i.e., shape model (4, 3)), where the autoencoder errorexhibits a clear binary pattern (higher error corresponds to high or lowvalues). Apparently, then, another factor besides the yield is correlatedwith the autoencoder error.

High-Yield High AE_Error

High-Yield Low AE_Error

Low-Yield Low AE_Error

High Forward_Error

(a) (b) (c) (d)

Fig. 8. X-ray images with different energy and error conditions.

We need to examine the difference between x-ray images correspond-ing to high/low yield and high/low error conditions, as the autoencodererror directly indicates how well the model can reconstruct the imagesand scalars from the latent representations. As shown in Fig. 8, we iden-tify two images (a, b) with high and low autoencoder error, respectively.Although both correspond to high yield conditions, image (a) has amuch more complex pattern, which is more challenging to reconstruct,thus leading to a larger autoencoder error when compared to the image(b). According to the physicists, the first parameter (shape model (4,3)) encodes the higher order shape information. As a result, the sim-ulator is expected to generate image patterns similar to (b) when theparameter value is closer to 0. Larger deviation from 0 induces morecomplex patterns, as we observe in the image (a).How many samples do we need to train the surrogate? As notedpreviously in Fig. 7 (b2), a number of local extrema in different partsof the domain are characterized by a high forward model error Flat(x).To analyze this behavior, we identify the corresponding images (seeFig. 8(d)), which exhibit high-frequency patterns around the centralcircular regions, making it harder to predict. The natural question is,does this mean we need more training data to better learn these patternsor did we not train the model until convergence?

To answer these questions, we carried out experiments by varyingthe training size and training time. By increasing the training time from40 to 80 epochs, we observe that the out-of-ordinary local extrema inFig. 7(b2) disappear (see Fig. 7(d2)), which indicates that the modelhad not converged sufficiently. This result was quite surprising sincethe average loss curve appeared to have reached a steady state, longbefore the 40 epochs, and we still chose to continue training. Based onfeedback from the physicists, we realized that the “cloud” around the

high-density center may not reflect the true physics, and could be anartifact of the simulator or the image rendering process. This discoveryis crucial because it is not otherwise feasible to examine these largedatasets for such anomalous behaviors.

10

0K

Tra

ined

1M

Tra

in 4

0-e

poch

1M

Tra

in 8

0-e

poch

parabola shapeparabola shape(a1)

(a)

(a2)

(b1) (b2)

(c1) (c2)

betti_v

shape_mode(2,1) shape_mode(4,3)

shape_mode(0,1)

(d1) (d2)

(d3)

(d4)

Fig. 9. The comparison of cyclic loss (a measure of surrogate self-consistency) among models with different training sets and training times.As shown in (a), the 100K samples trained model exhibits a peculiarparabola pattern, which indicates the lack of self-consistency for a certainparameter range. By increasing the training set (from 100K to 1 Million),we are able to eliminate these empty regions (see (b)), but the modelends up with larger errors, especially among outliers, which can bemitigated by increasing the training time (from 40 to 80 epochs, see (c)).In (d), we show the topological structure of the self-consistency loss of thebest model and reveal that the highly inconsistent region is characterizedby high values of physical parameter betti v of the surrogate input.

Similar to the training time, we found that changes to training sizealso drastically affect error behaviors. In Fig. 9, we show a comparisonof the cyclical errors in three different training setups (in the plot, they-axis is the error, and the x-axis is marked for each column at the topof the figure). The cyclic error G (x) provides a way to enforce/evaluatemodel self-consistency, which is crucial for building physically mean-ingful models. For the model trained using 100k samples (Fig. 9(a)),we can see the error exhibits a rather peculiar parabolic shape alongcertain dimensions (see (a1, a2)). The empty region in the error plotsreveals that for certain parameter combinations, the model will always

7

be inconsistent to a certain extent, which is not desirable. However, byincreasing the training set to 1 million samples (even without increas-ing the training epochs), as shown in Fig. 9(b), we no longer see thoseempty regions in the parameter space. However, with more data, theoverall errors do not improve; in fact, the outliers become more severe(all plots on the left panel have the same y-axis scale). As expected,we could reduce the discrepancies by increasing the training duration(Fig. 9(c)). The deep learning experts postulate that certain modesof the images (i.e., more complex ones) likely require larger numberexamples to learn, which the 100k training samples cannot provide.Finally, in Fig. 9(d), we show the topological structure of the cyclicerror of the (1M training samples, 80-epoch) model, where one of thepeaks corresponds to a high-error region characterized by large valuesfor one of the physics parameters betti v. As noted by our collaborators,the instability of the surrogate is most likely due to the volatile nature ofphysics around the point of implosion, where the diagnostic quantitiescan change drastically.

The above analysis for the ICF surrogate model clearly demonstratesthat scientific applications require a new suite of evaluation strategies,based on both statistical characterization and exploratory analysis. Theinsights provided by our approach cannot be obtained otherwise byusing aggregated statistics of errors. For analyzing similar surrogatemodels, as a general guideline, we believe it is crucial to first explorethe relationship between the input parameter space and the correspond-ing localized errors. We can then dive into the contributors of theoverall error and examine their causes to infer unintended behaviorsof the model. Lastly, to fully evaluate the model, we also need tounderstand how the error distribution in the input space impacts thetarget application. For example, a high concentration of localized errormay not always indicate an undesirable model, as the application mayrequire precision only where the model is highly accurate.

7 APPLICATION: MULTISCALE SIMULATION FOR CANCER RE-SEARCH

Although surrogate modeling is a natural use case for our approach,high-dimensional functions appear in many other applications, some-times in unexpected contexts. Here, we discuss another ongoing col-laboration with computational biologists, who are interested in un-derstanding a particular type of cancer-signaling chain through bothexperimental and computational means. Specifically, they focus onanalyzing how the RAS protein interacts with the human cell mem-brane to start a cascade of signals, leading to cell growth. Most types ofaggressive-cancers, such as pancreatic, are known to be linked to RASmutations that cause unregulated, i.e., cancerous, growth of the affectedcells. However, so far, the attempts to affect RAS function throughdrugs have led to complete disruption of signals and have ultimatelyproven fatal to the patient.

The cell membrane is made up of two layers of lipids that constantlymove, driven by complex dynamics dependent on lipid type, surround-ing proteins, local curvatures, etc. Scientists suspect that the locallipid composition underneath the RAS protein plays a major role inthe signaling cascade. Unfortunately, the relevant length scales arenot accessible experimentally. Therefore, molecular dynamics (MD)simulations are the only source of information. Using the latest gen-eration of computer models and resources, it is now possible to studythe interaction of the RAS protein with the cell membrane for both theatomistic and the so-called coarse-grained MD simulations. However,such simulations are costly and typically restricted to micro-secondintervals at nano-meter scales. The challenge is that at these time andlength scales, seeing changes in RAS configuration or unusual lipidmixtures is extremely rare, and thus the chances of simulating even oneevent of interest using a pseudo-random setup are very low. Instead,scientists are interested in simulating micro-meter-sized lipid bilayersfor up to seconds of time, which would not only make events of interestmore likely, but also reach experimentally observable scales, which iscrucial to calibrate and verify computational results. To address theseseemingly contradictory requirements, our collaborators are developinga multiresolution simulation framework (Fig. 10). At the coarse level,a continuum simulation is used that abstracts individual lipids into

spatial concentrations and models RAS proteins as individual parti-cles. Such simulations are capable of reaching biologically relevantlength and timescales and are expected to provide insights into locallipid concentrations as well as clustering of RAS proteins on the cellmembrane. However, such models cannot deliver an analysis of themolecular interactions. For such detailed insights, MD simulations areperformed using much smaller subsets of the bilayer in regions thatare considered “interesting.” The key challenge is to determine whichset of fine-scale simulations must be executed to provide the greatestchances for new insights.

7.1 Analysis of Importance SamplingAs shown in Fig. 10, the current approach uses an autoencoder-

style setup similar to the one discussed in the previous section. Eachlocal patch underneath a RAS protein in the continuum simulationis projected into a 15D latent space formed by the bottleneck layerof an autoencoder. In the context of this paper, the latent space is anonlinear dimensionality reduction of the space of all patches onto15 dimensions. Given limited computational resources, the goal isto determine which of these patches must be examined more closely,using MD simulations. To this end, our collaborators are using afarthest-point sampling approach in the latent space. As resourcesbecome available, they choose the patches (yellow) that are farthestaway in the latent space from all previously selected patches (green).The intuition is that this strategy, in the limit, will evenly samplethe space of all patches rather than repeatedly executing commonconfigurations. In this manner, it will provide maximal informationfor the available computing resources and drive the process towardrare configurations. In this context, “rare” refers to the simulationlength and timescales, which are still very small, i.e., seconds andmicrometers scale. The practical challenge is to determine how well thissampling strategy is working and what its practical effects are (see T1 inSection 3). Initial attempts used nonlinear embeddings, e.g., t-SNE, tocompare a hypothetical random sample to the optimized farthest-pointsampling. Due to the inherent nonlinear distortion, however, the resultswere inconclusive and difficult to interpret. Other straightforwardapproaches, e.g., comparing distribution of neighborhood distancesbetween random and optimized samples, also provided little insight.

Varia�onalAutoencoder

LatentSpace

Backmapping

�me �me

Con�nuumModel MolecularDynamicsSimula�ons

Patches

Fig. 10. Multiscale simulation of RAS-membrane biology for cancerresearch. As the continuum model (left) evolves, patches around theRAS proteins (shaded balls) are projected into a high-dimensional latentspace, which is used to drive an importance sampling. Patches that aredifferent from previous simulations (yellow) are used to initialize new MDsimulations, whereas patches too close to existing samples (red) arediscarded. Significant challenges exist in the analysis of high-dimensionallatent spaces to improve the sampling.

We use our tool to explore these sampling patterns. In particular, ourcollaborators estimate the point density in the latent space as the inverseof the mean distance to 20 nearest neighbors. The density is computedfor three sets of points: 1) all available patches (2 million points); 2) auniform random subselection of all patches (100K points); and 3) anoptimized subselection using farthest-point sampling (100K points).Given this density estimate, we compute the extrema graph for all threesets. As shown on the top row of Fig. 11, the set of all patches results inmultiple high-persistence plateaus (a2), which indicates well-separateddensity maxima, the smallest of which has a persistence of ∼ 50%.Note that these modes are not apparent in the parallel coordinates (a1)nor in the scatterplot of the latent dimension ((a3) shows the first two

8

To appear in IEEE Transactions on Visualization and Computer Graphics

Rand

om

Su

bse

tFu

ll D

ata

Ad

ap

tive S

ub

set

(a1) (a2)

(a3)

(b1) (b2)

(b3)

(c1) (c2)

(c3)

Fig. 11. Visualization of different sampling patterns (middle and bottom) when compared to the overall distribution (top) highlights that the adaptiveimportance-based approach has a wider coverage and lower and fewer modes, as required to execute the targeted multiscale simulation.

latent dimensions). The mid-row plots show the random selection,which creates fewer modes (b2) with lower persistence ∼ 30%, whichare also significantly less stable. The figure shows four major modes forillustration, but a more pragmatic interpretation is that there exists onlya single mode of density, and the remaining maxima are due to noise,which is surprising since the random sample is expected to mimic thefull distribution. In fact, our collaborators typically rely on analyzingthe random subset assuming their equivalence. One current hypothesisis that the modes are simply too small to be reliably sampled with100K points. Another possibility is that the high-dimensional densityestimation introduces artifacts. The bottom-row plots show the result ofthe adaptive importance-based sampling with even smaller modes andlower overall density peaks. Furthermore, the parallel coordinates showa substantially wider distribution, even though the highest density peak(∼ 0.82) remains, which is surprisingly similar to the one of the randomsample (∼ 0.95). Again, the similarity is likely a consequence of thedensity estimation (which applies a nonlinear scaling factor). Never-theless, our analysis and visual representations have provided directand intuitive evidence to our collaborators regarding the effectivenessof their sampling strategy, which was not apparent in their previousanalyses (e.g., tSNE plots). The team has executed this sampling aspart of a large multiscale simulation using a parallel workflow for sev-eral days utilizing all 176,000 CPUs and 16,000 GPUs of Sierra, thesecond-fastest supercomputer in the world, aggregating over 116,000MD simulations.

8 DISCUSSION AND FUTURE WORK

In this work, we have identified a set of common tasks often en-countered in analyzing data derived from computational pipelines forscientific discovery and introduced a scalable visual analytic tool toaddress these challenges via a joint exploration of both topologicaland geometric features. To achieve the analysis objectives and facil-itate large-scale applications, we employed a streaming constructionof extremum graphs and introduced the concept of topological-awaredatacube to aggregate large datasets according to their topologicalstructure for interactive query and analysis. We highlight the fact thatscientific deep learning requires a different set of evaluation and diag-

nostic schemes due to the drastically different objectives. In scientificapplications, it is often not sufficient to obtain the best average per-formance or identify the typical simulation results. Instead, it is moreimportant to provide insight and establish confidence in the model itselfand understand where or why the model may be unreliable for real-world scenarios. As demonstrated in the application, the proposed toolhelps us not only evaluate and finetune the surrogate model but alsoidentify a potential issue in the physical simulation code that wouldotherwise be omitted. We believe the application-aware error landscapeanalysis demonstrated in this work is both valuable and necessary formany similar deep learning applications in the scientific domain.

For future work, we plan to further exploit the parallelism in theneighborhood query process by partitioning the problem domain andthen merging the query results on the fly. Also, we plan to extendthe current topology-aware datacube to support online queries of awide range of joint distribution representations (e.g., parametric den-sity model, copulas) and incorporate efficient compression strategies.Finally, we are in the process of releasing the proposed tool as partof a general purpose high-dimensional data analysis package to betterfacilitate the analysis and evaluation of similar applications.

ACKNOWLEDGMENTS

This work was performed under the auspices of the U.S. Depart-ment of Energy by Lawrence Livermore National Laboratory underContract DE-AC52-07NA27344. The work was also supported in partby NSF:CGV Award: 1314896, NSF:IIP Award: 1602127, NSF:ACIAward:1649923, DOE/SciDAC DESC0007446, PSAAP CCMSC DE-NA0002375 and NSF:OAC Award: 1842042. Additional supportcomes from Intel Graphics and Visualization Institutes of XeLLENCEprogram.

REFERENCES

[1] Jag icf dataset for scientific machine learning. https://github.com/rushilanirudh/icf-jag-cycleGAN. Accessed: 2019-07-15.

[2] T. W. Anderson. An introduction to multivariate statistical analysis, vol. 2.Wiley New York, 1958.

[3] A. O. Artero, M. C. F. de Oliveira, and H. Levkowitz. Uncovering clustersin crowded parallel coordinates visualizations. In —, pp. 81–88. IEEE,

9

2004.[4] P. Baldi, S. Brunak, and F. Bach. Bioinformatics: the machine learning

approach. MIT press, 2001.[5] J. C. Bennett, H. Abbasi, P.-T. Bremer, R. Grout, A. Gyulassy, T. Jin,

S. Klasky, H. Kolla, M. Parashar, V. Pascucci, et al. Combining in-situ and in-transit processing to enable extreme-scale scientific analysis.In Proceedings of the International Conference on High PerformanceComputing, Networking, Storage and Analysis, p. 49. IEEE ComputerSociety Press, 2012.

[6] P.-T. Bremer, D. Maljovec, A. Saha, B. Wang, J. Gaffney, B. K. Spears,and V. Pascucci. Nddav: N-dimensional data analysis and visualizationanalysis for the national ignition campaign. Computing and Visualizationin Science, 17(1):1–18, 2015.

[7] P.-T. Bremer, V. Pascucci, and B. Hamann. Maximizing adaptivity inhierarchical topological models using cancellation trees. In T. Moeller,B. Hamann, and B. Russell, eds., Mathematical Foundations of ScientificVisualization, Computer Graphics, and Massive Data Exploration, p. toappear. Springer, 2006.

[8] K. T. Butler, D. W. Davies, H. Cartwright, O. Isayev, and A. Walsh. Ma-chine learning for molecular and materials science. Nature, 559(7715):547,2018.

[9] C. Correa and P. Lindstrom. Towards robust topology of sparsely sam-pled data. IEEE Transactions on Visualization and Computer Graphics,17(12):1852–1861, Dec. 2011. doi: 10.1109/TVCG.2011.245

[10] C. Correa, P. Lindstrom, and P.-T. Bremer. Topological spines: A structure-preserving visual representation of scalar fields. IEEE Transactions onVisualization and Computer Graphics, 17(12):1842–1851, Dec. 2011. doi:10.1109/TVCG.2011.244

[11] T. N. Dang, L. Wilkinson, and A. Anand. Stacking graphic elements toavoid over-plotting. IEEE Transactions on Visualization and ComputerGraphics, 16(6):1044–1052, 2010.

[12] J. Gaffney, P. Springer, and G. Collins. Thermodynamic modeling ofuncertainties in nif icf implosions due to underlying microphysics models.In APS Meeting Abstracts, 2014.

[13] S. Gerber, P.-T. Bremer, V. Pascucci, and R. Whitaker. Visual explorationof high dimensional scalar functions. IEEE transactions on visualizationand computer graphics, 16(6):1271, 2010.

[14] S. Gerber, P.-T. Bremer, V. Pascucci, and R. Whitaker. Visual explorationof high dimensional scalar functions. IEEE Transactions on Visualizationand Computer Graphics, 16(6):1271–1280, 2010.

[15] A. Gyulassy, P.-T. Bremer, B. Hamann, and V. Pascucci. A practicalapproach to morse-smale complex computation: Scalability and generality.IEEE Transactions on Visualization and Computer Graphics, 14(6), 2008.

[16] A. Gyulassy and V. Natarajan. Topology-based simplification for featureextraction from 3d scalar fields. In Visualization, 2005. VIS 05. IEEE, pp.535–542. IEEE, 2005.

[17] J. Krause, A. Perer, and K. Ng. Interacting with predictions: Visualinspection of black-box machine learning models. In Proceedings ofthe 2016 CHI Conference on Human Factors in Computing Systems, pp.5686–5697. ACM, 2016.

[18] A. G. Landge, V. Pascucci, A. Gyulassy, J. C. Bennett, H. Kolla, J. Chen,and P.-T. Bremer. In-situ feature extraction of large scale combustion sim-ulations using segmented merge trees. In High Performance Computing,Networking, Storage and Analysis, SC14: International Conference for,pp. 1020–1031. IEEE, 2014.

[19] L. Lins, J. T. Klosowski, and C. Scheidegger. Nanocubes for real-timeexploration of spatiotemporal datasets. IEEE Transactions on Visualizationand Computer Graphics, 19(12):2456–2465, 2013.

[20] M. Liu, S. Liu, H. Su, K. Cao, and J. Zhu. Analyzing the noise robustnessof deep neural networks. arXiv preprint arXiv:1810.03913, 2018.

[21] S. Liu, K. Humbird, L. Peterson, J. Thiagarajan, B. Spears, and P.-T.Bremer. Topology-driven analysis and exploration of high-dimensionalmodels. In Research Challenges and Opportunities at the interface ofMachine Learning and Uncertainty Quantification, 2018.

[22] S. Liu, Z. Li, T. Li, V. Srikumar, V. Pascucci, and P.-T. Bremer. Nlize: Aperturbation-driven visual interrogation tool for analyzing and interpretingnatural language inference models. IEEE transactions on visualizationand computer graphics, 25(1):651–660, 2019.

[23] Z. Liu, B. Jiang, and J. Heer. immens: Real-time visual querying of bigdata. In Computer Graphics Forum, vol. 32, pp. 421–430. Wiley OnlineLibrary, 2013.

[24] S. M. Lundberg and S.-I. Lee. A unified approach to interpreting modelpredictions. In Advances in Neural Information Processing Systems, pp.

4768–4777, 2017.[25] A. Mayorga and M. Gleicher. Splatterplots: Overcoming overdraw in

scatter plots. IEEE transactions on visualization and computer graphics,19(9):1526–1538, 2013.

[26] Y. Ming, H. Qu, and E. Bertini. Rulematrix: Visualizing and understandingclassifiers with rules. IEEE transactions on visualization and computergraphics, 25(1):342–352, 2019.

[27] E. Mjolsness and D. DeCoste. Machine learning for science: state of theart and future prospects. science, 293(5537):2051–2055, 2001.

[28] C. Olah, A. Mordvintsev, and L. Schubert. Feature visualization. Distill,2017. https://distill.pub/2017/feature-visualization. doi: 10.23915/distill.00007

[29] J. Peterson, K. Humbird, J. Field, S. Brandon, S. Langer, R. Nora,B. Spears, and P. Springer. Zonal flow generation in inertial confine-ment fusion implosions. Physics of Plasmas, 24(3):032702, 2017.

[30] M. T. Ribeiro, S. Singh, and C. Guestrin. Why should i trust you?:Explaining the predictions of any classifier. In Proceedings of the 22ndACM SIGKDD International Conference on Knowledge Discovery andData Mining, pp. 1135–1144. ACM, 2016.

[31] K. Simonyan, A. Vedaldi, and A. Zisserman. Deep inside convolutionalnetworks: Visualising image classification models and saliency maps.arXiv preprint arXiv:1312.6034, 2013.

[32] P. Springer, C. Cerjan, R. Betti, J. Caggiano, M. Edwards, J. Frenje, V. Y.Glebov, S. Glenzer, S. Glenn, N. Izumi, et al. Integrated thermodynamicmodel for ignition target performance. In EPJ Web of Conferences, vol. 59,p. 04001. EDP Sciences, 2013.

[33] D. M. Thomas and V. Natarajan. Detecting symmetry in scalar fields usingaugmented extremum graphs. IEEE Transactions on Visualization andComputer Graphics, 19(12):2663–2672, Dec 2013. doi: 10.1109/TVCG.2013.148

[34] I. Tolstikhin, O. Bousquet, S. Gelly, and B. Schoelkopf. Wassersteinauto-encoders. arXiv preprint arXiv:1711.01558, 2017.

[35] B. Van Essen, H. Kim, R. Pearce, K. Boakye, and B. Chen. Lbann:Livermore big artificial neural network hpc toolkit. In Proceedings ofthe Workshop on Machine Learning in High-Performance ComputingEnvironments, p. 5. ACM, 2015.

[36] J. Wang, L. Gou, H.-W. Shen, and H. Yang. Dqnviz: A visual analytics ap-proach to understand deep q-networks. IEEE transactions on visualizationand computer graphics, 25(1):288–298, 2019.

[37] J. Yosinski, J. Clune, A. Nguyen, T. Fuchs, and H. Lipson. Under-standing neural networks through deep visualization. arXiv preprintarXiv:1506.06579, 2015.

[38] M. D. Zeiler and R. Fergus. Visualizing and understanding convolutionalnetworks. In European conference on computer vision, pp. 818–833.Springer, 2014.

10