Figure 1: Current prototypeshowing biological data usinga spatially aware mobile and alarge interactive display.

Exploring Time-dependentScientific Data Using SpatiallyAware Mobiles and Large Displays

Hendrik SollichInteractive Media LabTechnische Universitat DresdenDresden, Germanyhendrik.sollich@tu-dresden.de

Pavel TomancakMPI-CBG DresdenDresden, Germanytomancak@mpi-cbg.de

Ulrich von ZadowInteractive Media LabTechnische Universitat DresdenDresden, Germanyuzadow@acm.org

Raimund DachseltInteractive Media LabTechnische Universitat DresdenDresden, Germanydachselt@acm.org

Tobias PietzschMPI-CBG DresdenDresden, Germanypietzsch@mpi-cbg.de

Permission to make digital or hard copies of part or all of this work for per-sonal or classroom use is granted without fee provided that copies are notmade or distributed for profit or commercial advantage and that copies bearthis notice and the full citation on the first page. Copyrights for third-partycomponents of this work must be honored. For all other uses, contact theOwner/Author.Copyright is held by the owner/author(s).ISS ’16 , November 06-09, 2016, Niagara Falls, ON, CanadaACM 978-1-4503-4248-3/16/11http://dx.doi.org/10.1145/2992154.2996779

AbstractScientific Visualization is generally based on very largeand often high-dimensional data sets. In many cases (e.g.,in the biomedical domain), high-resolution 3D data thatchanges over time and multiple data sets are involved.One important task in making sense of this data is findinginteresting 2D views in the 3D space and additionallyfinding ways to handle changes over time. We explorehow the combination of spatially aware mobiles and largedisplays can be used in this context. Specifically, wecontribute both a precise selection technique for 2D crosssections and a concept for time-dependent 4D bookmarksthat follow a point of interest over time. We implementedthese techniques in a prototype application that visualizesbiological Light Sheet Fluorescence Microscopy data andreport on initial user feedback.

Author KeywordsTangible views; large display; display wall; multi-displayenvironment; spatial interaction; information visualization

ACM Classification KeywordsH.5.2 [Information interfaces and presentation (e.g.,HCI)]: User Interfaces - Input devices and strategies;Interaction styles.

IntroductionExploration of time-dependent 3D data is an importanttopic in scientific visualization, e.g., in the biomedicalfield [7] and in fluid dynamics [2]. Challenges includefinding 2D cross sections in 3D and in time that displaysalient features of the data [6]. The combination of aspatially aware mobile display and a large contextualdisplay has been shown to support interaction with crosssections of 3D data well [2, 9]. In these cases, the mobiledisplay’s position and orientation in space are used tospecify a cross section to show and the large display servesto provide context and a frame of reference. We extendthis to cover effective selection using three-way clutchingand contribute a novel concept for selection andbookmarking in time-dependent data. Our interactionconcepts are based on observations of and discussions withdevelopmental biologists, and our prototype (Figure 1),which displays microscopy time-series, reflects this.

Figure 2: Conventionaluser interface based on Big-DataViewer [7] showing LSFMdata.

Related WorkOur work builds on previous work on tangible displays forinfovis. A salient example is Spindler et al.’s work onTangible Views [9], which used projection onto passivetangible surfaces tracked in 3D space for informationvisualization (later expanded to spatially-aware mobiledevices [8]). More recently, Besancon et al. [2]investigated hybrid tactile/tangible interaction in thecontext of fluid dynamics visualization, again using aspatially-aware mobile device to specify cutting planes.Both these works included an additional stationary displayfor context visualization. Closer to our applicationdomain, Bertelsen et al. [1] investigated medical volumevisualization using a larger display mounted on a movablemechanical arm. None of these works specifically considertime-dependent 3D data or bookmarking.

Additionally, there are several overviews that consider theuse of novel user interface techniques in the context ofscientific visualization. Keefe and Isenberg’s ’Reimaginingthe Scientific Visualization Interaction Paradigm’ [6] givesa good concise introduction, including an overview oflow-level interactions that these systems should support.Two overviews covering specific modalities are van Damet al.’s report on VR in scientific visualization [4] andIsenberg et al.’s paper on the use of data visualization oninteractive surfaces [5].

Application Scenario: Visualization of LSFMMicroscopy DataAlthough we designed them to be generalizable, ourinteraction concepts are based on the analysis of theworkflows of a specific group of users: DevelopmentalBiologists working with Light Sheet FluorescenceMicroscopy (LSFM) data. LSFM produceshigh-resolution, 3D time-series recordings of thedevelopment of living biological specimens, enablingscientists to examine anatomical structures close enoughto observe miniscule details as small as cell membranes ornuclei in motion during cell division. Our concepts andprototype were developed in an iterative fashion, involvingseveral meetings with prospective users and observationsof their work using current tools (e.g., Figure 2). Thefollowing user scenario is derived from these observations:

A developmental biologist, Alice, wants to analyze theeffects of different treatments on the development ofPlatynereis, a marine ragworm. Accordingly, her goal is tocompare certain biological aspects of these specimens.These aspects emerge in certain spatial regions of theembryo and at certain times in the development, so Aliceselects appropriate views (called samples: 2D crosssections at a point in time, i.e., a 2D plane in 4D space)

for comparison (Figure 3a). Additionally, to demonstratethe expected development of a specimen, she comparesthe same aspects in different stages of development of thesame specimen (Figure 3b). In both cases, the samplesshe selects must be located both in time and in space.

Figure 3: Comparing two samples: (a - red line) two speci-mens at the same stage of development or (b - blue line) twostages of development in the same specimen

Selection in time is aided by the fact that Platynereisdevelopment is stereotypic and roughly synchronousacross specimens, but the location in space involvesprecise selection of a cross section that involves threetranslational and three rotational degrees of freedom(DOF). This is more difficult, especially if using a 2Dinput device such as the mouse. An additional issueappears when she needs to be able to follow the aspectsover time: With the development of the organism, theinteresting points move in space. She therefore needs tobe able to adjust cross sections accordingly. Further, whenshe compares different specimens, it helps if she cantransfer sample locations from one specimen to another tohave an initial starting point.

After selecting samples for comparison alone, Alice wantsto present and discuss her analysis with colleagues. Forthis stage, Alice would like to have a larger screen forbetter overview of the data, as well as easy access tobookmarks.

Thus, challenges are the precise yet intuitive and quickselection of a cross section to display and seamlesssupport of time-dependent data, including bookmarking ofareas of interest that move over time.

Figure 4: Mobile user interface concept. (a) Sample (2Dview), (b) Timeline showing current point in time and book-mark keyframes, (c) Clutch for positioning in 3D, (d) Book-marks.

Interaction ConceptOur interaction concept supports the user scenario aboveand involves a spatially-aware mobile device as well as alarge touch-sensitive display. Users can intuitively selectcross sections and in general work with a single sample onthe mobile device, while the larger screen enables workwith multiple samples and specimens, comparison ofsamples, and group work.

The mobile UI is structured around a rendering of thecurrently selected sample (Figure 4a). In addition, itdisplays a timeline at the bottom (b) that allows selectingthe point in time to view. At the right and left sides ofthe screen – easily accessible by thumb if the device isheld in two hands – are further interface elements: A

clutch that controls cross section selection (c), and a listof bookmarks (d).

Figure 5: Clutch usage: (a)In neutral state, the datasetdoes not move in relation to thetablet’s screen, (b) Clutch down:Translation of the cross sectionwithin the dataset by moving thetablet along its z-axis, (c) Clutchup: Rotating the selected crosssection by tilting or rotating thehandheld.

The contextual display (Figure 7) is large and thereforecapable of displaying numerous samples (from the same ordifferent specimen) in multiple views. The individual wallviews are similar to the mobile UI: They have the sametimeline and bookmark selection functionality (but nofacility for specifying cross sections). In addition, they candisplay a volumetric view of the data that visualizes thepositions of the currently configured cross sections. Thecontextual display also supports a comparison view thatsuperimposes two samples.

Cross Section SelectionCross section selection – configuration of the 2D cuttingplane in the 3D volumetric data – is performed via spatialinput (Figure 5). In essence, the position and rotation ofthe mobile device in 3D space represents the cross sectionthe user wishes to view, and movement of the devicechanges the cross section (as in [2, 8]). However, it mustbe possible to enable and disable these view changes.Further, it is not effective to control 6 DOF at once.

We approach both issues by using a three-way clutch(Figure 5) that disables cross section changes when inneutral position (a). Two other modes can be enabled bysliding and holding the clutch down or up, respectivelyenabling translation (b) or rotation (c). This separation ofDOF allows more effective interaction [6]. Additionally, ifmore precision is needed, it should be straightforward tocombine three-way clutching with a mechanism thatmodifies control-display (CD) gain or with nudging.

An additional issue appears when the current cross sectionis visualized both on the mobile and the large stationaryscreen: While looking at the mobile, users expect the

plane to move through a stationary virtual object (theegocentric view [3]). However, when looking at astationary screen, users expect the inverse, with themobile representing the virtual object while the crosssection remains stationary in space (the exocentricview [3]). Thus, displaying the moving cross section onboth displays is confusing, and one view should bedisabled during the interaction.

Figure 6: Single bookmark with four keyframes at differentpoints in time

BookmarkingThe interface described in the previous section allowsspecifying cross sections for single points in time.However, one of our goals was to support working withpoints of interest that move over time. Our bookmarkconcept (Figure 6) enables this. Besides supportingtraditional bookmarks that reference single samples, itallows the user to specify multiple cross sections fordifferent points in time. These are then used as keyframesfor interpolation. An interface at the left side of thescreen (Figure 4d) allows the user to manage bookmarks.It supports adding and removing bookmarks andhighlights the currently selected bookmark. Keyframes aredisplayed and configured on the timeline (Figure 4b), withcolor coding used to identify different bookmarks.

PrototypeOur prototype implements the bulk of the conceptsdescribed above. We tested the prototype using aninteractive display wall comprised of twelve 55” displayswith a total size of 5x2 meters and a resolution of 24

megapixels running Ubuntu. As mobile device, we used aGoogle Nexus 7 tablet.

Figure 7: View of completemulti-device prototype.

Figure 8: Prototype mobile in-terface.

Figure 9: Comparison view onlarge display using color codingto show differences between sam-ples.

On the large display, the prototype supports multiple crosssection views (of the same or different specimen) as wellas volumetric views and a comparison view (Figure 9).The mobile device displays the currently selected sampleand supports cross section selection using clutching.Currently, we assume that the user looks at the mobiledevice when specifying a cross section and thereforeimplement an egocentric perspective. To avoid confusingusers, we disable view updates on the large screen as longas the clutch is active. Further, nudging and configurableCD gain are currently not implemented. Both mobile andlarge display views fully support bookmark management.

The UI of the prototype is implemented in Python usingthe media application framework libavg1. It is a singlemulti-device application: Both display wall and mobileinterfaces are rendered on the display wall workstation.The mobile interface is compressed to h264 in realtimeand streamed to the device. Conversely, touch data isstreamed from mobile to workstation. Mobile locationdata is obtained using a marker-based optical trackingsystem, OptiTrack.

For rendering of the actual LSFM images, we useBigDataViewer (BDV), an extensible open source solutionfor interactive visualization of very large volumes and timeseries of volumes [7]. Individual image stacks arearbitrarily arranged in global 3D coordinate space, andcan be displayed independently or as color composite. Thesoftware renders arbitrarily oriented virtual slices throughthe global space. An intelligent tile-based loading andcaching scheme allows smooth navigation in

1http://www.libavg.de

multi-terabyte image datasets. The data can be providedby both local and remote sources, and is easily extendedto handle new data sources such as 3rd party file formatsor online data stores.

BDV is implemented as a java library. To make it usablein our context, we implemented a libavg plugin in C++using the Java Native Interface. Using this plugin, anapplication can request rendered frames from BDV,specifying parameters such as timepoint, transformation,visible channels, etc. On the Java side, BDV fetches andcaches the data from an arbitrary backend (in ourexperiments we remotely access data provided by a serverat the MPI-CBG). Rendered frames are asynchronouslypassed back to libavg where they are loaded to texturesfor display.

Initial User FeedbackAs initial validation of our prototype, we solicitedfeedback from several domain experts (developmentalbiologists) who are potential users of our system. For thisfeedback session we used an LSFM dataset of Platynereisthat had a size of 1.4 terabytes. The session included apresentation of the interface, informal tests by the domainexperts, and a discussion about future directions.

The general reaction was very positive for bothtangible-based cross section selection and time-seriesbookmarking. There were few precision issues in crosssection selection with the current three-way clutch.However, users pointed out that this might change forhigher resolution data sets. Further, users would like theability to set a pivot point for rotations, as the currentrotation around the center of the specimen causes themto frequently lose their point of interest.

DiscussionBased on the initial user feedback, we are confident thatour concept is sound and that development is proceedingin the right direction in general. However, there are anumber of directions for future work. One very interestingtopic is the distribution of the interaction between themobile and the large display. While the mobile device isclearly more suitable for sample selection, the wall displayis good for work with multiple views and presentation ofresults. However, other interactions can be done on bothdevices, and sample selection could potentially be donewith the mobile device while viewing the results on thewall display in exocentric mode. Aspects such asnecessary attention switches, egocentric versus exocentricvisualization, and single- versus multi-user interaction allplay a role here, so we believe that a good solutionrequires the exploration of several alternatives.

ConclusionWe presented an interaction concept for exploration oflarge scientific time-dependent 3D data sets, validatedthrough a prototype and initial user feedback from domainexperts. The presented interactions include both a preciseselection technique for 2D cross sections and a conceptfor time-dependent bookmarks that allows following apoint of interest through time.

AcknowledgementsWe would like to thank the scientists of the Max PlanckInstitute of Molecular Cell Biology and Genetics whoprovided valuable domain knowledge and user feedback,namely Ulrik Gunther, Mette Handberg-Thorsager,Yu-Wen Hsieh, Akanksha Jain, Helena Jambor, andChristopher Schmied. Additional thanks to MetteHandberg-Thorsager and Yu-Wen Hsieh for providingLSFM data of Platynereis development.

References[1] Bertelsen, A., Irarrazaval, P., and Cadiz, R. F. Volume

visualization using a spatially aware mobile displaydevice. Computerized Medical Imaging and Graphics36, 1 (2012), 66 – 71.

[2] Besancon, L., Issartel, P., Ammi, M., and Isenberg, T.Hybrid tactile/tangible interaction for 3d dataexploration. IEEE Trans. Vis. Comp. Graph. PP, 99(2016), 1–10.

[3] Bowman, D. A., Kruijff, E., LaViola, J. J., andPoupyrev, I. 3D User Interfaces: Theory and Practice.Addison Wesley Longman Publishing Co., Inc.,Redwood City, CA, USA, 2004.

[4] Dam, A. v., Forsberg, A. S., Laidlaw, D. H., LaViola,J. J., and Simpson, R. M. Immersive VR for scientificvisualization: A progress report. IEEE Comput. Graph.Appl. 20, 6 (Nov. 2000), 26–52.

[5] Isenberg, P., Isenberg, T., Hesselmann, T., Lee, B.,von Zadow, U., and Tang, A. Data visualization oninteractive surfaces: A research agenda. IEEEComput. Graph. Appl. 33, 2 (2013).

[6] Keefe, D. F., and Isenberg, T. Reimagining thescientific visualization interaction paradigm. Computer46, 5 (May 2013), 51–57.

[7] Pietzsch, T., Saalfeld, S., Preibisch, S., andTomancak, P. BigDataViewer: visualization andprocessing for large image data sets. Nature Methods12, 6 (06 2015), 481–483.

[8] Spindler, M., Buschel, W., Winkler, C., and Dachselt,R. Tangible displays for the masses: Spatialinteraction with handheld displays by using consumerdepth cameras. Personal Ubiquitous Comput. 18, 5(June 2014), 1213–1225.

[9] Spindler, M., Tominski, C., Schumann, H., andDachselt, R. Tangible views for informationvisualization. In Proc. ITS, ACM (2010), 157–166.