STNexus: An Integrated Database and Visualization ...weaver/academic/... · Visual and Computational Requirements For information analysis, visualization tools must accomplish four

STNexus: An Integrated Database and Visualization Environment - 1

STNexus: An Integrated Database and Visualization Environment for Space-Time Information Exploitation

Chris Weaver, Donna Peuquet, Alan M. MacEachren

GeoVISTA Center www.geovista.psu.edu

Department of Geography 302 Walker

Penn State University University Park, PA 16802

Abstract Virtually all military and security intelligence analysis depends increasingly on interaction with space-time information. An integrated approach is needed for dealing with space-time components of evidence in flexible and holistic ways so as to facilitate the entire spectrum of intelligence tasks, from open-ended exploration and discovery to investigation to highly focused analysis. The STNexus space-time database visualization environment embodies this integrated approach. Currently in early development, STNexus will realize a unified conceptual scheme that combines a multi-representation database framework with diverse visualization techniques for synthesis and analysis of massive, complex information. Here we present the overall system architecture for STNexus as one step toward enabling the development, implementation, and effective application of this scheme. By its very nature, this process will involve fundamental advancements in both visualization and database elements. Introduction Understanding the space and time components of information is often critical for both inferential and descriptive analysis. Identifying terrorist threats involves looking for spatial and temporal patterns in the vulnerability and accessibility of potential targets in relation to movement and interaction of suspected perpetrators. Predicting an event often requires digesting substantial volumes of incomplete information about people, materials, vehicles, financial resources, etc. Visualization techniques must facilitate a range of analytic tasks: asking specific questions; gaining broad insight into the geographic structure and operation of organizations and processes; trolling large, multi-dimensional data collections to detect changes; finding relationships across heterogeneous information forms; and discovering previously unsuspected


patterns. Each of the multiple steps involved in intelligence analysis—searching data, marshaling evidence, deriving hypotheses, making decisions, presenting results—requires differing system capabilities. To meet these needs, visualization methods must be tightly integrated with an underlying data model capable of supporting space, time, attributes, and derived knowledge. However, there is currently a wide gap between visualization research and research on database representation. With a narrow focus on the merely visual, research and development of visualization software has essentially ignored database representation. Tools tend to be limited to flat files of simple tables. As a result, an increasingly wide range of complex visualizations that can quickly reveal or portray information structure cannot be used in conjunction with truly massive and heterogeneous data collections with any level of efficiency or effectiveness. The fact that different users and tasks require different (and often multiple) kinds of visualizations of the same data sets only exacerbates the problem. Existing research in this area is limited to information visualization approaches built on the relational data model [1-3]. While the relational model does represent many types of data well, its flexibility does not extend into the spatial realm. Moreover, these traditional data models are usually single-scale, treating the temporal dimension as an afterthought. A few sophisticated representations of time have been implemented for financial and other forms of business transactions (e.g., AT&T’s Daytona telephone record database). Major GIS software products can track point objects through space (e.g., ESRI’s Tracking Analyst). Although effective at the limited tasks for which they are designed, these systems are unable to represent the full potential complexities of space-time data, such as how spatial configurations of linear and polygonal objects may change over time, or the intricate, multi-scale space-time-object relationships confronted by all-source analysts attempting to recognize preparations for a terrorist attack or respond to such an event. The overall goal of the research described here is to address current DTO needs through development and implementation of a software architecture that combines powerful and flexible space-time database and information visualization capabilities. We view this as an enabling step toward effective use of diverse visualization techniques for both targeted analysis and unstructured data exploration. As such, the STNexus framework and prototype intrinsically involve continued fundamental advancements in space-time databases and visualization. In this paper, we focus on issues related to combining diverse visualization tools with space-time data representations in an integrated software architecture. First, we discuss design principles of an integrated environment that facilitates unstructured space-time data exploration. Following a conceptual overview of the capabilities of STNexus, we briefly describe its data and knowledge representation, its architecture components for coordinating multiple views of complex space-time information, and our approach for integrating its database and visualization components. We close with some summary comments that also suggest future work. Visual and Computational Requirements For information analysis, visualization tools must accomplish four complimentary goals. 1. Analysis of multi-source information requires an integrated data environment that includes GIS and imaging functionality, linked to a database that supports temporal as well as spatial information retrieval, coupled with flexible information visualization methods and tools. Analysis of geospatial data, imagery, documents, and other information currently


involves independent tools in which information moves sequentially from one tool (and typically from one analyst) to the next. For example, an image analyst might process raw imagery with image analysis software, passing the result to a geospatial analyst who uses a GIS to combine that imagery with other geospatial layers, who in turn passes the result to an intelligence analyst who attempts to relate the processed imagery and maps with a range of other information. 2. Analysis of multi-faceted information requires a flexible environment that supports interconnection of diverse methods and tools for visual and computational treatment of the geospatial, temporal, abstract, and knowledge components of information. Over the past decade, development of methods and tools for geospatial visualization has produced advances in fast, realistic rendering of terrain and landscapes and in abstract visualization of geo-located information (such as telephone records, health and demographic statistics, and multispectral image attributes [15]). Similar progress in information visualization has facilitated exploration, understanding, and decision-making involving temporal information [16] and knowledge, concepts, and arguments [17]. However, these developments have been largely independent. Only a few efforts (e.g., [18-20]) have been made to integrate diverse tools into coordinated systems that support exploration of spatial, temporal, and abstract relationships. 3. Analysis of multi-dimensional information requires a fluid environment that supports both open-ended exploration and deep examination by providing mechanisms for coordination in space (across qualitatively different kinds of views in a display) and time (over a viewing history of the discovery–proof–choice reasoning process of intelligence analysis [26]), both during and subsequent to analysis. Linking different visual displays to communicate context and correlated information will become increasingly important as multiple visualization methodologies are introduced into the analysts’ operational environment. Linked brushing and its variants are well established exploratory data analysis and information visualization methods [21, 22]. There have been several recent efforts to take a more comprehensive approach to the problem of coordinating multiple views that goes beyond simply cross-view linking. Work by Roberts, et al. [23, 24] and our own research team [25] involves coordinating various view parameters independently. North, et al. [1, 19] tightly couple visual coordination with database objects. These approaches, however, have not addressed the challenges of coordination to support spatiotemporal analysis or coordination across views that do not relate data objects through one-to-one mappings. Coordination of multiple views is an effective approach for representing both global patterns and local features. Visualization tools should support overview, zoom and filter, then details on demand, and do so for complex, heterogeneous space-time information. They should also seamlessly integrate computational methods that organize, process, filter, aggregate, cluster, and sort data in ways that leverage the power of human vision and expert knowledge. 4. Analysis of multi-scale information requires an expressive environment in which decisions can be realized about what information to emphasize, which features to show, and how to represent those features visually, from global context to minute detail. In a single visualization, multiple views can display different combinations of space, time, and abstract attributes. In particular, overviews provide an integrated perspective on the full


space of information attributes, thereby helping analysts develop initial hypotheses about complex multi-dimensional relationships. Overviews take three forms: an integrated display of all attributes in a single view [11-13]; a sequential ‘grand tour’ of computationally selected ‘interesting’ views of the information space [14]; and a multi-view layout in which different (groups of) attributes appear in separate, dynamically linked displays (e.g., scatterplot matrices and small multiples). Core Capabilities STNexus is intended to support analysts in descriptive and inferential tasks that draw upon a combination of maps, images, statistics, and raw data about people and objects in space and time from multiple sources in order to assemble evidence and assess potential threats. Our approach provides scientific advancements in the following areas:

• Encoding both time-varying geospatial data and high-level domain-specific knowledge, at multiple spatial and temporal scales, in a fundamentally new database representation.

• Coordinating multiple visualization components to enable relationships and perspectives to be recognized and contextualized within and across spatial and temporal scales.

• Facilitating knowledge construction and management—for identifying and understanding features, patterns, and relationships and for storing knowledge in accessible ways—by coupling information representation methods with coordination techniques.

• Translating component-based visualization into a variety of operational environments. • Designing and implementing production systems that help analysts exploit previously

captured domain knowledge by integrating storage, access, and visual analysis of data. STNexus is intended to test and demonstrate these advancements by combining an innovative, multi-representation database model with component-based visualization in a highly coordinated design. The STNexus prototype consists of database, visualization and knowledge acquisition components that can be used for a wide variety of applications. Since these components are cross-platform and open source, the entire environment is easily transportable. Data and Knowledge Representation The flexibility of relational databases in representing abstract data extends poorly into the spatial and temporal realms. It is a well-known principle that how data are represented within a database determines what can and cannot be done effectively with those data, especially in visual display. A classic example is storing space-time data as sequenced snapshots. This model coincides with digital movies—series of still images shown in quick succession to suggest movement and change through space over time. However, research indicates that animation alone is not a particularly effective way to promote understanding of process or function. No single image or database representation can support fully the range of data and applications that arise in intelligence analysis. What is needed is a flexible database representation that enables multi-perspective visualization and seamless interaction with information, one that does not require the analyst to generate a formal query every time insight prompts new questions. Peuquet [4] originally suggested a conceptual framework for representing space-time data. The Pyramid database model, built upon this approach, not only incorporates time but also is multi-representational. In the Pyramid model, an object (the ‘what’ component) represents any


entity. Entities can be geographically based (airports, cities), have locations that change over time (people, ships), or have no location at all (policies, laws). The ‘where’ component is a location-based view of entities. The ‘when’ component represents events. In contrast with traditional geometric (vector) representations, the Pyramid model offers greatly increased flexibility for representing dynamic objects by defining stored elements from a cognitive perspective. Its flexible, interlinked structure can accommodate differing views of represented phenomena and incorporate changes to those phenomena as new information is acquired through observation, computation, or annotation. The result in a more natural and intuitive representation for domain experts because it coincides with known aspects of cognition [5]. The Triad model—a subset of the Pyramid model without the knowledge component—was implemented as a proof-of-concept in a previous project (www.geovista.psu.edu/grants/apoala). Reserved capacity for a ‘shared memory’ of derived knowledge (including user-defined relationships among objects) is needed to preserve insights gained from expertise and analysis for use across domains in subsequent analyses. Knowledge representation languages—and the construction of ontologies (taxonomies of object types) that use them to describe features of the world—have garnered substantial attention [6-8]. What is largely missing from this work, however, is consideration of how knowledge is generated, promulgated, revised and retired. Knowledge representation implementations often focus on recording axioms about a domain without attempting to situate those axioms in the context of their creation or use. In building tools to help analysts express and explore the concepts they use to describe the world, our aim is to offer ways to structure and signify knowledge in forms that can be communicated and reused most efficiently. The goal is to enable analysts to move from simple concepts to more complex knowledge structures that link data, methods, theories and other elements. Our work will focus on representing both concepts and semantic relationships among concepts. Such concept-relationship representations can enable computational environments for visualizing and reasoning about concepts, by connecting and combining knowledge structures within an individual’s concept space as well as across multiple users and domains. By defining an ontology, the Pyramid model object component is the overlapping representational element in STNexus that brings together observational data, concepts used or constructed by analysts, and the knowledge base that integrates concepts and supports their use. Knowledge in the knowledge base can be obtained using a top-down or a bottom-up approach, or a combination of both based upon a general scheme described by Mennis and Peuquet [9, 10]. In the top-down approach, the domain expert (typically a human using prior knowledge) would create classes with predetermined membership criteria before defining sample (prototypical and atypical) member objects within each class. The bottom-up approach is more exploratory and is appropriate when the nature of the actual objects is unknown. Here, human and computational expertise can be applied independently or in combination. STNexus is designed to integrate key capabilities of several visual analytics development environments within an analytic workspace that connect five top-down processes: defining and browsing concepts ontologically; selecting specific concepts to use in an analysis exercise; operationalizing the concepts with classifiers—with the bottom-up processes; exploring data to help formulate concepts from emergent structures in the data; and modifying concepts, classifiers or data used when poor categories result (i.e. when categories misalign with mental concepts or are not clearly differentiable in the data).


Architectural Components In developing tools for knowledge representation and construction as part of an integrated database-visualization environment for information exploration, STNexus takes advantage of recent advances in geo/information visualization systems and toolkits. Specifically, STNexus builds upon two visualization application construction environments developed by our research group, GeoVISTA Studio and Improvise, as well as on GeoTools, an open source GIS library that we are core contributors to. Space-time Views: GeoVISTA Studio GeoVISTA Studio provides a suite of components for quickly building geovisualization applications. These applications are computationally enhanced visual analytics environments that support overview+detail analysis of highly multivariate space-time data as well as knowledge representation and management. Examples include ConceptVISTA (which supports concept mapping and connections between concept maps and domain ontologies) and the Visual Inquiry Toolkit (a joint product of this research and complementary research supported by NIH, developed to address the space-time data analysis challenge posed for the 2005 InfoVis contest). The innovation of the Visual Inquiry Toolkit is the ability to visualize multivariate aspects of space-time data in support of exploration of high-dimensional relationships and patterns. Linked views leverage human visual capabilities and novel computational strategies for removing noise and enhancing structure in complex space-time-attribute data. A map matrix, data matrix, parallel coordinate plot, and self-organizing map (SOM) allow an analyst to explore sales in 18 industries by state over time. Live interaction with clustering parameters, brushing, and color schemes adds analytical power. Space-time Coordination: Improvise Improvise is a live-design tool for building and browsing highly coordinated multi-view visualizations interactively. By coupling a declarative visual query language with a shared-object coordination model, users gain precise control over how navigation and selection affects the appearance of data across multiple views, using a potentially infinite number of variations on well-known coordination patterns such as synchronized scrolling, overview+detail, brushing, drill-down, and semantic zoom. As a result, it is practical to build visualizations with more views and richer coordination in Improvise than in other visualization systems. One Improvise application visualizes the guest registry for a hotel in Rebersburg, PA


from 1898-1900. A cyclic time view (top left) displays the number of guests on each day using color and text. Cycle length can be adjusted dynamically to reveal visit patterns; here, a 20 day cycle causes three-week intervals to appear along the diagonal. Characteristics of various natural and artificial calendars are encoded with additional graphics: months are outlined; weekdays and weekends are drawn as different shapes (with darker edges for selected holidays); a color gradient indicates yearly seasons (not shown). Travel patterns on selected days are dynamically linked with multiple layers in the map. Space-time Queries: GeoTools GeoTools is an open source library of GIS and image analysis tools that provides an abstraction layer between data storage (spatial database, flat files, memory, etc.) and usage (visualization, etc.). This abstraction layer—the Data Store API—currently supports over 15 data storage formats: key industry file formats including Shapefile and ArcSDE, spatially optimized database formats including Oracle Spatial, and recognized standards from the Open Geospatial Consortium (OGC) including Web Feature Services (WFS) and Geographical Markup Language. The Data Store implementation can translate shared operations used by visualization and analysis components into queries which best utilize the capabilities of each underlying store, while providing fallback implementation of capabilities for storage mechanisms unable to support them natively (such as spatial operations on Shapefiles). Space-time Database: Secondo Secondo is an extendable open source database engine with support for and optimization of application-specific query algebras and data models. Using space-time queries on a Secondo database, we visualized a subset of the APRS Ham radio data set consisting of 9000 transmissions originating from 14 call signs in fall 2001. In the Improvise visualization, multiple inset maps, time-series plots, and detail tables reveal spatial and temporal communication patterns, including a US-Canada border crossing by a mobile radio source (possibly a delivery truck) along a path that involves numerous side-trips and speed


changes. Arbitrary subsets of satellite, marine, vehicular, and stationary call signs can be selected to explore hypotheses about interaction between individuals or groups. Environment Prototype As an integrated architecture for building computationally-supported visualizations with multiple coordinated views in which analysts explore space-time information from multiple perspectives simultaneously, STNexus is intended to trail blaze the frontiers of massive data analysis by exploring highly-interactive parameterization of the entire source-to-display information path.

Current visual analysis tools provide a wide range of data processing. However, these tools typically maintain performance when handling large amounts of data by limiting, delaying, or altogether precluding interactive control. In contrast, Improvise supports full interactive control at all times along the entire information-processing path, at the cost of lower performance for larger volumes of data. By combining the geospatial visualization of GeoVISTA Studio, the expressive coordination of Improvise, the flexible data access of GeoTools, and the space-time query performance of Secondo into an integrated architecture, STNexus takes one step toward an ideal visual analytics environment that maximizes interactive control even for processing stages involving truly vast amounts of information. While it is clear that there is a need for database and visualization components to interact closely, STNexus is designed around a loosely coupled (modular, set-oriented) architecture in order to maximize flexibility and allow for the widest possible range of deployment scenarios. By avoiding unnecessary dependencies, additional and alternative visualization and database components can be developed, customized, and plugged-in for general deployment or as needed for particular analysis tasks. The layer of abstraction offered by this middleware approach is crucial as it isolates visualization and analysis software components from implementation issues associated with accessing the numerous and disparate information sources that are becoming increasingly important for intelligence analysis. Because the Pyramid model has already been implemented using the Data Store API, existing GeoTools-based visualization components can leverage Pyramid capabilities without modification. Moreover, the current reference implementation of the OGC WFS specification (GeoServer, http://cite.occamlab.com/reference/1055141916_13153.html) is built using GeoTools. By representing a Pyramid instance as an OGC Web Feature Service, any OGC complaint tool could exploit Pyramid capabilities without visualization components. Conversely, the Data Store API makes data translation between supported data formats and sources straightforward, allowing STNexus visualization components to work directly off a variety of legacy storage systems. While this would negate the performance and representational flexibility benefits of the Pyramid model for space-time querying, it would allow easy deployment of visualization components within existing infrastructure.


Summary Analysis of information that is multi-source, multi-faceted, multi-dimensional, and multi-scale requires an integrated architecture that combines data access, computation, coordination, and display effectively and efficiently. To address this goal, we are extending and integrating visual analytics methods and tools developed with both Improvise (emphasizing information visualization) and GeoVISTA Studio (emphasizing computationally-enabled, exploratory spatial data analysis). Our design incorporates the GeoTools library for geospatial processing, image analysis, and broad-spectrum data access. Development of the Pyramid model takes advantage of Secondo’s support for non-relational data models. Refitting the primary map view from GeoVISTA Studio for Improvise-style coordination is the last remaining step in achieving an initial working prototype that includes all four layers of the STNexus architecture. When completed, STNexus will provide an integrated and highly adaptable space-time visual analytics environment. Our effort, rather than supplying an incremental advance to current database and visualization methods, is intended to provide the scientific basis for redefining the state-of-the-art needed for the next generation of geospatial intelligence analysis tools. Several possibilities exist for future work to enhance the capabilities of STNexus and the integrated approach it demonstrates. Means to link automated ingest mechanisms for multiple data sources are needed; currently data must be identified and input by hand. Moreover, data in the cyberinfrastructure is not directly available for analysis. Integration between database and visualization components could be refined through more explicit and direct correspondence of their respective operations. There is a key tradeoff to be investigated here in that more direct linkages in the implementation would provide greater efficiency, but likely at the expense of interoperability across different software contexts. In addition, to support the analytical teams that are required to address complex information analysis and exploitation problems, a target for subsequent research is to add support for multi-user coordinated work. References [1] C. North, N. Conklin, and V. Saini. Visualization Schemas: Augmenting Relational Data Schemas for Coordinated Data and Visualization Design. in ACM Thirteenth Conference on Information and Knowledge Management (CIKM). 2002. Washington, D.C.: ACM. [2] C. North, Multiple Views and Tight Coupling in Visualization: A Language, Taxonomy, and System. in CSREA CISST 2001 Workshop of Fundamental Issues in Visualization. 2001. [3] D. Tang, C. Stolte, and R. Bosch, Design choices when architecting visualizations. Information Visualization, 2004. 3(2):65-79. [4] D. Peuquet, It's About Time: A Conceptual Framework for the Representation of Temporal Dynamics in Geographic Information Systems. Annals of the Association of American Geographers, 1994. 84(3):441-461. [5] L. Ungerleider and M. Mishkin, Two Cortical Visual Systems, in Analysis of Visual Behavior, D. Ingle, M. Goodale, and R. Mansfield, Editors. 1982, MIT Press: Cambridge, MA. 549-586. [6] N. Guarino, Understanding, Building, and Using Ontologies. Intl. Journal of Human and Computer Studies, 1997. 46(2/3):293-310. [7] F. Fonseca, et al., Using Ontologies for Integrated Geographic Information Systems. Transactions in GIS, 2002. 6(3):231-257. [8] A. Frank, Tiers of ontology and consistency constraints in geographical information systems. Intl. Journal of Geographical Information Science, 2001. 15(7):667-678.


[9] J. Mennis, D. Peuquet, and D. Guo. Model for The Exploration Of Spatio-Temporal Environmental Data. Proc. 4th Intl. Conference on Integrating GIS and Environmental Modeling (GIS/EM4): Problems, Prospects, and Research Needs. 2000. Banff, Alberta, Canada. [10] J. Mennis and D. Peuquet, The Role of Knowledge Representation in Geographic Knowledge Discovery: A Case Study. Transactions on GIS, 2003. 7(3):371-391. [11] G. Grinstein and S. Smith. The perceptualization of scientific data. in Extracting meaning from complex data: Processing, display, interaction. 1990. Santa Clara, CA: SPIE - The Intl. Society for Optical Engineering. [12] M. Gahegan and D. O'Brien, A strategy and architecture for the visualization of complex geographical datasets. Intl. Journal of Pattern Recognition and Artificial Intelligence, 1997. 11(2):239-261. [13] C. Healey and A. Enns. Building perceptual textures to visualize multidimensional datasets. in IEEE Visualization 1998. October 18-23, Research Triangle Par, NC: IEEE Computer Society Technical Committee on Computer Graphics. [14] E. Wegman and Q. Luo, High Dimensional Clustering Using Parallel Coordinates and the Grand Tour. Computing Science and Statistics, 1997. 28:352-360. [15] N. Andrienko, et al., Exploratory Analysis of Spatial Data Using Interactive Maps and Data Mining. Cartography & Geographic Information Science, 2001. 28(3). [16] S. Havre, B. Hetzler, and L. Nowell. ThemeRiver: Visualizing Theme Changes over Time. in Proc. IEEE Visualization 2000. 2000. Salt Lake City, Utah, Oct 8 - Oct 13, 2000.: IEEE. [17] K. Börner, C. Chen, and K.W. Boyack, Visualizing knowledge domains. Annual Review of Information Science & Technology, 2003. 37(5):179-255. [18] A. Fredrikson, et al. Temporal, geographical and categorical aggregations viewed through coordinated displays: a case study with highway incident data. in Proc. of the workshop on new paradigms in information visualization and manipulation. Kansas City, MO, Nov. 2-6, 1999. [19] C. North and B. Shneiderman, Snap-Together Visualization: Can Users Construct and Operate Coordinated Views? Intl. Journal of Human-Computer Studies, 2000. 53(5):715-739. [20] A. MacEachren, M. Gahegan, and W. Pike. Visualization for Constructing and Sharing Concepts. in National Academy of Science, invited paper for Sackler Colloquium on Mapping Knowledge Domains. 2003. Irvine, CA. [21] R. Becker and W. Cleveland, Brushing Scatterplots. Technometrics, 1987. 29:127-142. [22] C. Stolte and P. Hanrahan. Polaris: A system for query, analysis and visualization of multi-dimensional relational databases. in IEEE Information Visualization Symposium. 2000. Salt Lake City Utah, Oct. 9-10, 2000: IEEE. [23] J. Roberts, On Encouraging Coupled Views for Visualization Exploration. in Visual Data Exploration and Analysis VI, Proc. of SPIE. 1999: SPIE. [24] N. Boukhelifa and P. Rodgers, A model and software system for coordinated and multiple views in exploratory visualization. Information Visualization, 2003. 2(4):258-269. [25] F. Hardisty, Strategies for Designing Coordinated Geographic Visualization Software for Enumerated Data: A Component-Based Approach, in Department of Geography. 2003, Pennsylvania State University: University Park, PA. [26] F. Hughes, The science of judicial proof and the theory of argumentation: Foundations for the knowledge-creating discipline of intelligence analysis. 2004.

Documents

STNexus: An Integrated Database and Visualization ...weaver/academic/... · Visual and Computational Requirements For information analysis, visualization tools must accomplish four