Developing Visualization Techniques for Semantics-based Information Networks Rich Keller David Hall NASA Ames QSS Group, Inc. Information Sharing and Integration

Developing Visualization Techniques for Semantics-based Information Networks

Rich Keller David Hall NASA Ames QSS Group, Inc.

Information Sharing and Integration GroupComputational Sciences Division

NASA Ames Research CenterMoffett Field, CA

Virtual Iron Bird Workshop, Monterey, CA April 2, 2004

Goals of our Work

• Goal #2 (Engineering): Develop an effective visual interface for an existing NASA information / knowledge management tool

• Goal #1 (Scientific): Understand how semantic knowledge can be exploited to help visualize large network-structured information spaces (… like the Semantic Web)

Talk Outline

• SemanticOrganizer System

• Visualization Problem

• Proposed Semantic Approaches to Visualization Problem

Work in progress!

What is SemanticOrganizer?

• A semantics-based shared information space: designed to support distributed science and engineering project teams

• Facilitates information sharing, integration, correlation and dependency tracking

• Core is a digital information repository: users upload & download heterogeneous information (images, datasets, documents, and various types of scientific/engineering records)

• Features semantic cross-linkage: enables rapid intuitive access to interrelated information; permits linking facts and evidential information to scientific/engineering conclusions

• Serves as organizational memory: preserves details of investigative fieldwork, labwork, & associated data collection/ data analysis activities and processes

Operational Status

• First deployed in 2001

• Over 500 registered individual users from over 50 organizations within NASA

• Over 50 projects hosted

• Over 45,000 information nodes & 150,000 links in repository

• Over 14,000 electronic files stored (documents, image, datasets)

• Over 12,000 archived email messages

as of 4/1/04

Application Features

SemanticOrganizer Applications

SemanticOrganizer

ScienceOrganizer InvestigationOrganizer

NASA Astrobiology Institute

Mars Meteorite Research Team

NIH Malaria Study

Shuttle Columbia Investigation

Helios UAV Investigation

CONTOUR Spacecraft Investigation

MARTE Mars Analog Mission Moffett Airshow Investigation

MER Hypothesis Tracking

Mobile Agents Mars Exploration

ScienceOrganizer

• Real-time equipment control• Automated experimentation

Both• Collaborative image annotation

• Microsoft Office macros• Email lists & archive

InvestigationOrganizer

• Fault tree viewer• Event sequence editor

…

How is the Information Repository structured?

person

photo

measurement

siteinstrument

sample

document

• Links: defined relationships among resources

• Attached files: electronic products associated with resources (in almost any type of file format)

• Attributes: properties of resources (metadata)

• Nodes: key science or engineering resources (describing people, places, systems, hypotheses, evidence)

• date• size• format

Ontology:Specifies the types of nodes, attributes and links defined for each different application

(RDFS-type Representation)

Rules:Add/modify nodes, links & attributes in the network

DNA sequenceimage

field trip

culture

personsample

photographic image

SEM image

Scientific Investigation Ontology (partial)

other

experiment

Scientific Information Resouces

project

measurement

field site

equipment

camera

gas chromatograph

stub

O2 microsensor

N2 microsensorSEM

O2 concentration

N2 concentration

spectrometer

spectrograph

chromatogram

other

other

micrograph

cultivated-fromcultivated-by

has-genetic-sequence

pictured-in

researcher

lab tech

May-June 2001 Field Trip

BajaStudy Area

In Situ Diel6-4-01

Experiment

Measurements7

Images99

Documents3

EMERGProject

People15

Samples19

Brad’s Trip Planning

Document

ac

df

gh

i

j

a: has logisticsb: samples collected c: has objectivesd: conducted experimente: located atf: trip participantsg: destinationh: trip fori: measurements takenj: has photodocumentation

k: site forl: collected atm: experiment site forn: experiment staffo: has custodianp: pictured inq: has sequence infor: source ofs: has measurement

Links

b

Strawman for Focus group

Document

aa

t: employed in u: collected byv: led byw: authored byx: has subexperimentsy: has measurement passz: generated measurementsaa: associated documentsbb: has test point

Example Semantic Information Space

Pond 4 near 5Field Site

Projects2

Samples56

e k

l7

Experiments6

m

t x

ybb

Greenhouse Sulfate ManipulationExperiment

Test Point18

Experiments3

O2 Measurement

36

z

Thermal VarianceExperiment

Experiments2

x

SalinityExperiment

Diel CycleExperiment

MeasurementPass

2

bbs

s

People5

n

v

M13791-3Measurement

SC-8-11Culture

16S3 rRNADNA Sequence

Bebout, BradScientist

Carpenter, SteveLab Technician

o

r

qs

q

u

w

P4Mat-16Mat Sample

Images33

p

8

Instance space

Current SemanticOrganizer Interface

Links to Related

Records

create new records

modify recordicon identifies

record type

search for records

Right side displays metadata for the current repository record being inspected

Left side uses semantic links

to display all information

related to the repository

record shown on the right

semantic links

related records (click to

navigate)

current record

Interface Problems

• Graphical overview of information space needed for:

– Comprehension of information scope and context

– Non-local navigation

• Can’t display entire information space

– Over 45,000 nodes

– Over 150,000 links

• Can’t make sense of entire information space

Remedy: Filtering and Abstraction

• Filtering: Remove nodes/links

• Abstraction: Replace a set of nodes/links with a smaller number of nodes/links

Q: What is the basis for filtering or abstraction?

Sources of Knowledge for Filtering/Abstraction

• Graph-theoretic: based on topological properties of network (e.g. cut points)

• Content-based: using textual content stored in nodes (e.g., as in Web page clustering)

• Semantics-based:

– ontology (node-type, link-type, subsumption)

– auxiliary information:

• importance/intrinsicality of nodes/links

• usage context

Semantic Approaches to Simplifying Information Space Presentation

1. Contextual Filtering

2. Semantic Structure Abstraction

3. Semantic Navigation

1. Contextual Filtering

Observation: Not all nodes/links are relevant in a given context

Proposed Approach: Define explicit constraints that generate a meaningful subgraph of nodes in a specific context

Context examples:

• a specific scientific field trip

• a specific project

• a specific location (e.g., a scientific laboratory or field site)

Example: Using Constraints to define a Field Trip Context

FieldTripContext(f) = { {f} S M SA P R E FS I }

where: FieldTrip(f) // f is a node of type FieldTripS = {s | Sample(s) ∧ SamplesCollected(f, s)}M = {m | Measurement(m) ∧ MeasurementTaken(f, m)}SA = {sa | StudyArea(sa) ∧ Destination(f, sa)}P = {p | Person(p) ∧ TripParticipant(f, p)}R = {r | Project(r) ∧ TripFor(f, r)}E = {e | Experiment(e) ∧ ConductedExperiment(f, e)}FS = {fs | FieldSite(fs) ∧ CollectedAt(fs, s) ∧ sS}I = {i | Image(i) ∧ PicturedIn(s, i) ∧ sS}

a) subset of nodes linked directly to a field trip node+

b) images of samples gathered during that trip and field sites where those samples were collected

Field Trip Context =

a

b

Samples19May-June 2001

Field Trip

BajaStudy Area

In Situ Diel6-4-01

Experiment

Measurements7

EMERGProject

People15

P4Mat-16Mat Sample

Images33

M13791-3Measurement

Pond 4 near 5Field Site

a

df

gh

i

b

Results of Applying Field Trip Filter

p

s

e

Projects2

Experiments6

m

Samples56

aa

c

j

Images99

Documents3

Brad’s Trip Planning

Document

Strawman for Focus group

Document

8

u

SC-8-11Culture

16S3 rRNADNA Sequence

Bebout, BradScientist

Carpenter, SteveLab Technician

o

r

q

q

t x

ybb


Test Point18

Experiments3

O2 Measurement

36

z


Experiments2

x

SalinityExperiment


MeasurementPass

2

bbs

People5

n

v

8

w

k

l7

2. Semantic Structure Abstraction

Proposed Approach:

• apply semantic patterns to identify these substructures

• represent them as abstract nodes

• display them using familiar representation

Observation: Graphs can obscure structure! Certain graph substructures are better depicted using more familiar visual representations

• Hierarchical structures trees

• List structures arrays

• Cross-correlated structures tables

• Time sequences PERT charts

Semantic Structure Abstraction: Approach

x

ybb


Test Point5

Experiments3

O2 Measurement

10


Experiments2

x

SalinityExperiment


MeasurementPass

2

bbs

Gas FluxExperiment

Peak CycleExperiment

1. Recognize patterns


Test Pointx Measurement

x Pass

2. Represent as abstract nodes

Greenhouse Sulfate Manipulation

SalinityThermal Variance

Diel Cycle

Gas Flux

Peak cycle

3. Display appropriately

B C D E

Pass 1 2

m8m1 m2 m3 m4 m5

Test Point A

m6 m7 m9 m10

O2 Measurement{

Experiment Hierarchy

2-dimensional measurementindexing structure

3. Semantic Navigation

Proposed Approach:

• Move from current detailed, fine-grained interface to more abstract navigation interface

• Abstract away the specific links and present only clusters of nodes radiating out from a focal node

• Use a semantics-based focus+context style display (e.g., fisheye, hyperbolic)

Observation: High-level semantic categories in an ontology can help users visualize and navigate the information space in a more effective, rapid, intuitive fashion

More Abstract Interface:Bull’s-Eye Navigator

Related 2 nd

order nodes

Related 1 st

order nodes

Focal Node

Artifacts Activities

People Social Places

Activities

Places

Social

Artifacts

People

fieldtrip

artifacts relatedto “field trip”(e.g., sample-X)people related

to “field trip” artifacts (e.g., labtech who analyzed sample-X)

scientistslab techs

traverseexpts …samples, msmts

…

projectsorgs …

labssites …

(1 link away)

(2 links away)

Focal Region

Context Region

Compactrepresentation ofinformation space surroundingfocal node

docs

Semantic categories:• People• Places• Activities• Artifacts• Social Structures

Summary

• Large information spaces are difficult to comprehend and navigate

• Visualization can help

• Semantic information provides leverage for visualization

• Three examples:

– Contextual Filtering

– Semantic Structure Abstraction

– Semantic Navigation