View
216
Download
2
Category
Tags:
Preview:
Citation preview
A 2D Flow Visualization User StudyUsing
Explicit Flow Synthesis and Implicit Task Design
VisWeek 2011
Zhanping Liu
Shangshu Cai
J. Edward Swan II
Robert J. Moorhead II
Joel P. Martin
T. J. Jankun-Kelly
University of PennsylvaniaKentucky State University
University of Californiaat Santa Barbara
Mississippi State University
Mississippi State University
Lockheed Martin Corp.Army Research Lab
Mississippi State University
IEEE TVCG
Outline
Explicit Flow Synthesis
Diverse Evaluation Aspects
Implicit Task Design
VIS 2011
Experimental Components
Synthetic Flow Datasets
Flow Visualization Techniques
Flow Analysis Tasks
Brief Introduction
Test Results
Test Strategy
Concluding Remarks
Flow Representation
Geometry-based / glyph-based
Texture-based / image-based
arrow plots, streamlines, pathlines, streak lines, time lines
stream ribbons, stream tubes, stream surfaces, streak surfaces, ……
graphical primitives rendered for a sparse or discrete representation
good survey by McLoughlin et al (EuroGraphics 09)
topology-based methods use graphical primitives for a representation
spot noise, LIC, UFLIC, LEA, IBFV, IBFVS, ISA, UFAC, ……
texture convolution / advection for a dense continuous representation
good survey by Laramee et al (Computer Graphics Forum 04)
VIS 2011
Brief Introduction
Flow Visualization User Study
NIH-NSF report on Visualization Research Challenges (Johnson etc 06)
different techniques may be advantageous in different aspects
only a few have been evaluated to determine their effectiveness
the best methods might not have been integrated into vis. systems
domain scientists may not yet have access to cutting-edge techniques
insufficient user feedback for visualization researchers and developers
more user studies are needed to
examine flow representations
improve existing techniques
design innovative techniques
VIS 2011
bridge the long-lasting gaps
between research, development, and deployment
Brief Introduction
Flow Visualization User Study
VIS 2011
Previous work 2D flow visualization user study (Laidlaw et al, TVCG 05) 3D flow visualization user study (Forsberg et al, TVCG 09) …… insufficient research on effective user study methodologies
Brief Introduction
Flow Visualization User Study
VIS 2011
given a user study framework or platform
for evaluating flow visualization techniquesdistorted by various bias issues
the data collected and the analysisresults are distorted too, failingto provide objective conclusions
flow visualization techniques
Previous work
Brief Introduction
Flow Visualization User Study
Previous work 2D flow visualization user study (Laidlaw et al, TVCG 05) 3D flow visualization user study (Forsberg et al, TVCG 09) …… insufficient research on effective user study methodologies
VIS 2011
There is more to a flow visualization user study than the scenarios being considered
the techniques being evaluated
the flow features being examined
the specific yet usually ad-hoc conclusions being drawn
Brief Introduction
e.g., surface flows, volume flows, time-varying flows, ……
e.g., UFLIC, LEA, IBFV, IBFVS, ISA, UFAC, ……
e.g., separation, attachment, vortex core, periodic orbit, ……
Flow Visualization User Study
Conducting objective 2D flow visualization user studies even with traditional and well-known techniques
remains an open problem
requires valid methodologies — an anti-bias platform
refines our understanding of some 2D flow vis. techniques
offers quantitative support for qualitative evidence or anecdotal
advice in terms of the effectiveness of flow vis. techniques
VIS 2011
Brief Introduction
that is necessary for
carrying out convincing flow visualization user studies
with more complex configurations
helps formulate a general framework
VIS 2011
Brief Introduction
Our 2D Flow Visualization User Study — FlowVUS motivated by the necessity for and significance of effective flow visualization user study methodologies builds on Laidlaw et al’s work features new strategies and important improvements
explicit flow synthesis
implicit task design
flow data bias
task design bias
VIS 2011
Brief Introduction
Our 2D Flow Visualization User Study — FlowVUS
given a user study framework or platform
for evaluating flow visualization techniquesequipped with anti-bias methodologies
the data collected and the analysisresults are convincing, leading toa better understanding of techniques
flow visualization techniques
VIS 2011
Brief Introduction
Our 2D Flow Visualization User Study — FlowVUS
Major contributions explicit flow synthesis combats data-related bias by automatically generating many flows with similar topological complexities but with different structures
implicit task design reduces task-related bias by designing sample-free pattern-based flow analysis tasks that require thorough investigation of the flow direction diverse evaluation perspectives involve representation continuity, visual intuition, image contrast, and color mapping when selecting a set of representative vis. techniques
hybrid timing strategy uses two timing schemes (fixed duration / variable duration) to help reveal subtle differences in vis. effectiveness between techniques
refined statistical analysis processes outliers + Ryan REGWQ post-hoc homogeneous subset tests
VIS 2011
Experimental Components
SyntheticFlow
Datasets
FlowVisualizationTechniques
2D Flow Visualization User Study Pipeline
FlowAnalysis
Tasks
SyntheticFlow
Datasets
FlowVisualizationTechniques
FlowAnalysis
Tasks
three fundamentalcomponents of atypical flow vis.
user study
VIS 2011
Experimental Components
Synthetic Flow Datasets
Previous work on flow vis. user study uses implicit flow synthesis samples randomly selected and the associated vectors randomly assigned
a flow field is generated by vector interpolation between the samples
the topology of the resulting flow is unpredictable
— number of critical points, the locations, the types & overall complexity
A single dataset would introduce learning effect — unacceptable
Synthetic datasets are used for user study in medical imaging
Multiple datasets may incur data-dependent bias (in flow complexity)
Data-dependent bias can be suppressed to an acceptable degree
by synthesizing flows with similar topological complexities
implicit flow synthesis
VIS 2011
Experimental Components
Synthetic Flow Datasets
employs parameterized placement and configuration of critical points provides great flexibility and control in creating pseudo flow fields Basis Vector Field (BVF) flow synthesis method by van Wijk (TOG 02)
— a BVF is governed by a critical point with some parameters
— the entire flow results from the combination of multiple BVFs a survey and initialization-analysis-editing by Zhang et al (TOG 06)
Explicit Flow Synthesis
FlowVUS BVF FlowVUS is the first user study to value and apply explicit flow synthesis based on BVF for fast automatic generation of many synthetic flows centers and foci — Explicitly Specified Critical Points (ESCPs) saddles — derived from the interaction among centers and foci uses a force composition-attenuation method to govern the influence of
an
ESCP (with rotational force and radial force) or a BVF on an arbitrary point
VIS 2011
Experimental Components
Synthetic Flow Datasets
VIS 2011
Experimental Components
Synthetic Flow Datasets
Layout templates to synthesize flows with diverse structures yet with a relatively balanced layout of a fixed number of ESCPs + a slightly varying number of saddles
— to maintain nearly the same topological complexity between many flows
a primary ESCP is randomly placed & configured in each blue block and its mirror ESCP is placed based on a symmetry type yet with the sink / source type & clockwise/counter-clockwise orientation possibly different
— they may be geometrically symmetric but topologically asymmetric
location radial force rotational force
ESCP parameters force attenuation sink / source type clockwise / counter-clockwise orientation
to generate x- / y- / center-symmetric and dubiously asymmetric flows
— so as to support our pattern-based implicit flow analysis task design
4 pairs of x-symmetricESCP placement blocks
4 pairs of y-symmetricESCP placement blocks
4 pairs of center-symmetricESCP placement blocks
blue block: for primary ESCP placement; gray block: for mirror ESCP placement
VIS 2011
Experimental Components
Synthetic Flow Datasets
symmetric flows versus asymmetric flows
asymmetric x-symmetric asymmetric
center-symmetric asymmetric y-symmetric
VIS 2011
Experimental Components
Flow Visualization Techniques
direction — the positive and negative directions tangent to the flow orientation — the positive direction of the flow only (e.g., oriented LIC) velocity magnitude — a scalar quantity
Primitive flow characteristics
The most important a vector quantity providing the fundamental info that distinguishes a flow field from a scalar field and hence governs why / how flow visualization differs very much from scalar visualization in the working mechanism how well a flow vis. technique delineates the general, directional info largely determines its effectiveness in conveying specific flow features
An informal classification
Direct Feature-Extraction Based (DFEB) — e.g., topology extraction Indirect User-Exploration Based ( IUEB) — e.g., flow lines and LIC
many flow features (e.g., critical points) visually recognizable from them
— direction
from the flow reconstruction or visual analysis perspective
our focus
VIS 2011
Experimental Components
Flow Visualization Techniques
need more user studies than DFEB techniques do due to the human factors
user exploration visual analysis mental reconstruction
IUEB techniques
54 candidates — 3 families hedgehogs streamlines LIC
selected through a thorough intra- and inter-family investigation representative of many geometry-based and texture-based techniques
in terms of the aforementioned four major visual / evaluation aspects configured via iterative internal tests for optimal visualization results
7 techniques
involve several major visual factors
representation continuity (e.g., 0D / 1.5D / 2D) visual intuition
image contrast color mapping
— FlowVUS evaluation aspects
VIS 2011
Experimental Components
Flow Visualization Techniques
ArrowCM ArrowCW StreamCM
StreamCWBasicLICEnhancedLIC
OrientedLIC
VIS 2011
Experimental Components
Flow Analysis Tasks
impossible & unnecessary to enumerate specific / complex flow features and then design many flow analysis tasks (how many studies are enough?)
Some essential points
in order to reduce task-related bias, flow analysis tasks may take an indirect / implicit way
and a testable form
the performance of an average participant in visual flow analysis is expected to reflect the effectiveness of the IUEB technique (being used) in conveying the flow direction — the general fundamental information flow analysis tasks in a user study are not necessarily real or practical
flow analysis tasks are the way instead of (or at least more than) the goalfor example, synthetic tasks are often used for psychological user studiesby devising some seemingly irrelevant yet intrinsically coupled questions
(— do not directly ask the user to check the flow direction at a point)
(— questions are easy to understand but challenging to answer correctly)
VIS 2011
Experimental Components
Flow Analysis Tasks
used in previous work and susceptible to bias a typical example — directly ask to check the flow direction at a point the participant is shown a randomly placed circle (of which the center is hence a random sample) and asked to click on the point along the circle that a particle advected from the center is to hit
Explicit sample-based tasks
a methodology advocated and formulated in this paper to suppress bias use a simple form but indirectly require thorough investigation of the flow
Implicit pattern-based tasks
mouse pointing & clicking, irrelevant of judgment, affect the test result
the complexity of a flow usually varies with the location more difficult to do this task in turbulent areas than in laminar areas
the selection of the circle’s radius may further compound this issue
critical point recognition — detect patterns globally/across the whole domain critical point classification — match patterns locally/around an area of interest
VIS 2011
Experimental Components
Flow Analysis Tasks Implicit task design
to relieve non-expert participants from understanding complex, possibly domain-specific details in the form of easy-to-understand yet difficult-to-answer questions requiring intensive analysis of flow directions
using specific real tasks about well-known flow features critical point recognition (CPR) critical point classification (CPC) involving in-depth flow structures identification of separatrices identification of periodic orbits creating general synthetic tasks to reduce data-related bias resulting from flow sampling and mouse point-and-click operations
such as symmetric pattern categorization (SPC) — to examine the flow direction both globally and locally — to check the entire pattern: x-/y-/z-/center-symmetric or asymmetric
VIS 2011
Experimental Components
Flow Analysis Tasks Very challenging synthetic tasks
two or three critical points (centers, foci, and saddles) combined with
a variety of configurations to define some Composite Templates (CTs)
CT-based CPR-like pattern recognition
CT-based CPC-like pattern classification
checking if flow A and flow B have a CT pattern in common
judging if flow A is a rotational version of flow B
determining if flow A is exactly part of flow B
The selected implicit tasks CPR + CPC + SPC
integration of 2 real tasks and 1 synthetic task to demonstrate the types
the balance between the overall challenge degree and the test duration
— some synthetic tasks mentioned above would require more test time
VIS 2011
The Input
Test Strategy
7M images generated using the selected 7 techniques to visualize M synthetic flows
involving N x-symmetric, N y-symmetric, N center-symmetric, and
optionally N asymmetric flows — M = 3N or 4N (e.g., N = 30)
depending on the expected complexity and time duration of the test
Ground truth — one record per synthetic flow symmetry type of the overall pattern
the location and type of every ESCP (center / focus) from the synthesizer
the location of every derived saddle from Newton-Raphson root-finding
Task Session 1 CPR task (recognizing ALL critical points from an image) or
<= 30 CPC tasks or
<= 30 (without asymmetric flows) / 40 (with asymmetric flows) SPC tasks
VIS 2011
Task Management
Test Strategy
1 set = (1 CPR session + 1 CPC session + 1 SPC session) for one technique
1 cycle = 7 sets (one for each technique)
1 test = 3 cycles
1 session = 1 CPR task or (<= 30) CPC tasks or (<= 30/40) SPC tasks
= 21 sets = 63 sessions for each participant use 7 techniques thrice to produce 7 × 3 = 21 images (for 21 randomly- selected flows), with 1 image for each CPC session (3 per technique)
use each technique to produce 30 images (for 30 randomly-selected flows), with 1 randomly-selected critical point marked per image (with 10 marked for each critical point type: center, focus, saddle), for each CPC session
use each technique to visualize 30 or 40 randomly-selected flows (creating 10 images for each symmetry / asymmetric type) for each SPC session
21 CPR sessions + 21 CPC sessions + 21 SPC sessions = 63 sessions with a bank of images pre-generated for one time, 63 sessions are created using TestGen upon each test and are then delivered in random order
VIS 2011
Hybrid Timing
Test Strategy
Effectiveness metrics the effectiveness of a visualization technique is usually reflected by
answer correctness and response time a more effective technique allows the user to get a correct answer faster given a fixed amount of time, more correct answers tend to result from a more effective technique than from a less effective technique
Variable-duration session mouse click positions and response time are recorded for a session flow analysis (for recognizing a single critical point) is relatively quick the answer is precision-critical (despite a considerable error tolerance) seeks to “curb” the participant from hastiness and excessive inaccuracy
Fixed-duration session as many tasks as possible are presented to the participant one by one in a fixed amount of time (30s) and radio-button choices are recorded flow analysis is relatively slow and judgment-intensive intended to “push” the participant to accomplish more tasks
— for CPR
— for CPC & SPC (response time on average)
this hybrid timing strategy helps reveal the subtle differences
in visualization effectiveness that may exist between techniques
VIS 2011
Test Strategy
CPR — Critical Point Recognition
VIS 2011
Test Strategy
CPC — Critical Point Classification
VIS 2011
Test Strategy
SPC — Symmetric Pattern Categorization
VIS 2011
Test Results
Basic Facts 4 CFD experts + 16 graduate students in science & engineering expert and non-expert participants were not compared herein 5079 CPR trials + 7467 CPC trials + 4948 SPR trials were recorded
Processing Outliers the response time and the (CPR) location error each showed a skewed normal distribution in terms of the histogram outliers were determined case by case by investigating the tails of the distributions and noting values after conspicuous gaps each outlier was replaced with the median of the cell’s responses
the absolute differences in response time for CPR / CPC / SPC
turned out to be small, regardless of the statistical differences a higher priority assigned to correctness than to response speed to provide correctness-over-response-sorting (CORS) when evaluating the seven techniques in the overall visualization effectiveness
VIS 2011
Test Results
Statistical Analysis Chi-square tests and ANOVA (univariate analysis of variance) calculating post-hoc homogeneous subsets using Ryan REGWQ tests
FlowVUS Results CPR (Critical Point Recognition) — response time
mean time (in seconds) to recognize a critical point (5079 trials, F(6,115.3) = 19.9, p < 0.001)
means with the same letter are not significantly different at p 0.05 (Ryan REGWQ post-hoc hst)
VIS 2011
Test Results
FlowVUS Results CPR (Critical Point Recognition) — answer incorrectness
CORS sorting by CPR effectiveness in decreasing orderEnhancedLIC - StreamCM - BasicLIC - OrientedLIC - StreamCW -
ArrowCM - ArrowCW
336 errors, χ2(6) = 132, p < 0.001
VIS 2011
Test Results
FlowVUS Results CPC (Critical Point Classification) — response time
mean time (in seconds) to classify a critical point (7467 trials, F(6,116.2) = 30.9, p < 0.001)
means with the same letter are not significantly different at p 0.05 (Ryan REGWQ post-hoc hst)
VIS 2011
Test Results
FlowVUS Results CPC (Critical Point Classification) — answer incorrectness
CORS sorting by CPC effectiveness in decreasing orderEnhancedLIC - StreamCW - StreamCM - BasicLIC - OrientedLIC -
ArrowCW - ArrowCM
753 errors, χ2(6) = 772, p < 0.001
VIS 2011
Test Results
FlowVUS Results SPC (Symmetric Pattern Categorization) — response time
mean time (in sec.s) to categorize a symmetric pattern (4948 trials, F(6,123.1) = 8.74, p < 0.001)
means with the same letter are not significantly different at p 0.05 (Ryan REGWQ post-hoc hst)
VIS 2011
Test Results
FlowVUS Results SPC (Symmetric Pattern Categorization) — answer incorrectness
CORS sorting by SPC effectiveness in decreasing orderEnhancedLIC - StreamCM - BasicLIC - OrientedLIC - StreamCW –
ArrowCM - ArrowCW
323 errors, χ2(6) = 70.1, p < 0.001
VIS 2011
Test Results
CORS Sorting byCPR effectiveness
CORS Sorting byCPC effectiveness
CORS Sorting bySPC effectiveness
EnhancedLIC EnhancedLIC EnhancedLIC
StreamCM StreamCW StreamCM
BasicLIC StreamCM BasicLIC
OrientedLIC BasicLIC OrientedLIC
StreamCW OrientedLIC StreamCW
ArrowCM ArrowCW ArrowCM
ArrowCW ArrowCM ArrowCW a texture-based dense representation with accentuated flow streaks (EnhancedLIC) enables intuitive perception of the flow a geometry-based integral representation with uniform density control (StreamCM or StreamCW) exploits visual interpolation to facilitate mental reconstruction of the flow color mapping has a considerable influence on a geometry-based flow representation
VIS 2011
Concluding Remarks
Key Points Explicit flow synthesis
Implicit task design
Diverse evaluation perspectives
Hybrid timing strategy
Refined statistical analysis
— to reduce data-related bias template-based parameterized placement & configuration of critical points automatic synthesis of diverse flows with similar topological complexities
— to suppress task-related bias pattern-based (real tasks + synthetic tasks) the way more than the goal
— representative techniques representation continuity visual intuition image contrast color mapping
variable-duration session fixed-duration session to reveal the subtle differences in vis. effectiveness between techniques
processes outliers + Ryan REGWQ post-hoc homogeneous subset tests
— to reduce data-related bias
— to suppress task-related bias
Explicit flow synthesis
Implicit task design
Two important methodologies / concepts proposed as part of our anti-bias framework for conducting objective flow vis. user studies
VIS 2011
Concluding Remarks
Limitations & Lessons FlowVUS is bias-resistant but not bias-free
Varying a-priori familiarity with techniques
Varying a-priori familiarity with flow features
Real flows needed for introducing techniques
Care needed for predicting the time duration
bias is pervasive throughout the whole pipeline of a user study and hence we cannot totally eliminate it while we need to reduce it — cannot let it be
some participants were not familiar with the LICs upon the training session more user studies are needed to disseminate the latest vis. techniques care needs to be taken when evaluating more sophisticated / current ones
some participants needed extra help with some features during the training session many challenges facing an evaluation involving more complex features
synthetic flows are needed for formal tests while real flows (particularly with contextual boundaries) are needed, besides real flows, for the training session
VIS 2011
Concluding Remarks
Future Plans Anti-bias methodologies
user studies might otherwise be non-convincing & even worse misleading probably one way to help you judge between 2 contradicting conclusions as of now more important than scenarios, techniques, features,
conclusions require much research (e.g., explicit flow synthesis & implicit task design)
end users might not care about the underlying working mechanism they are interested in the resulting images and the associated visual aspects (such as image contrast, color map, intuition, continuity, etc)
neither possible nor necessary to evaluate every existing vis. technique
Evaluation aspects — representative visualization techniques
provide general guidelines for visualization research (algorithm design)
Interesting topics user studies on streamline placement algorithms user studies on surface flow visualization techniques user studies on volume flow visualization techniques
— to adopt the conclusions of a user study without necessary anti-bias methods?
controversialview
controversialview
Thank youfor your time and attention!
Recommended