A 2D Flow Visualization User StudyUsing

Explicit Flow Synthesis and Implicit Task Design

VisWeek 2011

Zhanping Liu

Shangshu Cai

J. Edward Swan II

Robert J. Moorhead II

Joel P. Martin

T. J. Jankun-Kelly

University of PennsylvaniaKentucky State University

University of Californiaat Santa Barbara

Mississippi State University

Mississippi State University

Lockheed Martin Corp.Army Research Lab

Mississippi State University

IEEE TVCG

Outline

Explicit Flow Synthesis

Diverse Evaluation Aspects

Implicit Task Design

VIS 2011

Experimental Components

Synthetic Flow Datasets

Flow Visualization Techniques

Flow Analysis Tasks

Brief Introduction

Test Results

Test Strategy

Concluding Remarks

Flow Representation

Geometry-based / glyph-based

Texture-based / image-based

arrow plots, streamlines, pathlines, streak lines, time lines

stream ribbons, stream tubes, stream surfaces, streak surfaces, ……

graphical primitives rendered for a sparse or discrete representation

good survey by McLoughlin et al (EuroGraphics 09)

topology-based methods use graphical primitives for a representation

spot noise, LIC, UFLIC, LEA, IBFV, IBFVS, ISA, UFAC, ……

texture convolution / advection for a dense continuous representation

good survey by Laramee et al (Computer Graphics Forum 04)

VIS 2011

Brief Introduction

Flow Visualization User Study

NIH-NSF report on Visualization Research Challenges (Johnson etc 06)

different techniques may be advantageous in different aspects

only a few have been evaluated to determine their effectiveness

the best methods might not have been integrated into vis. systems

domain scientists may not yet have access to cutting-edge techniques

insufficient user feedback for visualization researchers and developers

more user studies are needed to

examine flow representations

improve existing techniques

design innovative techniques

VIS 2011

bridge the long-lasting gaps

between research, development, and deployment

Brief Introduction

Flow Visualization User Study

VIS 2011

Previous work 2D flow visualization user study (Laidlaw et al, TVCG 05) 3D flow visualization user study (Forsberg et al, TVCG 09) …… insufficient research on effective user study methodologies

Brief Introduction

Flow Visualization User Study

VIS 2011

given a user study framework or platform

for evaluating flow visualization techniquesdistorted by various bias issues

the data collected and the analysisresults are distorted too, failingto provide objective conclusions

flow visualization techniques

Previous work

Brief Introduction

Flow Visualization User Study

Previous work 2D flow visualization user study (Laidlaw et al, TVCG 05) 3D flow visualization user study (Forsberg et al, TVCG 09) …… insufficient research on effective user study methodologies

VIS 2011

There is more to a flow visualization user study than the scenarios being considered

the techniques being evaluated

the flow features being examined

the specific yet usually ad-hoc conclusions being drawn

Brief Introduction

e.g., surface flows, volume flows, time-varying flows, ……

e.g., UFLIC, LEA, IBFV, IBFVS, ISA, UFAC, ……

e.g., separation, attachment, vortex core, periodic orbit, ……

Flow Visualization User Study

Conducting objective 2D flow visualization user studies even with traditional and well-known techniques

remains an open problem

requires valid methodologies — an anti-bias platform

refines our understanding of some 2D flow vis. techniques

offers quantitative support for qualitative evidence or anecdotal

advice in terms of the effectiveness of flow vis. techniques

VIS 2011

Brief Introduction

that is necessary for

carrying out convincing flow visualization user studies

with more complex configurations

helps formulate a general framework

VIS 2011

Brief Introduction

Our 2D Flow Visualization User Study — FlowVUS motivated by the necessity for and significance of effective flow visualization user study methodologies builds on Laidlaw et al’s work features new strategies and important improvements

explicit flow synthesis

implicit task design

flow data bias

task design bias

VIS 2011

Brief Introduction

Our 2D Flow Visualization User Study — FlowVUS

given a user study framework or platform

for evaluating flow visualization techniquesequipped with anti-bias methodologies

the data collected and the analysisresults are convincing, leading toa better understanding of techniques

flow visualization techniques

VIS 2011

Brief Introduction

Our 2D Flow Visualization User Study — FlowVUS

Major contributions explicit flow synthesis combats data-related bias by automatically generating many flows with similar topological complexities but with different structures

implicit task design reduces task-related bias by designing sample-free pattern-based flow analysis tasks that require thorough investigation of the flow direction diverse evaluation perspectives involve representation continuity, visual intuition, image contrast, and color mapping when selecting a set of representative vis. techniques

hybrid timing strategy uses two timing schemes (fixed duration / variable duration) to help reveal subtle differences in vis. effectiveness between techniques

refined statistical analysis processes outliers + Ryan REGWQ post-hoc homogeneous subset tests

VIS 2011

Experimental Components

SyntheticFlow

Datasets

FlowVisualizationTechniques

2D Flow Visualization User Study Pipeline

FlowAnalysis

Tasks

SyntheticFlow

Datasets

FlowVisualizationTechniques

FlowAnalysis

Tasks

three fundamentalcomponents of atypical flow vis.

user study

VIS 2011

Experimental Components

Synthetic Flow Datasets

Previous work on flow vis. user study uses implicit flow synthesis samples randomly selected and the associated vectors randomly assigned

a flow field is generated by vector interpolation between the samples

the topology of the resulting flow is unpredictable

— number of critical points, the locations, the types & overall complexity

A single dataset would introduce learning effect — unacceptable

Synthetic datasets are used for user study in medical imaging

Multiple datasets may incur data-dependent bias (in flow complexity)

Data-dependent bias can be suppressed to an acceptable degree

by synthesizing flows with similar topological complexities

implicit flow synthesis

VIS 2011

Experimental Components

Synthetic Flow Datasets

employs parameterized placement and configuration of critical points provides great flexibility and control in creating pseudo flow fields Basis Vector Field (BVF) flow synthesis method by van Wijk (TOG 02)

— a BVF is governed by a critical point with some parameters

— the entire flow results from the combination of multiple BVFs a survey and initialization-analysis-editing by Zhang et al (TOG 06)

Explicit Flow Synthesis

FlowVUS BVF FlowVUS is the first user study to value and apply explicit flow synthesis based on BVF for fast automatic generation of many synthetic flows centers and foci — Explicitly Specified Critical Points (ESCPs) saddles — derived from the interaction among centers and foci uses a force composition-attenuation method to govern the influence of

an

ESCP (with rotational force and radial force) or a BVF on an arbitrary point

VIS 2011

Experimental Components

Synthetic Flow Datasets

VIS 2011

Experimental Components

Synthetic Flow Datasets

Layout templates to synthesize flows with diverse structures yet with a relatively balanced layout of a fixed number of ESCPs + a slightly varying number of saddles

— to maintain nearly the same topological complexity between many flows

a primary ESCP is randomly placed & configured in each blue block and its mirror ESCP is placed based on a symmetry type yet with the sink / source type & clockwise/counter-clockwise orientation possibly different

— they may be geometrically symmetric but topologically asymmetric

location radial force rotational force

ESCP parameters force attenuation sink / source type clockwise / counter-clockwise orientation

to generate x- / y- / center-symmetric and dubiously asymmetric flows

— so as to support our pattern-based implicit flow analysis task design

4 pairs of x-symmetricESCP placement blocks

4 pairs of y-symmetricESCP placement blocks

4 pairs of center-symmetricESCP placement blocks

blue block: for primary ESCP placement; gray block: for mirror ESCP placement

VIS 2011

Experimental Components

Synthetic Flow Datasets

symmetric flows versus asymmetric flows

asymmetric x-symmetric asymmetric

center-symmetric asymmetric y-symmetric

VIS 2011

Experimental Components

Flow Visualization Techniques

direction — the positive and negative directions tangent to the flow orientation — the positive direction of the flow only (e.g., oriented LIC) velocity magnitude — a scalar quantity

Primitive flow characteristics

The most important a vector quantity providing the fundamental info that distinguishes a flow field from a scalar field and hence governs why / how flow visualization differs very much from scalar visualization in the working mechanism how well a flow vis. technique delineates the general, directional info largely determines its effectiveness in conveying specific flow features

An informal classification

Direct Feature-Extraction Based (DFEB) — e.g., topology extraction Indirect User-Exploration Based ( IUEB) — e.g., flow lines and LIC

many flow features (e.g., critical points) visually recognizable from them

— direction

from the flow reconstruction or visual analysis perspective

our focus

VIS 2011

Experimental Components

Flow Visualization Techniques

need more user studies than DFEB techniques do due to the human factors

user exploration visual analysis mental reconstruction

IUEB techniques

54 candidates — 3 families hedgehogs streamlines LIC

selected through a thorough intra- and inter-family investigation representative of many geometry-based and texture-based techniques

in terms of the aforementioned four major visual / evaluation aspects configured via iterative internal tests for optimal visualization results

7 techniques

involve several major visual factors

representation continuity (e.g., 0D / 1.5D / 2D) visual intuition

image contrast color mapping

— FlowVUS evaluation aspects

VIS 2011

Experimental Components

Flow Visualization Techniques

ArrowCM ArrowCW StreamCM

StreamCWBasicLICEnhancedLIC

OrientedLIC

VIS 2011

Experimental Components

Flow Analysis Tasks

impossible & unnecessary to enumerate specific / complex flow features and then design many flow analysis tasks (how many studies are enough?)

Some essential points

in order to reduce task-related bias, flow analysis tasks may take an indirect / implicit way

and a testable form

the performance of an average participant in visual flow analysis is expected to reflect the effectiveness of the IUEB technique (being used) in conveying the flow direction — the general fundamental information flow analysis tasks in a user study are not necessarily real or practical

flow analysis tasks are the way instead of (or at least more than) the goalfor example, synthetic tasks are often used for psychological user studiesby devising some seemingly irrelevant yet intrinsically coupled questions

(— do not directly ask the user to check the flow direction at a point)

(— questions are easy to understand but challenging to answer correctly)

VIS 2011

Experimental Components

Flow Analysis Tasks

used in previous work and susceptible to bias a typical example — directly ask to check the flow direction at a point the participant is shown a randomly placed circle (of which the center is hence a random sample) and asked to click on the point along the circle that a particle advected from the center is to hit

Explicit sample-based tasks

a methodology advocated and formulated in this paper to suppress bias use a simple form but indirectly require thorough investigation of the flow

Implicit pattern-based tasks

mouse pointing & clicking, irrelevant of judgment, affect the test result

the complexity of a flow usually varies with the location more difficult to do this task in turbulent areas than in laminar areas

the selection of the circle’s radius may further compound this issue

critical point recognition — detect patterns globally/across the whole domain critical point classification — match patterns locally/around an area of interest

VIS 2011

Experimental Components

Flow Analysis Tasks Implicit task design

to relieve non-expert participants from understanding complex, possibly domain-specific details in the form of easy-to-understand yet difficult-to-answer questions requiring intensive analysis of flow directions

using specific real tasks about well-known flow features critical point recognition (CPR) critical point classification (CPC) involving in-depth flow structures identification of separatrices identification of periodic orbits creating general synthetic tasks to reduce data-related bias resulting from flow sampling and mouse point-and-click operations

such as symmetric pattern categorization (SPC) — to examine the flow direction both globally and locally — to check the entire pattern: x-/y-/z-/center-symmetric or asymmetric

VIS 2011

Experimental Components

Flow Analysis Tasks Very challenging synthetic tasks

two or three critical points (centers, foci, and saddles) combined with

a variety of configurations to define some Composite Templates (CTs)

CT-based CPR-like pattern recognition

CT-based CPC-like pattern classification

checking if flow A and flow B have a CT pattern in common

judging if flow A is a rotational version of flow B

determining if flow A is exactly part of flow B

The selected implicit tasks CPR + CPC + SPC

integration of 2 real tasks and 1 synthetic task to demonstrate the types

the balance between the overall challenge degree and the test duration

— some synthetic tasks mentioned above would require more test time

VIS 2011

The Input

Test Strategy

7M images generated using the selected 7 techniques to visualize M synthetic flows

involving N x-symmetric, N y-symmetric, N center-symmetric, and

optionally N asymmetric flows — M = 3N or 4N (e.g., N = 30)

depending on the expected complexity and time duration of the test

Ground truth — one record per synthetic flow symmetry type of the overall pattern

the location and type of every ESCP (center / focus) from the synthesizer

the location of every derived saddle from Newton-Raphson root-finding

Task Session 1 CPR task (recognizing ALL critical points from an image) or

<= 30 CPC tasks or

<= 30 (without asymmetric flows) / 40 (with asymmetric flows) SPC tasks

VIS 2011

Task Management

Test Strategy

1 set = (1 CPR session + 1 CPC session + 1 SPC session) for one technique

1 cycle = 7 sets (one for each technique)

1 test = 3 cycles

1 session = 1 CPR task or (<= 30) CPC tasks or (<= 30/40) SPC tasks

= 21 sets = 63 sessions for each participant use 7 techniques thrice to produce 7 × 3 = 21 images (for 21 randomly- selected flows), with 1 image for each CPC session (3 per technique)

use each technique to produce 30 images (for 30 randomly-selected flows), with 1 randomly-selected critical point marked per image (with 10 marked for each critical point type: center, focus, saddle), for each CPC session

use each technique to visualize 30 or 40 randomly-selected flows (creating 10 images for each symmetry / asymmetric type) for each SPC session

21 CPR sessions + 21 CPC sessions + 21 SPC sessions = 63 sessions with a bank of images pre-generated for one time, 63 sessions are created using TestGen upon each test and are then delivered in random order

VIS 2011

Hybrid Timing

Test Strategy

Effectiveness metrics the effectiveness of a visualization technique is usually reflected by

answer correctness and response time a more effective technique allows the user to get a correct answer faster given a fixed amount of time, more correct answers tend to result from a more effective technique than from a less effective technique

Variable-duration session mouse click positions and response time are recorded for a session flow analysis (for recognizing a single critical point) is relatively quick the answer is precision-critical (despite a considerable error tolerance) seeks to “curb” the participant from hastiness and excessive inaccuracy

Fixed-duration session as many tasks as possible are presented to the participant one by one in a fixed amount of time (30s) and radio-button choices are recorded flow analysis is relatively slow and judgment-intensive intended to “push” the participant to accomplish more tasks

— for CPR

— for CPC & SPC (response time on average)

this hybrid timing strategy helps reveal the subtle differences

in visualization effectiveness that may exist between techniques

VIS 2011

Test Strategy

CPR — Critical Point Recognition

VIS 2011

Test Strategy

CPC — Critical Point Classification

VIS 2011

Test Strategy

SPC — Symmetric Pattern Categorization

VIS 2011

Test Results

Basic Facts 4 CFD experts + 16 graduate students in science & engineering expert and non-expert participants were not compared herein 5079 CPR trials + 7467 CPC trials + 4948 SPR trials were recorded

Processing Outliers the response time and the (CPR) location error each showed a skewed normal distribution in terms of the histogram outliers were determined case by case by investigating the tails of the distributions and noting values after conspicuous gaps each outlier was replaced with the median of the cell’s responses

the absolute differences in response time for CPR / CPC / SPC

turned out to be small, regardless of the statistical differences a higher priority assigned to correctness than to response speed to provide correctness-over-response-sorting (CORS) when evaluating the seven techniques in the overall visualization effectiveness

VIS 2011

Test Results

Statistical Analysis Chi-square tests and ANOVA (univariate analysis of variance) calculating post-hoc homogeneous subsets using Ryan REGWQ tests

FlowVUS Results CPR (Critical Point Recognition) — response time

mean time (in seconds) to recognize a critical point (5079 trials, F(6,115.3) = 19.9, p < 0.001)

means with the same letter are not significantly different at p 0.05 (Ryan REGWQ post-hoc hst)

VIS 2011

Test Results

FlowVUS Results CPR (Critical Point Recognition) — answer incorrectness

CORS sorting by CPR effectiveness in decreasing orderEnhancedLIC - StreamCM - BasicLIC - OrientedLIC - StreamCW -

ArrowCM - ArrowCW

336 errors, χ2(6) = 132, p < 0.001

VIS 2011

Test Results

FlowVUS Results CPC (Critical Point Classification) — response time

mean time (in seconds) to classify a critical point (7467 trials, F(6,116.2) = 30.9, p < 0.001)

means with the same letter are not significantly different at p 0.05 (Ryan REGWQ post-hoc hst)

VIS 2011

Test Results

FlowVUS Results CPC (Critical Point Classification) — answer incorrectness

CORS sorting by CPC effectiveness in decreasing orderEnhancedLIC - StreamCW - StreamCM - BasicLIC - OrientedLIC -

ArrowCW - ArrowCM

753 errors, χ2(6) = 772, p < 0.001

VIS 2011

Test Results

FlowVUS Results SPC (Symmetric Pattern Categorization) — response time

mean time (in sec.s) to categorize a symmetric pattern (4948 trials, F(6,123.1) = 8.74, p < 0.001)

means with the same letter are not significantly different at p 0.05 (Ryan REGWQ post-hoc hst)

VIS 2011

Test Results

FlowVUS Results SPC (Symmetric Pattern Categorization) — answer incorrectness

CORS sorting by SPC effectiveness in decreasing orderEnhancedLIC - StreamCM - BasicLIC - OrientedLIC - StreamCW –

ArrowCM - ArrowCW

323 errors, χ2(6) = 70.1, p < 0.001

VIS 2011

Test Results

CORS Sorting byCPR effectiveness

CORS Sorting byCPC effectiveness

CORS Sorting bySPC effectiveness

EnhancedLIC EnhancedLIC EnhancedLIC

StreamCM StreamCW StreamCM

BasicLIC StreamCM BasicLIC

OrientedLIC BasicLIC OrientedLIC

StreamCW OrientedLIC StreamCW

ArrowCM ArrowCW ArrowCM

ArrowCW ArrowCM ArrowCW a texture-based dense representation with accentuated flow streaks (EnhancedLIC) enables intuitive perception of the flow a geometry-based integral representation with uniform density control (StreamCM or StreamCW) exploits visual interpolation to facilitate mental reconstruction of the flow color mapping has a considerable influence on a geometry-based flow representation

VIS 2011

Concluding Remarks

Key Points Explicit flow synthesis

Implicit task design

Diverse evaluation perspectives

Hybrid timing strategy

Refined statistical analysis

— to reduce data-related bias template-based parameterized placement & configuration of critical points automatic synthesis of diverse flows with similar topological complexities

— to suppress task-related bias pattern-based (real tasks + synthetic tasks) the way more than the goal

— representative techniques representation continuity visual intuition image contrast color mapping

variable-duration session fixed-duration session to reveal the subtle differences in vis. effectiveness between techniques

processes outliers + Ryan REGWQ post-hoc homogeneous subset tests

— to reduce data-related bias

— to suppress task-related bias

Explicit flow synthesis

Implicit task design

Two important methodologies / concepts proposed as part of our anti-bias framework for conducting objective flow vis. user studies

VIS 2011

Concluding Remarks

Limitations & Lessons FlowVUS is bias-resistant but not bias-free

Varying a-priori familiarity with techniques

Varying a-priori familiarity with flow features

Real flows needed for introducing techniques

Care needed for predicting the time duration

bias is pervasive throughout the whole pipeline of a user study and hence we cannot totally eliminate it while we need to reduce it — cannot let it be

some participants were not familiar with the LICs upon the training session more user studies are needed to disseminate the latest vis. techniques care needs to be taken when evaluating more sophisticated / current ones

some participants needed extra help with some features during the training session many challenges facing an evaluation involving more complex features

synthetic flows are needed for formal tests while real flows (particularly with contextual boundaries) are needed, besides real flows, for the training session

VIS 2011

Concluding Remarks

Future Plans Anti-bias methodologies

user studies might otherwise be non-convincing & even worse misleading probably one way to help you judge between 2 contradicting conclusions as of now more important than scenarios, techniques, features,

conclusions require much research (e.g., explicit flow synthesis & implicit task design)

end users might not care about the underlying working mechanism they are interested in the resulting images and the associated visual aspects (such as image contrast, color map, intuition, continuity, etc)

neither possible nor necessary to evaluate every existing vis. technique

Evaluation aspects — representative visualization techniques

provide general guidelines for visualization research (algorithm design)

Interesting topics user studies on streamline placement algorithms user studies on surface flow visualization techniques user studies on volume flow visualization techniques

— to adopt the conclusions of a user study without necessary anti-bias methods?

controversialview

controversialview

Thank youfor your time and attention!