Pragmatic Issues in the Evaluation

of Interactive Visualization Systems

April 10, 2010

Youn-ah Kang, Carsten Görg, John Stasko

School of Interactive ComputingGeorgia Institute of Technology

A comparative lab study with an investigative analysis scenario

VAST 2009

Paper Desktop

Entity Jigsaw

Pragmatic Challenges

1. Student as Subject– Graduate students vs. domain experts– Difficulty in accessing domain experts – To what extent are the results valid?

2. Quantitative vs. Qualitative– Are the two really incompatible?– Using a mixed method to further

validate and enrich findings

Pragmatic Challenges

3. Statistical Power vs. Analysis Depth– Hard to have both large participant

population and in-depth analysis of each session

– Need to focus on either side

4. Scalability– Balancing between feasibility and

ecological validity– Chose the document size to make the study

feasible in a given time, but in reality…

Pragmatic Challenges

5. Familiarity with the Tool– Good usability is critical in a utility evaluation – Unfamiliarity with the tool blurs potential

benefits– Big learning curve but limited time

6. Comprehensiveness of the Task– Only targeting scenario for this study– Challenges in developing evaluation

strategies and measures for less focused scenarios

Answers

• ???