Cited by 187Data Visualization and AnalyticsUsability and User Interface DesignComputer Graphics and Visualization Techniques

In Pursuit of Error: A Survey of Uncertainty Visualization Evaluation

Hullman, Jessica, Qiao, Xiaoli, Correll, Michael, Kale, Alex, Kay, Matthew·2019·IEEE Transactions on Visualization and Computer Graphics

Abstract

Understanding and accounting for uncertainty is critical to effectively reasoning about visualized data. However, evaluating the impact of an uncertainty visualization is complex due to the difficulties that people have interpreting uncertainty and the challenge of defining correct behavior with uncertainty information. Currently, evaluators of uncertainty visualization must rely on general purpose visualization evaluation frameworks which can be ill-equipped to provide guidance with the unique difficulties of assessing judgments under uncertainty. To help evaluators navigate these complexities, we present a taxonomy for characterizing decisions made in designing an evaluation of an uncertainty visualization. Our taxonomy differentiates six levels of decisions that comprise an uncertainty visualization evaluation: the behavioral targets of the study, expected effects from an uncertainty visualization, evaluation goals, measures, elicitation techniques, and analysis approaches. Applying our taxonomy to 86 user studies of uncertainty visualizations, we find that existing evaluation practice, particularly in visualization research, focuses on Performance and Satisfaction-based measures that assume more predictable and statistically-driven judgment behavior than is suggested by research on human judgment and decision making. We reflect on common themes in evaluation practice concerning the interpretation and semantics of uncertainty, the use of confidence reporting, and a bias to…

Integrity signals

Registry

Crossref · 10.1109/tvcg.2018.2864889

Cites

90 of 90 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Cites

Flagged first, then by citations

3 of the 93 references couldn’t be matched to a known paper.

Cited by

Sorted by citations

Authors
Hullman, J., Qiao, X., Correll, M., Kale, A., & Kay, M.
Affiliations
  • Northwestern University (US)
  • Tableau Software (United States) (US)
  • University of Washington (US)
  • University of Michigan (US)
Venue
IEEE Transactions on Visualization and Computer Graphics
Published
2019-01
Volume
25
Issue
1
Pages
903-913
ISSN
  • 1077-2626 (ISSN-L)
  • 1941-0506
  • 2160-9306
OpenAlex
W2891628559
Cited by
187
References
93
Citations by year
2026: 18 · 2025: 20 · 2024: 32 · 2023: 22 · 2022: 23 · 2021: 37 · 2020: 23 · 2019: 11 · 2018: 1
FWCI
9.01
Type
journal article
Language
en
Open access
closed
Locations
  • IEEE Transactions on Visualization and Computer Graphics (publishedVersion)
  • PubMed (publishedVersion)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Data Visualization and Analytics
  • Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
  • Physical Sciences › Computer Science › Computer Graphics and Computer-Aided Design › Computer Graphics and Visualization Techniques
Keywords
uncertainty visualization, visualization evaluation, evaluation taxonomy, decision quality, performance indicators, user satisfaction, elicitation methods, evaluation goals

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review