The DOI matches its Crossref record; 90 of the 93 references it cites were found in a registry or an index.
In Pursuit of Error: A Survey of Uncertainty Visualization Evaluation
Abstract
Understanding and accounting for uncertainty is critical to effectively reasoning about visualized data. However, evaluating the impact of an uncertainty visualization is complex due to the difficulties that people have interpreting uncertainty and the challenge of defining correct behavior with uncertainty information. Currently, evaluators of uncertainty visualization must rely on general purpose visualization evaluation frameworks which can be ill-equipped to provide guidance with the unique difficulties of assessing judgments under uncertainty. To help evaluators navigate these complexities, we present a taxonomy for characterizing decisions made in designing an evaluation of an uncertainty visualization. Our taxonomy differentiates six levels of decisions that comprise an uncertainty visualization evaluation: the behavioral targets of the study, expected effects from an uncertainty visualization, evaluation goals, measures, elicitation techniques, and analysis approaches. Applying our taxonomy to 86 user studies of uncertainty visualizations, we find that existing evaluation practice, particularly in visualization research, focuses on Performance and Satisfaction-based measures that assume more predictable and statistically-driven judgment behavior than is suggested by research on human judgment and decision making. We reflect on common themes in evaluation practice concerning the interpretation and semantics of uncertainty, the use of confidence reporting, and a bias to…
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Hullman et al. [55] review methods for evaluating the success of uncertainty visualization.”
From Uncertainty as a Form of Transparency: Measuring, Communicating, and Using Uncertainty (2021) · cites this paper as [55]
Citations
Cites
Prospect Theory: An Analysis of Decision under Risk
Kahneman, D., & Tversky, A. (1979) · Econometrica · doi:10.2307/1914185 · Cited by 48,524
Prospect theory: An analysis of decision under risk
Kahneman, D. J., & Tversky, A. (1988) · Cambridge University Press eBooks · doi:10.1017/cbo9780511609220.014 · Cited by 32,720
Judgment under Uncertainty: Heuristics and Biases
Tversky, A., & Kahneman, D. (1975) · Utility, Probability, and Human Decision Making · doi:10.1007/978-94-010-1834-0_8 · Cited by 22,650
(2011) · Cited by 14,317
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
Availability: A heuristic for judging frequency and probability
Tversky, A., & Kahneman, D. (1973) · Cognitive Psychology · doi:10.1016/0010-0285(73)90033-9 · Cited by 9,984 · cited 1×
“Historically, such biases have been attributed to heuristics, the use of simple judgments as a proxy for difficult judgments [49, 90, 91, 92].”
BackgroundJudgment under Uncertainty: Heuristics and Biases
(1982) · doi:10.1017/cbo9780511809477 · Cited by 8,240
“Risk, Ambiguity, and the Savage Axioms.”
Ellsberg, D. (1961) · The Quarterly Journal of Economics · doi:10.2307/1884324 · Cited by 7,891
Anderson, N. H. (2001) · Psychology Press eBooks · doi:10.4324/9781410600004-24 · Cited by 6,939
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702
Research works printed but not linked: 9
- 2006, Completing the forecast Characterizing and communicating uncertainty for better decisions using weather and climate forecasts
- potter, 0, Uncertainty visualization and data references
- quinan, 2015, Towards ecological validity in evaluating uncertainty, Proceedings of Workshop on Visualization for Decision Making Under Uncertainty (VIS'15)
- harrower, 2003, Representing uncertainty Does it help people make better decisions
- hastie, 1982, Concepts in judgment and decision research-anderson bf deane dh hammond kr mcclelland gh shanteau jc
- 10.3758/s13423-012-0247-5 · DOI · Paper page
- 1997, Guide to the Expression of Uncertainty in Measurement
- deitrick, 2012, Evaluating implicit visualization of uncertainty for public policy decision, Proceedings AutoCarto
- 2017, Edelman 2017 edelman trust barometer
Cited by
Sorted by citations
Uncertainty as a Form of Transparency: Measuring, Communicating, and Using Uncertainty
Bhatt, U., Antorán, J., Zhang, Y., Liao, Q. V., Sattigeri, P., Fogliato, R., Melançon, G., Krishnan, R., Stanley, J., Tickoo, O., et al. (2021) · Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3461702.3462571 · Cited by 300
Effective Use of Likert Scales in Visualization Evaluations: A Systematic Review
South, L., Saffo, D., Vitek, O., Dunne, C., & Borkin, M. A. (2022) · Computer Graphics Forum · doi:10.1111/cgf.14521 · Cited by 242
Lee, C., Yang, T., Inchoco, G. D., Jones, G. M., & Satyanarayan, A. (2021) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445211 · Cited by 214
Why Authors Don't Visualize Uncertainty
Hullman, J. R. (2019) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2019.2934287 · Cited by 195
Self-citationEthical Dimensions of Visualization Research
Correll, M. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300418 · Cited by 150
Self-citation
Metadata
- Authors
- Hullman, J., Qiao, X., Correll, M., Kale, A., & Kay, M.
- Affiliations
- Northwestern University (US)
- Tableau Software (United States) (US)
- University of Washington (US)
- University of Michigan (US)
- Venue
- IEEE Transactions on Visualization and Computer Graphics
- Published
- 2019-01
- Volume
- 25
- Issue
- 1
- Pages
- 903-913
- ISSN
- 1077-2626 (ISSN-L)
- 1941-0506
- 2160-9306
- OpenAlex
- W2891628559
- Cited by
- 187
- References
- 93
- Citations by year
- 2026: 18 · 2025: 20 · 2024: 32 · 2023: 22 · 2022: 23 · 2021: 37 · 2020: 23 · 2019: 11
- FWCI
- 9.01
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- Locations
- IEEE Transactions on Visualization and Computer Graphics (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Graphics and Computer-Aided Design › Data Visualization and Analytics
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Physical Sciences › Computer Science › Computer Graphics and Computer-Aided Design › Computer Graphics and Visualization Techniques
- Keywords
- uncertainty visualization, visualization evaluation, evaluation taxonomy, decision quality, performance indicators, user satisfaction, elicitation methods, evaluation goals
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
