Cited by 187Data Visualization and AnalyticsUsability and User Interface DesignComputer Graphics and Visualization Techniques

The DOI matches its Crossref record; 90 of the 93 references it cites were found in a registry or an index.

In Pursuit of Error: A Survey of Uncertainty Visualization Evaluation

Hullman, Jessica, Qiao, Xiaoli, Correll, Michael, Kale, Alex, Kay, Matthew·2019·IEEE Transactions on Visualization and Computer Graphics

DOI 10.1109/tvcg.2018.2864889

Open this paper in CoChat — free account

Abstract

Understanding and accounting for uncertainty is critical to effectively reasoning about visualized data. However, evaluating the impact of an uncertainty visualization is complex due to the difficulties that people have interpreting uncertainty and the challenge of defining correct behavior with uncertainty information. Currently, evaluators of uncertainty visualization must rely on general purpose visualization evaluation frameworks which can be ill-equipped to provide guidance with the unique difficulties of assessing judgments under uncertainty. To help evaluators navigate these complexities, we present a taxonomy for characterizing decisions made in designing an evaluation of an uncertainty visualization. Our taxonomy differentiates six levels of decisions that comprise an uncertainty visualization evaluation: the behavioral targets of the study, expected effects from an uncertainty visualization, evaluation goals, measures, elicitation techniques, and analysis approaches. Applying our taxonomy to 86 user studies of uncertainty visualizations, we find that existing evaluation practice, particularly in visualization research, focuses on Performance and Satisfaction-based measures that assume more predictable and statistically-driven judgment behavior than is suggested by research on human judgment and decision making. We reflect on common themes in evaluation practice concerning the interpretation and semantics of uncertainty, the use of confidence reporting, and a bias to…

Integrity signals

Registry

Crossref · 10.1109/tvcg.2018.2864889

Cites

90 of 90 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Prospect Theory: An Analysis of Decision under Risk
Kahneman, D., & Tversky, A. (1979) · Econometrica · doi:10.2307/1914185 · Cited by 48,524
Prospect theory: An analysis of decision under risk
Kahneman, D. J., & Tversky, A. (1988) · Cambridge University Press eBooks · doi:10.1017/cbo9780511609220.014 · Cited by 32,720
Judgment under Uncertainty: Heuristics and Biases
Tversky, A., & Kahneman, D. (1975) · Utility, Probability, and Human Decision Making · doi:10.1007/978-94-010-1834-0_8 · Cited by 22,650
Untitled work
(2011) · Cited by 14,317
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
Availability: A heuristic for judging frequency and probability
Tversky, A., & Kahneman, D. (1973) · Cognitive Psychology · doi:10.1016/0010-0285(73)90033-9 · Cited by 9,984 · cited 1×
“Historically, such biases have been attributed to heuristics, the use of simple judgments as a proxy for difficult judgments [49, 90, 91, 92].”
Background
Judgment under Uncertainty: Heuristics and Biases
(1982) · doi:10.1017/cbo9780511809477 · Cited by 8,240
“Risk, Ambiguity, and the Savage Axioms.”
Ellsberg, D. (1961) · The Quarterly Journal of Economics · doi:10.2307/1884324 · Cited by 7,891
FOUNDATIONS OF STATISTICS
Anderson, N. H. (2001) · Psychology Press eBooks · doi:10.4324/9781410600004-24 · Cited by 6,939
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702

Show all 93 references

Research works printed but not linked: 9

2006, Completing the forecast Characterizing and communicating uncertainty for better decisions using weather and climate forecasts
potter, 0, Uncertainty visualization and data references
quinan, 2015, Towards ecological validity in evaluating uncertainty, Proceedings of Workshop on Visualization for Decision Making Under Uncertainty (VIS'15)
harrower, 2003, Representing uncertainty Does it help people make better decisions
hastie, 1982, Concepts in judgment and decision research-anderson bf deane dh hammond kr mcclelland gh shanteau jc
10.3758/s13423-012-0247-5 · DOI · Paper page
1997, Guide to the Expression of Uncertainty in Measurement
deitrick, 2012, Evaluating implicit visualization of uncertainty for public policy decision, Proceedings AutoCarto
2017, Edelman 2017 edelman trust barometer

Sorted by citations

Uncertainty as a Form of Transparency: Measuring, Communicating, and Using Uncertainty
Bhatt, U., Antorán, J., Zhang, Y., Liao, Q. V., Sattigeri, P., Fogliato, R., Melançon, G., Krishnan, R., Stanley, J., Tickoo, O., et al. (2021) · Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3461702.3462571 · Cited by 300
Effective Use of Likert Scales in Visualization Evaluations: A Systematic Review
South, L., Saffo, D., Vitek, O., Dunne, C., & Borkin, M. A. (2022) · Computer Graphics Forum · doi:10.1111/cgf.14521 · Cited by 242
Viral Visualizations: How Coronavirus Skeptics Use Orthodox Data Practices to Promote Unorthodox Science Online
Lee, C., Yang, T., Inchoco, G. D., Jones, G. M., & Satyanarayan, A. (2021) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445211 · Cited by 214
Why Authors Don't Visualize Uncertainty
Hullman, J. R. (2019) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2019.2934287 · Cited by 195
Self-citation
Ethical Dimensions of Visualization Research
Correll, M. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300418 · Cited by 150
Self-citation
Uncertainty Visualization
Padilla, L., Kay, M., & Hullman, J. R. (2021) · Wiley StatsRef: Statistics Reference Online · doi:10.1002/9781118445112.stat08296 · Cited by 87
Self-citation
The Unmet Data Visualization Needs of Decision Makers Within Organizations
Dimara, E., Zhang, H., Tory, M. K., & Franconeri, S. (2021) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2021.3074023 · Cited by 75
Recent advances and challenges in uncertainty visualization: a survey
Kamal, A., Dhakal, P., Javaid, A. Y., Devabhaktuni, V., Kaur, D., Zaientz, J. D., & Marinier, R. (2021) · Journal of Visualization · doi:10.1007/s12650-021-00755-1 · Cited by 71
A Critical Reflection on Visualization Research: Where Do Decision Making Tasks Hide?
Dimara, E., & Stasko, J. (2022) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2021.3114813 · Cited by 60
Visualizing Uncertainty and Alternatives in Event Sequence Predictions
Guo, S., Du, F., Malik, S., Koh, E., Kim, S., Liu, Z., Kim, Y., Hongyuan Zha, B., & Cao, N. (2019) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300803 · Cited by 56
Evaluating language models for mathematics through interactions
Collins, K. M., Jiang, A. Q., Frieder, S., Wong, L., Zilka, M., Bhatt, U., Lukasiewicz, T., Wu, Y., Tenenbaum, J. B., Hart, W., et al. (2024) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2318124121 · Cited by 108
Uncertainty Visualization: Concepts, Methods, and Applications in Biological Data Visualization
Weiskopf, D. (2022) · Frontiers in Bioinformatics · doi:10.3389/fbinf.2022.793819 · Cited by 50
A Visual Analytics Approach for Exploratory Causal Analysis: Exploration, Validation, and Applications
Xie, X., Du, F., & Wu, Y. (2020) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2020.3028957 · Cited by 52
Visualizing Privacy-Utility Trade-Offs in Differentially Private Data Releases
Nanayakkara, P., Bater, J., He, X. Q., Hullman, J. R., & Rogers, J. (2022) · Proceedings on Privacy Enhancing Technologies · doi:10.2478/popets-2022-0058 · Cited by 63
Self-citation
Designing for Interactive Exploratory Data Analysis Requires Theories of Graphical Inference
Hullman, J. R., & Gelman, A. (2021) · Harvard Data Science Review · doi:10.1162/99608f92.3ab8a587 · Cited by 62
Self-citation
Data Hunches: Incorporating Personal Knowledge into Visualizations
Lin, H., Akbaba, D., Meyer, M., & Lex, A. (2022) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2022.3209451 · Cited by 50
Data Comics for Reporting Controlled User Studies in Human-Computer Interaction
Wang, Z., Ritchie, J., Zhou, J., Chevalier, F., & Bach, B. (2020) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2020.3030433 · Cited by 42
Visualizing Uncertainty for Non-Expert End Users: The Challenge of the Deterministic Construal Error
Joslyn, S. L., & Savelli, S. (2021) · Frontiers in Computer Science · doi:10.3389/fcomp.2020.590232 · Cited by 41
Communicating health information with visual displays
Woloshin, S., Yang, Y., & Fischhoff, B. (2023) · Nature Medicine · doi:10.1038/s41591-023-02328-1 · Cited by 38
A Bayesian cognition approach for belief updating of correlation judgement through uncertainty visualizations
Karduni, A., Markant, D. B., Wesslen, R. S., & Dou, W. (2020) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2020.3029412 · Cited by 42
Can Visualization Alleviate Dichotomous Thinking? Effects of Visual Representations on the Cliff Effect
Helske, J., Helske, S., Cooper, M., Ynnerman, A., & Besancon, L. (2021) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2021.3073466 · Cited by 38
Uncertainty‐aware Visualization in Medical Imaging ‐ A Survey
Gillmann, C., Saur, D., Wischgoll, T., & Scheuermann, G. (2021) · Computer Graphics Forum · doi:10.1111/cgf.14333 · Cited by 38
Examining Effort in 1D Uncertainty Communication Using Individual Differences in Working Memory and NASA-TLX
Castro, S. C., Quinan, P. S., Hosseinpour, H., & Padilla, L. (2021) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2021.3114803 · Cited by 40
Is this Real?
Mannino, M., & Abouzied, A. (2019) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3332165.3347866 · Cited by 35
Interactive Cleaning for Progressive Visualization through Composite Questions
Luo, Y., Chai, C., Qin, X., Tang, N., & Li, G. (2020) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde48307.2020.00069 · Cited by 36
Effects of Uncertainty Visualization on Map-Based Decision Making Under Time Pressure
Korporaal, M., Ruginski, I. T., & Fabrikant, S. I. (2020) · Frontiers in Computer Science · doi:10.3389/fcomp.2020.00032 · Cited by 40
Improving Visualization Design for Effective Multi-Objective Decision Making
Dy, B., Ibrahim, N., Poorthuis, A., & Joyce, S. C. (2021) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2021.3065126 · Cited by 34
Information, incentives, and goals in election forecasts
Gelman, A., Hullman, J. R., Wlezien, C. B., & Morris, G. E. (2020) · Judgment and Decision Making · doi:10.1017/s1930297500007981 · Cited by 30
Self-citation
Everyday Uncertainty: How Blind People Use GenAI Tools for Information Access
Tang, X., Abdolrahmani, A., Gergle, D., & Piper, A. M. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713433 · Cited by 30
Visual‐Interactive Preprocessing of Multivariate Time Series Data
Bernard, J., Hutter, M., Reinemuth, H., Pfeifer, H., Bors, C., & Kohlhammer, J. (2019) · Computer Graphics Forum · doi:10.1111/cgf.13698 · Cited by 29
Vagueness visualization in building models across different design stages
Abualdenien, J., & Borrmann, A. (2020) · Advanced Engineering Informatics · doi:10.1016/j.aei.2020.101107 · Cited by 29
Robot-Assisted Decision-Making: Unveiling the Role of Uncertainty Visualisation and Embodiment
Schömbs, S., Pareek, S., Gonçalves, J., & Johal, W. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642911 · Cited by 29
Exploring Uncertainty in Digital Scheduling, and The Wider Implications of Unrepresented Temporalities in HCI
Bowler, R. D., Bach, B., & Pschetz, L. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3502107 · Cited by 27
Effective uncertainty visualization for aftershock forecast maps
Schneider, M., McDowell, M., Guttorp, P. M., Steel, E. A., & Fleischhut, N. (2022) · Natural hazards and earth system sciences · doi:10.5194/nhess-22-1499-2022 · Cited by 27
Uncertainty-Aware Seasonal-Trend Decomposition Based on Loess
Krake, T., Klötzl, D., Hägele, D., & Weiskopf, D. (2024) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2024.3364388 · Cited by 26
Accounting for uncertainty during a pandemic
Zelner, J., Riou, J., Etzioni, R., & Gelman, A. (2021) · Patterns · doi:10.1016/j.patter.2021.100310 · Cited by 30
Uncertainty Visualization of 2D Morse Complex Ensembles Using Statistical Summary Maps
Athawale, T. M., Maljovec, D., Yan, L., Johnson, C. R., Pascucci, V., & Wang, B. (2020) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2020.3022359 · Cited by 26
Uncertainty VisualizationPreprint
Padilla, L., Kay, M., & Hullman, J. R. (2020) · doi:10.31234/osf.io/ebd6r · Cited by 28
Self-citation
Methodological Considerations for Furthering the Understanding of Constraints in Applied Sports
Browne, P. R., Sweeting, A. J., Woods, C. T., & Robertson, S. J. (2021) · Sports Medicine - Open · doi:10.1186/s40798-021-00313-x · Cited by 28
Communicating Uncertainty in Digital Humanities Visualization Research
Panagiotidou, G., Lamqaddam, H., Poblome, J., Brosens, K., Verbert, K., & Moere, A. V. (2022) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2022.3209436 · Cited by 24
Uncertainty Visualisation: An Interactive Visual Survey
Jena, A., Engelke, U., Dwyer, T., Raiamanickam, V., & Paris, C. (2020) · IEEE Pacific Visualization Symposium · doi:10.1109/pacificvis48177.2020.1014 · Cited by 33
Evaluating the Impact of Uncertainty Visualization on Model Reliance
Zhao, J. L., Wang, Y., Mancenido, M. V., Chiou, E. K., & Maciejewski, R. (2023) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2023.3251950 · Cited by 26
Scenario design for infectious disease projections: Integrating concepts from decision analysis and experimental design
Runge, M. C., Shea, K., Howerton, E., Yan, K.-T., Hochheiser, H., Rosenstrom, E. T., Probert, W., Borchering, R., Marathe, M., Lewis, B., et al. (2024) · Epidemics · doi:10.1016/j.epidem.2024.100775 · Cited by 13
The illusion of data validity: Why numbers about people are likely wrong
Jansen, B. J. J., Salminen, J. O., Jung, S., & Almerekhi, H. (2022) · Data and Information Management · doi:10.1016/j.dim.2022.100020 · Cited by 26
Beyond Trust Building — Calibrating Trust in Visual Analytics
Han, W., & Schulz, H. (2020) · IEEE Workshop on TRust and EXpertise in Visual Analytics (TREX) · doi:10.1109/trex51495.2020.00006 · Cited by 21
A uncertainty visual analytics approach for bus travel time
Zhao, W., Wang, G., Wang, Z., Liu, L., Wei, X., & Ya, W. (2022) · Visual Informatics · doi:10.1016/j.visinf.2022.06.002 · Cited by 19
Communicating Uncertainty and Risk in Air Quality Maps
Preston, A., & Ma, K. (2022) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2022.3171443 · Cited by 18
Data visualisation for decision making under deep uncertainty: current challenges and opportunities
Hadjimichael, A., Schlumberger, J., & Haasnoot, M. (2024) · Environmental Research Letters · doi:10.1088/1748-9326/ad858b · Cited by 17
When do data visualizations persuade? The impact of prior attitudes on learning about correlations from scatterplot visualizations
Markant, D. B., Rogha, M., Karduni, A., Wesslen, R. S., & Wenwen Dou, A. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581330 · Cited by 16
Data visualization and domain knowledge: Insights through focus groups of researchers in Spain
Vázquez‐Ingelmo, A., García‐Holgado, A., Verdugo-Castro, S., Therón, R., & García‐Peñalvo, F. J. (2024) · Computers in Human Behavior · doi:10.1016/j.chb.2024.108162 · Cited by 20
Implicit Error, Uncertainty and Confidence in Visualization: An Archaeological Case Study
Panagiotidou, G., Vandam, R., Poblome, J., & Moere, A. V. (2021) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2021.3088339 · Cited by 19
Designing for Ambiguity: Visual Analytics in Avalanche Forecasting
Nowak, S., Bartram, L., & Haegeli, P. (2020) · Visualization · doi:10.1109/vis47514.2020.00023 · Cited by 15
Two graphs walk into a bar: Readout-based measurement reveals the Bar-Tip Limit error, a common, categorical misinterpretation of mean bar graphs
Kerns, S. H., & Wilmer, J. B. (2021) · Journal of Vision · doi:10.1167/jov.21.12.17 · Cited by 16
Uncertainty visualization: Fundamentals and recent developments
Hägele, D., Schulz, C., Beschle, C. A., Booth, H., Butt, M., Barth, A., Deußen, O., & Weiskopf, D. (2022) · it - Information Technology · doi:10.1515/itit-2022-0033 · Cited by 15
Exploring the role of conscientiousness on visualization-supported decision-making
Alves, T., Delgado, T., Henriques‐Calado, J., Gonçalves, D. J. V., & Gama, S. P. (2023) · Computers & Graphics · doi:10.1016/j.cag.2023.01.010 · Cited by 15
Interpretability Gone Bad: The Role of Bounded Rationality in How Practitioners Understand Machine Learning
Kaur, H., Conrad, M. R., Rule, D., Lampe, C. A. C., & Gilbert, É. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3637354 · Cited by 17
Evaluating the impact of visualization of risk upon emergency route-planning
Cheong, L., Kinkeldey, C., Burfurd, I., Bleisch, S., & Duckham, M. (2019) · International Journal of Geographical Information Systems · doi:10.1080/13658816.2019.1701677 · Cited by 15
Bayesian Methods for Interaction and Design
Williamson, J., Oulasvirta, A., Kristensson, P. O., & Banović, N. (2022) · Cambridge University Press eBooks · doi:10.1017/9781108874830 · Cited by 15
Subjective Probability Correction for Uncertainty Representations
Yang, F., Hedayati, M., & Kay, M. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580998 · Cited by 16
Self-citation
Comparing the Effectiveness of Visualizations of Different Data Distributions
Rodrigues, A. M. B., Barbosa, G. D. J., Lopes, H., & Barbosa, S. D. J. (2019) · SIBGRAPI Conference on Graphics, Patterns and Images (SIBGRAPI) · doi:10.1109/sibgrapi.2019.00020 · Cited by 14
Improving the Use of Hydrologic Probabilistic and Deterministic Information in Decision-Making
Carr, R., Semmens, K., Montz, B., & Maxfield, K. (2021) · Bulletin of The American Meteorological Society - (BAMS) · doi:10.1175/bams-d-21-0019.1 · Cited by 8
Should I Follow this Model? The Effect of Uncertainty Visualization on the Acceptance of Time Series Forecasts
Leffrang, D., & Muller, O. (2021) · IEEE Workshop on TRust and EXpertise in Visual Analytics (TREX) · doi:10.1109/trex53765.2021.00009 · Cited by 14
Empirical testing of the visualizations of climate change mitigation scenarios with citizens: A comparison among Germany, Poland, and France
Xexakis, G., & Trutnevyte, E. (2021) · Global Environmental Change · doi:10.1016/j.gloenvcha.2021.102324 · Cited by 18
Insights and Opportunities for HCI Research into Hurricane Risk Communication
Soden, R., Chilton, L. B., Miles, S. B., Bicksler, R., Villanueva, K. R., & Bica, M. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3502101 · Cited by 13
Milliways: Taming Multiverses through Principled Evaluation of Data Analysis Paths
Sarma, A., Hwang, K., Hullman, J. R., & Kay, M. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642375 · Cited by 13
Self-citation
Uncertainty Treemaps
Sondag, M., Meulemans, W., Schulz, C., Verbeek, K., Weiskopf, D., & Speckmann, B. (2020) · IEEE Pacific Visualization Symposium · doi:10.1109/pacificvis48177.2020.7614 · Cited by 13
Uncertainty-aware visual analytics: scope, opportunities, and challenges
Maack, R. G. C., Scheuermann, G., Hagen, H., Peñaloza, J. T. H., & Gillmann, C. (2022) · The Visual Computer · doi:10.1007/s00371-022-02733-6 · Cited by 13
Exploring the Effects of Aggregation Choices on Untrained Visualization Users' Generalizations From Data
Nguyen, F., Qiao, X., Heer, J., & Hullman, J. R. (2020) · Computer Graphics Forum · doi:10.1111/cgf.13902 · Cited by 15
Self-citation
Effect of uncertainty visualizations on myopic loss aversion and the equity premium puzzle in retirement investment decisions
Wesslen, R., Karduni, A., Markant, D., & Dou, W. (2021) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2021.3114692 · Cited by 15
Uncertainty-Aware Multidimensional Scaling
Hägele, D., Krake, T., & Weiskopf, D. (2022) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2022.3209420 · Cited by 13
Communicating Uncertainty in Machine Learning Explanations: A Visualization Analytics Approach for Predictive Process Monitoring
Mehdiyev, N., Majlatow, M., & Fettke, P. (2024) · Communications in computer and information science · doi:10.1007/978-3-031-63800-8_21 · Cited by 12
Challenges in Evaluating Interactive Visual Machine Learning Systems
Boukhelifa, N., Bezerianos, A., Chang, R., Collins, C., Drucker, S. M., Endert, A., Hullman, J. R., North, C. L., & Sedlmair, M. (2020) · IEEE Computer Graphics and Applications · doi:10.1109/mcg.2020.3017064 · Cited by 12
Self-citation
How Teams Communicate about the Quality of ML Models: A Case Study at an International Technology Company
Almahmoud, J., DeLine, R. A., & Drucker, S. M. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3463934 · Cited by 11
Studying Early Decision Making with Progressive Bar Charts
Patil, A., Richer, G., Jermaine, C., Moritz, D., & Fekete, J. (2022) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2022.3209426 · Cited by 11
Position Paper: Are We Making Progress In Visualization Research?
Correll, M. (2022) · IEEE Evaluation and Beyond - Methodological Approaches for Visualization (BELIV) · doi:10.1109/beliv57783.2022.00005 · Cited by 11
Self-citation
A Qualitative Interview Study of Distributed Tracing Visualisation: A Characterisation of Challenges and Opportunities
Davidson, T. J., Wall, E., & Mace, J. (2023) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2023.3241596 · Cited by 11
In Dice We Trust: Uncertainty Displays for Maintaining Trust in Election Forecasts Over Time
Yang, F., Mortenson, C. R., Nisbet, E. C., Diakopoulos, N. A., & Kay, M. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642371 · Cited by 19
Self-citation
Numerical and Visual Representations of Uncertainty Lead to Different Patterns of Decision Making
Matzen, L. E., Howell, B. C., Trumbo, M., & Divis, K. (2023) · IEEE Computer Graphics and Applications · doi:10.1109/mcg.2023.3299875 · Cited by 10
An empirical study of counterfactual visualization to support visual causal inference
Wang, A. Z., Borland, D., & Gotz, D. (2024) · Information Visualization · doi:10.1177/14738716241229437 · Cited by 11
Using AI Uncertainty Quantification to Improve Human Decision-MakingPreprint
Marusich, L. R., Bakdash, J. Z., Zhou, Y., & Kantarcıoğlu, M. (2023) · doi:10.31234/osf.io/6syfv · Cited by 13
A Knowledge Mobilization Framework: Toward Evidence-Based Statistical Communication Practices in Education Research
Fitzgerald, K. G., & Tipton, E. (2023) · Journal of Research on Educational Effectiveness · doi:10.1080/19345747.2023.2209082 · Cited by 14
An Analysis of the Interplay and Mutual Benefits of Grounded Theory and Visualization
Diehl, A., Abdul‐Rahman, A., Bach, B., El‐Assady, M., Kraus, M., Laramee, R. S., Keim, D. A., & Chen, M. (2024) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2024.3452985 · Cited by 17
Odds and Insights: Decision Quality in Exploratory Data Analysis Under Uncertainty
Sarma, A., Pu, X., Cui, Y., Correll, M., Brown, E. T., & Kay, M. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3641995 · Cited by 9
Self-citation
What Do We Actually Learn from Evaluations in the “Heroic Era” of Visualization? : Position Paper
Correll, M. (2020) · doi:10.1109/beliv51497.2020.00013 · Cited by 8
Self-citation
Is less more? Experimenting with visual stacking of coincident maps for spatial global sensitivity analysis in urban land-use change modeling
Şalap‐Ayça, S., Jankowski, P. L., Clarke, K., & Nara, A. (2021) · Environmental Modelling & Software · doi:10.1016/j.envsoft.2021.105181 · Cited by 8
Rare, but Valuable: Understanding Data-centered Talk in News Website Comment Sections
McInnis, B., Sun, L., Shin, J., & Dow, S. W. (2020) · Proc. ACM Hum. Comput. Interact. · doi:10.1145/3415245 · Cited by 10
Natural Expression of a Machine Learning Model's Uncertainty Through Verbal and Non-Verbal Behavior of Intelligent Virtual Agents
Schmidt, S., Rolff, T., Voigt, H., Offe, M., & Steinicke, F. (2024) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3654777.3676454 · Cited by 8
Who said only military officers can deal with uncertainty? On the importance of uncertainty in edtech data visualisations
Macgilchrist, F., & Jarke, J. (2025) · Discourse Studies in the Cultural Politics of Education · doi:10.1080/01596306.2025.2467365 · Cited by 10
Balancing the Unknown: Exploring Human Reliance on AI Advice under Aleatoric and Epistemic Uncertainty
Holstein, J., Böcking, L., Spitzer, P., Kühl, N., Vössing, M., & Satzger, G. (2025) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3762813 · Cited by 11
Complex model calibration through emulation, a worked example for a stochastic epidemic model
Dunne, M., Mohammadi, H., Challenor, P., Borgo, R., Porphyre, T., Vernon, I., Fırat, E. E., Turkay, Ç., Torsney-Weir, T., Goldstein, M., et al. (2022) · Epidemics · doi:10.1016/j.epidem.2022.100574 · Cited by 7
The Rational Agent Benchmark for Data Visualization
Wu, Y., Guo, Z., Mamakos, M., Hartline, J. D., & Hullman, J. R. (2023) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2023.3326513 · Cited by 11
Self-citation
What’s really wrong with bar graphs of mean values: variable and inaccurate communication of evidence on three key dimensionsPreprint
Wilmer, J. B., & Kerns, S. H. (2022) · doi:10.31219/osf.io/av5ey · Cited by 7
Communicating Consequences: Visual Narratives, Abstraction, and Polysemy in Rural Bangladesh
Sultana, S., Ahmed, S. I., & Rzeszotarski, J. M. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581149 · Cited by 6
'The AI is uncertain, so am I. What now?': Navigating Shortcomings of Uncertainty Representations in Human-AI Collaboration with Capability-focused Guidance
Schäfer, U., Sipos, L., & Müller-Birn, C. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3757451 · Cited by 10
Evaluating the Utility of Conformal Prediction Sets for AI-Advised Image Labeling
Zhang, D., Chatzimparmpas, A., Kamali, N., & Hullman, J. R. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642446 · Cited by 11
Self-citation
From Delays to Densities: Exploring Data Uncertainty through Speech, Text, and Visualization
Stokes, C., Sanker, C., Cogley, B., & Setlur, V. (2024) · Computer Graphics Forum · doi:10.1111/cgf.15100 · Cited by 6
Unmasking Dunning-Kruger Effect in Visual Reasoning & Judgment
Chen, M., Liu, Y., & Wall, E. (2024) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2024.3456326 · Cited by 9
Visualizing Uncertainty in Time Series Forecasts: The Impact of Uncertainty Visualization on Users' Confidence, Algorithmic Advice Utilization, and Forecasting Performance
Leffrang, D., & Müller, O. (2024) · Journal of Forecasting · doi:10.1002/for.3222 · Cited by 8
FairICP: identifying biases and increasing transparency at the point of care in post-implementation clinical decision support using inductive conformal prediction
Sun, X., Nakashima, M., Nguyen, C. T., Chen, P., Tang, W. H. W., Kwon, D. H., & Chen, D. J. (2025) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaf095 · Cited by 5
A systematic review of measurement uncertainty visualizations in the context of standardized assessments
Heltne, A., Frans, N., Hummelen, B., Falkum, E., Germans Selvik, S., & Paap, M. C. S. (2023) · Scandinavian Journal of Psychology · doi:10.1111/sjop.12918 · Cited by 5

Authors: Hullman, J., Qiao, X., Correll, M., Kale, A., & Kay, M.
Affiliations: Northwestern University (US)
Tableau Software (United States) (US)
University of Washington (US)
University of Michigan (US)
ORCID: Hullman, Jessica
Correll, Michael
Kale, Alex
Kay, Matthew
Venue: IEEE Transactions on Visualization and Computer Graphics
Publisher: Institute of Electrical and Electronics Engineers (IEEE)
Published: 2019-01
Volume: 25
Issue: 1
Pages: 903-913
ISSN: 1077-2626 (ISSN-L)
1941-0506
2160-9306
DOI: 10.1109/tvcg.2018.2864889
OpenAlex: W2891628559
Cited by: 187
References: 93
Citations by year: 2026: 18 · 2025: 20 · 2024: 32 · 2023: 22 · 2022: 23 · 2021: 37 · 2020: 23 · 2019: 11
FWCI: 9.01
Type: journal article
Language: en
Open access: closed
Licence: ieeexplore.ieee.org/Xplorehelp/downloads/license-information/IEEE.html (vor)
doi.org/10.15223/policy-029 (stm-asf)
doi.org/10.15223/policy-037 (stm-asf)
Locations: IEEE Transactions on Visualization and Computer Graphics (publishedVersion)
PubMed (publishedVersion)
Topics: Physical Sciences › Computer Science › Computer Graphics and Computer-Aided Design › Data Visualization and Analytics
Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
Physical Sciences › Computer Science › Computer Graphics and Computer-Aided Design › Computer Graphics and Visualization Techniques
Keywords: uncertainty visualization, visualization evaluation, evaluation taxonomy, decision quality, performance indicators, user satisfaction, elicitation methods, evaluation goals

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

In Pursuit of Error: A Survey of Uncertainty Visualization Evaluation

Abstract

Overview

Integrity signals

Verification

Claims the paper makes

Claims other papers make about this one

Citations

Cites

Cited by

Related

Metadata

Record

About these checks