Hypothetical Outcome Plots Help Untrained Observers Judge Trends in Ambiguous Data
Abstract
Animated representations of outcomes drawn from distributions (hypothetical outcome plots, or HOPs) are used in the media and other public venues to communicate uncertainty. HOPs greatly improve multivariate probability estimation over conventional static uncertainty visualizations and leverage the ability of the visual system to quickly, accurately, and automatically process the summary statistical properties of ensembles. However, it is unclear how well HOPs support applied tasks resembling real world judgments posed in uncertainty communication. We identify and motivate an appropriate task to investigate realistic judgments of uncertainty in the public domain through a qualitative analysis of uncertainty visualizations in the news. We contribute two crowdsourced experiments comparing the effectiveness of HOPs, error bars, and line ensembles for supporting perceptual decision-making from visualized uncertainty. Participants infer which of two possible underlying trends is more likely to have produced a sample of time series data by referencing uncertainty visualizations which depict the two trends with variability due to sampling error. By modeling each participant's accuracy as a function of the level of evidence presented over many repeated judgments, we find that observers are able to correctly infer the underlying trend in samples conveying a lower level of evidence when using HOPs rather than static aggregate uncertainty visualizations as a decision aid. Modeling approaches like ours contribute theoretically grounded and richly descriptive accounts of user perceptions to visualization evaluation.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Cites
Judgment under Uncertainty: Heuristics and Biases
Tversky, A., & Kahneman, D. (1975) · Utility, Probability, and Human Decision Making · doi:10.1007/978-94-010-1834-0_8 · Cited by 22,650 · see citing works
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505 · see citing works
(1966) · Medical Entomology and Zoology · Cited by 10,293 · see citing works
Signal detection theory and psychophysics
Ingleby, D. (1967) · Journal of Sound and Vibration · doi:10.1016/0022-460x(67)90197-6 · Cited by 8,616 · see citing works
The visual display of quantitative information
(1985) · General Pharmacology The Vascular System · doi:10.1016/0306-3623(85)90230-7 · Cited by 5,090 · see citing works
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · see citing works
Why a Diagram is (Sometimes) Worth Ten Thousand Words
Larkin, J. H., & Simon, H. A. (1987) · Cognitive Science · doi:10.1111/j.1551-6708.1987.tb00863.x · Cited by 3,741 · see citing works
Belief in the law of small numbers.
Tversky, A., & Kahneman, D. (1971) · Psychological Bulletin · doi:10.1037/h0031322 · Cited by 3,246 · see citing works
Jacob, M. (1989) · Health Education Journal · doi:10.1177/001789698904800401 · Cited by 3,112 · see citing works
Guidelines for evaluating and expressing the uncertainty of NIST measurement results
Taylor, B. N. (1994) · National Institute of Standards and Technology, NIST Technical Note 1297 · doi:10.6028/nist.tn.1297 · Cited by 2,833 · see citing works
Guidelines for evaluating and expressing the uncertainty of NIST measurement results
Taylor, B. N., & Kuyatt, C. E. (2018) · doi:10.6028/nist.tn.1297-1993 · Cited by 2,582 · see citing works
The psychometric function: I. Fitting, sampling, and goodness of fit
Wichmann, F. A., & Hill, N. J. (2001) · Perception & Psychophysics · doi:10.3758/bf03194544 · Cited by 2,359 · see citing works
How to improve Bayesian reasoning without instruction: Frequency formats.
Gigerenzer, G., & Hoffrage, U. (1995) · Psychological Review · doi:10.1037/0033-295x.102.4.684 · Cited by 2,128 · see citing works
Tversky, B., Morrison, J. B., & Bétrancourt, M. (2002) · International Journal of Human-Computer Studies · doi:10.1006/ijhc.2002.1017 · Cited by 1,677 · see citing works
Decisions from Experience and the Effect of Rare Events in Risky Choice
Hertwig, R., Barron, G., Weber, E. U., & Erev, I. (2004) · Psychological Science · doi:10.1111/j.0956-7976.2004.00715.x · Cited by 1,622 · see citing works
Uncertain Judgements: Eliciting Experts' Probabilities
O'Hagan, A., Buck, C. E., Daneshkhah, A., Eiser, J. R., Garthwaite, P. H., Jenkinson, D. J., Oakley, J. E., & Rakow, T. (2006) · doi:10.1002/0470033312 · Cited by 1,550 · see citing works
The Staircase-Method in Psychophysics
Cornsweet, T. N. (1962) · The American Journal of Psychology · doi:10.2307/1419876 · Cited by 1,346 · see citing works
Representations in Distributed Cognitive Tasks
Zhang, J., & Norman, D. A. (1994) · Cognitive Science · doi:10.1207/s15516709cog1801_3 · Cited by 1,203 · see citing works
Statistical Rethinking: A Bayesian Course with Examples in R and Stan
McElreath, R. L. (2015) · Helmholtz Centre for Ocean Research Kiel (GEOMAR) · Cited by 1,032 · see citing works
Adults' Statistical Literacy: Meanings, Components, Responsibilities
Gal, I. (2002) · International Statistical Review · doi:10.1111/j.1751-5823.2002.tb00336.x · Cited by 905 · see citing works
The visual display of quantitative information
Azen, S. P. (1983) · Computational Statistics & Data Analysis · doi:10.1016/0167-9473(83)90110-x · Cited by 882 · see citing works
Coping with Uncertainty: A Naturalistic Decision-Making Analysis
Lipshitz, R., & Strauss, O. (1997) · Organizational Behavior and Human Decision Processes · doi:10.1006/obhd.1997.2679 · Cited by 878 · see citing works
Statistical Rethinking: a Bayesian Course with Examples in R and Stan
Hewson, P. (2016) · Journal of the Royal Statistical Society Series A (Statistics in Society) · doi:10.1111/rssa.12221 · Cited by 850 · see citing works
Automatic processing of fundamental information: The case of frequency of occurrence.
Hasher, L., & Zacks, R. T. (1984) · American Psychologist · doi:10.1037//0003-066x.39.12.1372 · Cited by 818 · see citing works
Digital Color Imaging Handbook
Sharma, G. (2011) · CRC Press eBooks · doi:10.1201/9781420041484 · Cited by 651 · see citing works
Representing multiple objects as an ensemble enhances visual cognition
Alvarez, G. A. (2011) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2011.01.003 · Cited by 612 · see citing works
Automatic processing of fundamental information: The case of frequency of occurrence.
Hasher, L., & Zacks, R. T. (1984) · American Psychologist · doi:10.1037/0003-066x.39.12.1372 · Cited by 609 · see citing works
Using natural frequencies to improve diagnostic inferences
Hoffrage, U., & Gigerenzer, G. (1998) · Academic Medicine · doi:10.1097/00001888-199805000-00024 · Cited by 501 · see citing works
Forced-choice staircases with fixed step sizes: asymptotic and small-sample properties
García-Pérez, M. Á. (1998) · Vision Research · doi:10.1016/s0042-6989(97)00340-4 · Cited by 429 · see citing works
Robust misinterpretation of confidence intervals
Hoekstra, R., Morey, R. D., Rouder, J. N., & Wagenmakers, E.-J. (2014) · Psychonomic Bulletin & Review · doi:10.3758/s13423-013-0572-3 · Cited by 425 · see citing works
The Representation of Simple Ensemble Visual Features Outside the Focus of Attention
Alvarez, G. A., & Oliva, A. (2008) · Psychological Science · doi:10.1111/j.1467-9280.2008.02098.x · Cited by 394 · see citing works
Researchers Misunderstand Confidence Intervals and Standard Error Bars.
Belia, S., Fidler, F., Williams, J., & Cumming, G. (2005) · Psychological Methods · doi:10.1037/1082-989x.10.4.389 · Cited by 390 · see citing works
The Challenge of Developing Statistical Reasoning
Garfield, J. (2002) · Journal of Statistics Education · doi:10.1080/10691898.2002.11910676 · Cited by 389 · see citing works
Seeing the mean: Ensemble coding for sets of faces.
Haberman, J. M., & Whitney, D. C. (2009) · Journal of Experimental Psychology Human Perception & Performance · doi:10.1037/a0013899 · Cited by 365 · see citing works
Representation construction, externalised cognition and individual differences
Cox, R. (1999) · Learning and Instruction · doi:10.1016/s0959-4752(98)00051-6 · Cited by 348 · see citing works
Error Bars Considered Harmful: Exploring Alternate Encodings for Mean and Error
Correll, M., & Gleicher, M. (2014) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2014.2346298 · Cited by 258 · see citing works
Signatures of a Statistical Computation in the Human Sense of Confidence
Sanders, J. I., Hangya, B., & Kepecs, A. (2016) · Neuron · doi:10.1016/j.neuron.2016.03.025 · Cited by 253 · see citing works
Kay, M., Kola, T., Hullman, J. R., & Munson, S. A. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858558 · Cited by 239 · see citing works
Self-citationModeling Color Difference for Visualization Design
Szafir, D. A. (2017) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2017.2744359 · Cited by 213 · see citing works
Hullman, J., Resnick, P., & Adar, E. (2015) · PLOS ONE · doi:10.1371/journal.pone.0142444 · Cited by 194 · see citing works
Self-citationSelecting Semantically‐Resonant Colors for Data Visualization
Lin, S., Fortuna, J., Kulkarni, C. E., Stone, M., & Heer, J. (2013) · Computer Graphics Forum · doi:10.1111/cgf.12127 · Cited by 188 · see citing works
From Quantification to Visualization: A Taxonomy of Uncertainty Visualization Approaches
Potter, K. C., Rosen, P., & Johnson, C. R. (2012) · IFIP International Federation for Information Processing/IFIP · doi:10.1007/978-3-642-32677-6_15 · Cited by 186 · see citing works
Uncertainty forecasts improve weather-related decisions and attenuate the effects of forecast error.
Joslyn, S. L., & LeClerc, J. E. (2011) · Journal of Experimental Psychology Applied · doi:10.1037/a0025185 · Cited by 186 · see citing works
Variables on Scatterplots Look More Highly Correlated When the Scales Are Increased
Cleveland, W. S., Diaconis, P. W., & McGill, R. (1982) · Science · doi:10.1126/science.216.4550.1138 · Cited by 177 · see citing works
Uncertainty Displays Using Quantile Dotplots or CDFs Improve Transit Decision-Making
Fernandes, M., Walls, L., Munson, S., Hullman, J., & Kay, M. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173718 · Cited by 175 · see citing works
Self-citationGeographic information systems and science: today and tomorrow
Goodchild, M. F. (2009) · Procedia Earth and Planetary Science · doi:10.1016/j.proeps.2009.09.160 · Cited by 170 · see citing works
Visualizing spatial data uncertainty using animation
Ehlschlaeger, C., Shortridge, A. M., & Goodchild, M. F. (1997) · Computers & Geosciences · doi:10.1016/s0098-3004(97)00005-8 · Cited by 160 · see citing works
Benefitting InfoVis with Visual Difficulties
Hullman, J., Adar, E., & Shah, P. (2011) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2011.175 · Cited by 159 · see citing works
Self-citationUncertainty Visualization in Medical Volume Rendering Using Probabilistic Animation
Lundström, C. F., Ljung, P., Persson, A., & Ynnerman, A. (2007) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2007.70518 · Cited by 137 · see citing works
Dynamic display of spatial data-reliability: Does it benefit the map user?
Evans, B. J. (1997) · Computers & Geosciences · doi:10.1016/s0098-3004(97)00011-3 · Cited by 120 · see citing works
Lay understanding of probability distributions
Goldstein, D. G., & Rothschild, D. M. (2014) · Judgment and Decision Making · doi:10.1017/s1930297500004940 · Cited by 116 · see citing works
Perceptually Averaging in a Continuous Visual World
Albrecht, A. R., & Scholl, B. J. (2010) · Psychological Science · doi:10.1177/0956797610363543 · Cited by 110 · see citing works
Effects of ensemble and summary displays on interpretations of geospatial uncertainty data
Padilla, L., Ruginski, I. T., & Creem-Regehr, S. H. (2017) · Cognitive Research Principles and Implications · doi:10.1186/s41235-017-0076-1 · Cited by 105 · see citing works
Bar graphs depicting averages are perceptually misinterpreted: The within-the-bar bias
Newman, G. E., & Scholl, B. J. (2012) · Psychonomic Bulletin & Review · doi:10.3758/s13423-012-0247-5 · Cited by 101 · see citing works
Fast ensemble representations for abstract visual impressions
Leib, A. Y., Kosovicheva, A. A., & Whitney, D. C. (2016) · Nature Communications · doi:10.1038/ncomms13186 · Cited by 94 · see citing works
Not all summary statistics are made equal: Evidence from extracting summaries across time
Hubert‐Wallander, B., & Boynton, G. M. (2015) · Journal of Vision · doi:10.1167/15.4.5 · Cited by 92 · see citing works
The illusion of predictability: How regression statistics mislead experts
Soyer, E., & Hogarth, R. M. (2012) · International Journal of Forecasting · doi:10.1016/j.ijforecast.2012.02.002 · Cited by 84 · see citing works
Shen, Y., & Richards, V. M. (2012) · The Journal of the Acoustical Society of America · doi:10.1121/1.4733540 · Cited by 81 · see citing works
Uncertainty Visualization by Representative Sampling from Prediction Ensembles
Liu, L., Boone, A. P., Ruginski, I. T., Padilla, L., Hegarty, M., Creem-Regehr, S. H., Thompson, W. B., Yuksel, C., & House, D. H. (2016) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2016.2607204 · Cited by 76 · see citing works
Hullman, J., Kay, M., Kim, Y.-S., & Shrestha, S. (2018) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2017.2743898 · Cited by 55 · see citing works
Self-citationCognitive Efficiency Considerations for Good Graphic Design
Casner, S. M., & Larkin, J. H. (1989) · doi:10.21236/ada218976 · Cited by 43 · see citing works
The influence of different graphical displays on nonexpert decision making under uncertainty.
Padilla, L., Hansen, G., Ruginski, I. T., Kramer, H., Thompson, W. B., & Creem-Regehr, S. H. (2014) · Journal of Experimental Psychology Applied · doi:10.1037/xap0000037 · Cited by 39 · see citing works
Binder, A. R., Hillback, E. D., & Brossard, D. (2016) · Risk Analysis · doi:10.1111/risa.12462 · Cited by 38 · see citing works
Evaluating the Perception of Uncertainty in Alternative Visualization Strategies
Blenkinsop, S., Fisher, P. F., Bastin, L., & Wood, J. (2000) · Cartographica The International Journal for Geographic Information and Geovisualization · doi:10.3138/3645-4v22-0m23-3t52 · Cited by 36 · see citing works
(2016) · doi:10.1016/c2012-0-01278-1 · Cited by 21 · see citing works
Excessive confidence in visually-based estimates
Andrade, E. B. (2011) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2011.07.002 · Cited by 17 · see citing works
The representation of ensemble visual features outside the focus of attention
Alvarez, G. A., & Oliva, A. (2010) · Journal of Vision · doi:10.1167/7.9.129 · Cited by 11 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
The Science of Visual Data Communication: What Works
Franconeri, S., Padilla, L., Shah, P., Zacks, J. M., & Hullman, J. R. (2021) · Gothic.net · doi:10.1177/15291006211051956 · Cited by 398 · see citing works
Self-citationIn Pursuit of Error: A Survey of Uncertainty Visualization Evaluation
Hullman, J., Qiao, X., Correll, M., Kale, A., & Kay, M. (2019) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2018.2864889 · Cited by 187 · see citing works
Self-citationWhy Authors Don't Visualize Uncertainty
Hullman, J. R. (2019) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2019.2934287 · Cited by 177 · see citing works
Self-citationggdist: Visualizations of Distributions and Uncertainty in the Grammar of Graphics
Kay, M. (2023) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2023.3327195 · Cited by 185 · see citing works
Self-citationIncreasing the Transparency of Research Papers with Explorable Multiverse Analyses
Dragicevic, P., Jansen, Y., Sarma, A., Kay, M., & Chevalier, F. (2019) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300295 · Cited by 123 · see citing works
Self-citationVisual Reasoning Strategies for Effect Size Judgments and Decisions
Kale, A., Kay, M., & Hullman, J. (2021) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2020.3030335 · Cited by 109 · see citing works
Self-citationPadilla, L., Kay, M., & Hullman, J. R. (2021) · Wiley StatsRef: Statistics Reference Online · doi:10.1002/9781118445112.stat08296 · Cited by 87 · see citing works
Self-citationA Bayesian Cognition Approach to Improve Data Visualization
Kim, Y.-S., Walls, L. A., Krafft, P., & Hullman, J. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300912 · Cited by 76 · see citing works
Self-citationPadilla, L. M. K., Powell, M., Kay, M., & Hullman, J. (2021) · Frontiers in Psychology · doi:10.3389/fpsyg.2020.579267 · Cited by 74 · see citing works
Self-citationIllusion of Causality in Visualized Data
Xiong, C., Shapiro, J., Hullman, J., & Franconeri, S. (2020) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2019.2934399 · Cited by 71 · see citing works
Self-citationWhat Makes a Data-GIF Understandable?
Shu, X., Wu, A., Tang, J., Bach, B., Wu, Y., & Qu, H. (2020) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2020.3030396 · Cited by 64 · see citing works
Gemini: A Grammar and Recommender System for Animated Transitions in Statistical Graphics
Kim, Y.-H., & Heer, J. (2020) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2020.3030360 · Cited by 57 · see citing works
A Critical Reflection on Visualization Research: Where Do Decision Making Tasks Hide?
Dimara, E., & Stasko, J. (2022) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2021.3114813 · Cited by 60 · see citing works
Hofman, J. M., Goldstein, D. G., & Hullman, J. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376454 · Cited by 75 · see citing works
Self-citationMan & machine: Adaptive tools for the contemporary performance analyst
Robertson, S. J. (2020) · Journal of Sports Sciences · doi:10.1080/02640414.2020.1774143 · Cited by 57 · see citing works
Uncertainty Visualization: Concepts, Methods, and Applications in Biological Data Visualization
Weiskopf, D. (2022) · Frontiers in Bioinformatics · doi:10.3389/fbinf.2022.793819 · Cited by 50 · see citing works
Visualizing Privacy-Utility Trade-Offs in Differentially Private Data Releases
Nanayakkara, P., Bater, J., He, X. Q., Hullman, J. R., & Rogers, J. (2022) · Proceedings on Privacy Enhancing Technologies · doi:10.2478/popets-2022-0058 · Cited by 52 · see citing works
Self-citationSeeing What You Believe or Believing What You See? Belief Biases Correlation Estimation
Xiong, C., Stokes, C., Kim, Y., & Franconeri, S. (2022) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2022.3209405 · Cited by 56 · see citing works
A Design Space of Vision Science Methods for Visualization Research
Elliott, M., Nothelfer, C. E., Xiong, C., & Szafir, D. A. (2020) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2020.3029413 · Cited by 47 · see citing works
Designing for Interactive Exploratory Data Analysis Requires Theories of Graphical Inference
Hullman, J. R., & Gelman, A. (2021) · Harvard Data Science Review · doi:10.1162/99608f92.3ab8a587 · Cited by 50 · see citing works
Self-citationPadilla, L., Fygenson, R., Castro, S. C., & Bertini, E. (2022) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2022.3209457 · Cited by 45 · see citing works
Visualizing Uncertainty for Non-Expert End Users: The Challenge of the Deterministic Construal Error
Joslyn, S. L., & Savelli, S. (2021) · Frontiers in Computer Science · doi:10.3389/fcomp.2020.590232 · Cited by 41 · see citing works
Bayesian-Assisted Inference from Visualized Data
Kim, Y., Kayongo, P., Grunde-McLaughlin, M., & Hullman, J. R. (2020) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2020.3028984 · Cited by 43 · see citing works
Self-citationDecision-Making Under Uncertainty in Research Synthesis
Kale, A. M., Kay, M., & Hullman, J. R. (2019) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300432 · Cited by 49 · see citing works
Self-citationKarduni, A., Markant, D. B., Wesslen, R. S., & Wenwen Dou, A. (2020) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2020.3029412 · Cited by 37 · see citing works
Castro, S. C., Quinan, P. S., Hosseinpour, H., & Padilla, L. (2021) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2021.3114803 · Cited by 37 · see citing works
Animated Vega-Lite: Unifying Animation with a Grammar of Interactive Graphics
Zong, J., Pollock, J., Wootton, D., & Satyanarayan, A. (2022) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2022.3209369 · Cited by 38 · see citing works
Helske, J., Helske, S., Cooper, M., Ynnerman, A., & Besancon, L. (2021) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2021.3073466 · Cited by 38 · see citing works
Datamations: Animated Explanations of Data Analysis Pipelines
Pu, X., Kross, S., Hofman, J. M., & Goldstein, D. G. (2021) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445063 · Cited by 32 · see citing works
How Do We Measure Trust in Visual Data Communication?
Elhamdadi, H., Gaba, A., Kim, Y., & Xiong, C. (2022) · IEEE Evaluation and Beyond - Methodological Approaches for Visualization (BELIV) · doi:10.1109/beliv57783.2022.00014 · Cited by 28 · see citing works
Impact of Cognitive Biases on Progressive Visualization
Procopio, M., Mosca, A., Scheidegger, C., Wu, E., & Chang, R. (2021) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2021.3051013 · Cited by 25 · see citing works
Uncertainty VisualizationPreprint
Padilla, L., Kay, M., & Hullman, J. R. (2020) · doi:10.31234/osf.io/ebd6r · Cited by 28 · see citing works
Self-citationMethodological Considerations for Furthering the Understanding of Constraints in Applied Sports
Browne, P. R., Sweeting, A. J., Woods, C. T., & Robertson, S. J. (2021) · Sports Medicine - Open · doi:10.1186/s40798-021-00313-x · Cited by 28 · see citing works
Communicating Uncertainty in Digital Humanities Visualization Research
Panagiotidou, G., Lamqaddam, H., Poblome, J., Brosens, K., Verbert, K., & Moere, A. V. (2022) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2022.3209436 · Cited by 24 · see citing works
Holder, E., & Xiong, C. (2022) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2022.3209377 · Cited by 25 · see citing works
Show or suppress? Managing input uncertainty in machine learning model explanations
Wang, D., Zhang, W., & Lim, B. Y. (2021) · Artificial Intelligence · doi:10.1016/j.artint.2021.103456 · Cited by 22 · see citing works
Revealing Perceptual Proxies with Adversarial Examples
Ondov, B., Yang, F., Kay, M., Elmqvist, N., & Franconeri, S. (2020) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2020.3030429 · Cited by 22 · see citing works
Self-citationA review of uncertainty visualization errors: Working memory as an explanatory theory
Padilla, L. M. K., Castro, S., & Hosseinpour, H. (2021) · The Psychology of learning and motivation/The psychology of learning and motivation · doi:10.1016/bs.plm.2021.03.001 · Cited by 17 · see citing works
Davis, R., Pu, X., Ding, Y., Hall, B. D., Bonilla, K., Feng, M., Kay, M., & Harrison, L. (2022) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2022.3226463 · Cited by 19 · see citing works
Self-citationVisualizing Uncertainty in Probabilistic Graphs with Network Hypothetical Outcome Plots (NetHOPs)
Zhang, D., Adar, E., & Hullman, J. R. (2021) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2021.3114679 · Cited by 17 · see citing works
Self-citationGemini2: Generating Keyframe-Oriented Animated Transitions Between Statistical Graphics
Kim, Y.-H., & Heer, J. (2021) · Visualization · doi:10.1109/vis49827.2021.9623291 · Cited by 20 · see citing works
Sarma, A., Guo, S., Hoffswell, J., Rossi, R. A., Du, F., Koh, E., & Kay, M. (2022) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2022.3209348 · Cited by 17 · see citing works
Self-citationEVM: Incorporating Model Checking into Exploratory Visual Analysis
Kale, A. M., Guo, Z., Qiao, X. L., Heer, J., & Hullman, J. R. (2023) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2023.3326516 · Cited by 18 · see citing works
Self-citationMarkant, D. B., Rogha, M., Karduni, A., Wesslen, R. S., & Wenwen Dou, A. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581330 · Cited by 16 · see citing works
Designing for Ambiguity: Visual Analytics in Avalanche Forecasting
Nowak, S., Bartram, L., & Haegeli, P. (2020) · Visualization · doi:10.1109/vis47514.2020.00023 · Cited by 15 · see citing works
Generative Models: An Interdisciplinary Perspective
Sankaran, K., & Holmes, S. (2022) · Annual Review of Statistics and Its Application · doi:10.1146/annurev-statistics-033121-110134 · Cited by 14 · see citing works
Subjective Probability Correction for Uncertainty Representations
Yang, F., Hedayati, M., & Kay, M. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580998 · Cited by 12 · see citing works
Self-citationLeffrang, D., & Muller, O. (2021) · IEEE Workshop on TRust and EXpertise in Visual Analytics (TREX) · doi:10.1109/trex53765.2021.00009 · Cited by 14 · see citing works
Xexakis, G., & Trutnevyte, E. (2021) · Global Environmental Change · doi:10.1016/j.gloenvcha.2021.102324 · Cited by 15 · see citing works
Sondag, M., Meulemans, W., Schulz, C., Verbeek, K., Weiskopf, D., & Speckmann, B. (2020) · IEEE Pacific Visualization Symposium · doi:10.1109/pacificvis48177.2020.7614 · Cited by 13 · see citing works
Evaluating Gradient Perception in Color-Coded Scalar Fields
Reda, K., & Papka, M. E. (2019) · Visualization · doi:10.1109/visual.2019.8933760 · Cited by 13 · see citing works
Nguyen, F., Qiao, X., Heer, J., & Hullman, J. R. (2020) · Computer Graphics Forum · doi:10.1111/cgf.13902 · Cited by 13 · see citing works
Self-citationWesslen, R., Karduni, A., Markant, D., & Dou, W. (2021) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2021.3114692 · Cited by 15 · see citing works
Simple methods for improving the communication of uncertainty in species’ temporal trends
Pescott, O., Stroh, P. A., Humphrey, T., & Walker, K. (2022) · Ecological Indicators · doi:10.1016/j.ecolind.2022.109117 · Cited by 12 · see citing works
Karjack, S., Brudzinski, M. R., & Shipley, T. F. (2022) · Seismological Research Letters · doi:10.1785/0220210318 · Cited by 11 · see citing works
Stressors affecting the ecological status of temporary rivers in the Mediterranean region
Lorenz, A. W., Kaijser, W., Acuña, V., Austnes, K., Bonada, N., Dörflinger, G., Ferreira, T., Karaouzas, I., Rico, A., & Hering, D. (2023) · The Science of The Total Environment · doi:10.1016/j.scitotenv.2023.166254 · Cited by 12 · see citing works
Decision-Making Under Uncertainty in Research Synthesis: Designing for the Garden of Forking PathsPreprint
Kale, A. M., Kay, M., & Hullman, J. R. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.02957 · Cited by 12 · see citing works
Self-citationCiccione, L., Sablé-Meyer, M., Boissin, E., Josserand, M., Watkins, C. P., Caparos, S., & Dehaene, S. (2023) · Scientific Reports · doi:10.1038/s41598-023-37172-3 · Cited by 11 · see citing works
Studying Early Decision Making with Progressive Bar Charts
Patil, A., Richer, G., Jermaine, C., Moritz, D., & Fekete, J. (2022) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2022.3209426 · Cited by 11 · see citing works
Modelling the Influence of Task Constraints on Goal Kicking Performance in Australian Rules Football
Browne, P. R., Sweeting, A. J., & Robertson, S. J. (2022) · Sports Medicine - Open · doi:10.1186/s40798-021-00393-9 · Cited by 10 · see citing works
Masson, D., Malacria, S., Casiez, G., & Vogel, D. (2023) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3586183.3606762 · Cited by 11 · see citing works
Some Prior(s) Experience Necessary
Phelan, C., Hullman, J. R., Kay, M., & Resnick, P. (2019) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300709 · Cited by 10 · see citing works
Self-citationWhat are Table Cartograms Good for Anyway? An Algebraic Analysis
McNutt, A. M. (2021) · Computer Graphics Forum · doi:10.1111/cgf.14289 · Cited by 10 · see citing works
Wang, X., Li, Y., & Xue, C. (2024) · Electronics · doi:10.3390/electronics13153004 · Cited by 8 · see citing works
Capture, analyse, visualise: An exemplar of performance analysis in practice in field hockey
Lord, F., Pyne, D. B., Welvaert, M., & Mara, J. K. (2022) · PLoS ONE · doi:10.1371/journal.pone.0268171 · Cited by 8 · see citing works
Odds and Insights: Decision Quality in Exploratory Data Analysis Under Uncertainty
Sarma, A., Pu, X., Cui, Y., Correll, M., Brown, E. T., & Kay, M. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3641995 · Cited by 9 · see citing works
Self-citationElection predictions in the news: how users perceive and respond to visual election forecasts
Witzenberger, B., & Diakopoulos, N. A. (2023) · Information Communication & Society · doi:10.1080/1369118x.2023.2230267 · Cited by 7 · see citing works
Beecham, R., & Lovelace, R. (2022) · Geographical Analysis · doi:10.1111/gean.12338 · Cited by 7 · see citing works
Decoupling Judgment and Decision Making: A Tale of Two Tails
Oral, B., Dragicevic, P., Telea, A. C., & Dimara, E. (2023) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2023.3346640 · Cited by 7 · see citing works
MetaExplorer : Facilitating Reasoning with Epistemic Uncertainty in Meta-analysis
Kale, A. M., Lee, S., Goan, T., Tipton, E., & Hullman, J. R. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580869 · Cited by 7 · see citing works
Self-citationSeeing Through the Fog: The Ability to Resolve Ambiguity Reduces Dishonesty
Puntiroli, M., Kandul, S., Bezençon, V., & Lanz, B. (2025) · Personality and Social Psychology Bulletin · doi:10.1177/01461672241305687 · Cited by 5 · see citing works
Scalable Interactive Machine Learning for Future Command and Control
Madison, A., Novoseller, E., Goecks, V. G., Files, B. T., Waytowich, N. R., Yu, A., Lawhern, V. J., Thurman, S. M., Kelshaw, C., & McDowell, K. (2024) · International Conference on Military Communications and Information Systems (ICMCIS) · doi:10.1109/icmcis61231.2024.10540933 · Cited by 7 · see citing works
Communicating qualitative uncertainty in data visualization
Panagiotidou, G., & Moere, A. V. (2022) · Information Design Journal · doi:10.1075/idj.22014.pan · Cited by 4 · see citing works
Perceptual and Cognitive Foundations of Information Visualization
Schloss, K. B. (2025) · Annual Review of Vision Science · doi:10.1146/annurev-vision-110323-110009 · Cited by 7 · see citing works
Motion-Based Visual Encoding Can Improve Performance on Perceptual Tasks with Dynamic Time Series
Hu, S., Jiang, O., Riedmiller, J., & Xiong, C. (2024) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2024.3456405 · Cited by 4 · see citing works
Visualization of Emergency Needs Posted on Social Media by Metaphor Map
Ma, X., Cui, K., Ji, C., & Xue, P. (2020) · Data and Information Management · doi:10.2478/dim-2020-0021 · Cited by 4 · see citing works
Vázquez‐Ingelmo, A., García‐Holgado, A., García‐Peñalvo, F. J., & Therón, R. (2021) · Advances in intelligent systems and computing · doi:10.1007/978-3-030-72651-5_54 · Cited by 4 · see citing works
Visualising Temporal Uncertainty: A Taxonomy and Call for Systematic Evaluation
Grewal, Y. S., Goodwin, S., & Dwyer, T. (2021) · IEEE Pacific Visualization Symposium · doi:10.1109/pacificvis52677.2021.00013 · Cited by 4 · see citing works
Impact of Vertical Scaling on Normal Probability Density Function Plots
Fygenson, R., & Padilla, L. (2024) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2024.3456396 · Cited by 3 · see citing works
Ehlers, H., Pahr, D., Bartolomeo, S. D., Filipov, V., Wu, H., & Raidou, R. G. (2025) · Computers & Graphics · doi:10.1016/j.cag.2025.104290 · Cited by 3 · see citing works
Evaluation of 1D Selection Techniques for Mobile Visualizations
Schwab, M., Pandey, A., & Borkin, M. A. (2021) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3411763.3451650 · Cited by 3 · see citing works
Wang, X., Hu, R., & Xue, C. (2024) · Symmetry · doi:10.3390/sym16080986 · Cited by 2 · see citing works
Noa-Yarasca, E., Babbar-Sebens, M., Chiou, M., & Macuga, K. L. (2025) · Journal of Cognitive Engineering and Decision Making · doi:10.1177/15553434251324537 · Cited by 2 · see citing works
Designing Shared Information Displays for Agents of Varying Strategic Sophistication
Zhang, D., Hartline, J. D., & Hullman, J. R. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3637319 · Cited by 3 · see citing works
Self-citationPlatform-based Adaptive Experimental Research in Education
Леонидович, М. И., Reza, M., Song, H., Moore, S., Chen, P., Kumar, H., Li, T., Stamper, J. C., Bier, N., Rafferty, A. N., et al. (2025) · International Conference on Learning Analytics and Knowledge (LAK) · doi:10.1145/3706468.3706471 · Cited by 3 · see citing works
A design space of animating data-driven transitions in data videos
Zheng, C., Gao, T., Guo, S., Shi, Y., Ma, X., & Cao, N. (2025) · Journal of Visualization · doi:10.1007/s12650-025-01066-5 · Cited by 2 · see citing works
Pushing Visualization Research Frontiers: Essential Topics Not Addressed by Machine Learning
Ma, K. (2023) · IEEE Computer Graphics and Applications · doi:10.1109/mcg.2022.3225692 · Cited by 2 · see citing works
To design interfaces for exploratory data analysis, we need theories of graphical inferencePreprint
Hullman, J. R., & Gelman, A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.02015 · Cited by 2 · see citing works
Self-citationAniVis: Generating Animated Transitions Between Statistical Charts with a Tree ModelPreprint
Li, W., Wang, Y., Huang, H., Cui, W., Zhang, H., Qu, H., & Zhang, D. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.14313 · Cited by 2 · see citing works
Gemini2: Generating Keyframe-Oriented Animated Transitions Between Statistical GraphicsPreprint
Kim, Y., & Heer, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.04385 · Cited by 2 · see citing works
Matzen, L. E., Rogers, A., & Howell, B. C. (2023) · Springer eBooks · doi:10.1007/978-3-031-34738-2_14 · Cited by 1 · see citing works
Bancilhon, M., Ottley, A., & Jordan, A. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713096 · Cited by 1 · see citing works
Macrophytes and their sedimentary phosphorus niche in lowland rivers
Kaijser, W., Brauer, V. S., Schürings, C., & Lorenz, A. W. (2025) · PLoS ONE · doi:10.1371/journal.pone.0330460 · Cited by 1 · see citing works
Interacting with Sensor Data to Understand Uncertainty in Air Quality Monitoring
Pondaven, E., Liu, Z., Eri Saikawa, B., Jacques, J. T., & Williams, K. (2025) · International Conference on ICT for Sustainability (ICT4S) · doi:10.1109/ict4s68164.2025.00029 · Cited by 1 · see citing works
Evaluating Judgment of Spatial Correlation in Visual Displays of Scalar Field Distributions
Zhao, Y., & Berger, M. (2025) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2025.3633830 · Cited by 1 · see citing works
Uncertainty displays in pandemic dashboards: a review of COVID-19 use cases
Tessema, T., Shahezian, M., Vasquez, H. M., Rosella, L., & Donmez, B. (2026) · Frontiers in Digital Health · doi:10.3389/fdgth.2026.1869108 · Cited by 1 · see citing works
Let's Gamble: Uncovering the Impact of Visualization on Risk Perception and Decision-MakingPreprint
Bancilhon, M., Liu, Z., & Ottley, A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.09725 · Cited by 1 · see citing works
Data visualization tool for student dropouts in Tanzania
Kayanda, A. M. (2020) · doi:10.58694/20.500.12479/901 · Cited by 1 · see citing works
Dynamic Glyphs: Appropriating Causality Perception in Multivariate Visual AnalysisPreprint
Reda, K., Potts, C., & Childers, T. (2019) · doi:10.31219/osf.io/s5q9n · Cited by 1 · see citing works
- Authors
- Kale, A., Nguyen, F., Kay, M., & Hullman, J.
- Affiliations
- University of Washington (US)
- University of Michigan (US)
- Northwestern University (US)
- Venue
- IEEE Transactions on Visualization and Computer Graphics
- Published
- 2019-01
- Volume
- 25
- Issue
- 1
- Pages
- 892-902
- ISSN
- 1077-2626 (ISSN-L)
- 1941-0506
- 2160-9306
- OpenAlex
- W2888554701
- Cited by
- 122
- References
- 67
- Citations by year
- 2026: 8 · 2025: 11 · 2024: 10 · 2023: 14 · 2022: 20 · 2021: 27 · 2020: 19 · 2019: 12 · 2018: 1
- FWCI
- 7.38
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- Locations
- IEEE Transactions on Visualization and Computer Graphics (publishedVersion, public-domain)
- PubMed (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Data Visualization and Analytics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Visual and Cognitive Learning Processes
- Physical Sciences › Mathematics › Statistics and Probability › Statistics Education and Methodologies
- Keywords
- uncertainty communication, perceptual decision making, error bars, sampling error, news visualization, visualization evaluation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
