The DOI matches its Crossref record; 67 of the 67 references it cites were found in a registry or an index.
Hypothetical Outcome Plots Help Untrained Observers Judge Trends in Ambiguous Data
Abstract
Animated representations of outcomes drawn from distributions (hypothetical outcome plots, or HOPs) are used in the media and other public venues to communicate uncertainty. HOPs greatly improve multivariate probability estimation over conventional static uncertainty visualizations and leverage the ability of the visual system to quickly, accurately, and automatically process the summary statistical properties of ensembles. However, it is unclear how well HOPs support applied tasks resembling real world judgments posed in uncertainty communication. We identify and motivate an appropriate task to investigate realistic judgments of uncertainty in the public domain through a qualitative analysis of uncertainty visualizations in the news. We contribute two crowdsourced experiments comparing the effectiveness of HOPs, error bars, and line ensembles for supporting perceptual decision-making from visualized uncertainty. Participants infer which of two possible underlying trends is more likely to have produced a sample of time series data by referencing uncertainty visualizations which depict the two trends with variability due to sampling error. By modeling each participant's accuracy as a function of the level of evidence presented over many repeated judgments, we find that observers are able to correctly infer the underlying trend in samples conveying a lower level of evidence when using HOPs rather than static aggregate uncertainty visualizations as a decision aid. Modeling approaches like ours contribute theoretically grounded and richly descriptive accounts of user perceptions to visualization evaluation.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Judgment under Uncertainty: Heuristics and Biases
Tversky, A., & Kahneman, D. (1975) · Utility, Probability, and Human Decision Making · doi:10.1007/978-94-010-1834-0_8 · Cited by 22,650
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
(1966) · Medical Entomology and Zoology · Cited by 10,293
Signal detection theory and psychophysics
Ingleby, D. (1967) · Journal of Sound and Vibration · doi:10.1016/0022-460x(67)90197-6 · Cited by 8,616
The visual display of quantitative information
(1985) · General Pharmacology The Vascular System · doi:10.1016/0306-3623(85)90230-7 · Cited by 5,090
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · cited 1×
“Among scientists, misunderstandings of sampling error and the likelihood of replicating experimental results contribute to rampant use of underpowered studies and the “replication crisis” [37, 62].”
BackgroundWhy a Diagram is (Sometimes) Worth Ten Thousand Words
Larkin, J. H., & Simon, H. A. (1987) · Cognitive Science · doi:10.1111/j.1551-6708.1987.tb00863.x · Cited by 3,855
Belief in the law of small numbers.
Tversky, A., & Kahneman, D. (1971) · Psychological Bulletin · doi:10.1037/h0031322 · Cited by 3,246
Jacob, M. (1989) · Health Education Journal · doi:10.1177/001789698904800401 · Cited by 3,123
Guidelines for evaluating and expressing the uncertainty of NIST measurement results
Taylor, B. N. (1994) · National Institute of Standards and Technology, NIST Technical Note 1297 · doi:10.6028/nist.tn.1297 · Cited by 2,833
Guidelines for evaluating and expressing the uncertainty of NIST measurement results
Taylor, B. N., & Kuyatt, C. E. (2018) · doi:10.6028/nist.tn.1297-1993 · Cited by 2,582
The psychometric function: I. Fitting, sampling, and goodness of fit
Wichmann, F. A., & Hill, N. J. (2001) · Perception & Psychophysics · doi:10.3758/bf03194544 · Cited by 2,359
How to improve Bayesian reasoning without instruction: Frequency formats.
Gigerenzer, G., & Hoffrage, U. (1995) · Psychological Review · doi:10.1037/0033-295x.102.4.684 · Cited by 2,190
Tversky, B., Morrison, J. B., & Bétrancourt, M. (2002) · International Journal of Human-Computer Studies · doi:10.1006/ijhc.2002.1017 · Cited by 1,677
Uncertain Judgements: Eliciting Experts' Probabilities
O'Hagan, A., Buck, C. E., Daneshkhah, A., Eiser, J. R., Garthwaite, P. H., Jenkinson, D. J., Oakley, J. E., & Rakow, T. (2006) · doi:10.1002/0470033312 · Cited by 1,550
Adults' Statistical Literacy: Meanings, Components, Responsibilities
Gal, I. (2002) · International Statistical Review · doi:10.1111/j.1751-5823.2002.tb00336.x · Cited by 1,350
The Staircase-Method in Psychophysics
Cornsweet, T. N. (1962) · The American Journal of Psychology · doi:10.2307/1419876 · Cited by 1,346
Decisions from Experience and the Effect of Rare Events in Risky Choice
Hertwig, R., Barron, G., Weber, E. U., & Erev, I. (2004) · Psychological Science · doi:10.1111/j.0956-7976.2004.00715.x · Cited by 1,225
Representations in Distributed Cognitive Tasks
Zhang, J., & Norman, D. A. (1994) · Cognitive Science · doi:10.1207/s15516709cog1801_3 · Cited by 1,203
Statistical Rethinking: A Bayesian Course with Examples in R and Stan
McElreath, R. L. (2015) · Helmholtz Centre for Ocean Research Kiel (GEOMAR) · Cited by 996
Coping with Uncertainty: A Naturalistic Decision-Making Analysis
Lipshitz, R., & Strauss, O. (1997) · Organizational Behavior and Human Decision Processes · doi:10.1006/obhd.1997.2679 · Cited by 922
The visual display of quantitative information
Azen, S. P. (1983) · Computational Statistics & Data Analysis · doi:10.1016/0167-9473(83)90110-x · Cited by 882
Statistical Rethinking: a Bayesian Course with Examples in R and Stan
Hewson, P. (2016) · Journal of the Royal Statistical Society Series A (Statistics in Society) · doi:10.1111/rssa.12221 · Cited by 850
Automatic processing of fundamental information: The case of frequency of occurrence.
Hasher, L., & Zacks, R. T. (1984) · American Psychologist · doi:10.1037//0003-066x.39.12.1372 · Cited by 818
Digital Color Imaging Handbook
Sharma, G. (2011) · CRC Press eBooks · doi:10.1201/9781420041484 · Cited by 651
Representing multiple objects as an ensemble enhances visual cognition
Alvarez, G. A. (2011) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2011.01.003 · Cited by 612
Automatic processing of fundamental information: The case of frequency of occurrence.
Hasher, L., & Zacks, R. T. (1984) · American Psychologist · doi:10.1037/0003-066x.39.12.1372 · Cited by 609
Using natural frequencies to improve diagnostic inferences
Hoffrage, U., & Gigerenzer, G. (1998) · Academic Medicine · doi:10.1097/00001888-199805000-00024 · Cited by 501
Forced-choice staircases with fixed step sizes: asymptotic and small-sample properties
García-Pérez, M. Á. (1998) · Vision Research · doi:10.1016/s0042-6989(97)00340-4 · Cited by 429
The Challenge of Developing Statistical Reasoning
Garfield, J. (2002) · Journal of Statistics Education · doi:10.1080/10691898.2002.11910676 · Cited by 429
Robust misinterpretation of confidence intervals
Hoekstra, R., Morey, R. D., Rouder, J. N., & Wagenmakers, E.-J. (2014) · Psychonomic Bulletin & Review · doi:10.3758/s13423-013-0572-3 · Cited by 425
The Representation of Simple Ensemble Visual Features Outside the Focus of Attention
Alvarez, G. A., & Oliva, A. (2008) · Psychological Science · doi:10.1111/j.1467-9280.2008.02098.x · Cited by 394
Researchers Misunderstand Confidence Intervals and Standard Error Bars.
Belia, S., Fidler, F., Williams, J., & Cumming, G. (2005) · Psychological Methods · doi:10.1037/1082-989x.10.4.389 · Cited by 390
Seeing the mean: Ensemble coding for sets of faces.
Haberman, J. M., & Whitney, D. C. (2009) · Journal of Experimental Psychology Human Perception & Performance · doi:10.1037/a0013899 · Cited by 386
Representation construction, externalised cognition and individual differences
Cox, R. (1999) · Learning and Instruction · doi:10.1016/s0959-4752(98)00051-6 · Cited by 348
Error Bars Considered Harmful: Exploring Alternate Encodings for Mean and Error
Correll, M., & Gleicher, M. (2014) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2014.2346298 · Cited by 258
Signatures of a Statistical Computation in the Human Sense of Confidence
Sanders, J. I., Hangya, B., & Kepecs, A. (2016) · Neuron · doi:10.1016/j.neuron.2016.03.025 · Cited by 253
Kay, M., Kola, T., Hullman, J. R., & Munson, S. A. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858558 · Cited by 239
Self-citationModeling Color Difference for Visualization Design
Szafir, D. A. (2017) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2017.2744359 · Cited by 213
Hullman, J., Resnick, P., & Adar, E. (2015) · PLOS ONE · doi:10.1371/journal.pone.0142444 · Cited by 194
Self-citationSelecting Semantically‐Resonant Colors for Data Visualization
Lin, S., Fortuna, J., Kulkarni, C. E., Stone, M., & Heer, J. (2013) · Computer Graphics Forum · doi:10.1111/cgf.12127 · Cited by 188
From Quantification to Visualization: A Taxonomy of Uncertainty Visualization Approaches
Potter, K. C., Rosen, P., & Johnson, C. R. (2012) · IFIP International Federation for Information Processing/IFIP · doi:10.1007/978-3-642-32677-6_15 · Cited by 186
Uncertainty forecasts improve weather-related decisions and attenuate the effects of forecast error.
Joslyn, S. L., & LeClerc, J. E. (2011) · Journal of Experimental Psychology Applied · doi:10.1037/a0025185 · Cited by 186
Variables on Scatterplots Look More Highly Correlated When the Scales Are Increased
Cleveland, W. S., Diaconis, P. W., & McGill, R. (1982) · Science · doi:10.1126/science.216.4550.1138 · Cited by 177
Uncertainty Displays Using Quantile Dotplots or CDFs Improve Transit Decision-Making
Fernandes, M., Walls, L., Munson, S., Hullman, J., & Kay, M. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173718 · Cited by 175
Self-citationGeographic information systems and science: today and tomorrow
Goodchild, M. F. (2009) · Procedia Earth and Planetary Science · doi:10.1016/j.proeps.2009.09.160 · Cited by 170
Visualizing spatial data uncertainty using animation
Ehlschlaeger, C., Shortridge, A. M., & Goodchild, M. F. (1997) · Computers & Geosciences · doi:10.1016/s0098-3004(97)00005-8 · Cited by 160
Benefitting InfoVis with Visual Difficulties
Hullman, J., Adar, E., & Shah, P. (2011) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2011.175 · Cited by 159
Self-citationUncertainty Visualization in Medical Volume Rendering Using Probabilistic Animation
Lundström, C. F., Ljung, P., Persson, A., & Ynnerman, A. (2007) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2007.70518 · Cited by 137
Dynamic display of spatial data-reliability: Does it benefit the map user?
Evans, B. J. (1997) · Computers & Geosciences · doi:10.1016/s0098-3004(97)00011-3 · Cited by 120
Lay understanding of probability distributions
Goldstein, D. G., & Rothschild, D. M. (2014) · Judgment and Decision Making · doi:10.1017/s1930297500004940 · Cited by 120
Perceptually Averaging in a Continuous Visual World
Albrecht, A. R., & Scholl, B. J. (2010) · Psychological Science · doi:10.1177/0956797610363543 · Cited by 110
Effects of ensemble and summary displays on interpretations of geospatial uncertainty data
Padilla, L., Ruginski, I. T., & Creem-Regehr, S. H. (2017) · Cognitive Research Principles and Implications · doi:10.1186/s41235-017-0076-1 · Cited by 105
Bar graphs depicting averages are perceptually misinterpreted: The within-the-bar bias
Newman, G. E., & Scholl, B. J. (2012) · Psychonomic Bulletin & Review · doi:10.3758/s13423-012-0247-5 · Cited by 101
Fast ensemble representations for abstract visual impressions
Leib, A. Y., Kosovicheva, A. A., & Whitney, D. C. (2016) · Nature Communications · doi:10.1038/ncomms13186 · Cited by 94
Not all summary statistics are made equal: Evidence from extracting summaries across time
Hubert‐Wallander, B., & Boynton, G. M. (2015) · Journal of Vision · doi:10.1167/15.4.5 · Cited by 92
The illusion of predictability: How regression statistics mislead experts
Soyer, E., & Hogarth, R. M. (2012) · International Journal of Forecasting · doi:10.1016/j.ijforecast.2012.02.002 · Cited by 84
Shen, Y., & Richards, V. M. (2012) · The Journal of the Acoustical Society of America · doi:10.1121/1.4733540 · Cited by 81
Uncertainty Visualization by Representative Sampling from Prediction Ensembles
Liu, L., Boone, A. P., Ruginski, I. T., Padilla, L., Hegarty, M., Creem-Regehr, S. H., Thompson, W. B., Yuksel, C., & House, D. H. (2016) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2016.2607204 · Cited by 76
Hullman, J., Kay, M., Kim, Y.-S., & Shrestha, S. (2018) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2017.2743898 · Cited by 55
Self-citationCognitive Efficiency Considerations for Good Graphic Design
Casner, S. M., & Larkin, J. H. (1989) · doi:10.21236/ada218976 · Cited by 43
The influence of different graphical displays on nonexpert decision making under uncertainty.
Padilla, L., Hansen, G., Ruginski, I. T., Kramer, H., Thompson, W. B., & Creem-Regehr, S. H. (2014) · Journal of Experimental Psychology Applied · doi:10.1037/xap0000037 · Cited by 39
Binder, A. R., Hillback, E. D., & Brossard, D. (2016) · Risk Analysis · doi:10.1111/risa.12462 · Cited by 38
Evaluating the Perception of Uncertainty in Alternative Visualization Strategies
Blenkinsop, S., Fisher, P. F., Bastin, L., & Wood, J. (2000) · Cartographica The International Journal for Geographic Information and Geovisualization · doi:10.3138/3645-4v22-0m23-3t52 · Cited by 36
(2016) · doi:10.1016/c2012-0-01278-1 · Cited by 21
Excessive confidence in visually-based estimates
Andrade, E. B. (2011) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2011.07.002 · Cited by 17
The representation of ensemble visual features outside the focus of attention
Alvarez, G. A., & Oliva, A. (2010) · Journal of Vision · doi:10.1167/7.9.129 · Cited by 11
Research works printed but not linked: 7
- irwin, 2014, How Not toBe Misled by the Jobs Report, The New York Times
- aisch, 2016, Live Presidential Forecast
- quealy, 2015, The first g.o.p.debate: Who's in, who's out and the role of chance, The New York Times
- bui, 2017, What the tax bill would look like for 25,000 middle-class families, The New York Times
- franconeri, 0, Personal communication
- bostock, 2014, Who Will Win The Senate?, The New York Times
- kay, 2016, When (ish) is my bus? user-centered visualizations of uncertainty in everyday, mobile predictive systems, Proc CHI2016
Cited by
Sorted by citations
The Science of Visual Data Communication: What Works
Franconeri, S., Padilla, L. M. K., Shah, P., Zacks, J. M., & Hullman, J. (2021) · Psychological Science in the Public Interest · doi:10.1177/15291006211051956 · Cited by 394
Self-citationIn Pursuit of Error: A Survey of Uncertainty Visualization Evaluation
Hullman, J., Qiao, X., Correll, M., Kale, A., & Kay, M. (2019) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2018.2864889 · Cited by 187
Self-citationWhy Authors Don't Visualize Uncertainty
Hullman, J. R. (2019) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2019.2934287 · Cited by 195
Self-citationggdist: Visualizations of Distributions and Uncertainty in the Grammar of Graphics
Kay, M. (2023) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2023.3327195 · Cited by 185
Self-citationIncreasing the Transparency of Research Papers with Explorable Multiverse Analyses
Dragicevic, P., Jansen, Y., Sarma, A., Kay, M., & Chevalier, F. (2019) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300295 · Cited by 144
Self-citation
Metadata
- Authors
- Kale, A., Nguyen, F., Kay, M., & Hullman, J.
- Affiliations
- University of Washington (US)
- University of Michigan (US)
- Northwestern University (US)
- Venue
- IEEE Transactions on Visualization and Computer Graphics
- Published
- 2019-01
- Volume
- 25
- Issue
- 1
- Pages
- 892-902
- ISSN
- 1077-2626 (ISSN-L)
- 1941-0506
- 2160-9306
- OpenAlex
- W2888554701
- Cited by
- 122
- References
- 67
- Citations by year
- 2026: 8 · 2025: 11 · 2024: 10 · 2023: 14 · 2022: 20 · 2021: 27 · 2020: 19 · 2019: 12
- FWCI
- 7.38
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- Locations
- IEEE Transactions on Visualization and Computer Graphics (publishedVersion, public-domain)
- PubMed (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Graphics and Computer-Aided Design › Data Visualization and Analytics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Visual and Cognitive Learning Processes
- Physical Sciences › Mathematics › Statistics and Probability › Statistics Education and Methodologies
- Keywords
- uncertainty communication, perceptual decision making, error bars, sampling error, news visualization, visualization evaluation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
