The DOI matches its Crossref record; 67 of the 67 references it cites were found in a registry or an index.
Hypothetical Outcome Plots Help Untrained Observers Judge Trends in Ambiguous Data
Abstract
Animated representations of outcomes drawn from distributions (hypothetical outcome plots, or HOPs) are used in the media and other public venues to communicate uncertainty. HOPs greatly improve multivariate probability estimation over conventional static uncertainty visualizations and leverage the ability of the visual system to quickly, accurately, and automatically process the summary statistical properties of ensembles. However, it is unclear how well HOPs support applied tasks resembling real world judgments posed in uncertainty communication. We identify and motivate an appropriate task to investigate realistic judgments of uncertainty in the public domain through a qualitative analysis of uncertainty visualizations in the news. We contribute two crowdsourced experiments comparing the effectiveness of HOPs, error bars, and line ensembles for supporting perceptual decision-making from visualized uncertainty. Participants infer which of two possible underlying trends is more likely to have produced a sample of time series data by referencing uncertainty visualizations which depict the two trends with variability due to sampling error. By modeling each participant's accuracy as a function of the level of evidence presented over many repeated judgments, we find that observers are able to correctly infer the underlying trend in samples conveying a lower level of evidence when using HOPs rather than static aggregate uncertainty visualizations as a decision aid. Modeling approaches like ours contribute theoretically grounded and richly descriptive accounts of user perceptions to visualization evaluation.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Judgment under Uncertainty: Heuristics and Biases
Tversky, A., & Kahneman, D. (1975) · Utility, Probability, and Human Decision Making · doi:10.1007/978-94-010-1834-0_8 · Cited by 22,650
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
(1966) · Medical Entomology and Zoology · Cited by 10,293
Signal detection theory and psychophysics
Ingleby, D. (1967) · Journal of Sound and Vibration · doi:10.1016/0022-460x(67)90197-6 · Cited by 8,616
The visual display of quantitative information
(1985) · General Pharmacology The Vascular System · doi:10.1016/0306-3623(85)90230-7 · Cited by 5,090
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · cited 1×
“Among scientists, misunderstandings of sampling error and the likelihood of replicating experimental results contribute to rampant use of underpowered studies and the “replication crisis” [37, 62].”
BackgroundWhy a Diagram is (Sometimes) Worth Ten Thousand Words
Larkin, J. H., & Simon, H. A. (1987) · Cognitive Science · doi:10.1111/j.1551-6708.1987.tb00863.x · Cited by 3,855
Belief in the law of small numbers.
Tversky, A., & Kahneman, D. (1971) · Psychological Bulletin · doi:10.1037/h0031322 · Cited by 3,246
Jacob, M. (1989) · Health Education Journal · doi:10.1177/001789698904800401 · Cited by 3,123
Guidelines for evaluating and expressing the uncertainty of NIST measurement results
Taylor, B. N. (1994) · National Institute of Standards and Technology, NIST Technical Note 1297 · doi:10.6028/nist.tn.1297 · Cited by 2,833
Research works printed but not linked: 7
- irwin, 2014, How Not toBe Misled by the Jobs Report, The New York Times
- aisch, 2016, Live Presidential Forecast
- quealy, 2015, The first g.o.p.debate: Who's in, who's out and the role of chance, The New York Times
- bui, 2017, What the tax bill would look like for 25,000 middle-class families, The New York Times
- franconeri, 0, Personal communication
- bostock, 2014, Who Will Win The Senate?, The New York Times
- kay, 2016, When (ish) is my bus? user-centered visualizations of uncertainty in everyday, mobile predictive systems, Proc CHI2016
Cited by
Sorted by citations
The Science of Visual Data Communication: What Works
Franconeri, S., Padilla, L. M. K., Shah, P., Zacks, J. M., & Hullman, J. (2021) · Psychological Science in the Public Interest · doi:10.1177/15291006211051956 · Cited by 394
Self-citationIn Pursuit of Error: A Survey of Uncertainty Visualization Evaluation
Hullman, J., Qiao, X., Correll, M., Kale, A., & Kay, M. (2019) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2018.2864889 · Cited by 187
Self-citationWhy Authors Don't Visualize Uncertainty
Hullman, J. R. (2019) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2019.2934287 · Cited by 195
Self-citationggdist: Visualizations of Distributions and Uncertainty in the Grammar of Graphics
Kay, M. (2023) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2023.3327195 · Cited by 185
Self-citationIncreasing the Transparency of Research Papers with Explorable Multiverse Analyses
Dragicevic, P., Jansen, Y., Sarma, A., Kay, M., & Chevalier, F. (2019) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300295 · Cited by 144
Self-citation
Metadata
- Authors
- Kale, A., Nguyen, F., Kay, M., & Hullman, J.
- Affiliations
- University of Washington (US)
- University of Michigan (US)
- Northwestern University (US)
- Venue
- IEEE Transactions on Visualization and Computer Graphics
- Published
- 2019-01
- Volume
- 25
- Issue
- 1
- Pages
- 892-902
- ISSN
- 1077-2626 (ISSN-L)
- 1941-0506
- 2160-9306
- OpenAlex
- W2888554701
- Cited by
- 122
- References
- 67
- Citations by year
- 2026: 8 · 2025: 11 · 2024: 10 · 2023: 14 · 2022: 20 · 2021: 27 · 2020: 19 · 2019: 12
- FWCI
- 7.38
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- Locations
- IEEE Transactions on Visualization and Computer Graphics (publishedVersion, public-domain)
- PubMed (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Graphics and Computer-Aided Design › Data Visualization and Analytics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Visual and Cognitive Learning Processes
- Physical Sciences › Mathematics › Statistics and Probability › Statistics Education and Methodologies
- Keywords
- uncertainty communication, perceptual decision making, error bars, sampling error, news visualization, visualization evaluation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
