Imagining Replications: Graphical Prediction & Discrete Visualizations Improve Recall & Estimation of Effect Uncertainty
Abstract
People often have erroneous intuitions about the results of uncertain processes, such as scientific experiments. Many uncertainty visualizations assume considerable statistical knowledge, but have been shown to prompt erroneous conclusions even when users possess this knowledge. Active learning approaches been shown to improve statistical reasoning, but are rarely applied in visualizing uncertainty in scientific reports. We present a controlled study to evaluate the impact of an interactive, graphical uncertainty prediction technique for communicating uncertainty in experiment results. Using our technique, users sketch their prediction of the uncertainty in experimental effects prior to viewing the true sampling distribution from an experiment. We find that having a user graphically predict the possible effects from experiment replications is an effective way to improve one's ability to make predictions about replications of new experiments. Additionally, visualizing uncertainty as a set of discrete outcomes, as opposed to a continuous probability distribution, can improve recall of a sampling distribution from a single experiment. Our work has implications for various applications where it is important to elicit peoples' estimates of probability distributions and to communicate uncertainty effectively.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Cites
On Information and Sufficiency
Kullback, S., & Leibler, R. A. (1951) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177729694 · Cited by 20,193 · see citing works
(2011) · Cited by 14,317 · see citing works
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505 · see citing works
Accommodation of a scientific conception: Toward a theory of conceptual change
Posner, G. J., Strike, K. A., Hewson, P. W., & Gertzog, W. A. (1982) · Science Education · doi:10.1002/sce.3730660207 · Cited by 5,187 · see citing works
Active learning : creating excitement in the classroom
Bonwell, C. C., & Eison, J. A. (1991) · Cited by 5,014 · see citing works
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · see citing works
Learning Theories: An Educational Perspective
Schunk, D. H. (1991) · Cited by 3,725 · see citing works
How to improve Bayesian reasoning without instruction: Frequency formats.
Gigerenzer, G., & Hoffrage, U. (1995) · Psychological Review · doi:10.1037/0033-295x.102.4.684 · Cited by 2,128 · see citing works
Believe it or not: how much can we rely on published data on potential drug targets?
Prinz, F., Schlange, T., & Asadullah, K. (2011) · Nature Reviews Drug Discovery · doi:10.1038/nrd3439-c1 · Cited by 1,888 · see citing works
Casler, K., Bickel, L., & Hackett, E. (2013) · Computers in Human Behavior · doi:10.1016/j.chb.2013.05.009 · Cited by 1,662 · see citing works
Uncertain Judgements: Eliciting Experts' Probabilities
O'Hagan, A., Buck, C. E., Daneshkhah, A., Eiser, J. R., Garthwaite, P. H., Jenkinson, D. J., Oakley, J. E., & Rakow, T. (2006) · doi:10.1002/0470033312 · Cited by 1,550 · see citing works
Pashler, H., & Wagenmakers, E. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612465253 · Cited by 1,321 · see citing works
Cosmides, L. (1996) · Cognition · doi:10.1016/0010-0277(95)00664-8 · Cited by 1,293 · see citing works
Rational choice in an uncertain world the psychology of judgment and decision making
Hastie, R., & Dawes, R. M. (2010) · Cited by 910 · see citing works
Statistical Rethinking: a Bayesian Course with Examples in R and Stan
Hewson, P. (2016) · Journal of the Royal Statistical Society Series A (Statistics in Society) · doi:10.1111/rssa.12221 · Cited by 850 · see citing works
The use of statistical heuristics in everyday inductive reasoning
Nisbett, R., Krantz, D., Jepson, C., & Kunda, Z. (1983) · Psychological Review · doi:10.1037/0033-295x.90.4.339 · Cited by 802 · see citing works
Schwartz, D. L., & Martin, T. (2004) · Cognition and Instruction · doi:10.1207/s1532690xci2202_1 · Cited by 763 · see citing works
Measuring Risk Literacy: The Berlin Numeracy Test
Cokely, E. T., Galešić, M., Schulz, E., Ghazal, S., & García‐Retamero, R. (2012) · Judgment and Decision Making · doi:10.1017/s1930297500001819 · Cited by 752 · see citing works
The effects of statistical training on thinking about everyday problems
Fong, G. T., Krantz, D. H., & Nisbett, R. E. (1986) · Cognitive Psychology · doi:10.1016/0010-0285(86)90001-0 · Cited by 597 · see citing works
The Challenge of Developing Statistical Literacy, Reasoning and Thinking
Ben‐Zvi, D., & Garfield, J. (2004) · Springer eBooks · doi:10.1007/1-4020-2278-6 · Cited by 587 · see citing works
Using natural frequencies to improve diagnostic inferences
Hoffrage, U., & Gigerenzer, G. (1998) · Academic Medicine · doi:10.1097/00001888-199805000-00024 · Cited by 501 · see citing works
New Pedagogy and New Content: The Case of Statistics
Moore, D. S. (1997) · International Statistical Review · doi:10.1111/j.1751-5823.1997.tb00390.x · Cited by 455 · see citing works
Robust misinterpretation of confidence intervals
Hoekstra, R., Morey, R. D., Rouder, J. N., & Wagenmakers, E.-J. (2014) · Psychonomic Bulletin & Review · doi:10.3758/s13423-013-0572-3 · Cited by 425 · see citing works
Researchers Misunderstand Confidence Intervals and Standard Error Bars.
Belia, S., Fidler, F., Williams, J., & Cumming, G. (2005) · Psychological Methods · doi:10.1037/1082-989x.10.4.389 · Cited by 390 · see citing works
The Challenge of Developing Statistical Reasoning
Garfield, J. (2002) · Journal of Statistics Education · doi:10.1080/10691898.2002.11910676 · Cited by 389 · see citing works
Teaching Bayesian reasoning in less than two hours.
Sedlmeier, P., & Gigerenzer, G. (2001) · Journal of Experimental Psychology General · doi:10.1037/0096-3445.130.3.380 · Cited by 366 · see citing works
Representation construction, externalised cognition and individual differences
Cox, R. (1999) · Learning and Instruction · doi:10.1016/s0959-4752(98)00051-6 · Cited by 348 · see citing works
Clarke, P. B., Fu, D. S., Jakubovic, A., & Fibiger, H. C. (1988) · The Journal of Pharmacology and Experimental Therapeutics · doi:10.1016/s0022-3565(25)22152-8 · Cited by 318 · see citing works
Depth of processing, retrieval cues, and uniqueness of encoding as factors in recall
Moscovitch, M., & Craik, F. I. M. (1976) · Journal of Verbal Learning and Verbal Behavior · doi:10.1016/s0022-5371(76)90040-2 · Cited by 277 · see citing works
Communicating Health Risks With Visual Aids
García‐Retamero, R., & Cokely, E. T. (2013) · Current Directions in Psychological Science · doi:10.1177/0963721413491570 · Cited by 277 · see citing works
Spatial Visualization and Gender Differences in High School Geometry
Battista, M. T. (1990) · Journal for Research in Mathematics Education · doi:10.5951/jresematheduc.21.1.0047 · Cited by 265 · see citing works
Error Bars Considered Harmful: Exploring Alternate Encodings for Mean and Error
Correll, M., & Gleicher, M. (2014) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2014.2346298 · Cited by 258 · see citing works
Designing Incentives for Inexpert Human Raters
Shaw, A., Horton, J. J., & Chen, D. L. (2010) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/1958824.1958865 · Cited by 253 · see citing works
Kay, M., Kola, T., Hullman, J. R., & Munson, S. A. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858558 · Cited by 239 · see citing works
Self-citationUsing Computer Simulation Methods to Teach Statistics: A Review of the Literature
Jamie, D. M. (2002) · Journal of Statistics Education · doi:10.1080/10691898.2002.11910548 · Cited by 234 · see citing works
delMas, R. C., Garfield, J., & Chance, B. (1999) · Journal of Statistics Education · doi:10.1080/10691898.1999.12131279 · Cited by 230 · see citing works
Spatial Visualization and Gender Differences in High School Geometry
Battista, M. T. (1990) · Journal for Research in Mathematics Education · doi:10.2307/749456 · Cited by 204 · see citing works
Hullman, J., Resnick, P., & Adar, E. (2015) · PLOS ONE · doi:10.1371/journal.pone.0142444 · Cited by 194 · see citing works
Self-citationGarcía‐Retamero, R., & Hoffrage, U. (2013) · Social Science & Medicine · doi:10.1016/j.socscimed.2013.01.034 · Cited by 193 · see citing works
Choosing Outcomes versus Choosing Products: Consumer-Focused Retirement Investment Advice
Goldstein, D. G., Johnson, E. J., & Sharpe, W. F. (2008) · Journal of Consumer Research · doi:10.1086/589562 · Cited by 165 · see citing works
Assessing the Effect of Visualizations on Bayesian Reasoning through Crowdsourcing
Micallef, L., Dragicevic, P., & Fekete, J. (2012) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2012.199 · Cited by 163 · see citing works
Reasoning about Sampling Distribitions
Chance, B., del Mas, R. C., & Garfield, J. (2004) · Springer eBooks · doi:10.1007/1-4020-2278-6_13 · Cited by 145 · see citing works
Pictorial representations in statistical reasoning
Brase, G. L. (2008) · Applied Cognitive Psychology · doi:10.1002/acp.1460 · Cited by 140 · see citing works
Gelman, A., & Weakliem, D. L. (2009) · American Scientist · doi:10.1511/2009.79.310 · Cited by 137 · see citing works
Assessment and Statistics Education: Current Challenges and Directions
Garfield, J., & Gal, I. (1999) · International Statistical Review · doi:10.1111/j.1751-5823.1999.tb00377.x · Cited by 122 · see citing works
Improving Statistical Reasoning: Theoretical Models and Practical Implications
Sedlmeier, P. (1999) · Cited by 117 · see citing works
Lay understanding of probability distributions
Goldstein, D. G., & Rothschild, D. M. (2014) · Judgment and Decision Making · doi:10.1017/s1930297500004940 · Cited by 116 · see citing works
Decisions With Uncertainty: The Glass Half Full
Joslyn, S., & LeClerc, J. (2013) · Current Directions in Psychological Science · doi:10.1177/0963721413481473 · Cited by 109 · see citing works
Kim, Y., Reinecke, K., & Hullman, J. R. (2017) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3025453.3025592 · Cited by 105 · see citing works
Self-citationMeasuring the Productivity of Computer Systems Development Activities with Function Points
Behrens, C. (1983) · IEEE Transactions on Software Engineering · doi:10.1109/tse.1983.235429 · Cited by 103 · see citing works
Wilson, M. L., Resnick, P., Coyle, D., & Ed, H. (2013) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/2468356.2479636 · Cited by 84 · see citing works
Improving Statistical Reasoning
Sedlmeier, P. (1999) · Psychology Press eBooks · doi:10.4324/9781410601247 · Cited by 80 · see citing works
Divided attention during encoding and retrieval: Differential control effects?
Craik, F. I. M., Naveh‐Benjamin, M., Ishaik, G., & Anderson, N. D. (2000) · Journal of Experimental Psychology Learning Memory and Cognition · doi:10.1037//0278-7393.26.6.1744 · Cited by 71 · see citing works
Brase, G. L. (2002) · Journal of Behavioral Decision Making · doi:10.1002/bdm.421 · Cited by 69 · see citing works
The Perception of Visual UncertaintyRepresentation by Non-Experts
Tak, S., Toet, A., & van Erp, J. B. F. (2014) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2013.247 · Cited by 57 · see citing works
RepliCHI - CHI should be replicating and validating results more
Wilson, M. L., Mackay, W. E., Ed, H., Bernstein, M. S., Russell, D., & Thimbleby, H. W. (2011) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/1979742.1979491 · Cited by 56 · see citing works
Effects of active information processing on the understanding of risk information
Natter, H. M., & Berry, D. C. (2004) · Applied Cognitive Psychology · doi:10.1002/acp.1068 · Cited by 55 · see citing works
Prediction Interval: What to Expect When You’re Expecting … A Replication
Spence, J. R., & Stanley, D. J. (2016) · PLoS ONE · doi:10.1371/journal.pone.0162874 · Cited by 52 · see citing works
Divided attention during encoding and retrieval: Differential control effects?
Craik, F. I. M., Naveh‐Benjamin, M., Ishaik, G., et al. (2000) · Journal of Experimental Psychology Learning Memory and Cognition · doi:10.1037/0278-7393.26.6.1744 · Cited by 49 · see citing works
Special Interest Group on Transparent Statistics in HCI
Kay, M., Haroz, S., Guha, S., & Dragicevic, P. (2016) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/2851581.2886442 · Cited by 31 · see citing works
Self-citationWhat Makes Everyday Scientific Reasoning So Challenging?
(2017) · Psychology of Learning and Motivation · doi:10.1016/bs.plm.2016.11.006 · Cited by 29 · see citing works
Schwarz, C. J., & Sutherland, J. M. (1997) · Journal of Statistics Education · doi:10.1080/10691898.1997.11910523 · Cited by 17 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
In Pursuit of Error: A Survey of Uncertainty Visualization Evaluation
Hullman, J., Qiao, X., Correll, M., Kale, A., & Kay, M. (2019) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2018.2864889 · Cited by 187 · see citing works
Self-citationHypothetical Outcome Plots Help Untrained Observers Judge Trends in Ambiguous Data
Kale, A., Nguyen, F., Kay, M., & Hullman, J. (2019) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2018.2864909 · Cited by 122 · see citing works
Self-citationA Bayesian Cognition Approach to Improve Data Visualization
Kim, Y.-S., Walls, L. A., Krafft, P., & Hullman, J. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300912 · Cited by 76 · see citing works
Self-citationIllusion of Causality in Visualized Data
Xiong, C., Shapiro, J., Hullman, J., & Franconeri, S. (2020) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2019.2934399 · Cited by 71 · see citing works
Self-citationHeltne, A., Frans, N., Hummelen, B., Falkum, E., Germans Selvik, S., & Paap, M. C. S. (2023) · Scandinavian Journal of Psychology · doi:10.1111/sjop.12918 · Cited by 5 · see citing works
- Authors
- Hullman, J., Kay, M., Kim, Y.-S., & Shrestha, S.
- Affiliations
- University of Washington (US)
- University of Michigan (US)
- Vassar College (US)
- Venue
- IEEE Transactions on Visualization and Computer Graphics
- Published
- 2018-01
- Volume
- 24
- Issue
- 1
- Pages
- 446-456
- ISSN
- 1077-2626 (ISSN-L)
- OpenAlex
- W2750605487
- Cited by
- 55
- References
- 62
- Citations by year
- 2026: 2 · 2025: 5 · 2024: 3 · 2023: 6 · 2022: 10 · 2021: 10 · 2020: 8 · 2019: 5 · 2018: 6
- FWCI
- 2.16
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- IEEE Transactions on Visualization and Computer Graphics (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Data Visualization and Analytics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Visual and Cognitive Learning Processes
- Physical Sciences › Mathematics › Statistics and Probability › Statistics Education and Methodologies
- Keywords
- uncertainty visualization, replication of experiments, sampling distribution, effect size estimation, statistical reasoning, active learning, scientific reporting
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review