An illusion of predictability in scientific results: Even experts confuse inferential uncertainty and outcome variability
Open accessAbstract
Traditionally, scientists have placed more emphasis on communicating inferential uncertainty (i.e., the precision of statistical estimates) compared to outcome variability (i.e., the predictability of individual outcomes). Here, we show that this can lead to sizable misperceptions about the implications of scientific results. Specifically, we present three preregistered, randomized experiments where participants saw the same scientific findings visualized as showing only inferential uncertainty, only outcome variability, or both and answered questions about the size and importance of findings they were shown. Our results, composed of responses from medical professionals, professional data scientists, and tenure-track faculty, show that the prevalent form of visualizing only inferential uncertainty can lead to significant overestimates of treatment effects, even among highly trained experts. In contrast, we find that depicting both inferential uncertainty and outcome variability leads to more accurate perceptions of results while appearing to leave other subjective impressions of the results unchanged, on average.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Prior work on statistical misperceptions shows that even expert judgements vary with how data are presented.
“Prior work on statistical misperceptions 84 shows that even expert judgements vary with how data are presented”
From Quantifying the prevalence and impact of overreaching causal claims in social science (2026) · cites this paper as [84]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Greedy function approximation: A gradient boosting machine
Friedman, J. H. (2001) · The Annals of Statistics · doi:10.1214/aos/1013203451 · Cited by 30,731 · see citing works
Sontrop, J. M., Dixon, S. N., Garg, A. X., Jiménez, I. B., Dohein, O., Huang, S. S., & Clark, W. F. (2013) · American Journal of Nephrology · doi:10.1159/000350377 · Cited by 5,903 · see citing works
The earth is round (p < .05).
Cohen, J. (1994) · American Psychologist · doi:10.1037/0003-066x.49.12.997 · Cited by 4,003 · see citing works
Confidence intervals rather than P values: estimation rather than hypothesis testing.
Gardner, M. J., & Altman, D. G. (1986) · BMJ · doi:10.1136/bmj.292.6522.746 · Cited by 2,014 · see citing works
Video games and aggressive thoughts, feelings, and behavior in the laboratory and in life.
Anderson, C. A., & Dill, K. E. (2000) · Journal of Personality and Social Psychology · doi:10.1037//0022-3514.78.4.772 · Cited by 1,778 · see citing works
Video games and aggressive thoughts, feelings, and behavior in the laboratory and in life.
Anderson, C. A., & Dill, K. E. (2000) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.78.4.772 · Cited by 1,778 · see citing works
Moving beyond P values: data analysis with estimation graphics
Ho, J., Tumkaya, T., Aryal, S., Choi, H., & Claridge‐Chang, A. (2019) · Nature Methods · doi:10.1038/s41592-019-0470-3 · Cited by 1,770 · see citing works
Publication Manual of the American Psychological Association
Yao, R. (2011) · Family and Consumer Sciences Research Journal · doi:10.1111/j.1552-3934.2011.02081.x · Cited by 1,459 · see citing works
Inference by Eye: Confidence Intervals and How to Read Pictures of Data.
Cumming, G., & Finch, S. (2005) · American Psychologist · doi:10.1037/0003-066x.60.2.170 · Cited by 1,449 · cited 2× · see citing works
“We presented the sample sizes, the sample means, and the standard errors on these estimates for each experiment [3], and manipulated whether participants saw visualizations depicting only inferential uncertainty or inferential uncertainty and outcome variability depending on their experimental…”
BackgroundMethodRaincloud plots: a multi-platform tool for robust data visualization
Allen, M., Poggiali, D., Whitaker, K. J., Marshall, T. R., & Kievit, R. (2019) · Wellcome Open Research · doi:10.12688/wellcomeopenres.15191.1 · Cited by 1,125 · see citing works
Why Psychologists Should by Default Use Welch’s t-test Instead of Student’s t-test
Delacre, M., Lakens, D., & Leys, C. (2017) · International Review of Social Psychology · doi:10.5334/irsp.82 · Cited by 1,037 · see citing works
Error bars in experimental biology
Cumming, G., Fidler, F., & Vaux, D. L. (2007) · The Journal of Cell Biology · doi:10.1083/jcb.200611141 · Cited by 940 · see citing works
A common language effect size statistic
McGraw, K. O., & Wong, S. P. (1992) · Psychological Bulletin · doi:10.1037/0033-2909.111.2.361 · Cited by 934 · cited 1× · see citing works
“We chose this measure because it was developed to aid in the communication of effect sizes and thus provides an easy and effective means of eliciting effect sizes from participants (27, 28).”
MethodUnderstanding The New Statistics
Cumming, G. (2013) · Routledge eBooks · doi:10.4324/9780203807002 · Cited by 839 · see citing works
Beyond Bar and Line Graphs: Time for a New Data Presentation Paradigm
Weissgerber, T. L., Milic, N. M., Winham, S. J., & Garovic, V. D. (2015) · PLoS Biology · doi:10.1371/journal.pbio.1002128 · Cited by 713 · see citing works
Visualizing samples with box plots
Krzywinski, M., & Altman, N. (2014) · Nature Methods · doi:10.1038/nmeth.2813 · Cited by 576 · see citing works
Registered Reports: A new publishing initiative at Cortex
Chambers, C. (2012) · Cortex · doi:10.1016/j.cortex.2012.12.016 · Cited by 574 · see citing works
Robust misinterpretation of confidence intervals
Hoekstra, R., Morey, R. D., Rouder, J. N., & Wagenmakers, E.-J. (2014) · Psychonomic Bulletin & Review · doi:10.3758/s13423-013-0572-3 · Cited by 425 · see citing works
A probability-based measure of effect size: Robustness to base rates and other factors.
Ruscio, J. (2008) · Psychological Methods · doi:10.1037/1082-989x.13.1.19 · Cited by 398 · see citing works
Researchers Misunderstand Confidence Intervals and Standard Error Bars.
Belia, S., Fidler, F., Williams, J., & Cumming, G. (2005) · Psychological Methods · doi:10.1037/1082-989x.10.4.389 · Cited by 390 · see citing works
Integrating explanation and prediction in computational social science
Hofman, J. M., Watts, D. J., Athey, S., Garip, F., Griffiths, T. L., Kleinberg, J., Margetts, H., Mullainathan, S., Salganik, M. J., Vazire, S., et al. (2021) · Nature · doi:10.1038/s41586-021-03659-0 · Cited by 371 · see citing works
Self-citationKrzywinski, M. I., & Altman, N. (2013) · Nature Methods · doi:10.1038/nmeth.2659 · Cited by 294 · see citing works
Error Bars Considered Harmful: Exploring Alternate Encodings for Mean and Error
Correll, M., & Gleicher, M. (2014) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2014.2346298 · Cited by 258 · see citing works
Kay, M., Kola, T., Hullman, J. R., & Munson, S. A. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858558 · Cited by 239 · see citing works
Hullman, J., Resnick, P., & Adar, E. (2015) · PLOS ONE · doi:10.1371/journal.pone.0142444 · Cited by 194 · see citing works
Replication and Researchers' Understanding of Confidence Intervals and Standard Error Bars
Cumming, G., Williams, J., & Fidler, F. M. (2004) · Understanding Statistics · doi:10.1207/s15328031us0304_5 · Cited by 172 · see citing works
Choosing Outcomes versus Choosing Products: Consumer-Focused Retirement Investment Advice
Goldstein, D. G., Johnson, E. J., & Sharpe, W. F. (2008) · Journal of Consumer Research · doi:10.1086/589562 · Cited by 165 · see citing works
Self-citationCumming, G., Fidler, F. M., Kalinowski, P., & Lai, J. (2011) · Australian Journal of Psychology · doi:10.1111/j.1742-9536.2011.00037.x · Cited by 140 · see citing works
Proceedings of the 34th Session of the International Statistical Institute.
F., W. T. (1965) · Biometrics · doi:10.2307/2528381 · Cited by 121 · see citing works
Nägele, P. (2003) · British Journal of Anaesthesia · doi:10.1093/bja/aeg087 · Cited by 116 · see citing works
Visual Reasoning Strategies for Effect Size Judgments and Decisions
Kale, A., Kay, M., & Hullman, J. (2021) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2020.3030335 · Cited by 109 · see citing works
Bar graphs depicting averages are perceptually misinterpreted: The within-the-bar bias
Newman, G. E., & Scholl, B. J. (2012) · Psychonomic Bulletin & Review · doi:10.3758/s13423-012-0247-5 · Cited by 101 · see citing works
The illusion of predictability: How regression statistics mislead experts
Soyer, E., & Hogarth, R. M. (2012) · International Journal of Forecasting · doi:10.1016/j.ijforecast.2012.02.002 · Cited by 84 · see citing works
Hofman, J. M., Goldstein, D. G., & Hullman, J. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376454 · Cited by 75 · see citing works
Self-citationCoplots, nonparametric regression, and conditionally parametric fits
Cleveland, W. S. (1994) · Lecture notes-monograph series · doi:10.1214/lnms/1215463783 · Cited by 30 · see citing works
Teaching Confidence Intervals: Problems and Potential Solutions
Fidler, F. M., & Cumming, G. (2005) · Cited by 22 · see citing works
Putting scientific results in perspective: Improving the communication of standardized effect sizes
Kim, Y., Hofman, J. M., & Goldstein, D. G. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3502053 · Cited by 16 · see citing works
Self-citation(2017) · Nature Biomedical Engineering · doi:10.1038/s41551-017-0079 · Cited by 7 · see citing works
ANDRE, Q. (2016) · Figshare · doi:10.5281/zenodo.166736 · Cited by 5 · see citing works
An illusion of predictability in scientific resultsPreprint
Zhang, S., Heck, P. R., Meyer, M. N., Chabris, C. F., Goldstein, D. G., & Hofman, J. M. (2022) · doi:10.31235/osf.io/5tcgs · Cited by 1 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 2
- [16]JAMA: Instructions for Authors, (2022). https://web.archive.org/web/20220412040639/https://jamanetwork.com/journals/jama/pages/instructions-for-authors. Accessed 4 December 2022.
- [17]New England Journal of Medicine: Statistical Reporting Guidelines, (2022). https://web.archive.org/web/20220405233315/https://www.nejm.org/author-center/new-manuscripts. Accessed 4 December 2022.
Cited by
Sorted by citations
Jack, C. R., Wiste, H. J., Algeciras‐Schimnich, A., Weigand, S. D., Figdore, D. J., Lowe, V. J., Vemuri, P., Graff‐Radford, J., Ramanan, V. K., Knopman, D. S., et al. (2024) · Alzheimer s & Dementia · doi:10.1002/alz.13651 · Cited by 43 · see citing works
Favero, M., Carlucci, S., Chinazzo, G., Møller, J. K., Schweiker, M., Vellei, M., & Sonta, A. J. (2024) · Building and Environment · doi:10.1016/j.buildenv.2024.111903 · Cited by 22 · see citing works
The origins of unpredictability in life outcome prediction tasks
Lundberg, I., Brown‐Weinstock, R., Clampet‐Lundquist, S., Pachman, S., Nelson, T. J., Yang, V., Edin, K. J., & Salganik, M. (2024) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2322973121 · Cited by 21 · see citing works
Analytic racecraft: Race-based averages create illusory group differences in perceptions of racismPreprint
Martinez, J. E. (2020) · doi:10.31234/osf.io/kfpjg · Cited by 14 · see citing works
Collecting Qualitative Data at Scale with Large Language Models: A Case Study
Cuevas, A., Scurrell, J. V., Brown, E. M., Entenmann, J., & Daepp, M. I. G. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3710947 · Cited by 23 · see citing works
Hofmann, R., Rozgonjuk, D., Soto, C. J., Ostendorf, F., & Mõttus, R. (2025) · Journal of Research in Personality · doi:10.1016/j.jrp.2025.104582 · Cited by 18 · see citing works
Modeling the transmission mitigation impact of testing for infectious diseases
Middleton, C. E., & Larremore, D. B. (2024) · Science Advances · doi:10.1126/sciadv.adk5108 · Cited by 11 · see citing works
Odds and Insights: Decision Quality in Exploratory Data Analysis Under Uncertainty
Sarma, A., Pu, X., Cui, Y., Correll, M., Brown, E. T., & Kay, M. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3641995 · Cited by 9 · see citing works
Messer, D. J., Henry, L. A., & Danielsson, H. (2023) · Brain Science · doi:10.3390/brainsci13101377 · Cited by 6 · see citing works
Policy makers believe money motivates more than it does
Jilke, S., Keppeler, F., Ternovski, J., Vogel, D., & Yoeli, E. (2024) · Scientific Reports · doi:10.1038/s41598-024-51590-x · Cited by 6 · see citing works
Hofmann, R., Rozgonjuk, D., Soto, C. J., Ostendorf, F., & Mõttus, R. (2023) · doi:10.31234/osf.io/cedwk · Cited by 5 · see citing works
Mana, J., Bezdíček, O., Růžička, F., Lasica, A., Šmídová, A., Klempířová, O., Nikolai, T., Uhrová, T., Růžička, E., Urgošík, D., et al. (2024) · European Journal of Neuroscience · doi:10.1111/ejn.16521 · Cited by 4 · see citing works
Invasive grass species do not have priority effects
Matos, C., & Ward, D. (2025) · Journal of Plant Ecology · doi:10.1093/jpe/rtaf053 · Cited by 2 · see citing works
Platform-based Adaptive Experimental Research in Education
Леонидович, М. И., Reza, M., Song, H., Moore, S., Chen, P., Kumar, H., Li, T., Stamper, J. C., Bier, N., Rafferty, A. N., et al. (2025) · International Conference on Learning Analytics and Knowledge (LAK) · doi:10.1145/3706468.3706471 · Cited by 3 · see citing works
Kühlwein, F., Merk, S., Schneider, J. M., & Schmidt, K. A. (2025) · Learning and Instruction · doi:10.1016/j.learninstruc.2025.102179 · Cited by 1 · see citing works
Sharifan, A., Dobrescu, A. I., Harrod, C. S., Klerings, I., Ong, A. Y., Ngeh, E. N., Xiao, Y., & Gartlehner, G. (2025) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2025.112083 · Cited by 1 · see citing works
A primer on large-sample statistical inference for epidemiologists
Shook‐Sa, B. E., Cole, S. R., Zivich, P. N., Edwards, J. K., Krajewski, T., Feeney, T., Adimora, A. A., & Hudgens, M. G. (2026) · Epidemiologic Methods · doi:10.1515/em-2025-0036 · Cited by 1 · see citing works
Quantifying the prevalence and impact of overreaching causal claims in social science
Isch, C., Dörr, T., Fasching, N., Jennings, G., & Watts, D. J. (2026) · Nature Human Behaviour · doi:10.1038/s41562-026-02553-x
A Taxonomy for Understanding and Identifying Uncertainty in AI-Generated ResponsesPreprint
Prabhudesai, S., Goldstein, D. G., Hofman, J. M., & Rothschild, D. M. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4836380
Self-citationRelação entre desenho de pesquisa e análise de dados em estudos de bilinguismoPreprint
Garcia, G. D. (2024) · doi:10.31219/osf.io/th4zb
(2024) · Elsevier eBooks · doi:10.1016/b978-0-443-15632-8.09989-5
Critically evaluating systematic reviews, meta-analyses, and alternate reviews
Wilczynski, S. (2024) · Elsevier eBooks · doi:10.1016/b978-0-443-15632-8.00015-0
Recruiting Healthcare Workers on the International Labor Market: A MegastudyPreprint
Keppeler, F., Linos, E., Sciepura, B., Lacey, K., & Jacobsen, C. B. (2024) · doi:10.31219/osf.io/6pwxe
DECISIONS, VARIABILITY, AND VISUALIZATION: A NOVEL INSTRUMENT FOR DECISION MAKING UNDER VARIABILITYPreprint
del Rosario, Z., Saur, E., Jin, Lin, J., & Wilmer, J. B. (2025) · doi:10.31234/osf.io/vujta_v1
Information sampling and Bayesian belief formation in statistical judgment
He, L., Wang, H., Bian, Y., Zhang, X., & Bhatia, S. (2025) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2517302122 · Cited by 1 · see citing works
Rakow, T., & Blunt, M. (2026) · Judgment and Decision Making · doi:10.1017/jdm.2026.10037
Thu, K. S., Jain, R., Htoo, S. E. N. Z., & Naing, Y. Y. (2026) · IGI Global eBooks · doi:10.4018/979-8-3373-5052-3.ch003
Linguistic Uncertainty Markers for Trust Calibration in AI-Assisted Decision-Making
Spillner, L., Rockstroh, J., Wenig, N., Zargham, N., Kiefner, L., Porzel, R., & Malaka, R. (2026) · ACM Conference on Conversational User Interfaces (CUI) · doi:10.1145/3816046.3816231
Kühlwein, F., Merk, S., Schneider, J., & Schmidt, K. (2026) · Learning and Instruction · doi:10.1016/j.learninstruc.2026.102427
Day, E., Trevino, S. D., Chinn, L. K., Schweer-Collins, M., Grant, S., & Tanner‐Smith, E. E. (2026) · Evaluation Review · doi:10.1177/0193841x261475375
- Authors
- Zhang, S., Heck, P. R., Meyer, M. N., Chabris, C. F., Goldstein, D. G., & Hofman, J. M.
- Affiliations
- University of Colorado Boulder (US)
- Consumer Financial Protection Bureau (US)
- Geisinger Health System (US)
- Microsoft Research New York City (United States) (US)
- Corresponding author
- Zhang, Sam
- Hofman, Jake M.
- ORCID
- Venue
- Proceedings of the National Academy of Sciences
- Publisher
- National Academy of Sciences
- Published
- 2023-08-09
- Volume
- 120
- Issue
- 33
- Pages
- e2302491120-e2302491120
- ISSN
- 0027-8424 (ISSN-L)
- 1091-6490
- PMC
- PMC10438372
- OpenAlex
- W4385693083
- Cited by
- 30
- References
- 40
- Citations by year
- 2026: 7 · 2025: 8 · 2024: 12 · 2023: 2 · 2020: 1
- FWCI
- 2.63
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Proceedings of the National Academy of Sciences (publishedVersion, cc-by-nc-nd)
- PubMed Central (publishedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- Funders
- National Science Foundation (DGE 2040434)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Data Visualization and Analytics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistics Education and Methodologies
- Keywords
- outcome variability, statistical data visualization, overestimation of treatment effects, expert opinion, medical professionals, data scientists, faculty researchers, confidence intervals, prediction intervals
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
