Better P-curves: Making P-curve analysis more robust to errors, fraud, and ambitious P-hacking: A reply to Ulrich and Miller (2015)
Open accessAbstract
When studies examine true effects, they generate right-skewed p-curves, distributions of statistically significant results with more low (.01 s) than high (.04 s) p values. What else can cause a right-skewed p-curve? First, we consider the possibility that researchers report only the smallest significant p value (as conjectured by Ulrich & Miller, 2015), concluding that it is a very uncommon problem. We then consider more common problems, including (a) p-curvers selecting the wrong p values, (b) fake data, (c) honest errors, and (d) ambitiously p-hacked (beyond p < .05) results. We evaluate the impact of these common problems on the validity of p-curve analysis, and provide practical solutions that substantially increase its robustness.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
1 retracted
Sanna, L. J., Chang, E. C., Miceli, P. M., & Lundberg, K. B. (2010) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2010.12.013 · Cited by 27 · see citing works
RetractedTversky, A., & Kahneman, D. (1974) · Science · doi:10.1126/science.185.4157.1124 · Cited by 28,810 · see citing works
The case for motivated reasoning.
Kunda, Z. (1990) · Psychological Bulletin · doi:10.1037/0033-2909.108.3.480 · Cited by 9,022 · see citing works
Measuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling
John, L. K., Loewenstein, G., & Prelec, D. (2012) · Psychological Science · doi:10.1177/0956797611430953 · Cited by 2,239 · see citing works
Publication bias in the social sciences: Unlocking the file drawer
Franco, A., Malhotra, N., & Simonovits, G. (2014) · Science · doi:10.1126/science.1255484 · Cited by 1,542 · see citing works
P-curve: A key to the file-drawer.
Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2014) · Journal of Experimental Psychology: General · doi:10.1037/a0033242 · Cited by 1,500 · see citing works
Self-citationPuzzlingly High Correlations in fMRI Studies of Emotion, Personality, and Social Cognition
Vul, E., Harris, C., Winkielman, P., & Pashler, H. (2009) · Perspectives on Psychological Science · doi:10.1111/j.1745-6924.2009.01125.x · Cited by 1,475 · see citing works
The Extent and Consequences of P-Hacking in Science
Head, M. L., Holman, L., Lanfear, R., Kahn, A. T., & Jennions, M. D. (2015) · PLOS Biology · doi:10.1371/journal.pbio.1002106 · Cited by 1,388 · see citing works
The Statistical Crisis in Science
Gelman, A., & Loken, E. (2014) · American Scientist · doi:10.1511/2014.111.460 · Cited by 963 · see citing works
Statistics As Principled Argument
Abelson, R. P. (2012) · Psychology Press eBooks · doi:10.4324/9781410601155 · Cited by 473 · see citing works
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · PsycEXTRA Dataset · doi:10.1037/e519702015-014 · Cited by 462 · see citing works
Self-citationThe (mis)reporting of statistical results in psychology journals
Bakker, M., & Wicherts, J. M. (2011) · Behavior Research Methods · doi:10.3758/s13428-011-0089-5 · Cited by 356 · see citing works
Just Post It: The Lesson From Two Cases of Fabricated Data Detected by Statistics Alone
Simonsohn, U. (2013) · Psychological Science · doi:10.1177/0956797613480366 · Cited by 216 · see citing works
Self-citationPreferences for symmetry in faces change across the menstrual cycle
Little, A. C., Jones, B. C., Burt, D. M., & Perrett, D. I. (2007) · Biological Psychology · doi:10.1016/j.biopsycho.2007.08.003 · Cited by 117 · see citing works
Gildersleeve, K. A., Haselton, M. G., & Fales, M. R. (2014) · Psychological Bulletin · doi:10.1037/a0037714 · Cited by 91 · see citing works
Ulrich, R., & Miller, J. O. (2015) · Journal of Experimental Psychology General · doi:10.1037/xge0000086 · Cited by 87 · see citing works
The use of Statistics in the British Journal of Psychiatry
Mcguigan, S. (1995) · The British Journal of Psychiatry · doi:10.1192/bjp.167.5.683 · Cited by 73 · see citing works
Measuring the Prevalence of Questionable Research Practices with Incentives for Truth-TellingPreprint
John, L. K., Loewenstein, G. F., & Prelec, D. (2012) · SSRN Electronic Journal · doi:10.2139/ssrn.1996631 · Cited by 56 · see citing works
p-curve: A Key to the File Drawer
Simonsohn, U., Nelson, L. D., & Simmons, J. (2012) · PsycEXTRA Dataset · doi:10.1037/e519682015-077 · Cited by 53 · see citing works
Self-citationInconsistencies between reported test statistics and <i>p‐</i>values in two psychiatry journals
Berle, D., & Starcevic, V. (2007) · International Journal of Methods in Psychiatric Research · doi:10.1002/mpr.225 · Cited by 38 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Debunking: A Meta-Analysis of the Psychological Efficacy of Messages Countering Misinformation
Chan, M.-P. S., Jones, C. R., Hall Jamieson, K., & Albarracín, D. (2017) · Psychological Science · doi:10.1177/0956797617714579 · Cited by 898 · see citing works
How Much Does Education Improve Intelligence? A Meta-Analysis
Ritchie, S. J., & Tucker-Drob, E. M. (2018) · Psychological Science · doi:10.1177/0956797618774253 · Cited by 680 · see citing works
Nelson, L. D., Simmons, J., & Simonsohn, U. (2018) · Annual Review of Psychology · doi:10.1146/annurev-psych-122216-011836 · Cited by 548 · see citing works
Self-citationCorrecting for Bias in Psychology: A Comparison of Meta-Analytic Methods
Carter, E. C., Schönbrodt, F. D., Gervais, W. M., & Hilgard, J. (2019) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245919847196 · Cited by 588 · see citing works
Theory of mind and prosocial behavior in childhood: A meta-analytic review.
Imuta, K., Henry, J. D., Slaughter, V., Selcuk, B., & Ruffman, T. (2016) · Developmental Psychology · doi:10.1037/dev0000140 · Cited by 474 · see citing works
A Meta-Analytic Review of Social Identification and Health in Organizational Contexts
Steffens, N. K., Haslam, S. A., Schuh, S. C., Jetten, J., & van Dick, R. (2017) · Personality and Social Psychology Review · doi:10.1177/1088868316656701 · Cited by 311 · see citing works
Working memory training in typically developing children: A meta-analysis of the available evidence.
Sala, G., & Gobet, F. (2017) · Developmental Psychology · doi:10.1037/dev0000265 · Cited by 216 · see citing works
When Problem Solving Followed by Instruction Works: Evidence for Productive Failure
Sinha, T., & Kapur, M. (2021) · Review of Educational Research · doi:10.3102/00346543211019105 · Cited by 212 · see citing works
“04s) or an equal probability of different significance levels (Simonsohn et al., 2015).”
InfluentialA historical review of publication bias
Marks‐Anglin, A., & Chen, Y. (2020) · Research Synthesis Methods · doi:10.1002/jrsm.1452 · Cited by 165 · see citing works
A meta-analysis of social network site use and social support
Liu, D., Wright, K. B., & Hu, B. (2018) · Computers & Education · doi:10.1016/j.compedu.2018.08.024 · Cited by 127 · see citing works
P-hacking and publication bias interact to distort meta-analytic effect size estimates
Friese, M., & Frankenbach, J. (2020) · Psychological Methods · doi:10.1037/met0000246 · Cited by 122 · see citing works
Boggero, I. A., Hostinar, C. E., Haak, E. A., Murphy, M. L. M., & Segerstrom, S. C. (2017) · Biological Psychology · doi:10.1016/j.biopsycho.2017.08.058 · Cited by 97 · see citing works
“Thus, by assembling the p-values for any set of significant findings and calculating the slope of that line, evidential value can be determined (Simonsohn et al., 2014a, 2014b, 2015).”
BackgroundMethod10 years of Bayesian theories of autism: A comprehensive review
Angeletos Chrysaitis, N., & Seriès, P. (2023) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2022.105022 · Cited by 92 · see citing works
Bishop, D. V. M., & Thompson, P. A. (2016) · PeerJ · doi:10.7717/peerj.1715 · Cited by 90 · see citing works
Challenging the challenge hypothesis on testosterone in fathers: Limited meta-analytic support
Meijer, W. M., van IJzendoorn, M. H., & Bakermans - Kranenburg, M. J. (2019) · Psychoneuroendocrinology · doi:10.1016/j.psyneuen.2019.104435 · Cited by 49 · see citing works
Lutz, M. C., Kok, R., Verveer, I., Malbec, M., Koot, S., van Lier, P. A. C., & Franken, I. H. A. (2021) · Journal of Psychiatry and Neuroscience · doi:10.1503/jpn.200031 · Cited by 44 · see citing works
Schindler, S., Hilgard, J., Fritsche, I., Burke, B., & Pfattheicher, S. (2023) · Personality and Social Psychology Review · doi:10.1177/10888683221107267 · Cited by 34 · see citing works
“To address the possibility of “ambitious p-hacking,” that is, exploitation of researcher degrees of freedom to reach p values substantially below .05, Simonsohn et al. (2015) proposed a p-curve of only p values less than .025 (i.e., in our case 16 p values).”
BackgroundChen, L., Benjamin, R., Guo, Y., Lai, A., & Heine, S. J. (2023) · Research Square · doi:10.21203/rs.3.rs-1254756/v1 · Cited by 5 · see citing works
The effectiveness of implementation intentions in children: A systematic review and meta‐analysis
Breitwieser, J., & Reinelt, T. (2026) · British Journal of Psychology · doi:10.1111/bjop.70065 · Cited by 1 · see citing works
- Authors
- Simonsohn, U., Simmons, J. P., & Nelson, L. D.
- Affiliations
- Esade (ES)
- University of Pennsylvania (US)
- University of California, Berkeley (US)
- Venue
- Journal of Experimental Psychology General
- Published
- 2015-12
- Volume
- 144
- Issue
- 6
- Pages
- 1146-1152
- ISSN
- 0096-3445 (ISSN-L)
- 1939-2222
- OpenAlex
- W2339493379
- Cited by
- 343
- References
- 20
- Citations by year
- 2026: 6 · 2025: 28 · 2024: 22 · 2023: 27 · 2022: 28 · 2021: 52 · 2020: 59 · 2019: 47 · 2018: 25 · 2017: 37 · 2016: 11 · 2014: 1
- FWCI
- 22.16
- Type
- journal article
- Language
- en
- Open access
- green
- repository.upenn.edu
- Licence
- other-oa
- Locations
- Journal of Experimental Psychology General (publishedVersion)
- PubMed (publishedVersion)
- ScholarlyCommons (University of Pennsylvania) (submittedVersion, other-oa)
- ScholarlyCommons (University of Pennsylvania) (submittedVersion)
- SSRN Electronic Journal (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Social Sciences › Safety Research › Academic integrity and plagiarism
- Keywords
- p-curve, p-hacking, scientific fraud, fake data, significance testing
- MeSH terms
- Data Interpretation, Statistical, Humans, Research Design, Scientific Misconduct, Publication Bias
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
