p-Curve and p-Hacking in Observational Research
Open accessThe DOI matches its Crossref record; 44 of the 44 references it cites were found in a registry or an index.
Abstract
The p-curve, the distribution of statistically significant p-values of published studies, has been used to make inferences on the proportion of true effects and on the presence of p-hacking in the published literature. We analyze the p-curve for observational research in the presence of p-hacking. We show by means of simulations that even with minimal omitted-variable bias (e.g., unaccounted confounding) p-curves based on true effects and p-curves based on null-effects with p-hacking cannot be reliably distinguished. We also demonstrate this problem using as practical example the evaluation of the effect of malaria prevalence on economic growth between 1960 and 1996. These findings call recent studies into question that use the p-curve to infer that most published research findings are based on true effects in the medical literature and in a wide range of disciplines. p-values in observational research may need to be empirically calibrated to be interpretable with respect to the commonly used significance threshold of 0.05. Violations of randomization in experimental studies may also result in situations where the use of p-curves is similarly unreliable.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures



Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“The use of statistical methods has been demonstrated to be suboptimal in some fields.”
From How to design preclinical studies in nanomedicine and cell therapy to maximize the prospects of clinical translation (2018) · cites this paper as [22]
In the citing paper
“Methodological criticisms suggest that these and similar studies may be underestimating the true impact of P -hacking (11, 12).”
From Is science really facing a reproducibility crisis, and do we need it to? (2018) · cites this paper as [11]
In the citing paper
“P-hacking22,23 (where investigators perform multiple analyses sequentially, for example by adding more samples or excluding some potential outliers until the desired thresholds of statistical significance are achieved)”
From How to design preclinical studies in nanomedicine and cell therapy to maximize the prospects of clinical translation (2018) · cites this paper as [22]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
Self-citationWitten, I. H. (2008) · Cited by 9,143
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · cited 1×
“The p -curve was also used to infer that most studies actually analyze true effects challenging previous claims [36].”
MethodSelf-citationGreene, W. (2000) · University of Maribor digital library (University of Maribor) · Cited by 3,746
Measuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling
John, L. K., Loewenstein, G., & Prelec, D. (2012) · Psychological Science · doi:10.1177/0956797611430953 · Cited by 2,239 · cited 3×
“However, there is evidence that the incentive system of academic publishing fosters scientists to even engage in questionable research practices to search for and select statistically significant results [9,10].”
BackgroundLet's Take the Con out of Econometrics
Leamer, E. E. (2026) · Journal of Business Cycle Research · doi:10.1007/s41549-026-00137-2 · Cited by 2,202
Determinants of Long-Term Growth: A Bayesian Averaging of Classical Estimates (BACE) Approach
Sala-i-Martin, X., Doppelhofer, G., & Miller, R. I. (2004) · American Economic Review · doi:10.1257/0002828042002570 · Cited by 2,112
Fanelli, D. (2009) · PLoS ONE · doi:10.1371/journal.pone.0005738 · Cited by 1,911 · cited 1×
“However, there is evidence that the incentive system of academic publishing fosters scientists to even engage in questionable research practices to search for and select statistically significant results [9,10].”
BackgroundA Reality Check for Data Snooping
White, H. (2000) · Econometrica · doi:10.1111/1468-0262.00152 · Cited by 1,851 · cited 1×
“This type of p -hacking is sometimes known as multiple modelling [14], data snooping [15] or data-mining [16].”
BackgroundP-curve: A key to the file-drawer.
Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2014) · Journal of Experimental Psychology: General · doi:10.1037/a0033242 · Cited by 1,500 · cited 9×
“Simonsohn et al. [1] who coined the distribution of statistically significant p -values “ p - curve ” have argued that true effects generate right-skewed p -curves whereas null effects should result in uniformly distributed p -curves.”
BackgroundMethodInfluentialWhy Most Discovered True Associations Are Inflated
Ioannidis, J. P. A. (2008) · Epidemiology · doi:10.1097/ede.0b013e31818131e7 · Cited by 1,495 · cited 1×
“It may cause what is called a vibration of effects [17] and it is well known to be a key threat to the validity of inferences in observational research [14,18,19].”
BackgroundSelf-citationContradicted and Initially Stronger Effects in Highly Cited Clinical Research
Ioannidis, J. P. A. (2005) · JAMA · doi:10.1001/jama.294.2.218 · Cited by 1,465 · cited 1×
“This may explain the extremely high failure of major inferences from observational studies to replicate in randomized trials [14,29].”
BackgroundSelf-citationThe Extent and Consequences of P-Hacking in Science
Head, M. L., Holman, L., Lanfear, R., Kahn, A. T., & Jennions, M. D. (2015) · PLOS Biology · doi:10.1371/journal.pbio.1002106 · Cited by 1,388 · cited 2×
“…[1], the distribution of statistically significant p -values, has been used to infer that most studies actually analyze true relationships in the medical sciences [2] and in a wide range of disciplines [3] irrespective of whether these studies use experimental or observational research designs.”
BackgroundInfluentialInvestigating Variation in Replicability: A “Many Labs” Replication Project
Klein, R. A., Ratliff, K. A., Vianello, M., Adams, R. B., Bahník, Š., Bernstein, M. J., Bocian, K., Brandt, M. J., Brooks, B., Brumbaugh, C. C., et al. (2014) · Social Psychology · doi:10.1027/1864-9335/a000178 · Cited by 1,208
Savović, J., Jones, H. E., Altman, D. G., Harris, R. J., Jüni, P., Pildal, J., Als-Nielsen, B., Balk, E. M., Gluud, C., Gluud, L. L., et al. (2012) · Annals of Internal Medicine · doi:10.7326/0003-4819-157-6-201209180-00537 · Cited by 1,049 · cited 1×
“But there is extensive literature that shows that a large proportion of seemingly experimental, randomized studies in fact are not properly randomized, or have many other biases that subvert randomization with substantial impact on their results [33].”
BackgroundSelf-citationDurlauf, S. N., Johnson, P. A., & Temple, J. R. W. (2005) · Handbook of Economic Growth · doi:10.1016/s1574-0684(05)01008-7 · Cited by 994
Econometrics: Alchemy or Science?: Essays in Econometric Methodology
Hendry, D. F. (2000) · doi:10.1093/0198293542.001.0001 · Cited by 634 · cited 1×
“It may cause what is called a vibration of effects [17] and it is well known to be a key threat to the validity of inferences in observational research [14,18,19].”
Econometrics-Alchemy or Science?
Hendry, D. F. (1980) · Economica · doi:10.2307/2553385 · Cited by 580
Khoury, M. J., & Ioannidis, J. P. A. (2014) · Science · doi:10.1126/science.aaa2709 · Cited by 387 · cited 1×
“It also should give us pause as to what extent observational big data can be trusted, when it is practically impossible to exclude the presence of such tiny biases that can totally invalidate the results [30].”
Self-citationLovell, M. C. (1983) · The Review of Economics and Statistics · doi:10.2307/1924403 · Cited by 354
Excess Significance Bias in the Literature on Brain Volume Abnormalities
Ioannidis, J. P. A. (2011) · Archives of General Psychiatry · doi:10.1001/archgenpsychiatry.2011.28 · Cited by 299
Self-citationEvaluation of excess significance bias in animal studies of neurological diseases
Tsilidis, K. K., Panagiotou, O. A., Sena, E. S., Aretouli, E., Evangelou, E., Howells, D. W., Salman, R. A.-S., Macleod, M. R., & Ioannidis, J. P. A. (2013) · PLoS Biology · doi:10.1371/journal.pbio.1001609 · Cited by 295 · cited 1×
“…surveys have documented an increased prevalence of p -values of 0.041 – 0.049 in the scientific literature over time [4,5], and the spurious excess of statistically significant findings in various types of both observational and experimental research [6,7] that have been attributed mostly to bias.”
BackgroundInfluentialSelf-citationPatel, C. J., Burford, B., & Ioannidis, J. P. A. (2015) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2015.05.029 · Cited by 274 · cited 1×
“We illustrate the range of estimates of β that can be obtained by varying sets of adjusting variables and samples of countries by using a vibration plot [25].”
MethodSelf-citationDeming, Data and Observational Studies
Young, S. S., & Karr, A. (2011) · Significance · doi:10.1111/j.1740-9713.2011.00506.x · Cited by 235 · cited 3×
“This may explain the extremely high failure of major inferences from observational studies to replicate in randomized trials [14,29].”
BackgroundInterpreting observational studies: why empirical calibration is needed to correct p‐values
Schuemie, M. J., Ryan, P. B., DuMouchel, W., Suchard, M. A., & Madigan, D. (2014) · Statistics in Medicine · doi:10.1002/sim.5925 · Cited by 234 · cited 4×
“Our findings are consistent with Schuemie et al. [13].”
MethodResultInfluentialAn estimate of the science-wise false discovery rate and application to the top medical literature
Jager, L. R., & Leek, J. T. (2014) · Biostatistics · doi:10.1093/biostatistics/kxt007 · Cited by 179 · cited 4×
“Jager and Leek [2] argue in a similar direction and describe the p -curve as a mixture of a uniform distribution and a beta distribution.”
BackgroundInfluentialMaking Prospective Registration of Observational Research a Reality
Dal-Ré, R., Ioannidis, J. P., Bracken, M. B., Buffler, P. A., Chan, A.-W., Franco, E. L., La Vecchia, C., & Weiderpass, E. (2014) · Science Translational Medicine · doi:10.1126/scitranslmed.3007513 · Cited by 123 · cited 1×
“Observational studies comprise the large majority of the scientific literature with almost 300,000 observational studies compared to 20,000 randomized ones (experimental research) per year in PubMed [8].”
BackgroundSelf-citationBishop, D. V. M., & Thompson, P. A. (2016) · PeerJ · doi:10.7717/peerj.1715 · Cited by 90
Ulrich, R., & Miller, J. (2015) · Journal of Experimental Psychology: General · doi:10.1037/xge0000086 · Cited by 85
de Winter, J. C., & Dodou, D. (2015) · PeerJ · doi:10.7717/peerj.733 · Cited by 78
Saquib, N., Saquib, J., & Ioannidis, J. P. A. (2013) · BMJ · doi:10.1136/bmj.f4313 · Cited by 68 · cited 2×
“…in the best clinical journals, the analysis of the primary outcome (adjusted or unadjusted) was different in the protocol versus the published papers and whenever only one of multiple analyses gave statistically significant results, this was almost always the analysis preferred by the authors [35].”
BackgroundResultSelf-citationRegistration practices for observational studies on ClinicalTrials.gov indicated low adherence
Boccia, S., Rothman, K. J., Panić, N., Flacco, M. E., Rosso, A., Pastorino, R., Manzoli, L., Vecchia, C. L., Villari, P., Boffetta, P., et al. (2015) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2015.09.009 · Cited by 63 · cited 1×
“A large empirical survey of observational study protocols shows that even when these protocols are registered, their statistical analysis plans are almost never pre-specified [28], so there is plenty of room for improvement in this.”
BackgroundSelf-citationIoannidis, J. P. A. (2014) · Biostatistics · doi:10.1093/biostatistics/kxt036 · Cited by 60
Self-citationComment: What p-hacking really looks like: A comment on Masicampo and LaLande (2012)
Lakens, D. (2015) · Quarterly Journal of Experimental Psychology · doi:10.1080/17470218.2014.982664 · Cited by 56
Gelman, A., & O'Rourke, K. (2014) · Biostatistics · doi:10.1093/biostatistics/kxt034 · Cited by 48 · cited 1×
“Further problems in the analysis have been also discussed [37,38].”
BackgroundOn the challenges of drawing conclusions from p -values just below 0.05
Lakens, D. (2015) · PeerJ · doi:10.7717/peerj.1142 · Cited by 44 · cited 1×
“However, other empirical surveys have documented an increased prevalence of p -values of 0.041 – 0.049 in the scientific literature over time [4,5], and the spurious excess of statistically significant findings in various types of both observational and experimental research [6,7] that have been…”
BackgroundInfluentialGrowth econometrics for agnostics and true believers
Rockey, J. C., & Temple, J. R. W. (2015) · European Economic Review · doi:10.1016/j.euroecorev.2015.05.010 · Cited by 41 · cited 1×
“Recent reviews of the literature do not consider malaria prevalence to be a genuine determinant of economic growth [24].”
BackgroundSerghiou, S., Patel, C. J., Tan, Y. Y., Koay, P., & Ioannidis, J. P. A. (2016) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2015.09.004 · Cited by 34 · cited 1×
“An empirical assessment of all 60 studies on pterygium risk factor epidemiology showed that there were no two studies that adjusted their models for the same variables [27].”
BackgroundSelf-citationSchuemie, M. J., Ryan, P. B., Suchard, M. A., Shahn, Z., & Madigan, D. (2013) · Biostatistics · doi:10.1093/biostatistics/kxt037 · Cited by 18 · cited 1×
“[26] further demonstrate that this increased rate of false positives is characterized by right-skewed p-curves.”
BackgroundProblems in using text-mining and p-curve analysis to detect rate of p-hackingPreprint
Bishop, D., & Thompson, P. A. (2015) · doi:10.7287/peerj.preprints.1266v1 · Cited by 6
Problems in using text-mining and p-curve analysis to detect rate of p-hackingPreprint
Bishop, D., & Thompson, P. (2015) · doi:10.7287/peerj.preprints.1266v2 · Cited by 6 · cited 3×
“Bishop and Thompson [31] illustrate that right-skewed p-curves may occur if only 25% of the considered studies analyze true effects.”
BackgroundThe Fragility of Meta-Regression Models in Observational ResearchPreprint
Bruns, S. B. (2016) · RePEc: Research Papers in Economics · doi:10.17192/es2024.0516 · Cited by 1
Self-citation
Cited by
Meta-evaluation of meta-analysis: ten appraisal questions for biologists
Nakagawa, S., Noble, D. W. A., Senior, A. M., & Lagisz, M. (2017) · BMC Biology · doi:10.1186/s12915-017-0357-7 · Cited by 579
Heisig, J. P., & Schaeffer, M. (2019) · European Sociological Review · doi:10.1093/esr/jcy053 · Cited by 552
“Second, and more importantly, a narrow interpretation of p-curve analysis has come under attack by Bruns and Ioannidis (2016), who argue that in observational studies omitted variable biases may create right skewed p-curves even in the absence of an underlying causal effect.”
BackgroundThe past, present and future of Registered Reports
Chambers, C. D., & Tzavella, L. (2021) · Nature Human Behaviour · doi:10.1038/s41562-021-01193-7 · Cited by 499
Detecting and avoiding likely false‐positive findings – a practical guide
Forstmeier, W., Wagenmakers, E., & Parker, T. H. (2017) · Biological Reviews · doi:10.1111/brv.12315 · Cited by 436
Is science really facing a reproducibility crisis, and do we need it to?
Fanelli, D. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1708272114 · Cited by 416
Metadata
- Authors
- Bruns, S. B., & Ioannidis, J. P. A.
- Affiliations
- University of Kassel (DE)
- Stanford University (US)
- Corresponding author
- Ioannidis, John P. A.
- Venue
- PLOS ONE
- Publisher
- Public Library of Science (PLoS)
- Published
- 2016-02-17
- Volume
- 11
- Issue
- 2
- Pages
- e0149144
- ISSN
- 1932-6203 (ISSN-L)
- PMC
- PMC4757561
- OpenAlex
- W2275632435
- Cited by
- 210
- References
- 44
- Citations by year
- 2026: 8 · 2025: 15 · 2024: 13 · 2023: 15 · 2022: 19 · 2021: 24 · 2020: 37 · 2019: 20 · 2018: 25 · 2017: 23 · 2016: 11
- FWCI
- 24.57
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- PLoS ONE (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Document Server@UHasselt (UHasselt) (submittedVersion)
- Figshare (submittedVersion, cc-by)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Keywords
- p-curve, p-hacking, omitted variable bias, confounding, observational studies, null effect, malaria prevalence, economic growth, p-value calibration, significance threshold
- MeSH terms
- Humans, Malaria, Research, Statistics as Topic, Vibration, Prevalence, Sample Size, Observational Studies as Topic
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.