p-Curve and p-Hacking in Observational Research
Open accessThe DOI matches its Crossref record; 44 of the 44 references it cites were found in a registry or an index.
Abstract
The p-curve, the distribution of statistically significant p-values of published studies, has been used to make inferences on the proportion of true effects and on the presence of p-hacking in the published literature. We analyze the p-curve for observational research in the presence of p-hacking. We show by means of simulations that even with minimal omitted-variable bias (e.g., unaccounted confounding) p-curves based on true effects and p-curves based on null-effects with p-hacking cannot be reliably distinguished. We also demonstrate this problem using as practical example the evaluation of the effect of malaria prevalence on economic growth between 1960 and 1996. These findings call recent studies into question that use the p-curve to infer that most published research findings are based on true effects in the medical literature and in a wide range of disciplines. p-values in observational research may need to be empirically calibrated to be interpretable with respect to the commonly used significance threshold of 0.05. Violations of randomization in experimental studies may also result in situations where the use of p-curves is similarly unreliable.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures



Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“The use of statistical methods has been demonstrated to be suboptimal in some fields.”
From How to design preclinical studies in nanomedicine and cell therapy to maximize the prospects of clinical translation (2018) · cites this paper as [22]
In the citing paper
“Methodological criticisms suggest that these and similar studies may be underestimating the true impact of P -hacking (11, 12).”
From Is science really facing a reproducibility crisis, and do we need it to? (2018) · cites this paper as [11]
In the citing paper
“P-hacking22,23 (where investigators perform multiple analyses sequentially, for example by adding more samples or excluding some potential outliers until the desired thresholds of statistical significance are achieved)”
From How to design preclinical studies in nanomedicine and cell therapy to maximize the prospects of clinical translation (2018) · cites this paper as [22]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
Self-citationWitten, I. H. (2008) · Cited by 9,143
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · cited 1×
“The p -curve was also used to infer that most studies actually analyze true effects challenging previous claims [36].”
MethodSelf-citationGreene, W. (2000) · University of Maribor digital library (University of Maribor) · Cited by 3,746
Measuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling
John, L. K., Loewenstein, G., & Prelec, D. (2012) · Psychological Science · doi:10.1177/0956797611430953 · Cited by 2,239 · cited 3×
“However, there is evidence that the incentive system of academic publishing fosters scientists to even engage in questionable research practices to search for and select statistically significant results [9,10].”
BackgroundLet's Take the Con out of Econometrics
Leamer, E. E. (2026) · Journal of Business Cycle Research · doi:10.1007/s41549-026-00137-2 · Cited by 2,202
Determinants of Long-Term Growth: A Bayesian Averaging of Classical Estimates (BACE) Approach
Sala-i-Martin, X., Doppelhofer, G., & Miller, R. I. (2004) · American Economic Review · doi:10.1257/0002828042002570 · Cited by 2,112
Fanelli, D. (2009) · PLoS ONE · doi:10.1371/journal.pone.0005738 · Cited by 1,911 · cited 1×
“However, there is evidence that the incentive system of academic publishing fosters scientists to even engage in questionable research practices to search for and select statistically significant results [9,10].”
Background
Cited by
Items 201–230 of 230
Variable Coding, Additional Statistical Tests, and Interview Data
Pinckney, J. C. (2020) · Oxford University Press eBooks · doi:10.1093/oso/9780190097301.005.0001
Pinckney, J. C. (2020) · Oxford University Press eBooks · doi:10.1093/oso/9780190097301.002.0004
Pinckney, J. C. (2020) · Oxford University Press eBooks · doi:10.1093/oso/9780190097301.002.0003
Pinckney, J. C. (2020) · Oxford University Press eBooks · doi:10.1093/oso/9780190097301.002.0007
Data Validation for Data Science Projects
Demchenko, Y., Cuadrado-Gallego, J. J., Chertov, О., & Aleksandrova, M. (2024) · Springer eBooks · doi:10.1007/978-3-031-69366-3_14
Bayesian evaluation of effect size after replicating an original studyPreprint
van Aert, R. C. M., & van Assen, M. A. L. M. (2017) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/g3sne_v1
Beaulieu‐Guay, L., Béchard, B., Ouimet, M., Claveau, F., & Montpetit, É. (2025) · SocArXiv (OSF Preprints) · doi:10.31235/osf.io/kwe94_v3
Kaplan, D. (2025) · Springer eBooks · doi:10.1007/978-3-032-08228-2_3
Societal restraint of behavior during the pre-vaccine pandemic saved working-age men but not women
Catalano, R. A. (2025) · Science Advances · doi:10.1126/sciadv.ady7033
“…seasonality and other forms of autocorrelation (e.g., quarterly cycles, week- of-month cycles, and persistence of high or low scores in week t into week t + n ) while avoiding “p-hacking” or shrinking confidence intervals around estimated associations by increasing the number of cases ( 21 ).”
Tail Risk Protection via reproducible data-adaptive strategies
Spilak, B. (2024) · edoc Publication server (Humboldt University of Berlin) · doi:10.18452/28076
Ayorinde, A., Williams, I., Mannion, R., Song, F., Skrybant, M., Lilford, R. J., & Chen, Y.-F. (2020) · Figshare · doi:10.6084/m9.figshare.12413774
Ayorinde, A., Williams, I., Mannion, R., Song, F., Skrybant, M., Lilford, R. J., & Chen, Y.-F. (2020) · Figshare · doi:10.6084/m9.figshare.12413768
Approximate Reasoning and the Ethnomathematics of Cooking
Mani, A. (2023) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.8024608
Non-market valuation of global change impacts on ecosystems and biodiversity
Nobel, A. (2021) · Document Server@UHasselt (UHasselt)
Meta-analyses on Environmental and Energy Economics
Schütt, M. (2023) · MACAU: Open Access Repository of Kiel University
What really drives financial inclusion? Evidence from a meta-analysis of 3,817 estimates
Eshun, S. F., & Kočenda, E. (2025) · Econstor (Econstor)
Cheong, F. S. (2026) · Econometrics · doi:10.3390/econometrics14020019
“Researchers who prioritize statistically significant results may rationally prefer weaker, misspecified propensity score models that preserve overlap and sample size, even if those models yield biased estimates (Bruns & Ioannidis, 2016).”
Réévaluation de l'apport du nalméfène au traitement de l'alcoolodépendance
Palpacuer, C. (2018) · theses.fr (ABES) · doi:10.70675/aa77e495z482ez45b0z9aabz3b0862e3413e
Reliable inference: benefits of open raw data may be universal in meta-analysisPreprint
Žanko, D., Geček, S., Tafro, A., Puljak, L., & Čulina, A. (2026) · EcoEvoRxiv · doi:10.32942/x2dm3p
Holbert, R. L., Bowman, N. D., & Walter, N. (2026) · De Gruyter eBooks · doi:10.1515/9783111087139-004
Sakagami, S., Yoshida, Y., Uemura, S., Kumamoto, T., Tsukamoto, R., Nishi, T., Kawano, S., Fukuoka, K., Ino, H., Hamamura, K., et al. (2026) · Clinical and Translational Science · doi:10.1111/cts.70650
“However, in a comprehensive screening of this scale, a standardized framework is essential to maintain analytical feasibility and ensure comparability across candidate drugs; applying drug-specific covariate sets to each of the 86 drugs would not have been practically feasible [54, 55].”
Schwinn, T., Hirschmiller, J., Brähler, E., Wiltink, J., Zwerenz, R., Beutel, M. E., Ernst, M., & Krakau, L. (2026) · BMC Psychology · doi:10.1186/s40359-026-05021-3
“As the sample size increases, even very small effects are more likely to reach statistical significance [53].”
Reporting of Statistical Results in the Field of Pain Research
Chinaka, T. (2025) · University of the Witwatersrand, Johannesburg Institutional Repository on DSpace (University of the Witwatersrand, Johannesburg)
Grover, A., Reis‐Pardal, J., Ellis, R. J., Ioannidis, J. P. A., & Patel, C. J. (2026) · medRxiv · doi:10.64898/2026.08.28.26361666
Self-citationDoes p-Hacking Mitigate or Exacerbate the Effects of Publication Bias?Preprint
Cai, Y., Pernoud, A., & Xu, B. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.05372
Below the Surface: Creep, Corrosion, and Tipping Points in the Social Resilience of SciencePreprint
Sobkowicz, P. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.22863
“The phenomenon is widely present in psychology (John et al., 2012), but may be found in other disciplines (Head et al., 2015) and in observational studies (Bruns and Ioannidis, 2016).”
Ben-Menachem, J., Galper, A., & Fishman, N. (2026) · Sociological Methods & Research · doi:10.1177/00491241261490850
Financial Inclusion: A Meta-analytic Evidence from Global Research
Eshun, S. F., & Kočenda, E. (2026) · CESifo · doi:10.65864/ha4apy70jp
Metadata
- Authors
- Bruns, S. B., & Ioannidis, J. P. A.
- Affiliations
- University of Kassel (DE)
- Stanford University (US)
- Corresponding author
- Ioannidis, John P. A.
- Venue
- PLOS ONE
- Publisher
- Public Library of Science (PLoS)
- Published
- 2016-02-17
- Volume
- 11
- Issue
- 2
- Pages
- e0149144
- ISSN
- 1932-6203 (ISSN-L)
- PMC
- PMC4757561
- OpenAlex
- W2275632435
- Cited by
- 210
- References
- 44
- Citations by year
- 2026: 8 · 2025: 15 · 2024: 13 · 2023: 15 · 2022: 19 · 2021: 24 · 2020: 37 · 2019: 20 · 2018: 25 · 2017: 23 · 2016: 11
- FWCI
- 24.57
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- PLoS ONE (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Document Server@UHasselt (UHasselt) (submittedVersion)
- Figshare (submittedVersion, cc-by)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Keywords
- p-curve, p-hacking, omitted variable bias, confounding, observational studies, null effect, malaria prevalence, economic growth, p-value calibration, significance threshold
- MeSH terms
- Humans, Malaria, Research, Statistics as Topic, Vibration, Prevalence, Sample Size, Observational Studies as Topic
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.