The DOI matches its Crossref record; 44 of the 44 references it cites were found in a registry or an index.
p-Curve and p-Hacking in Observational Research
Open accessAbstract
The p-curve, the distribution of statistically significant p-values of published studies, has been used to make inferences on the proportion of true effects and on the presence of p-hacking in the published literature. We analyze the p-curve for observational research in the presence of p-hacking. We show by means of simulations that even with minimal omitted-variable bias (e.g., unaccounted confounding) p-curves based on true effects and p-curves based on null-effects with p-hacking cannot be reliably distinguished. We also demonstrate this problem using as practical example the evaluation of the effect of malaria prevalence on economic growth between 1960 and 1996. These findings call recent studies into question that use the p-curve to infer that most published research findings are based on true effects in the medical literature and in a wide range of disciplines. p-values in observational research may need to be empirically calibrated to be interpretable with respect to the commonly used significance threshold of 0.05. Violations of randomization in experimental studies may also result in situations where the use of p-curves is similarly unreliable.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures



Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“The use of statistical methods has been demonstrated to be suboptimal in some fields.”
From How to design preclinical studies in nanomedicine and cell therapy to maximize the prospects of clinical translation (2018) · cites this paper as [22]
In the citing paper
“Methodological criticisms suggest that these and similar studies may be underestimating the true impact of P -hacking (11, 12).”
From Is science really facing a reproducibility crisis, and do we need it to? (2018) · cites this paper as [11]
In the citing paper
“P-hacking22,23 (where investigators perform multiple analyses sequentially, for example by adding more samples or excluding some potential outliers until the desired thresholds of statistical significance are achieved)”
From How to design preclinical studies in nanomedicine and cell therapy to maximize the prospects of clinical translation (2018) · cites this paper as [22]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
Self-citationWitten, I. H. (2008) · Cited by 9,143
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · cited 1×
“The p -curve was also used to infer that most studies actually analyze true effects challenging previous claims [36].”
MethodSelf-citationGreene, W. (2000) · University of Maribor digital library (University of Maribor) · Cited by 3,746
Measuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling
John, L. K., Loewenstein, G., & Prelec, D. (2012) · Psychological Science · doi:10.1177/0956797611430953 · Cited by 2,239
Let's Take the Con out of Econometrics
Leamer, E. E. (2026) · Journal of Business Cycle Research · doi:10.1007/s41549-026-00137-2 · Cited by 2,202
Determinants of Long-Term Growth: A Bayesian Averaging of Classical Estimates (BACE) Approach
Sala-i-Martin, X., Doppelhofer, G., & Miller, R. I. (2004) · American Economic Review · doi:10.1257/0002828042002570 · Cited by 2,112
Fanelli, D. (2009) · PLoS ONE · doi:10.1371/journal.pone.0005738 · Cited by 1,911 · cited 1×
“However, there is evidence that the incentive system of academic publishing fosters scientists to even engage in questionable research practices to search for and select statistically significant results [9,10].”
Background
Cited by
Meta-evaluation of meta-analysis: ten appraisal questions for biologists
Nakagawa, S., Noble, D. W. A., Senior, A. M., & Lagisz, M. (2017) · BMC Biology · doi:10.1186/s12915-017-0357-7 · Cited by 579
Heisig, J. P., & Schaeffer, M. (2019) · European Sociological Review · doi:10.1093/esr/jcy053 · Cited by 552
The past, present and future of Registered Reports
Chambers, C. D., & Tzavella, L. (2021) · Nature Human Behaviour · doi:10.1038/s41562-021-01193-7 · Cited by 499
Detecting and avoiding likely false‐positive findings – a practical guide
Forstmeier, W., Wagenmakers, E., & Parker, T. H. (2017) · Biological Reviews · doi:10.1111/brv.12315 · Cited by 436
Is science really facing a reproducibility crisis, and do we need it to?
Fanelli, D. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1708272114 · Cited by 416
Metadata
- Authors
- Bruns, S. B., & Ioannidis, J. P. A.
- Affiliations
- University of Kassel (DE)
- Stanford University (US)
- Corresponding author
- Ioannidis, John P. A.
- Venue
- PLOS ONE
- Publisher
- Public Library of Science (PLoS)
- Published
- 2016-02-17
- Volume
- 11
- Issue
- 2
- Pages
- e0149144
- ISSN
- 1932-6203 (ISSN-L)
- PMC
- PMC4757561
- OpenAlex
- W2275632435
- Cited by
- 210
- References
- 44
- Citations by year
- 2026: 8 · 2025: 15 · 2024: 13 · 2023: 15 · 2022: 19 · 2021: 24 · 2020: 37 · 2019: 20 · 2018: 25 · 2017: 23 · 2016: 11
- FWCI
- 24.57
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- PLoS ONE (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Document Server@UHasselt (UHasselt) (submittedVersion)
- Figshare (submittedVersion, cc-by)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Keywords
- p-curve, p-hacking, omitted variable bias, confounding, observational studies, null effect, malaria prevalence, economic growth, p-value calibration, significance threshold
- MeSH terms
- Humans, Malaria, Research, Statistics as Topic, Vibration, Prevalence, Sample Size, Observational Studies as Topic
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: p-Curve and p-Hacking in Observational Research. https://app.cochat.ai/papers/10.1371/journal.pone.0149144
- The paper
- Bruns, S. B., & Ioannidis, J. P. A. (2016). p-Curve and p-Hacking in Observational Research. PLOS ONE. https://doi.org/10.1371/journal.pone.0149144
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.