The Extent and Consequences of P-Hacking in Science
Open accessThe DOI matches its Crossref record; 70 of the 71 references it cites were found in a registry or an index; 2 of the 13 sources its claims cite were checked for support.
Abstract
A focus on novel, confirmatory, and statistically significant results leads to substantial bias in the scientific literature. One type of bias, known as "p-hacking," occurs when researchers collect or select data or statistical analyses until nonsignificant results become significant. Here, we use text-mining to demonstrate that p-hacking is widespread throughout science. We then illustrate how one can test for p-hacking when performing a meta-analysis and show that, while p-hacking is probably common, its effect seems to be weak relative to the real effect sizes being measured. This result suggests that p-hacking probably does not drastically alter scientific consensuses drawn from meta-analyses.
Overview
CoChat Summary
## TL;DR
This paper argues that p-hacking is widespread in the scientific literature and that it inflates published effect sizes. Using p-curves from text-mined p-values and from published meta-analyses, the authors find strong evidential value but also signs of p-hacking in both broad literature samples and focused meta-analytic datasets.
## Key findings
- In the text-mined dataset, pooled p-curves showed strong evidential value in both Results and Abstracts: estimated proportion of p-values in the upper bin was 0.257 for Results and 0.262 for Abstracts, both with p < 0.001.
- The same text-mined data also showed evidence of p-hacking: estimated proportion of p-values in the upper bin was 0.546 for Results and 0.537 for Abstracts, both with p < 0.001.
- Across the meta-analysis datasets, there was evidence for evidential value: the pooled estimated proportion in the upper bin was 0.202 with p < 0.001 across 12 datasets; 9 of 12 individual p-curves were also significant for evidential value.
- For meta-analyses, evidence of p-hacking depended on whether misreported p-values were included: including them gave a pooled upper-bin proportion of 0.615 with p = 0.033, while excluding them gave 0.489 with p = 0.443; the paper states that significant p-hacking was seen in only one of the 12 questions when considering the individual datasets.
## Methods
- The authors used text-mining to extract reported p-values from all Open Access papers in PubMed, then constructed p-curves separately for p-values from the Results and Abstracts across disciplines.
- They also collected p-values from studies used in published meta-analyses in evolutionary biology/sexual selection, then built p-curves for those datasets.
- Analysis used binomial tests / binomial generalized linear models on counts of p-values in bins near the significance threshold, comparing 0 ≤ p < 0.025 vs 0.025 ≤ p < 0.05 for evidential value and 0.04 < p < 0.045 vs 0.045 < p < 0.05 for p-hacking.
## Limitations
- The paper states that text-mining combines many different types of questions across disciplines, so it remains unclear whether some subfields or questions have a flat p-curve and therefore zero average effect size.
- The paper notes that Abstracts are optional for p-value reporting, so they may disproportionately contain only the strongest results; this can bias analyses and make p-hacking harder to detect.
- Evident from the text: the meta-analysis sample is not a representative sample of all research questions, since it only includes questions important enough to warrant meta-analysis; the authors explicitly say this when discussing generalizability.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures


![Fig 3 Evidence for p-hacking across scientific disciplines. A) Evidence for p-hacking from p-values obtained from Results sections. B) Evidence for p-hacking from p-values obtained from Abstracts. The strength of p-hacking is presented as the proportion of p-values in the upper bin (0.045 < p < 0.05) with one-tailed 95% confidence intervals (calculated following Clopper and Pearson [47] using the binom.test function in R). Only disciplines where text-mining of the Results sections returned more than 25 p-values between 0.04 and 0.05 are presented. Marker colour is shaded according to the sample size: with white indicating low samples sizes and red indicating larger sample sizes.](https://figures.cochat.ai/paper-figures/c0961868545043479c9439de894f8966/the-extent-and-consequences-of-p-hacking-in-science-figure-3.jpg)

Verification
Claims the paper makes
2 Valid · 0 Not valid · 2 Not checked · 13 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“there is good evidence that journals, especially prestigious ones with higher impact factors, disproportionately publish statistically significant results [ 4 – 10 ].”
What the sources say
- Not checked yetcites [4] CB Begg, 1988, Publication bias—a problem in interpreting medical data, J R Stat Soc Ser A Stat Soc source
- Partly supportedcites [5] K Dwan, 2008, Systematic review of the empirical evidence of study publication bias and outcome reporting bias, PLoS ONE sourceExperimental — accuracy not yet measured
Dwan et al. (2008) strongly support the general half of the claim: the Abstract Conclusions state 'There is strong evidence of an association between significant results and publication; studies that report positive or significant results are more likely to be published', and Results > Publication and trial findings reports five of the empirical studies finding significant studies more likely to be published. However, the 'especially prestigious ones with higher impact factors' element appears in our copy only as one included cohort's summary in Table 1 (Easterbrook 1991: significant studies 'more likely... to be published in journals with a high citation impact factor'), which the review's own synthesis does not carry forward; the review instead notes in Results that 'studies remain unpublished due largely to failure to submit rather than rejection by journals' and in Discussion that 'None of the inception cohorts addressed the question as to whether the significance determined whether a submitted paper was accepted or not'. So a journal-selection framing ('journals disproportionately publish') overstates what this source establishes. (LLM, conf=0.78)
- Not checked yetcites [6] D Fanelli, 2012, Negative results are disappearing from most disciplines and countries, Scientometrics source
- Not checked yetcites [7] R Rosenthal, 1979, The file drawer problem and tolerance for null results, Psychol Bull source
- Not matchedcites [8] F Song, 2000, Publication and related biases, Health technology assessment (Winchester, England)
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [9] TD Sterling, 1959, Publication decisions and their possible effects on inferences drawn from tests of significance—or vice versa, J Am Stat Assoc
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not checked yetcites [10] JM Stern, 1997, Publication bias: Evidence of delayed publication in a cohort study of clinical research projects, Br Med J source
boostedstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“However, meta-analyses are compromised if the studies being synthesized do not reflect the true distribution of effect sizes [ 5 , 35 – 37 ].”
What the sources say
- Supportedcites [5] K Dwan, 2008, Systematic review of the empirical evidence of study publication bias and outcome reporting bias, PLoS ONE source
The Introduction states that 'Study publication bias will lead to overestimation of treatment effects; it has been recognised as a threat to the validity of meta-analysis and can make the readily available evidence unreliable for decision making,' and the Abstract Background repeats this. The Introduction also frames 'the bias from missing outcome data that may affect a meta-analysis' as operating at both study level (non-publication) and outcome level (selective non-reporting), and the Discussion advises that 'Meta-analyses of outcomes where several relevant trials have missing data should be seen with extra caution.' Together these support the paraphrase that a synthesis is compromised when the included studies are not a representative sample of the true effect distribution. (LLM, conf=0.85)
- Not checked yetcites [35] MD Jennions, 2004, Meta-analysis can "fail": Reply to Kotiaho and Tomkins, Oikos source
- Not checked yetcites [36] JS Kotiaho, 2002, Meta-analysis, can it ever fail?, Oikos source
- Not checked yetcites [37] AR Palmer, 2000, Quasireplication and the contract of error: Lessons from sex ratios, heritabilities and fluctuating asymmetry, Annu Rev Ecol Sys source
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For instance, we found a statistically significant indication of p-hacking in only one of the 12 questions examined in published meta-analyses ( Fig. 4 ).”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We then examined each dataset separately and found statistically significant evidential value for 9 of the 12 p-curves ( Table 3 ).”
plainn = 12 ·- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“and the Abstracts (binomial glm: estimated proportion of p-values in the upper bin (0.045 < p < 0.05) (lower CI) = 0.537 (0.518), p < 0.001, n = 10 disciplines).”
hedgedn = 10 ·- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
5 more claims
Create a free account to see all 17 claims and to verify the 9 references still open.
- It may state two separate claims as one. 7
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“At the individual study level, internal validity may be weak due to uncontrolled confounding variables, improper use of statistical methods, and use of cherry-picked data and methods to achieve statistically significant results [17].”
From Causal language and strength of inference in academic and media articles shared in social media (CLAIMS): A systematic review (2018) · cites this paper as [17]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Moreover, the same analysis found a much stronger ' evidential value ' in the literature of all disciplines”
From Is science really facing a reproducibility crisis, and do we need it to? (2018) · cites this paper as [10]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“A related issue arises when the selection of the study population or statistical model is chosen based on the P value (‘P hacking’) [64].”
From COSMOS-E: Guidance on conducting systematic reviews and meta-analyses of observational studies of etiology (2019) · cites this paper as [64]
The cited source says what the claim attributes to it.
In the citing paper
“Publication bias remains an important issue in psychological and medical research (Head et al., 2015; Thornton & Lee, 2000).”
From Trauma-related dissociation and the autonomic nervous system: a systematic literature review of psychophysiological correlates of dissociative experiencing in PTSD patients (2022) · cites this paper as Head et al., 2015
The cited source says what the claim attributes to it.
In the citing paper
“Studies that claim to test a hypothesis but do not specify a predefined primary outcome measure or those that change the primary outcome measure after data were collected (also known as primary outcome switching) are liable to selectively report only statistically significant results, favouring more positive findings [94].”
From Reporting animal research: Explanation and elaboration for the ARRIVE guidelines 2.0 (2020) · cites this paper as [94]
4 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Statistical Methods for Research Workers
(1945) · American Journal of Clinical Pathology · doi:10.1093/ajcp/15.11.552b · Cited by 10,880
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
The file drawer problem and tolerance for null results.
Rosenthal, R. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.3.638 · Cited by 9,068 · cited 1×
“There are two widely recognized types of researcher-driven publication bias: selection (also known as the “ file drawer effect ” , where studies with nonsignificant results have lower publication rates [7]) and inflation [12].”
BackgroundSimmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
THE USE OF CONFIDENCE OR FIDUCIAL LIMITS ILLUSTRATED IN THE CASE OF THE BINOMIAL
CLOPPER, C. J., & PEARSON, E. S. (1934) · Biometrika · doi:10.1093/biomet/26.4.404 · Cited by 5,003
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · cited 4×
“Results from prespecified studies offer far more convincing evidence than those from exploratory research [2].”
BackgroundResultInfluentialEffect size, confidence interval and statistical significance: a practical guide for biologists
Nakagawa, S., & Cuthill, I. C. (2007) · Biological Reviews · doi:10.1111/j.1469-185x.2007.00027.x · Cited by 3,835 · cited 1×
“Similarly, a low p -value does not mean that a finding is of major clinical or biological interest [28].”
BackgroundThe New Statistics: Why and How
Cumming, G. (2014) · Psychological Science · doi:10.1177/0956797613504966 · Cited by 3,200
A practical solution to the pervasive problems ofp values
Wagenmakers, E.-J. (2007) · Psychonomic Bulletin & Review · doi:10.3758/bf03194105 · Cited by 2,490 · cited 1×
“• Encourage and/or provide platforms for method prespecification [13,65].”
BackgroundMeasuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling
John, L. K., Loewenstein, G., & Prelec, D. (2012) · Psychological Science · doi:10.1177/0956797611430953 · Cited by 2,239 · cited 5×
“John et al. [16] measured the prevalence of questionable research practices in psychology.”
BackgroundInfluential
Cited by
Items 101–122 of 122
Elsherif, M. M., Middleton, S. L., Phan, J. M., Azevedo, F., Iley, B. J., Grose-Hodge, M., Tyler, S. L., Kapp, S. K., Gourdon-Kanhukamwe, A., Grafton-Clarke, D., et al. (2022) · doi:10.31222/osf.io/k7a9p · Cited by 91
Foy, B. H., Sundt, T. M., Carlson, J. C. T., Aguirre, A. D., & Higgins, J. M. (2022) · Nature Communications · doi:10.1038/s41467-022-32222-2 · Cited by 84
Digital open science—Teaching digital tools for reproducible and transparent research
Toelch, U., & Ostwald, D. (2018) · PLOS Biology · doi:10.1371/journal.pbio.2006022 · Cited by 80
“We discussed how preregistration potentially prevents p - hacking [16] and Hypothesising After Results are Known (HARKing) [17], two major threats for the reproducibility of science [18].”
BackgroundMethodStress and cognition: A user’s guide to designing and interpreting studies
Shields, G. S. (2020) · Psychoneuroendocrinology · doi:10.1016/j.psyneuen.2019.104475 · Cited by 80
Bishop, D. V. M., & Thompson, P. A. (2016) · PeerJ · doi:10.7717/peerj.1715 · Cited by 90
“The script yields tables that contain information similar to that reported by Head et al. (2015): the number of runs with p-values in specific frequency bins.”
BackgroundMethodInfluentialConstruct Validity in Software Engineering Research and Software Metrics
Ralph, P., & Tempero, E. (2018) · Proceedings of the 22nd International Conference on Evaluation and Assessment in Software Engineering 2018 · doi:10.1145/3210459.3210461 · Cited by 78
“In contrast, reworking the nomological net (e.g. dropping indicators, changing hypotheses) after the main study is p-hacking [13] and therefore unacceptable.”
BackgroundBeutler, S., Mertens, Y. L., Ladner, L., Schellong, J., Croy, I., & Daniels, J. K. (2022) · European Journal of Psychotraumatology · doi:10.1080/20008066.2022.2132599 · Cited by 64
“Publication bias remains an important issue in psychological and medical research (Head et al., 2015; Thornton & Lee, 2000).”
BackgroundLehr, S. A., Caliskan, A., Liyanage, S., & Banaji, M. R. (2024) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2404328121 · Cited by 66
“One source of these issues is unquestionably poor statistical practices by generally well - intentioned scientists: Studies show that poor statistical practices are widely prevalent across scientific research (31, 32).”
Biomedical text mining for research rigor and integrity: tasks, challenges, directions
Kilicoglu, H. (2017) · Briefings in Bioinformatics · doi:10.1093/bib/bbx057 · Cited by 61
Effectiveness of multidisciplinary programmes for clinical pain conditions: an umbrella review.
Dragioti, E., Evangelou, E., Larsson, B., & Gerdle, B. (2018) · Journal of Rehabilitation Medicine · doi:10.2340/16501977-2377 · Cited by 53
Kumpfer, K. L., Scheier, L. M., & Brown, J. (2020) · Evaluation & the Health Professions · doi:10.1177/0163278718772886 · Cited by 41
Human Minds and Animal Stories: How Narratives Make Us Care About Other Species
Małecki, W., Sorokowski, P., Pawłowski, B., & Cieński, M. (2019) · doi:10.4324/9780429061424 · Cited by 30
Magnetic Stirring May Cause Irreproducible Results in Chemical Reactions
Cherepanova, V. A., Gordeev, E. G., & Ananikov, V. P. (2025) · JACS Au · doi:10.1021/jacsau.5c00412 · Cited by 25
Allegiance effects in mindfulness-based interventions for psychiatric disorders: A meta-re-analysis
Goldberg, S. B., & Tucker, R. P. (2020) · Psychotherapy Research · doi:10.1080/10503307.2019.1664783 · Cited by 21
The rodent object-in-context task: A systematic review and meta-analysis of important variables
Sep, M. S. C., Vellinga, M., Sarabdjitsingh, R. A., & Joëls, M. (2021) · PLOS ONE · doi:10.1371/journal.pone.0249102 · Cited by 17
FEP Protocol Builder: Optimization of Free Energy Perturbation Protocols Using Active Learning
de Oliveira, C., Leswing, K., Feng, S., Kanters, R., Abel, R., & Bhat, S. (2023) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.3c00681 · Cited by 15
Assessing Open Science practices in physical activity behaviour change intervention evaluations
Norris, E., Sulevani, I., Finnerty, A. N., & Castro, O. (2022) · BMJ Open Sport & Exercise Medicine · doi:10.1136/bmjsem-2021-001282 · Cited by 14
Reproducibility failure in biomedical research: problems and solutions
Haven, T. L., & Ioannidis, J. P. A. (2026) · Annual Review of Medicine · doi:10.1146/annurev-med-050124-050859 · Cited by 9
Statistical messages from ARRIVE 2.0 guidelines
Kang, H. (2021) · The Korean Journal of Pain · doi:10.3344/kjp.2021.34.1.1 · Cited by 6
Epidemiologische Beiträge zur Klinischen Psychologie
Beesdo-Baum, K., Höfler, M., Jacobi, F., & Wittchen, H.-U. (2020) · Klinische Psychologie & Psychotherapie · doi:10.1007/978-3-662-61814-1_3 · Cited by 6
Levy, N. (2021) · Bad Beliefs · doi:10.1093/oso/9780192895325.003.0005 · Cited by 2
Yang, Y., Lagisz, M., Williams, C., Pan, J., Noble, D., & Nakagawa, S. (2023) · doi:10.32942/x20g6q · Cited by 2
Metadata
- Authors
- Head, M. L., Holman, L., Lanfear, R., Kahn, A. T., & Jennions, M. D.
- Affiliations
- Australian National University (AU)
- Corresponding author
- Head, Megan L.
- Venue
- PLOS Biology
- Publisher
- Public Library of Science (PLoS)
- Published
- 2015-03-13
- Volume
- 13
- Issue
- 3
- Pages
- e1002106
- ISSN
- 1544-9173 (ISSN-L)
- 1545-7885
- PMC
- PMC4359000
- OpenAlex
- W2164005910
- Semantic Scholar
- a07a64ba110e0f9f7156f3bd1e376f0d2e1cddf1
- Cited by
- 1,388
- References
- 71
- Citations by year
- 2026: 66 · 2025: 124 · 2024: 104 · 2023: 157 · 2022: 151 · 2021: 167 · 2020: 178 · 2019: 141 · 2018: 121 · 2017: 91 · 2016: 64 · 2015: 24
- FWCI
- 91.09
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- PLoS Biology (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Minerva Access (University of Melbourne) (submittedVersion, cc-by)
- Edinburgh Napier Research Repository (Edinburgh Napier University) (submittedVersion, cc-by)
- ANU Open Research (Australian National University) (submittedVersion, cc-by)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Education › Academic integrity and plagiarism
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Keywords
- p-hacking, research bias, statistical significance, meta-analysis, text mining, publication bias, effect size, scientific consensus
- MeSH terms
- Humans, Science, Statistics as Topic, Meta-Analysis as Topic, Publication Bias
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (qwen3.8-omni-flash).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.