The Extent and Consequences of P-Hacking in Science
Open accessThe DOI matches its Crossref record; 70 of the 71 references it cites were found in a registry or an index; 2 of the 13 sources its claims cite were checked for support.
Abstract
A focus on novel, confirmatory, and statistically significant results leads to substantial bias in the scientific literature. One type of bias, known as "p-hacking," occurs when researchers collect or select data or statistical analyses until nonsignificant results become significant. Here, we use text-mining to demonstrate that p-hacking is widespread throughout science. We then illustrate how one can test for p-hacking when performing a meta-analysis and show that, while p-hacking is probably common, its effect seems to be weak relative to the real effect sizes being measured. This result suggests that p-hacking probably does not drastically alter scientific consensuses drawn from meta-analyses.
Overview
CoChat Summary
## TL;DR
This paper argues that p-hacking is widespread in the scientific literature and that it inflates published effect sizes. Using p-curves from text-mined p-values and from published meta-analyses, the authors find strong evidential value but also signs of p-hacking in both broad literature samples and focused meta-analytic datasets.
## Key findings
- In the text-mined dataset, pooled p-curves showed strong evidential value in both Results and Abstracts: estimated proportion of p-values in the upper bin was 0.257 for Results and 0.262 for Abstracts, both with p < 0.001.
- The same text-mined data also showed evidence of p-hacking: estimated proportion of p-values in the upper bin was 0.546 for Results and 0.537 for Abstracts, both with p < 0.001.
- Across the meta-analysis datasets, there was evidence for evidential value: the pooled estimated proportion in the upper bin was 0.202 with p < 0.001 across 12 datasets; 9 of 12 individual p-curves were also significant for evidential value.
- For meta-analyses, evidence of p-hacking depended on whether misreported p-values were included: including them gave a pooled upper-bin proportion of 0.615 with p = 0.033, while excluding them gave 0.489 with p = 0.443; the paper states that significant p-hacking was seen in only one of the 12 questions when considering the individual datasets.
## Methods
- The authors used text-mining to extract reported p-values from all Open Access papers in PubMed, then constructed p-curves separately for p-values from the Results and Abstracts across disciplines.
- They also collected p-values from studies used in published meta-analyses in evolutionary biology/sexual selection, then built p-curves for those datasets.
- Analysis used binomial tests / binomial generalized linear models on counts of p-values in bins near the significance threshold, comparing 0 ≤ p < 0.025 vs 0.025 ≤ p < 0.05 for evidential value and 0.04 < p < 0.045 vs 0.045 < p < 0.05 for p-hacking.
## Limitations
- The paper states that text-mining combines many different types of questions across disciplines, so it remains unclear whether some subfields or questions have a flat p-curve and therefore zero average effect size.
- The paper notes that Abstracts are optional for p-value reporting, so they may disproportionately contain only the strongest results; this can bias analyses and make p-hacking harder to detect.
- Evident from the text: the meta-analysis sample is not a representative sample of all research questions, since it only includes questions important enough to warrant meta-analysis; the authors explicitly say this when discussing generalizability.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures


![Fig 3 Evidence for p-hacking across scientific disciplines. A) Evidence for p-hacking from p-values obtained from Results sections. B) Evidence for p-hacking from p-values obtained from Abstracts. The strength of p-hacking is presented as the proportion of p-values in the upper bin (0.045 < p < 0.05) with one-tailed 95% confidence intervals (calculated following Clopper and Pearson [47] using the binom.test function in R). Only disciplines where text-mining of the Results sections returned more than 25 p-values between 0.04 and 0.05 are presented. Marker colour is shaded according to the sample size: with white indicating low samples sizes and red indicating larger sample sizes.](https://figures.cochat.ai/paper-figures/c0961868545043479c9439de894f8966/the-extent-and-consequences-of-p-hacking-in-science-figure-3.jpg)

Verification
Claims the paper makes
2 Valid · 0 Not valid · 2 Not checked · 13 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“there is good evidence that journals, especially prestigious ones with higher impact factors, disproportionately publish statistically significant results [ 4 – 10 ].”
What the sources say
- Not checked yetcites [4] CB Begg, 1988, Publication bias—a problem in interpreting medical data, J R Stat Soc Ser A Stat Soc source
- Partly supportedcites [5] K Dwan, 2008, Systematic review of the empirical evidence of study publication bias and outcome reporting bias, PLoS ONE sourceExperimental — accuracy not yet measured
Dwan et al. (2008) strongly support the general half of the claim: the Abstract Conclusions state 'There is strong evidence of an association between significant results and publication; studies that report positive or significant results are more likely to be published', and Results > Publication and trial findings reports five of the empirical studies finding significant studies more likely to be published. However, the 'especially prestigious ones with higher impact factors' element appears in our copy only as one included cohort's summary in Table 1 (Easterbrook 1991: significant studies 'more likely... to be published in journals with a high citation impact factor'), which the review's own synthesis does not carry forward; the review instead notes in Results that 'studies remain unpublished due largely to failure to submit rather than rejection by journals' and in Discussion that 'None of the inception cohorts addressed the question as to whether the significance determined whether a submitted paper was accepted or not'. So a journal-selection framing ('journals disproportionately publish') overstates what this source establishes. (LLM, conf=0.78)
- Not checked yetcites [6] D Fanelli, 2012, Negative results are disappearing from most disciplines and countries, Scientometrics source
- Not checked yetcites [7] R Rosenthal, 1979, The file drawer problem and tolerance for null results, Psychol Bull source
- Not matchedcites [8] F Song, 2000, Publication and related biases, Health technology assessment (Winchester, England)
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [9] TD Sterling, 1959, Publication decisions and their possible effects on inferences drawn from tests of significance—or vice versa, J Am Stat Assoc
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not checked yetcites [10] JM Stern, 1997, Publication bias: Evidence of delayed publication in a cohort study of clinical research projects, Br Med J source
boostedstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“However, meta-analyses are compromised if the studies being synthesized do not reflect the true distribution of effect sizes [ 5 , 35 – 37 ].”
What the sources say
- Supportedcites [5] K Dwan, 2008, Systematic review of the empirical evidence of study publication bias and outcome reporting bias, PLoS ONE source
The Introduction states that 'Study publication bias will lead to overestimation of treatment effects; it has been recognised as a threat to the validity of meta-analysis and can make the readily available evidence unreliable for decision making,' and the Abstract Background repeats this. The Introduction also frames 'the bias from missing outcome data that may affect a meta-analysis' as operating at both study level (non-publication) and outcome level (selective non-reporting), and the Discussion advises that 'Meta-analyses of outcomes where several relevant trials have missing data should be seen with extra caution.' Together these support the paraphrase that a synthesis is compromised when the included studies are not a representative sample of the true effect distribution. (LLM, conf=0.85)
- Not checked yetcites [35] MD Jennions, 2004, Meta-analysis can "fail": Reply to Kotiaho and Tomkins, Oikos source
- Not checked yetcites [36] JS Kotiaho, 2002, Meta-analysis, can it ever fail?, Oikos source
- Not checked yetcites [37] AR Palmer, 2000, Quasireplication and the contract of error: Lessons from sex ratios, heritabilities and fluctuating asymmetry, Annu Rev Ecol Sys source
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For instance, we found a statistically significant indication of p-hacking in only one of the 12 questions examined in published meta-analyses ( Fig. 4 ).”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We then examined each dataset separately and found statistically significant evidential value for 9 of the 12 p-curves ( Table 3 ).”
plainn = 12 ·- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“and the Abstracts (binomial glm: estimated proportion of p-values in the upper bin (0.045 < p < 0.05) (lower CI) = 0.537 (0.518), p < 0.001, n = 10 disciplines).”
hedgedn = 10 ·- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
5 more claims
Create a free account to see all 17 claims and to verify the 9 references still open.
- It may state two separate claims as one. 7
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“At the individual study level, internal validity may be weak due to uncontrolled confounding variables, improper use of statistical methods, and use of cherry-picked data and methods to achieve statistically significant results [17].”
From Causal language and strength of inference in academic and media articles shared in social media (CLAIMS): A systematic review (2018) · cites this paper as [17]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Moreover, the same analysis found a much stronger ' evidential value ' in the literature of all disciplines”
From Is science really facing a reproducibility crisis, and do we need it to? (2018) · cites this paper as [10]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“A related issue arises when the selection of the study population or statistical model is chosen based on the P value (‘P hacking’) [64].”
From COSMOS-E: Guidance on conducting systematic reviews and meta-analyses of observational studies of etiology (2019) · cites this paper as [64]
The cited source says what the claim attributes to it.
In the citing paper
“Publication bias remains an important issue in psychological and medical research (Head et al., 2015; Thornton & Lee, 2000).”
From Trauma-related dissociation and the autonomic nervous system: a systematic literature review of psychophysiological correlates of dissociative experiencing in PTSD patients (2022) · cites this paper as Head et al., 2015
The cited source says what the claim attributes to it.
In the citing paper
“Studies that claim to test a hypothesis but do not specify a predefined primary outcome measure or those that change the primary outcome measure after data were collected (also known as primary outcome switching) are liable to selectively report only statistically significant results, favouring more positive findings [94].”
From Reporting animal research: Explanation and elaboration for the ARRIVE guidelines 2.0 (2020) · cites this paper as [94]
4 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Statistical Methods for Research Workers
(1945) · American Journal of Clinical Pathology · doi:10.1093/ajcp/15.11.552b · Cited by 10,880
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
The file drawer problem and tolerance for null results.
Rosenthal, R. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.3.638 · Cited by 9,068 · cited 1×
“There are two widely recognized types of researcher-driven publication bias: selection (also known as the “ file drawer effect ” , where studies with nonsignificant results have lower publication rates [7]) and inflation [12].”
BackgroundSimmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
THE USE OF CONFIDENCE OR FIDUCIAL LIMITS ILLUSTRATED IN THE CASE OF THE BINOMIAL
CLOPPER, C. J., & PEARSON, E. S. (1934) · Biometrika · doi:10.1093/biomet/26.4.404 · Cited by 5,003
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · cited 4×
“Results from prespecified studies offer far more convincing evidence than those from exploratory research [2].”
BackgroundResultInfluentialEffect size, confidence interval and statistical significance: a practical guide for biologists
Nakagawa, S., & Cuthill, I. C. (2007) · Biological Reviews · doi:10.1111/j.1469-185x.2007.00027.x · Cited by 3,835 · cited 1×
“Similarly, a low p -value does not mean that a finding is of major clinical or biological interest [28].”
BackgroundThe New Statistics: Why and How
Cumming, G. (2014) · Psychological Science · doi:10.1177/0956797613504966 · Cited by 3,200
A practical solution to the pervasive problems ofp values
Wagenmakers, E.-J. (2007) · Psychonomic Bulletin & Review · doi:10.3758/bf03194105 · Cited by 2,490 · cited 1×
“• Encourage and/or provide platforms for method prespecification [13,65].”
BackgroundMeasuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling
John, L. K., Loewenstein, G., & Prelec, D. (2012) · Psychological Science · doi:10.1177/0956797611430953 · Cited by 2,239 · cited 5×
“John et al. [16] measured the prevalence of questionable research practices in psychology.”
BackgroundInfluential
Cited by
Reporting animal research: Explanation and elaboration for the ARRIVE guidelines 2.0
Percie du Sert, N., Ahluwalia, A., Alam, S., Avey, M. T., Baker, M., Browne, W. J., Clark, A., Cuthill, I. C., Dirnagl, U., Emerson, M., et al. (2020) · PLOS Biology · doi:10.1371/journal.pbio.3000411 · Cited by 2,677
“Analysing the data using different methods and selectively reporting those with statistically significant results constitutes p-hacking and introduces bias in the literature [90,94].”
MethodResultMethods for the detection of carelessly invalid responses in survey data
Curran, P. G. (2016) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2015.07.006 · Cited by 1,356
Experimental design and analysis and their reporting: new guidance for publication in BJP
Curtis, M. J., Bond, R. A., Spina, D., Ahluwalia, A., Alexander, S. P. A., Giembycz, M. A., Gilchrist, A., Hoyer, D., Insel, P. A., Izzo, A. A., et al. (2015) · British Journal of Pharmacology · doi:10.1111/bph.12856 · Cited by 1,033
“The first point to make is that one should not collect data then try out various statistical tests until one obtains the ‘desired’ result, known as ‘P hacking’ (Head et al ., 2015).”
BackgroundThe contact hypothesis re-evaluated
PALUCK, E. L., GREEN, S. A., & GREEN, D. P. (2019) · Behavioural Public Policy · doi:10.1017/bpp.2018.25 · Cited by 856
“The same may be said for our analysis of p-hacking (Simonsohn et al., 2014; Head et al., 2015), which is symptomatic of research discretion that favors significant relationships.”
O'Dea, R. E., Lagisz, M., Jennions, M. D., Koricheva, J., Noble, D. W. A., Parker, T. H., Gurevitch, J., Page, M. J., Stewart, G., Moher, D., et al. (2021) · Biological Reviews · doi:10.1111/brv.12721 · Cited by 697
“Forexample,givenmultiplechoices instatistical analyses, researchersmight favour paths leading to statistical significance (i.e. ‘P-hacking’; Simmons, Nelson & Simonsohn, 2011; Head et al., 2015).”
BackgroundSelf-citation
Metadata
- Authors
- Head, M. L., Holman, L., Lanfear, R., Kahn, A. T., & Jennions, M. D.
- Affiliations
- Australian National University (AU)
- Corresponding author
- Head, Megan L.
- Venue
- PLOS Biology
- Publisher
- Public Library of Science (PLoS)
- Published
- 2015-03-13
- Volume
- 13
- Issue
- 3
- Pages
- e1002106
- ISSN
- 1544-9173 (ISSN-L)
- 1545-7885
- PMC
- PMC4359000
- OpenAlex
- W2164005910
- Semantic Scholar
- a07a64ba110e0f9f7156f3bd1e376f0d2e1cddf1
- Cited by
- 1,388
- References
- 71
- Citations by year
- 2026: 66 · 2025: 124 · 2024: 104 · 2023: 157 · 2022: 151 · 2021: 167 · 2020: 178 · 2019: 141 · 2018: 121 · 2017: 91 · 2016: 64 · 2015: 24
- FWCI
- 91.09
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- PLoS Biology (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Minerva Access (University of Melbourne) (submittedVersion, cc-by)
- Edinburgh Napier Research Repository (Edinburgh Napier University) (submittedVersion, cc-by)
- ANU Open Research (Australian National University) (submittedVersion, cc-by)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Education › Academic integrity and plagiarism
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Keywords
- p-hacking, research bias, statistical significance, meta-analysis, text mining, publication bias, effect size, scientific consensus
- MeSH terms
- Humans, Science, Statistics as Topic, Meta-Analysis as Topic, Publication Bias
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (qwen3.8-omni-flash).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: The Extent and Consequences of P-Hacking in Science. https://app.cochat.ai/papers/10.1371/journal.pbio.1002106
- The paper
- Head, M. L., Holman, L., Lanfear, R., Kahn, A. T., & Jennions, M. D. (2015). The Extent and Consequences of P-Hacking in Science. PLOS Biology. https://doi.org/10.1371/journal.pbio.1002106
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.