CoChat

The Extent and Consequences of P-Hacking in Science

Open access
Head, Megan L., Holman, Luke, Lanfear, Rob, Kahn, Andrew T., Jennions, Michael D. · 2015 · PLOS Biology

The DOI matches its Crossref record; 70 of the 71 references it cites were found in a registry or an index; 2 of the 13 sources its claims cite were checked for support.

Abstract

A focus on novel, confirmatory, and statistically significant results leads to substantial bias in the scientific literature. One type of bias, known as "p-hacking," occurs when researchers collect or select data or statistical analyses until nonsignificant results become significant. Here, we use text-mining to demonstrate that p-hacking is widespread throughout science. We then illustrate how one can test for p-hacking when performing a meta-analysis and show that, while p-hacking is probably common, its effect seems to be weak relative to the real effect sizes being measured. This result suggests that p-hacking probably does not drastically alter scientific consensuses drawn from meta-analyses.

Overview

CoChat Summary

## TL;DR

This paper argues that p-hacking is widespread in the scientific literature and that it inflates published effect sizes. Using p-curves from text-mined p-values and from published meta-analyses, the authors find strong evidential value but also signs of p-hacking in both broad literature samples and focused meta-analytic datasets.

## Key findings

  • In the text-mined dataset, pooled p-curves showed strong evidential value in both Results and Abstracts: estimated proportion of p-values in the upper bin was 0.257 for Results and 0.262 for Abstracts, both with p < 0.001.
  • The same text-mined data also showed evidence of p-hacking: estimated proportion of p-values in the upper bin was 0.546 for Results and 0.537 for Abstracts, both with p < 0.001.
  • Across the meta-analysis datasets, there was evidence for evidential value: the pooled estimated proportion in the upper bin was 0.202 with p < 0.001 across 12 datasets; 9 of 12 individual p-curves were also significant for evidential value.
  • For meta-analyses, evidence of p-hacking depended on whether misreported p-values were included: including them gave a pooled upper-bin proportion of 0.615 with p = 0.033, while excluding them gave 0.489 with p = 0.443; the paper states that significant p-hacking was seen in only one of the 12 questions when considering the individual datasets.

## Methods

  • The authors used text-mining to extract reported p-values from all Open Access papers in PubMed, then constructed p-curves separately for p-values from the Results and Abstracts across disciplines.
  • They also collected p-values from studies used in published meta-analyses in evolutionary biology/sexual selection, then built p-curves for those datasets.
  • Analysis used binomial tests / binomial generalized linear models on counts of p-values in bins near the significance threshold, comparing 0 ≤ p < 0.025 vs 0.025 ≤ p < 0.05 for evidential value and 0.04 < p < 0.045 vs 0.045 < p < 0.05 for p-hacking.

## Limitations

  • The paper states that text-mining combines many different types of questions across disciplines, so it remains unclear whether some subfields or questions have a flat p-curve and therefore zero average effect size.
  • The paper notes that Abstracts are optional for p-value reporting, so they may disproportionately contain only the strongest results; this can bias analyses and make p-hacking harder to detect.
  • Evident from the text: the meta-analysis sample is not a representative sample of all research questions, since it only includes questions important enough to warrant meta-analysis; the authors explicitly say this when discussing generalizability.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1371/journal.pbio.1002106

Cites

70 of 70 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Fig 1 The effect of publication bias on the distribution of p-values around the significance threshold of 0.05. A) Black line shows distribution of p-values when there is no evidential value and the red line shows how publication bias influences this distribution. B) Black line shows distribution of p-values when there is evidential value and red line shows how publication bias influences this distribution. Tests for publication bias due to a file-drawer effect often compare the number of p-values in each of the bins either side of 0.05.
Fig 1 The effect of publication bias on the distribution of p-values around the significance threshold of 0.05. A) Black line shows distribution of p-values when there is no evidential value and the red line shows how publication bias influences this distribution. B) Black line shows distribution of p-values when there is evidential value and red line shows how publication bias influences this distribution. Tests for publication bias due to a file-drawer effect often compare the number of p-values in each of the bins either side of 0.05.CC BY · PubMed Central
Fig 2 The effect of p-hacking on the distribution of p-values in the range of significance. A) Black line shows distribution of p-values when there is no evidential value and the red line shows how p-hacking influences this distribution. B) Black line shows distribution of p-values when there is evidential value and the red line shows how p-hacking influences this distribution. Tests for p-hacking often compare the number of p-values in two adjacent bins just below 0.05.
Fig 2 The effect of p-hacking on the distribution of p-values in the range of significance. A) Black line shows distribution of p-values when there is no evidential value and the red line shows how p-hacking influences this distribution. B) Black line shows distribution of p-values when there is evidential value and the red line shows how p-hacking influences this distribution. Tests for p-hacking often compare the number of p-values in two adjacent bins just below 0.05.CC BY · PubMed Central
Fig 3 Evidence for p-hacking across scientific disciplines. A) Evidence for p-hacking from p-values obtained from Results sections. B) Evidence for p-hacking from p-values obtained from Abstracts. The strength of p-hacking is presented as the proportion of p-values in the upper bin (0.045 < p < 0.05) with one-tailed 95% confidence intervals (calculated following Clopper and Pearson [47] using the binom.test function in R). Only disciplines where text-mining of the Results sections returned more than 25 p-values between 0.04 and 0.05 are presented. Marker colour is shaded according to the sample size: with white indicating low samples sizes and red indicating larger sample sizes.
Fig 3 Evidence for p-hacking across scientific disciplines. A) Evidence for p-hacking from p-values obtained from Results sections. B) Evidence for p-hacking from p-values obtained from Abstracts. The strength of p-hacking is presented as the proportion of p-values in the upper bin (0.045 < p < 0.05) with one-tailed 95% confidence intervals (calculated following Clopper and Pearson [47] using the binom.test function in R). Only disciplines where text-mining of the Results sections returned more than 25 p-values between 0.04 and 0.05 are presented. Marker colour is shaded according to the sample size: with white indicating low samples sizes and red indicating larger sample sizes.CC BY · PubMed Central
Fig 4 The distribution of p-values associated with the meta-analysis conducted by Jiang et al. (2013). The p-curve shows evidence for evidential value (strong right skew) and p-hacking (rise in p-values just below 0.05).
Fig 4 The distribution of p-values associated with the meta-analysis conducted by Jiang et al. (2013). The p-curve shows evidence for evidential value (strong right skew) and p-hacking (rise in p-values just below 0.05).CC BY · PubMed Central

Verification

Claims the paper makes

2 Valid · 0 Not valid · 2 Not checked · 13 Own results and interpretations

5 more claims

Create a free account to see all 17 claims and to verify the 9 references still open.

See all claims — free account
  • It may state two separate claims as one. 7
  • The quote only partly states this claim. 1

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

4 more claims about this paper

Create a free account to see every claim other papers make about this one.

See all claims — free account

Citations

Cites

Show all 70 references

Cited by

Metadata

Authors
Head, M. L., Holman, L., Lanfear, R., Kahn, A. T., & Jennions, M. D.
Affiliations
Australian National University (AU)
Corresponding author
Head, Megan L.
Venue
PLOS Biology
Published
2015-03-13
Volume
13
Issue
3
Pages
e1002106
ISSN
  • 1544-9173 (ISSN-L)
  • 1545-7885
OpenAlex
W2164005910
Cited by
1,388
References
71
Citations by year
2026: 66 · 2025: 124 · 2024: 104 · 2023: 157 · 2022: 151 · 2021: 167 · 2020: 178 · 2019: 141 · 2018: 121 · 2017: 91 · 2016: 64 · 2015: 24
FWCI
91.09
Type
journal article
Language
en
Open access
Locations
  • PLoS Biology (publishedVersion, cc-by)
  • Europe PMC (PubMed Central) (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • RePEc: Research Papers in Economics (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • Minerva Access (University of Melbourne) (submittedVersion, cc-by)
  • Edinburgh Napier Research Repository (Edinburgh Napier University) (submittedVersion, cc-by)
  • ANU Open Research (Australian National University) (submittedVersion, cc-by)
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Social Sciences › Education › Academic integrity and plagiarism
  • Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
Keywords
p-hacking, research bias, statistical significance, meta-analysis, text mining, publication bias, effect size, scientific consensus
MeSH terms
Humans, Science, Statistics as Topic, Meta-Analysis as Topic, Publication Bias

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (qwen3.8-omni-flash).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.