The Extent and Consequences of P-Hacking in Science
Open accessThe DOI matches its Crossref record; 70 of the 71 references it cites were found in a registry or an index; 2 of the 13 sources its claims cite were checked for support.
Abstract
A focus on novel, confirmatory, and statistically significant results leads to substantial bias in the scientific literature. One type of bias, known as "p-hacking," occurs when researchers collect or select data or statistical analyses until nonsignificant results become significant. Here, we use text-mining to demonstrate that p-hacking is widespread throughout science. We then illustrate how one can test for p-hacking when performing a meta-analysis and show that, while p-hacking is probably common, its effect seems to be weak relative to the real effect sizes being measured. This result suggests that p-hacking probably does not drastically alter scientific consensuses drawn from meta-analyses.
Overview
CoChat Summary
## TL;DR
This paper argues that p-hacking is widespread in the scientific literature and that it inflates published effect sizes. Using p-curves from text-mined p-values and from published meta-analyses, the authors find strong evidential value but also signs of p-hacking in both broad literature samples and focused meta-analytic datasets.
## Key findings
- In the text-mined dataset, pooled p-curves showed strong evidential value in both Results and Abstracts: estimated proportion of p-values in the upper bin was 0.257 for Results and 0.262 for Abstracts, both with p < 0.001.
- The same text-mined data also showed evidence of p-hacking: estimated proportion of p-values in the upper bin was 0.546 for Results and 0.537 for Abstracts, both with p < 0.001.
- Across the meta-analysis datasets, there was evidence for evidential value: the pooled estimated proportion in the upper bin was 0.202 with p < 0.001 across 12 datasets; 9 of 12 individual p-curves were also significant for evidential value.
- For meta-analyses, evidence of p-hacking depended on whether misreported p-values were included: including them gave a pooled upper-bin proportion of 0.615 with p = 0.033, while excluding them gave 0.489 with p = 0.443; the paper states that significant p-hacking was seen in only one of the 12 questions when considering the individual datasets.
## Methods
- The authors used text-mining to extract reported p-values from all Open Access papers in PubMed, then constructed p-curves separately for p-values from the Results and Abstracts across disciplines.
- They also collected p-values from studies used in published meta-analyses in evolutionary biology/sexual selection, then built p-curves for those datasets.
- Analysis used binomial tests / binomial generalized linear models on counts of p-values in bins near the significance threshold, comparing 0 ≤ p < 0.025 vs 0.025 ≤ p < 0.05 for evidential value and 0.04 < p < 0.045 vs 0.045 < p < 0.05 for p-hacking.
## Limitations
- The paper states that text-mining combines many different types of questions across disciplines, so it remains unclear whether some subfields or questions have a flat p-curve and therefore zero average effect size.
- The paper notes that Abstracts are optional for p-value reporting, so they may disproportionately contain only the strongest results; this can bias analyses and make p-hacking harder to detect.
- Evident from the text: the meta-analysis sample is not a representative sample of all research questions, since it only includes questions important enough to warrant meta-analysis; the authors explicitly say this when discussing generalizability.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures


![Fig 3 Evidence for p-hacking across scientific disciplines. A) Evidence for p-hacking from p-values obtained from Results sections. B) Evidence for p-hacking from p-values obtained from Abstracts. The strength of p-hacking is presented as the proportion of p-values in the upper bin (0.045 < p < 0.05) with one-tailed 95% confidence intervals (calculated following Clopper and Pearson [47] using the binom.test function in R). Only disciplines where text-mining of the Results sections returned more than 25 p-values between 0.04 and 0.05 are presented. Marker colour is shaded according to the sample size: with white indicating low samples sizes and red indicating larger sample sizes.](https://figures.cochat.ai/paper-figures/c0961868545043479c9439de894f8966/the-extent-and-consequences-of-p-hacking-in-science-figure-3.jpg)

Verification
Claims the paper makes
2 Valid · 0 Not valid · 2 Not checked · 13 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“there is good evidence that journals, especially prestigious ones with higher impact factors, disproportionately publish statistically significant results [ 4 – 10 ].”
What the sources say
- Not checked yetcites [4] CB Begg, 1988, Publication bias—a problem in interpreting medical data, J R Stat Soc Ser A Stat Soc source
- Partly supportedcites [5] K Dwan, 2008, Systematic review of the empirical evidence of study publication bias and outcome reporting bias, PLoS ONE sourceExperimental — accuracy not yet measured
Dwan et al. (2008) strongly support the general half of the claim: the Abstract Conclusions state 'There is strong evidence of an association between significant results and publication; studies that report positive or significant results are more likely to be published', and Results > Publication and trial findings reports five of the empirical studies finding significant studies more likely to be published. However, the 'especially prestigious ones with higher impact factors' element appears in our copy only as one included cohort's summary in Table 1 (Easterbrook 1991: significant studies 'more likely... to be published in journals with a high citation impact factor'), which the review's own synthesis does not carry forward; the review instead notes in Results that 'studies remain unpublished due largely to failure to submit rather than rejection by journals' and in Discussion that 'None of the inception cohorts addressed the question as to whether the significance determined whether a submitted paper was accepted or not'. So a journal-selection framing ('journals disproportionately publish') overstates what this source establishes. (LLM, conf=0.78)
- Not checked yetcites [6] D Fanelli, 2012, Negative results are disappearing from most disciplines and countries, Scientometrics source
- Not checked yetcites [7] R Rosenthal, 1979, The file drawer problem and tolerance for null results, Psychol Bull source
- Not matchedcites [8] F Song, 2000, Publication and related biases, Health technology assessment (Winchester, England)
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [9] TD Sterling, 1959, Publication decisions and their possible effects on inferences drawn from tests of significance—or vice versa, J Am Stat Assoc
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not checked yetcites [10] JM Stern, 1997, Publication bias: Evidence of delayed publication in a cohort study of clinical research projects, Br Med J source
boostedstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“However, meta-analyses are compromised if the studies being synthesized do not reflect the true distribution of effect sizes [ 5 , 35 – 37 ].”
What the sources say
- Supportedcites [5] K Dwan, 2008, Systematic review of the empirical evidence of study publication bias and outcome reporting bias, PLoS ONE source
The Introduction states that 'Study publication bias will lead to overestimation of treatment effects; it has been recognised as a threat to the validity of meta-analysis and can make the readily available evidence unreliable for decision making,' and the Abstract Background repeats this. The Introduction also frames 'the bias from missing outcome data that may affect a meta-analysis' as operating at both study level (non-publication) and outcome level (selective non-reporting), and the Discussion advises that 'Meta-analyses of outcomes where several relevant trials have missing data should be seen with extra caution.' Together these support the paraphrase that a synthesis is compromised when the included studies are not a representative sample of the true effect distribution. (LLM, conf=0.85)
- Not checked yetcites [35] MD Jennions, 2004, Meta-analysis can "fail": Reply to Kotiaho and Tomkins, Oikos source
- Not checked yetcites [36] JS Kotiaho, 2002, Meta-analysis, can it ever fail?, Oikos source
- Not checked yetcites [37] AR Palmer, 2000, Quasireplication and the contract of error: Lessons from sex ratios, heritabilities and fluctuating asymmetry, Annu Rev Ecol Sys source
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For instance, we found a statistically significant indication of p-hacking in only one of the 12 questions examined in published meta-analyses ( Fig. 4 ).”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We then examined each dataset separately and found statistically significant evidential value for 9 of the 12 p-curves ( Table 3 ).”
plainn = 12 ·- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“and the Abstracts (binomial glm: estimated proportion of p-values in the upper bin (0.045 < p < 0.05) (lower CI) = 0.537 (0.518), p < 0.001, n = 10 disciplines).”
hedgedn = 10 ·- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
5 more claims
Create a free account to see all 17 claims and to verify the 9 references still open.
- It may state two separate claims as one. 7
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“At the individual study level, internal validity may be weak due to uncontrolled confounding variables, improper use of statistical methods, and use of cherry-picked data and methods to achieve statistically significant results [17].”
From Causal language and strength of inference in academic and media articles shared in social media (CLAIMS): A systematic review (2018) · cites this paper as [17]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Moreover, the same analysis found a much stronger ' evidential value ' in the literature of all disciplines”
From Is science really facing a reproducibility crisis, and do we need it to? (2018) · cites this paper as [10]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“A related issue arises when the selection of the study population or statistical model is chosen based on the P value (‘P hacking’) [64].”
From COSMOS-E: Guidance on conducting systematic reviews and meta-analyses of observational studies of etiology (2019) · cites this paper as [64]
The cited source says what the claim attributes to it.
In the citing paper
“Publication bias remains an important issue in psychological and medical research (Head et al., 2015; Thornton & Lee, 2000).”
From Trauma-related dissociation and the autonomic nervous system: a systematic literature review of psychophysiological correlates of dissociative experiencing in PTSD patients (2022) · cites this paper as Head et al., 2015
The cited source says what the claim attributes to it.
In the citing paper
“Studies that claim to test a hypothesis but do not specify a predefined primary outcome measure or those that change the primary outcome measure after data were collected (also known as primary outcome switching) are liable to selectively report only statistically significant results, favouring more positive findings [94].”
From Reporting animal research: Explanation and elaboration for the ARRIVE guidelines 2.0 (2020) · cites this paper as [94]
4 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Statistical Methods for Research Workers
(1945) · American Journal of Clinical Pathology · doi:10.1093/ajcp/15.11.552b · Cited by 10,880
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
The file drawer problem and tolerance for null results.
Rosenthal, R. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.3.638 · Cited by 9,068 · cited 1×
“There are two widely recognized types of researcher-driven publication bias: selection (also known as the “ file drawer effect ” , where studies with nonsignificant results have lower publication rates [7]) and inflation [12].”
BackgroundSimmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
THE USE OF CONFIDENCE OR FIDUCIAL LIMITS ILLUSTRATED IN THE CASE OF THE BINOMIAL
CLOPPER, C. J., & PEARSON, E. S. (1934) · Biometrika · doi:10.1093/biomet/26.4.404 · Cited by 5,003
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · cited 4×
“Results from prespecified studies offer far more convincing evidence than those from exploratory research [2].”
BackgroundResultInfluentialEffect size, confidence interval and statistical significance: a practical guide for biologists
Nakagawa, S., & Cuthill, I. C. (2007) · Biological Reviews · doi:10.1111/j.1469-185x.2007.00027.x · Cited by 3,835 · cited 1×
“Similarly, a low p -value does not mean that a finding is of major clinical or biological interest [28].”
BackgroundThe New Statistics: Why and How
Cumming, G. (2014) · Psychological Science · doi:10.1177/0956797613504966 · Cited by 3,200
A practical solution to the pervasive problems ofp values
Wagenmakers, E.-J. (2007) · Psychonomic Bulletin & Review · doi:10.3758/bf03194105 · Cited by 2,490 · cited 1×
“• Encourage and/or provide platforms for method prespecification [13,65].”
BackgroundMeasuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling
John, L. K., Loewenstein, G., & Prelec, D. (2012) · Psychological Science · doi:10.1177/0956797611430953 · Cited by 2,239 · cited 5×
“John et al. [16] measured the prevalence of questionable research practices in psychology.”
BackgroundInfluential
Cited by
Reporting animal research: Explanation and elaboration for the ARRIVE guidelines 2.0
Percie du Sert, N., Ahluwalia, A., Alam, S., Avey, M. T., Baker, M., Browne, W. J., Clark, A., Cuthill, I. C., Dirnagl, U., Emerson, M., et al. (2020) · PLOS Biology · doi:10.1371/journal.pbio.3000411 · Cited by 2,677
“Analysing the data using different methods and selectively reporting those with statistically significant results constitutes p-hacking and introduces bias in the literature [90,94].”
MethodResultMethods for the detection of carelessly invalid responses in survey data
Curran, P. G. (2016) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2015.07.006 · Cited by 1,356
Experimental design and analysis and their reporting: new guidance for publication in BJP
Curtis, M. J., Bond, R. A., Spina, D., Ahluwalia, A., Alexander, S. P. A., Giembycz, M. A., Gilchrist, A., Hoyer, D., Insel, P. A., Izzo, A. A., et al. (2015) · British Journal of Pharmacology · doi:10.1111/bph.12856 · Cited by 1,033
“The first point to make is that one should not collect data then try out various statistical tests until one obtains the ‘desired’ result, known as ‘P hacking’ (Head et al ., 2015).”
BackgroundThe contact hypothesis re-evaluated
PALUCK, E. L., GREEN, S. A., & GREEN, D. P. (2019) · Behavioural Public Policy · doi:10.1017/bpp.2018.25 · Cited by 856
“The same may be said for our analysis of p-hacking (Simonsohn et al., 2014; Head et al., 2015), which is symptomatic of research discretion that favors significant relationships.”
O'Dea, R. E., Lagisz, M., Jennions, M. D., Koricheva, J., Noble, D. W. A., Parker, T. H., Gurevitch, J., Page, M. J., Stewart, G., Moher, D., et al. (2021) · Biological Reviews · doi:10.1111/brv.12721 · Cited by 697
“Forexample,givenmultiplechoices instatistical analyses, researchersmight favour paths leading to statistical significance (i.e. ‘P-hacking’; Simmons, Nelson & Simonsohn, 2011; Head et al., 2015).”
BackgroundSelf-citationInterpretable Machine Learning – A Brief History, State-of-the-Art and Challenges
Molnar, C., Casalicchio, G., & Bischl, B. (2020) · Communications in Computer and Information Science · doi:10.1007/978-3-030-65965-3_28 · Cited by 677
A solution to minimum sample size for regressions
Jenkins, D. G., & Quintana-Ascencio, P. F. (2020) · PLOS ONE · doi:10.1371/journal.pone.0229345 · Cited by 665
“05 criterion for statistical "significance" and its numerous work-arounds have been widely discussed, including post hoc hypothesis formation, data dredging, and p-hacking [30,31,32].”
BackgroundReporting animal research: Explanation and Elaboration for the ARRIVE guidelines 2019
Percie du Sert, N., Ahluwalia, A., Alam, S., Avey, M. T., Baker, M., Browne, W. J., Clark, A., Cuthill, I. C., Dirnagl, U., Emerson, M., et al. (2019) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/703355 · Cited by 559
“Studies that claim to test a hypothesis but do not specify a pre-defined primary outcome measure, or those that change the primary outcome measure after data were collected (also known as primary outcome switching) are liable to selectively report only statistically significant results, favouring more positive finding…”
ResultGoligher, E. C., Tomlinson, G., Hajage, D., Wijeysundera, D. N., Fan, E., Jüni, P., Brodie, D., Slutsky, A. S., & Combes, A. (2018) · JAMA · doi:10.1001/jama.2018.14276 · Cited by 534
The National COVID Cohort Collaborative (N3C): Rationale, design, infrastructure, and deployment
Haendel, M. A., Chute, C. G., Bennett, T. D., Eichmann, D. A., Guinney, J., Kibbe, W. A., Payne, P. R. O., Pfaff, E. R., Robinson, P. N., Saltz, J. H., et al. (2021) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaa196 · Cited by 553
Dekkers, O. M., Vandenbroucke, J. P., Cevallos, M., Renehan, A. G., Altman, D. G., & Egger, M. (2019) · PLOS Medicine · doi:10.1371/journal.pmed.1002742 · Cited by 525
Machine Learning for Materials Scientists: An Introductory Guide toward Best Practices
Wang, A. Y.-T., Murdock, R. J., Kauwe, S. K., Oliynyk, A. O., Gurlo, A., Brgoch, J., Persson, K. A., & Sparks, T. D. (2020) · Chemistry of Materials · doi:10.1021/acs.chemmater.0c01907 · Cited by 509
Presidential Address: The Scientific Outlook in Financial Economics
HARVEY, C. R. (2017) · The Journal of Finance · doi:10.1111/jofi.12530 · Cited by 461
How Computational Modeling Can Force Theory Building in Psychological Science
Guest, O., & Martin, A. E. (2021) · Perspectives on Psychological Science · doi:10.1177/1745691620970585 · Cited by 460
Is science really facing a reproducibility crisis, and do we need it to?
Fanelli, D. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1708272114 · Cited by 416
Misinformation in and about science
West, J. D., & Bergstrom, C. T. (2021) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1912444117 · Cited by 391
Evolution of Reporting P Values in the Biomedical Literature, 1990-2015
Chavalarias, D., Wallach, J. D., Li, A. H. T., & Ioannidis, J. P. A. (2016) · JAMA · doi:10.1001/jama.2016.1952 · Cited by 370
“197 921 60 317 (736-3700) Clinical trial 52 639 16 018 (30-2960) Meta-analysis 1 396 965 17 125 (176-2245) Review”
Zhang, N., Wang, J., Li, Y., & Jiang, B. (2021) · Scientific Reports · doi:10.1038/s41598-021-92971-w · Cited by 353
On the scientific superiority of conceptual replications for scientific progress
Crandall, C. S., & Sherman, J. W. (2016) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2015.10.002 · Cited by 344
Publication bias examined in meta-analyses from psychology and medicine: A meta-meta-analysis
van Aert, R. C. M., Wicherts, J. M., & van Assen, M. A. L. M. (2019) · PLOS ONE · doi:10.1371/journal.pone.0215052 · Cited by 334
Sensitivity Analysis for Publication Bias in Meta-Analyses
Mathur, M. B., & VanderWeele, T. J. (2020) · Journal of the Royal Statistical Society Series C: Applied Statistics · doi:10.1111/rssc.12440 · Cited by 354
“…that we modelled selection by using a single cut-off at p = 0:05 both for ease of interpretation and because it conforms well to empirical findings on how applied researchers and statisticians interpret and report p-values (Head et al., 2015; Masicampo and Lalande, 2012; McShane and Gal, 2017).”
ResultGorrochategui, E., Jaumot, J., Lacorte, S., & Tauler, R. (2016) · TrAC Trends in Analytical Chemistry · doi:10.1016/j.trac.2016.07.004 · Cited by 352
Over-optimization of academic publishing metrics: observing Goodhart's Law in action
Fire, M., & Guestrin, C. (2019) · GigaScience · doi:10.1093/gigascience/giz053 · Cited by 351
“…of the measures can be compromised, and unwelcome and unethical behaviors may develop, such as salami publications [17], ghost author-ships [18], p-hacking [19], metrics manipulation [20], faking research data [21], faking of peer reviews [22], and even plagiariz-ing by a peer reviewer [23].”
Simonsohn, U., Simmons, J. P., & Nelson, L. D. (2015) · Journal of Experimental Psychology General · doi:10.1037/xge0000104 · Cited by 343
What’s in a p? Reassessing best practices for conducting and reporting hypothesis-testing research
Meyer, K. E., van Witteloostuijn, A., & Beugelsdijk, S. (2017) · Journal of International Business Studies · doi:10.1057/s41267-017-0078-8 · Cited by 318
Outcome-Wide Longitudinal Designs for Causal Inference: A New Template for Empirical Studies
VanderWeele, T. J., Mathur, M. B., & Chen, Y. (2020) · Statistical Science · doi:10.1214/19-sts728 · Cited by 310
“05 is having numerous false positive published in the literature (Head et al. 2015).”
BackgroundInfluentialSchloss, P. D. (2018) · mBio · doi:10.1128/mbio.00525-18 · Cited by 280
“In reality, questions change and scientists can fall into the traps of the “Garden of Many Forking Paths,” where they go looking for a desired result (32), or “P-hacking,” where large numbers of statistical hypothesis tests are attempted without adequately correcting for performing multiple tests (33).”
BackgroundNo raw data, no science: another possible source of the reproducibility crisis
Miyakawa, T. (2020) · Molecular Brain · doi:10.1186/s13041-020-0552-2 · Cited by 268
The FAIR guiding principles for data stewardship: fair enough?
Boeckhout, M., Zielhuis, G. A., & Bredenoord, A. L. (2018) · European Journal of Human Genetics · doi:10.1038/s41431-018-0160-0 · Cited by 290
Misrepresentation and distortion of research in biomedical literature
Boutron, I., & Ravaud, P. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1710755115 · Cited by 273
“P-hacking, a detrimental practice, is defined as the misreporting of true effect sizes in published reports after researchers perform several statistical analyses and selectively choose to report or focus on those that produce statistically significant results (20).”
BackgroundOn-Line Analysis of Exhaled Breath: Focus Review
Bruderer, T., Gaisl, T., Gaugg, M. T., Nowak, N., Streckenbach, B., Müller, S., Moeller, A., Kohler, M., & Zenobi, R. (2019) · Chemical Reviews · doi:10.1021/acs.chemrev.9b00005 · Cited by 278
Publication bias and the canonization of false facts
Nissen, S. B., Magidson, T., Gross, K., & Bergstrom, C. T. (2016) · eLife · doi:10.7554/elife.21451 · Cited by 266
A guide to Bayesian model checking for ecologists
Conn, P. B., Johnson, D. S., Williams, P. J., Melin, S. R., & Hooten, M. B. (2018) · Ecological Monographs · doi:10.1002/ecm.1314 · Cited by 257
Moustafa, N., Koroniotis, N., Keshk, M., Zomaya, A. Y., & Tari, Z. (2023) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2023.3280465 · Cited by 266
“Otherwise, XAI has to face statistical testing problems such as p-hacking [211].”
BackgroundEvidence of experimental bias in the life sciences: why we need blind data recording
Holman, L., Head, M. L., Lanfear, R., & Jennions, M. D. (2015) · PLOS Biology · doi:10.1371/journal.pbio.1002190 · Cited by 263
Self-citationSample Size, Statistical Power, and False Conclusions in Infant Looking‐Time Research
Oakes, L. M. (2017) · Infancy · doi:10.1111/infa.12186 · Cited by 248
Improving Methodological Standards in Behavioral Interventions for Cognitive Enhancement
Shawn Green, C., Bavelier, D., Kramer, A. F., Vinogradov, S., Ansorge, U., Ball, K. K., Bingel, U., Chein, J. M., Colzato, L. S., Edwards, J. D., et al. (2019) · Journal of Cognitive Enhancement · doi:10.1007/s41465-018-0115-y · Cited by 235
“The other bias is p-hacking (Head et al. 2015).”
Coles, N. A., Larsen, J. T., & Lench, H. C. (2019) · Psychological Bulletin · doi:10.1037/bul0000194 · Cited by 228
Hagger, M. S., Luszczynska, A., de Wit, J., Benyamini, Y., Burkert, S., Chamberland, P.-E., Chater, A., Dombrowski, S. U., van Dongen, A., French, D. P., et al. (2016) · Psychology & Health · doi:10.1080/08870446.2016.1146719 · Cited by 212
The home math environment and math achievement: A meta-analysis.
Daucourt, M. C., Napoli, A. R., Quinn, J. M., Wood, S. G., & Hart, S. A. (2021) · Psychological Bulletin · doi:10.1037/bul0000330 · Cited by 218
Donaldson, S. I., Dormanesh, A., Perez, C., Majmundar, A., & Allem, J.-P. (2022) · JAMA Pediatrics · doi:10.1001/jamapediatrics.2022.2223 · Cited by 213
p-Curve and p-Hacking in Observational Research
Bruns, S. B., & Ioannidis, J. P. A. (2016) · PLOS ONE · doi:10.1371/journal.pone.0149144 · Cited by 210
“…[1], the distribution of statistically significant p -values, has been used to infer that most studies actually analyze true relationships in the medical sciences [2] and in a wide range of disciplines [3] irrespective of whether these studies use experimental or observational research designs.”
BackgroundInfluentialSeven lessons for interdisciplinary research on interactive digital health interventions
Blandford, A., Gibbs, J., Newhouse, N., Perski, O., Singh, A., & Murray, E. (2018) · DIGITAL HEALTH · doi:10.1177/2055207618770325 · Cited by 209
Towards reproducible computational drug discovery
Schaduangrat, N., Lampa, S., Simeon, S., Gleeson, M. P., Spjuth, O., & Nantasenamat, C. (2020) · Journal of Cheminformatics · doi:10.1186/s13321-020-0408-x · Cited by 208
Comparison of text preprocessing methods
Chai, C. P. (2023) · Natural Language Engineering · doi:10.1017/s1351324922000213 · Cited by 212
“Such strategy is essentially p-hacking and hurts reproducibility in research (Head et al. 2015).”
BackgroundTransparency in Ecology and Evolution: Real Problems, Real Solutions
Parker, T. H., Forstmeier, W., Koricheva, J., Fidler, F., Hadfield, J. D., Chee, Y. E., Kelly, C. D., Gurevitch, J., & Nakagawa, S. (2016) · Trends in Ecology & Evolution · doi:10.1016/j.tree.2016.07.002 · Cited by 201
“This practice has become known as ‘p-hacking’ (see glossary) 73 [4, 5].”
BackgroundMethodBest practices for analyzing large-scale health data from wearables and smartphone apps
Hicks, J. L., Althoff, T., Sosic, R., Kuhar, P., Bostjancic, B., King, A. C., Leskovec, J., & Delp, S. L. (2019) · npj Digital Medicine · doi:10.1038/s41746-019-0121-1 · Cited by 183
Problematic Machine Behavior: A Systematic Literature Review of Algorithm Audits
Bandy, J. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3449148 · Cited by 193
Communicating sentiment and outlook reverses inaction against collective risks
Wang, Z., Jusup, M., Guo, H., Shi, L., Geček, S., Anand, M., Perc, M., Bauch, C. T., Kurths, J., Boccaletti, S., et al. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1922345117 · Cited by 185
“Appendix to avoid spurious results (31).”
Editorial: Evidence on Questionable Research Practices: The Good, the Bad, and the Ugly
Banks, G. C., Rogelberg, S. G., Woznyj, H. M., Landis, R. S., & Rupp, D. E. (2016) · Journal of Business and Psychology · doi:10.1007/s10869-016-9456-7 · Cited by 173
Dissimilarity measures in detrital geochronology
Vermeesch, P. (2018) · Earth-Science Reviews · doi:10.1016/j.earscirev.2017.11.027 · Cited by 179
“This point of view is gradually gaining acceptance in many fields of science (e.g., Gelman and Stern, 2006; Head et al., 2015; Stang et al., 2010).”
BackgroundLazari, A., & Lipp, I. (2021) · NeuroImage · doi:10.1016/j.neuroimage.2021.117744 · Cited by 197
“Mottershead et al. (2003) Relaxometry PD, T1, T2 Human fresh LFB -”
BackgroundHow computational modeling can force theory building in psychological sciencePreprint
Guest, O., & Martin, A. E. (2020) · doi:10.31234/osf.io/rybh9 · Cited by 175
Kretser, A., Murphy, D., Bertuzzi, S., Abraham, T., Allison, D. B., Boor, K. J., Dwyer, J., Grantham, A., Harris, L. J., Hollander, R., et al. (2019) · Science and Engineering Ethics · doi:10.1007/s11948-019-00094-3 · Cited by 165
Using Blockchain Technology to Manage Clinical Trials Data: A Proof-of-Concept Study
Maslove, D. M., Klein, J., Brohman, M. K., & Martin, P. (2018) · JMIR Medical Informatics · doi:10.2196/11949 · Cited by 167
“This need reflects a third trend, highlighting the potential negative consequences of multiple post hoc analyses (also referred to as “data dredging”) [22,23].”
BackgroundGeeleher, P., Zhang, Z., Wang, F., Gruener, R. F., Nath, A., Morrison, G., Bhutra, S., Grossman, R. L., & Huang, R. S. (2017) · Genome Research · doi:10.1101/gr.221077.117 · Cited by 138
“Here, we have explicitly avoided fitting multiple models to the data and selecting that which performed best, a practice that leads to irrepro-ducible results (Head et al. 2015).”
Navigating the garden of forking paths for data exclusions in fear conditioning research
Lonsdorf, T. B., Klingelhöfer-Jens, M., Andreatta, M., Beckers, T., Chalkia, A., Gerlicher, A., Jentsch, V. L., Meir Drexler, S., Mertens, G., Richter, J., et al. (2019) · eLife · doi:10.7554/elife.52465 · Cited by 164
“favorable results (e.g., ‘p-hacking’; Head et al., 2015): the decision to follow a certain path may be based on unawareness of alternative paths (due to lack of specific background knowledge) or the researcher following the most obvious path from an individual perspective.”
BackgroundInfluentialSex bias in preclinical research and an exploration of how to change the status quo
Karp, N. A., & Reavey, N. (2019) · British Journal of Pharmacology · doi:10.1111/bph.14539 · Cited by 159
“This will also address the issue of ‘p-hacking’ or ‘data-dredging’ which is the process of amending your statistical plan until statistical significance is obtained which is recognized to contribute to the replication crisis (Head et al., 2015).”
BackgroundZhang, N., Wang, J., Chen, B., Li, Y., & Jiang, B. (2021) · Frontiers in Medicine · doi:10.3389/fmed.2020.624179 · Cited by 158
“Publication bias was calculated using the Funnel plots (17, 18), P-curve analysis (19), and Egger test (17) (p < 0.”
MethodReplicability and Reproducibility in Comparative Psychology
Stevens, J. R. (2017) · Frontiers in Psychology · doi:10.3389/fpsyg.2017.00862 · Cited by 190
“This “phacking” (Simmons et al., 2011; Simonsohn et al., 2014) generates an over-reporting of results with p-values just under 0.05 (Masicampo and Lalande, 2012) and is a particular problem for psychology (Head et al., 2015).”
BackgroundInfluentialBad Beliefs: Why They Happen to Good People
Levy, N. L. (2021) · doi:10.1093/oso/9780192895325.001.0001 · Cited by 295
von Kortzfleisch, V. T., Karp, N. A., Palme, R., Kaiser, S., Sachser, N., & Richter, S. H. (2020) · Scientific Reports · doi:10.1038/s41598-020-73503-4 · Cited by 154
More than 3Rs: The importance of scientific validity for harm-benefit analysis of animal research
Würbel, H. (2017) · Lab Animal · doi:10.1038/laban.1220 · Cited by 152
Sibling Constructs: What Are They, Why Do They Matter, and How Should You Handle Them?
Lawson, K. M., & Robins, R. W. (2021) · Personality and Social Psychology Review · doi:10.1177/10888683211047101 · Cited by 144
“Running multiple analyses that handle sibling constructs differently and then reporting only the significant results constitutes p-hacking (Head et al., 2015), and should obviously be avoided.”
BackgroundHeeke, C., Kampisiou, C., Niemeyer, H., & Knaevelsrud, C. (2017) · European Journal of Psychotraumatology · doi:10.1080/20008198.2019.1583524 · Cited by 175
Technology acceptance research: Meta-analysis
Marikyan, D., Papagiannidis, S., & Stewart, G. (2026) · Journal of Information Science · doi:10.1177/01655515231191177 · Cited by 140
“To confirm that studies publish true effects as expected, first, the skewness of the solid line should be similar to the dashed line, and, second, 80% of pooled effects or more should be at a p -value of 0.01 or 0.02 [168].”
InfluentialMichel, M. C., Murphy, T. J., & Motulsky, H. J. (2020) · Molecular Pharmacology · doi:10.1124/mol.119.118927 · Cited by 127
Selecting the “Best” Factor Structure and Moving Measurement Validation Forward: An Illustration
Schmitt, T. A., Sass, D. A., Chappelle, W. L., & Thompson, W. T. (2018) · Journal of Personality Assessment · doi:10.1080/00223891.2018.1449116 · Cited by 163
Climate change and infectious disease: a review of evidence and research trends
de Vuurst, P. V., & Escobar, L. E. (2023) · Infectious Diseases of Poverty · doi:10.1186/s40249-023-01102-2 · Cited by 164
Cohen, M. X. (2021) · NeuroImage · doi:10.1016/j.neuroimage.2021.118809 · Cited by 128
Spectacle: An interactive resource for ocular single-cell RNA sequencing data analysis
Voigt, A. P., Whitmore, S. S., Lessing, N. D., DeLuca, A. P., Tucker, B. A., Stone, E. M., Mullins, R. F., & Scheetz, T. E. (2020) · Experimental Eye Research · doi:10.1016/j.exer.2020.108204 · Cited by 114
“Lastly, highly interactive differential expression is powerful for identifying enriched genes across different cell types and biological conditions; however, such interactivity may permit p-hacking in data analysis (Head et al., 2015).”
BackgroundNakagawa, S., & Parker, T. H. (2015) · BMC Biology · doi:10.1186/s12915-015-0196-3 · Cited by 110
“of freedom’ [20] seems pervasive across many scientific fields [21].”
BackgroundBig little lies: a compendium and simulation of p -hacking strategies
Stefan, A. M., & Schönbrodt, F. D. (2023) · Royal Society Open Science · doi:10.1098/rsos.220346 · Cited by 206
P-hacking and publication bias interact to distort meta-analytic effect size estimates
Friese, M., & Frankenbach, J. (2020) · Psychological Methods · doi:10.1037/met0000246 · Cited by 122
Bombaywala, S., Mandpe, A., Paliya, S., & Kumar, S. (2021) · Environmental Science and Pollution Research · doi:10.1007/s11356-021-13143-x · Cited by 128
Caldwell, A. R., Vigotsky, A. D., Nuckols, G. L., Boardley, I. D., Schmidt, J., Tenan, M. S., Škarabot, J., Radel, R., Naughton, M. R., Schoenfeld, B. J., et al. (2020) · Sports Medicine · doi:10.1007/s40279-019-01227-1 · Cited by 127
Chester, D. S., & Lasko, E. N. (2020) · Perspectives on Psychological Science · doi:10.1177/1745691620950684 · Cited by 158
Dissanayaka, T., Zoghi, M., Farrell, M. J., Egan, G. F., & Jaberzadeh, S. (2017) · European Journal of Neuroscience · doi:10.1111/ejn.13640 · Cited by 127
Designing a rigorous microscopy experiment: Validating methods and avoiding bias
Jost, A. P.-T., & Waters, J. C. (2019) · The Journal of Cell Biology · doi:10.1083/jcb.201812109 · Cited by 116
An overview and update on the epidemiology of flavonoid intake and cardiovascular disease risk
Parmenter, B. H., Croft, K. D., Hodgson, J. M., Dalgaard, F., Bondonno, C. P., Lewis, J. R., Cassidy, A., Scalbert, A., & Bondonno, N. P. (2020) · Food & Function · doi:10.1039/d0fo01118e · Cited by 126
Chicco, D., Sichenze, A., & Jurman, G. (2025) · BioData Mining · doi:10.1186/s13040-025-00465-6 · Cited by 277
Leckey, C., van Dyk, N., Doherty, C., Lawlor, A., & Delahunt, E. (2024) · British Journal of Sports Medicine · doi:10.1136/bjsports-2024-108576 · Cited by 198
“Furthermore, by declaring only one ML method, these studies are susceptible to ‘p-hacking’ whereby researchers conduct numerous tests with multiple combinations of parameters until a statistically significant result is found, reporting it in isolation.”
Cooke, S. J., Birnie‐Gauvin, K., Lennox, R. J., Taylor, J. J., Rytwinski, T., Rummer, J. L., Franklin, C. E., Bennett, J., & Haddaway, N. (2017) · Conservation Physiology · doi:10.1093/conphys/cox043 · Cited by 115
“Otherwise, bias can be introduced by serial implementation of possible statistical methods until one is identi fi ed that yields desired results (p-hacking: Simonsohn et al. , 2014; Head et al. , 2015).”
MethodP-Values: Misunderstood and Misused
Vidgen, B., & Yasseri, T. (2016) · Frontiers in Physics · doi:10.3389/fphy.2016.00006 · Cited by 143
“Several commentators have recommended that researchers pre-register all studies with initiatives such as the Open Science Framework [18, 1, 7, 14, 25].”
BackgroundMcGreivy, N., & Hakim, A. H. (2024) · Nature Machine Intelligence · doi:10.1038/s42256-024-00897-5 · Cited by 269
“Types of reporting biases include publication bias [134], spin bias [135], and different flavors of outcome reporting bias [136] such as p-hacking [137], selective reporting [138], outcome switching [139], and data-dredging [140].”
BackgroundBooksmythe, I., Mautz, B. S., Davis, J., Nakagawa, S., & Jennions, M. D. (2015) · Biological reviews/Biological reviews of the Cambridge Philosophical Society · doi:10.1111/brv.12220 · Cited by 112
Self-citationMethodology over metrics: current scientific standards are a disservice to patients and society
Van Calster, B., Wynants, L., Riley, R. D., van Smeden, M., & Collins, G. S. (2021) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2021.05.018 · Cited by 105
Fluctuating Asymmetry of Human Populations: A Review
Graham, J. H., & Özener, B. (2016) · Symmetry · doi:10.3390/sym8120154 · Cited by 115
“Studies of fluctuating asymmetry are especially vulnerable to these abuses, which include p-hacking ([295], but see [296]), overuse of one-tailed tests [297,298], confirmation bias [299], lack of effect-size estimates and confidence intervals [295], poor experimental design, small sample sizes, low statistical power,…”
BackgroundMetaresearch for Evaluating Reproducibility in Ecology and Evolution
Fidler, F. M., Chee, Y. E., Wintle, B. C., Burgman, M. A., Mccarthy, M. A., & Gordon, A. (2016) · BioScience · doi:10.1093/biosci/biw159 · Cited by 123
Machine learning and modeling: Data, validation, communication challenges
Naqa, I. E., Ruan, D., Valdés, G., Dekker, A. L. A. J., McNutt, T., Ge, Y., Wu, Q., Oh, J. H., Thor, M., Smith, W. P., et al. (2018) · Medical Physics · doi:10.1002/mp.12811 · Cited by 128
“overfitting the data, via multiple testing adjustments, false discovery analysis, or other methods to avoid data dredging or p-hacking problems.(46) Similar considerations should inform which metrics are used to evaluate performance, as they will be used to judge suitability of the model.”
Haber, N., Smith, E. R., Moscoe, E., Andrews, K., Audy, R., Bell, W., Brennan, A. T., Breskin, A., Kane, J. C., Karra, M., et al. (2018) · PLoS ONE · doi:10.1371/journal.pone.0196346 · Cited by 97
Han, H., & Park, J. (2018) · Frontiers in Neuroinformatics · doi:10.3389/fninf.2018.00001 · Cited by 103
Ellis, B. J., Horn, A. J., Carter, C. S., van IJzendoorn, M. H., & Bakermans-Kranenburg, M. J. (2021) · Clinical Psychology Review · doi:10.1016/j.cpr.2021.101985 · Cited by 101
Advancing AI in healthcare: A comprehensive review of best practices
Polevikov, S. (2023) · Clinica Chimica Acta · doi:10.1016/j.cca.2023.117519 · Cited by 140
York, L. M. (2018) · Journal of Experimental Botany · doi:10.1093/jxb/ery379 · Cited by 108
Pollet, T. V., & van der Meij, L. (2016) · Adaptive Human Behavior and Physiology · doi:10.1007/s40750-016-0050-z · Cited by 122
“…prominently in the literature (e.g., Bakker et al. 2012; Button et al. 2013; Ioannidis et al. 2014) and that many researchers, reviewers, and editors decide that findings are important if p ≤ .05 (e.g., in psychophysics: Hoekstra et al. 2006; also see: Head et al. 2015; Leggett et al. 2013).”
BackgroundIsbell, D. R., BROWN, D., Chen, M., Derrick, D. J., Ghanem, R., Arvizu, M. N. G., Schnur, E., Zhang, M., & Plonsky, L. (2022) · Modern Language Journal · doi:10.1111/modl.12760 · Cited by 113
Michel, M. C., Murphy, T. J., & Motulsky, H. J. (2019) · Journal of Pharmacology and Experimental Therapeutics · doi:10.1124/jpet.119.264143 · Cited by 97
Common scientific and statistical errors in obesity research
George, B. J., Beasley, T. M., Brown, A. W., Dawson, J., Dimova, R. B., Divers, J., Goldsby, T. U., Heo, M., Kaiser, K. A., Keith, S. W., et al. (2016) · Obesity · doi:10.1002/oby.21449 · Cited by 104
“More commonly, the preference is to reanalyze data until a “statistically significant” result is discovered under the assumption that validly conducted NHST with nonsignificant results will not lead to publication (52,53).”
BackgroundBradley, S. H., DeVito, N., Lloyd, K. E., Richards, G. C., Rombey, T., Wayant, C., & Gill, P. J. (2020) · Journal of the Royal Society of Medicine · doi:10.1177/0141076820956799 · Cited by 112
“been dubbed p-hacking.(44) P-hacking in turn facilitates so called HARKing (Hypothesising After the Result is Known), whereby researchers can retrospectively generate a matching hypothesis to a significant p value.”
Background
Metadata
- Authors
- Head, M. L., Holman, L., Lanfear, R., Kahn, A. T., & Jennions, M. D.
- Affiliations
- Australian National University (AU)
- Corresponding author
- Head, Megan L.
- Venue
- PLOS Biology
- Publisher
- Public Library of Science (PLoS)
- Published
- 2015-03-13
- Volume
- 13
- Issue
- 3
- Pages
- e1002106
- ISSN
- 1544-9173 (ISSN-L)
- 1545-7885
- PMC
- PMC4359000
- OpenAlex
- W2164005910
- Semantic Scholar
- a07a64ba110e0f9f7156f3bd1e376f0d2e1cddf1
- Cited by
- 1,388
- References
- 71
- Citations by year
- 2026: 66 · 2025: 124 · 2024: 104 · 2023: 157 · 2022: 151 · 2021: 167 · 2020: 178 · 2019: 141 · 2018: 121 · 2017: 91 · 2016: 64 · 2015: 24
- FWCI
- 91.09
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- PLoS Biology (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Minerva Access (University of Melbourne) (submittedVersion, cc-by)
- Edinburgh Napier Research Repository (Edinburgh Napier University) (submittedVersion, cc-by)
- ANU Open Research (Australian National University) (submittedVersion, cc-by)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Education › Academic integrity and plagiarism
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Keywords
- p-hacking, research bias, statistical significance, meta-analysis, text mining, publication bias, effect size, scientific consensus
- MeSH terms
- Humans, Science, Statistics as Topic, Meta-Analysis as Topic, Publication Bias
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (qwen3.8-omni-flash).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.