The Extent and Consequences of P-Hacking in Science
Open accessThe DOI matches its Crossref record; 70 of the 71 references it cites were found in a registry or an index; 2 of the 13 sources its claims cite were checked for support.
Abstract
A focus on novel, confirmatory, and statistically significant results leads to substantial bias in the scientific literature. One type of bias, known as "p-hacking," occurs when researchers collect or select data or statistical analyses until nonsignificant results become significant. Here, we use text-mining to demonstrate that p-hacking is widespread throughout science. We then illustrate how one can test for p-hacking when performing a meta-analysis and show that, while p-hacking is probably common, its effect seems to be weak relative to the real effect sizes being measured. This result suggests that p-hacking probably does not drastically alter scientific consensuses drawn from meta-analyses.
Overview
CoChat Summary
## TL;DR
This paper argues that p-hacking is widespread in the scientific literature and that it inflates published effect sizes. Using p-curves from text-mined p-values and from published meta-analyses, the authors find strong evidential value but also signs of p-hacking in both broad literature samples and focused meta-analytic datasets.
## Key findings
- In the text-mined dataset, pooled p-curves showed strong evidential value in both Results and Abstracts: estimated proportion of p-values in the upper bin was 0.257 for Results and 0.262 for Abstracts, both with p < 0.001.
- The same text-mined data also showed evidence of p-hacking: estimated proportion of p-values in the upper bin was 0.546 for Results and 0.537 for Abstracts, both with p < 0.001.
- Across the meta-analysis datasets, there was evidence for evidential value: the pooled estimated proportion in the upper bin was 0.202 with p < 0.001 across 12 datasets; 9 of 12 individual p-curves were also significant for evidential value.
- For meta-analyses, evidence of p-hacking depended on whether misreported p-values were included: including them gave a pooled upper-bin proportion of 0.615 with p = 0.033, while excluding them gave 0.489 with p = 0.443; the paper states that significant p-hacking was seen in only one of the 12 questions when considering the individual datasets.
## Methods
- The authors used text-mining to extract reported p-values from all Open Access papers in PubMed, then constructed p-curves separately for p-values from the Results and Abstracts across disciplines.
- They also collected p-values from studies used in published meta-analyses in evolutionary biology/sexual selection, then built p-curves for those datasets.
- Analysis used binomial tests / binomial generalized linear models on counts of p-values in bins near the significance threshold, comparing 0 ≤ p < 0.025 vs 0.025 ≤ p < 0.05 for evidential value and 0.04 < p < 0.045 vs 0.045 < p < 0.05 for p-hacking.
## Limitations
- The paper states that text-mining combines many different types of questions across disciplines, so it remains unclear whether some subfields or questions have a flat p-curve and therefore zero average effect size.
- The paper notes that Abstracts are optional for p-value reporting, so they may disproportionately contain only the strongest results; this can bias analyses and make p-hacking harder to detect.
- Evident from the text: the meta-analysis sample is not a representative sample of all research questions, since it only includes questions important enough to warrant meta-analysis; the authors explicitly say this when discussing generalizability.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures


![Fig 3 Evidence for p-hacking across scientific disciplines. A) Evidence for p-hacking from p-values obtained from Results sections. B) Evidence for p-hacking from p-values obtained from Abstracts. The strength of p-hacking is presented as the proportion of p-values in the upper bin (0.045 < p < 0.05) with one-tailed 95% confidence intervals (calculated following Clopper and Pearson [47] using the binom.test function in R). Only disciplines where text-mining of the Results sections returned more than 25 p-values between 0.04 and 0.05 are presented. Marker colour is shaded according to the sample size: with white indicating low samples sizes and red indicating larger sample sizes.](https://figures.cochat.ai/paper-figures/c0961868545043479c9439de894f8966/the-extent-and-consequences-of-p-hacking-in-science-figure-3.jpg)

Verification
Claims the paper makes
2 Valid · 0 Not valid · 2 Not checked · 13 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“there is good evidence that journals, especially prestigious ones with higher impact factors, disproportionately publish statistically significant results [ 4 – 10 ].”
What the sources say
- Not checked yetcites [4] CB Begg, 1988, Publication bias—a problem in interpreting medical data, J R Stat Soc Ser A Stat Soc source
- Partly supportedcites [5] K Dwan, 2008, Systematic review of the empirical evidence of study publication bias and outcome reporting bias, PLoS ONE sourceExperimental — accuracy not yet measured
Dwan et al. (2008) strongly support the general half of the claim: the Abstract Conclusions state 'There is strong evidence of an association between significant results and publication; studies that report positive or significant results are more likely to be published', and Results > Publication and trial findings reports five of the empirical studies finding significant studies more likely to be published. However, the 'especially prestigious ones with higher impact factors' element appears in our copy only as one included cohort's summary in Table 1 (Easterbrook 1991: significant studies 'more likely... to be published in journals with a high citation impact factor'), which the review's own synthesis does not carry forward; the review instead notes in Results that 'studies remain unpublished due largely to failure to submit rather than rejection by journals' and in Discussion that 'None of the inception cohorts addressed the question as to whether the significance determined whether a submitted paper was accepted or not'. So a journal-selection framing ('journals disproportionately publish') overstates what this source establishes. (LLM, conf=0.78)
- Not checked yetcites [6] D Fanelli, 2012, Negative results are disappearing from most disciplines and countries, Scientometrics source
- Not checked yetcites [7] R Rosenthal, 1979, The file drawer problem and tolerance for null results, Psychol Bull source
- Not matchedcites [8] F Song, 2000, Publication and related biases, Health technology assessment (Winchester, England)
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [9] TD Sterling, 1959, Publication decisions and their possible effects on inferences drawn from tests of significance—or vice versa, J Am Stat Assoc
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not checked yetcites [10] JM Stern, 1997, Publication bias: Evidence of delayed publication in a cohort study of clinical research projects, Br Med J source
boostedstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“However, meta-analyses are compromised if the studies being synthesized do not reflect the true distribution of effect sizes [ 5 , 35 – 37 ].”
What the sources say
- Supportedcites [5] K Dwan, 2008, Systematic review of the empirical evidence of study publication bias and outcome reporting bias, PLoS ONE source
The Introduction states that 'Study publication bias will lead to overestimation of treatment effects; it has been recognised as a threat to the validity of meta-analysis and can make the readily available evidence unreliable for decision making,' and the Abstract Background repeats this. The Introduction also frames 'the bias from missing outcome data that may affect a meta-analysis' as operating at both study level (non-publication) and outcome level (selective non-reporting), and the Discussion advises that 'Meta-analyses of outcomes where several relevant trials have missing data should be seen with extra caution.' Together these support the paraphrase that a synthesis is compromised when the included studies are not a representative sample of the true effect distribution. (LLM, conf=0.85)
- Not checked yetcites [35] MD Jennions, 2004, Meta-analysis can "fail": Reply to Kotiaho and Tomkins, Oikos source
- Not checked yetcites [36] JS Kotiaho, 2002, Meta-analysis, can it ever fail?, Oikos source
- Not checked yetcites [37] AR Palmer, 2000, Quasireplication and the contract of error: Lessons from sex ratios, heritabilities and fluctuating asymmetry, Annu Rev Ecol Sys source
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For instance, we found a statistically significant indication of p-hacking in only one of the 12 questions examined in published meta-analyses ( Fig. 4 ).”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We then examined each dataset separately and found statistically significant evidential value for 9 of the 12 p-curves ( Table 3 ).”
plainn = 12 ·- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“and the Abstracts (binomial glm: estimated proportion of p-values in the upper bin (0.045 < p < 0.05) (lower CI) = 0.537 (0.518), p < 0.001, n = 10 disciplines).”
hedgedn = 10 ·- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
5 more claims
Create a free account to see all 17 claims and to verify the 9 references still open.
- It may state two separate claims as one. 7
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“At the individual study level, internal validity may be weak due to uncontrolled confounding variables, improper use of statistical methods, and use of cherry-picked data and methods to achieve statistically significant results [17].”
From Causal language and strength of inference in academic and media articles shared in social media (CLAIMS): A systematic review (2018) · cites this paper as [17]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Moreover, the same analysis found a much stronger ' evidential value ' in the literature of all disciplines”
From Is science really facing a reproducibility crisis, and do we need it to? (2018) · cites this paper as [10]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“A related issue arises when the selection of the study population or statistical model is chosen based on the P value (‘P hacking’) [64].”
From COSMOS-E: Guidance on conducting systematic reviews and meta-analyses of observational studies of etiology (2019) · cites this paper as [64]
The cited source says what the claim attributes to it.
In the citing paper
“Publication bias remains an important issue in psychological and medical research (Head et al., 2015; Thornton & Lee, 2000).”
From Trauma-related dissociation and the autonomic nervous system: a systematic literature review of psychophysiological correlates of dissociative experiencing in PTSD patients (2022) · cites this paper as Head et al., 2015
The cited source says what the claim attributes to it.
In the citing paper
“Studies that claim to test a hypothesis but do not specify a predefined primary outcome measure or those that change the primary outcome measure after data were collected (also known as primary outcome switching) are liable to selectively report only statistically significant results, favouring more positive findings [94].”
From Reporting animal research: Explanation and elaboration for the ARRIVE guidelines 2.0 (2020) · cites this paper as [94]
4 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Statistical Methods for Research Workers
(1945) · American Journal of Clinical Pathology · doi:10.1093/ajcp/15.11.552b · Cited by 10,880
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
The file drawer problem and tolerance for null results.
Rosenthal, R. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.3.638 · Cited by 9,068 · cited 1×
“There are two widely recognized types of researcher-driven publication bias: selection (also known as the “ file drawer effect ” , where studies with nonsignificant results have lower publication rates [7]) and inflation [12].”
BackgroundSimmons, J. P., Nelson, L. D., & Simonsohn, U. (2011) · Psychological Science · doi:10.1177/0956797611417632 · Cited by 6,567
THE USE OF CONFIDENCE OR FIDUCIAL LIMITS ILLUSTRATED IN THE CASE OF THE BINOMIAL
CLOPPER, C. J., & PEARSON, E. S. (1934) · Biometrika · doi:10.1093/biomet/26.4.404 · Cited by 5,003
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · cited 4×
“Results from prespecified studies offer far more convincing evidence than those from exploratory research [2].”
BackgroundResultInfluentialEffect size, confidence interval and statistical significance: a practical guide for biologists
Nakagawa, S., & Cuthill, I. C. (2007) · Biological Reviews · doi:10.1111/j.1469-185x.2007.00027.x · Cited by 3,835 · cited 1×
“Similarly, a low p -value does not mean that a finding is of major clinical or biological interest [28].”
BackgroundThe New Statistics: Why and How
Cumming, G. (2014) · Psychological Science · doi:10.1177/0956797613504966 · Cited by 3,200
A practical solution to the pervasive problems ofp values
Wagenmakers, E.-J. (2007) · Psychonomic Bulletin & Review · doi:10.3758/bf03194105 · Cited by 2,490 · cited 1×
“• Encourage and/or provide platforms for method prespecification [13,65].”
BackgroundMeasuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling
John, L. K., Loewenstein, G., & Prelec, D. (2012) · Psychological Science · doi:10.1177/0956797611430953 · Cited by 2,239 · cited 5×
“John et al. [16] measured the prevalence of questionable research practices in psychology.”
BackgroundInfluentialNull Hypothesis Testing: Problems, Prevalence, and an Alternative
Anderson, D. R., Burnham, K. P., & Thompson, W. L. (2000) · The Journal of Wildlife Management · doi:10.2307/3803199 · Cited by 1,930 · cited 1×
“, [29,30]).”
(2000) · Queen Mary Research Online (Queen Mary University of London) · Cited by 1,795
Systematic Review of the Empirical Evidence of Study Publication Bias and Outcome Reporting Bias
Dwan, K., Altman, D. G., Arnaiz, J. A., Bloom, J., Chan, A.-W., Cronin, E., Decullier, E., Easterbrook, P. J., Von Elm, E., Gamble, C., et al. (2008) · PLoS ONE · doi:10.1371/journal.pone.0003081 · Cited by 1,546 · cited 2×
“Although methods and results in publications do not always match their prespecified protocols [5,66], prespecification allows readers to assess the risk of p-hacking and adjust their confidence in the reported outcomes accordingly.”
BackgroundMethodScientific method: Statistical errors
Nuzzo, R. (2014) · Nature · doi:10.1038/506150a · Cited by 1,536
P-curve: A key to the file-drawer.
Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2014) · Journal of Experimental Psychology: General · doi:10.1037/a0033242 · Cited by 1,500 · cited 8×
“…Abstracts are more likely to contain p -values that relate to primary hypotheses, which are expected to be more strongly p-hacked than p -values from less interesting, ancillary tests [41], lower power and reporting bias may impede detection of p-hacking using p -values obtained from Abstracts.”
BackgroundMethodResultInfluentialSterne, J. A. C. (2001) · BMJ · doi:10.1136/bmj.322.7280.226 · Cited by 1,466 · cited 1×
“NHST is the most widely used data analysis method in most scientific disciplines [22,23].”
MethodNosek (2012) · doi:10.1177/1745691612459058 · Cited by 1,301
Negative results are disappearing from most disciplines and countries
Fanelli, D. (2012) · Scientometrics · doi:10.1007/s11192-011-0494-7 · Cited by 1,248
Toward Evidence-Based Medical Statistics. 1: The P Value Fallacy
Goodman, S. N. (1999) · Annals of Internal Medicine · doi:10.7326/0003-4819-130-12-199906150-00008 · Cited by 1,185 · cited 1×
“NHST is the most widely used data analysis method in most scientific disciplines [22,23].”
MethodNull hypothesis significance testing: a review of an old and continuing controversy
Nickerson, R. S. (2000) · Psychological Methods · doi:10.1037/1082-989x.5.2.241 · Cited by 1,090
Sterling, T. D. (1959) · Journal of the American Statistical Association · doi:10.1080/01621459.1959.10501497 · Cited by 964 · cited 1×
“, “positive”) results, and there is good evidence that journals, especially prestigious ones with higher impact factors, disproportionately publish statistically significant results [4–10].”
BackgroundPublication bias: evidence of delayed publication in a cohort study of clinical research projects
Stern, J. M., & Simes, R. J. (1997) · BMJ · doi:10.1136/bmj.315.7109.640 · Cited by 818 · cited 1×
“…that current scientific practices create strong incentives to publish statistically significant (i.e., “ positive ” ) results, and there is good evidence that journals, especially prestigious ones with higher impact factors, disproportionately publish statistically significant results [4 – 10].”
BackgroundPublication bias—a problem in interpreting medical data
Begg, C. B., & Berlin, J. A. (1988) · Journal of the Royal Statistical Society. Series A (Statistics in Society) · doi:10.2307/2982993 · Cited by 814 · cited 1×
“…that current scientific practices create strong incentives to publish statistically significant (i.e., “ positive ” ) results, and there is good evidence that journals, especially prestigious ones with higher impact factors, disproportionately publish statistically significant results [4 – 10].”
Backgroundp -Curve and Effect Size: Correcting for Publication Bias Using Only Significant Results
Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2014) · Perspectives on Psychological Science · doi:10.1177/1745691614553988 · Cited by 717
False Discovery Rate–Adjusted Multiple Confidence Intervals for Selected Parameters
Benjamini, Y., & Yekutieli, D. (2005) · Journal of the American Statistical Association · doi:10.1198/016214504000001907 · Cited by 671 · cited 1×
“However, others note that many of the problems with publication bias reoccur with other approaches, such as reporting effect sizes and their confidence intervals [31] or Bayesian credible intervals [32].”
BackgroundStatistical guidelines for contributors to medical journals.
Altman, D. G., Gore, S. M., Gardner, M. J., & Pocock, S. J. (1983) · BMJ · doi:10.1136/bmj.286.6376.1489 · Cited by 661 · cited 1×
“For instance, stating that it is necessary to report effect sizes whether small or large, to report all p -values to three decimal places [27,64], to report samples sizes, and, most importantly, to be explicit about the entire analysis process (not just the final tests used to generate reported p…”
BackgroundSterling, T. D. (1959) · Journal of the American Statistical Association · doi:10.2307/2282137 · Cited by 524
The effectiveness and cost-effectiveness of prophylactic removal of wisdom teeth.
Song, O'Meara, Wilson, Golder, & Kleijnen (2000) · Health Technology Assessment · doi:10.3310/hta4150 · Cited by 478
Publication and Related Biases
Song, F., Eastwood, A., Gilbody, S., Duley, L., & Sutton, A. (2001) · The Advanced Handbook of Methods in Evidence Based Healthcare · doi:10.4135/9781848608344.n21 · Cited by 455
Cumming, G. (2008) · Perspectives on Psychological Science · doi:10.1111/j.1745-6924.2008.00079.x · Cited by 451 · cited 1×
“On the other hand, when the true effect size is nonzero, the expected distribution of p-values is exponential with a right skew [39–42] (Black line, Fig.”
BackgroundJiang, Y., Bolnick, D. I., & Kirkpatrick, M. (2013) · The American Naturalist · doi:10.1086/670160 · Cited by 427 · cited 1×
“However, this study [56] also showed strong evidential value and p -values in the 0.045 – 0.05 bin were only a small proportion of published significant p -values.”
BackgroundInfluentialPublication and related biases.
Song, Eastwood, Gilbody, Duley, & Sutton (2000) · Health Technology Assessment · doi:10.3310/hta4100 · Cited by 427
Palmer, A. R. (2000) · Annual Review of Ecology and Systematics · doi:10.1146/annurev.ecolsys.31.1.441 · Cited by 385 · cited 1×
“However, meta-analyses are compromised if the studies being synthesized do not reflect the true distribution of effect sizes [5,35 – 37].”
BackgroundPsychology Will Be a Much Better Science When We Change the Way We Analyze Data
Loftus, G. R. (1996) · Current Directions in Psychological Science · doi:10.1111/1467-8721.ep11512376 · Cited by 361 · cited 1×
“Many researchers have advocated abolishing NHST (e.g., [29,30]).”
BackgroundThe Influence of Operational Sex Ratio on the Intensity of Competition for Mates
Weir, L. K., Grant, J. W. A., & Hutchings, J. A. (2011) · The American Naturalist · doi:10.1086/657918 · Cited by 300 · cited 1×
“To demonstrate this procedure, we obtained p -values from studies subject to meta-analyses by evolutionary bi-ologists studying sexual selection [53 – 61] (see S1 Text).”
MethodExtra-pair paternity in birds: review of the genetic benefits
Akçay, E., & Roughgarden, J. E. (2007) · ScholarlyCommons (University of Pennsylvania) · Cited by 270
A peculiar prevalence of p values just below .05
Masicampo, E. J., & Lalande, D. R. (2012) · Quarterly Journal of Experimental Psychology · doi:10.1080/17470218.2012.711335 · Cited by 266
Meta-analysis using effect size distributions of only statistically significant studies.
van Assen, M. A. L. M., van Aert, R. C. M., & Wicherts, J. M. (2015) · Psychological Methods · doi:10.1037/met0000025 · Cited by 265 · cited 1×
“, using only the significant p-values they find), a method which has been proposed to account for publication biases and to offer a conservative estimate of the true effect when there is p-hacking [62,63].”
MethodSexual selection and speciation: the comparative evidence revisited
Kraaijeveld, K., Kraaijeveld-Smit, F. J. L., & Maan, M. E. (2011) · Biological Reviews · doi:10.1111/j.1469-185x.2010.00150.x · Cited by 255
Gerber, A. S., & Malhotra, N. (2008) · Sociological Methods & Research · doi:10.1177/0049124108318973 · Cited by 246
Bias in Meta-Analysis Due to Outcome Variable Selection Within Studies
Hutton, J. L., & Williamson, P. R. (2000) · Journal of the Royal Statistical Society Series C: Applied Statistics · doi:10.1111/1467-9876.00197 · Cited by 194 · cited 1×
“…experiments to decide whether to continue collecting data [15,16]; recording many response variables and deciding which to report postanalysis [16,17], deciding whether to include or drop outliers postanalyses [16], excluding, combining, or splitting treatment groups postanalysis [2], including…”
BackgroundAn estimate of the science-wise false discovery rate and application to the top medical literature
Jager, L. R., & Leek, J. T. (2014) · Biostatistics · doi:10.1093/biostatistics/kxt007 · Cited by 179 · cited 3×
“We chose p 0.05 as the cutoff for our upper bin (following [3]), rather than p = 0.05 (see [46]) because we suspect that many authors do not regard p = 0.05 as significant.”
BackgroundMethodInfluentialKelly, C. D. (2006) · The Quarterly Review of Biology · doi:10.1086/506236 · Cited by 167
Pereira, T. V., & Ioannidis, J. P. A. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.12.012 · Cited by 160 · cited 2×
“However, meta-analyses are only as good as the data they use, and a recent study estimated that up to 37% of meta-analyses of clinical trials reporting a significant mean effect size represent false positives [34].”
BackgroundTrafimow, D. (2003) · Psychological Review · doi:10.1037/0033-295x.110.3.526 · Cited by 158 · cited 1×
“With NHST, one computes the probability (i.e., p) of finding an effect at least or more extreme than the observed finding if the null hypothesis is true [24,25].”
BackgroundDominance and plumage traits: meta-analysis and metaregression analysis
Santos, E. S. A., Scheck, D., & Nakagawa, S. (2011) · Animal Behaviour · doi:10.1016/j.anbehav.2011.03.022 · Cited by 132
META-ANALYSIS SUGGESTS CHOOSY FEMALES GET SEXY SONS MORE THAN “GOOD GENES”
Prokop, Z. M., Michalczyk, Ł., Drobniak, S. M., Herdegen‐Radwan, M., & Radwan, J. (2012) · Evolution · doi:10.1111/j.1558-5646.2012.01654.x · Cited by 129 · cited 1×
“To demonstrate this procedure, we obtained p-values from studies subject to meta-analyses by evolutionary biologists studying sexual selection [53–61] (see S1 Text).”
MethodThe influence of male age on within‐pair and extra‐pair paternity in passerines
Cleasby, I. R., & Nakagawa, S. (2012) · Ibis · doi:10.1111/j.1474-919x.2011.01209.x · Cited by 124 · cited 1×
“To demonstrate this procedure, we obtained p-values from studies subject to meta-analyses by evolutionary biologists studying sexual selection [53–61] (see S1 Text).”
Method1. Place of Meta-analysis among Other Methods of Research Synthesis
(2013) · Handbook of Meta-analysis in Ecology and Evolution · doi:10.1515/9781400846184-003 · Cited by 121
Wishful Thinking: Belief, Desire, and the Motivated Evaluation of Scientific Evidence
Bastardi, A., Uhlmann, E. L., & Ross, L. D. (2011) · Psychological Science · doi:10.1177/0956797611406447 · Cited by 119
Compounding Probabilities from Independent Significance Tests
Wallis, W. A. (1942) · Econometrica · doi:10.2307/1905466 · Cited by 113 · cited 1×
“On the other hand, when the true effect size is nonzero, the expected distribution of p -values is exponential with a right skew [39 – 42] (Black line, Fig.”
BackgroundGildersleeve, K. A., Haselton, M. G., & Fales, M. R. (2014) · Psychological Bulletin · doi:10.1037/a0037714 · Cited by 108
Predicting publication success for biologists
Laurance, W. F., Useche, D. C., Laurance, S. G., & Bradshaw, C. J. A. (2013) · BioScience · doi:10.1525/bio.2013.63.10.9 · Cited by 100
Kelly, C. D. (2007) · Behavioral Ecology and Sociobiology · doi:10.1007/s00265-007-0518-8 · Cited by 99
Hahn, S., Williamson, P. R., & Hutton, J. L. (2002) · Journal of Evaluation in Clinical Practice · doi:10.1046/j.1365-2753.2002.00314.x · Cited by 97 · cited 1×
“Although methods and results in publications do not always match their prespecified protocols [5,66], prespecification allows readers to assess the risk of p-hacking and adjust their confidence in the reported outcomes accordingly.”
MethodData from Investigating Variation in Replicability: A “Many Labs” Replication Project
(2014) · Journal of Open Psychology Data · doi:10.5334/jopd.ad · Cited by 86
Barch, D. M., & Yarkoni, T. (2013) · Cognitive, Affective, & Behavioral Neuroscience · doi:10.3758/s13415-013-0201-7 · Cited by 67 · cited 1×
“There is increasing concern that many published results are false positives [1,2] (but see [3]).”
BackgroundIoannidis, J. P. A. (2013) · Biostatistics · doi:10.1093/biostatistics/kxt036 · Cited by 66
The Life of p : “Just Significant” Results are on the Rise
Leggett, N. C., Thomas, N. A., Loetscher, T., & Nicholls, M. E. R. (2013) · Quarterly Journal of Experimental Psychology · doi:10.1080/17470218.2013.863371 · Cited by 64 · cited 3×
“1B) is interpreted as evidence for publication bias (e.g., [40,43 – 45]).”
BackgroundInfluentialMeasuring mating competition correctly: available evidence supports operational sex ratio theory
de Jong, K., Forsgren, E., Sandvik, H., et al. (2012) · Behavioral Ecology · doi:10.1093/beheco/ars094 · Cited by 64
Gadbury, G. L., & Allison, D. B. (2012) · PLoS ONE · doi:10.1371/journal.pone.0046363 · Cited by 61
Benjamini, Y., & Hechtlinger, Y. (2013) · Biostatistics · doi:10.1093/biostatistics/kxt032 · Cited by 57
Meta-analysis, can it ever fail?
Kotiaho, J. S., & Tomkins, J. L. (2002) · Oikos · doi:10.1034/j.1600-0706.2002.960316.x · Cited by 52 · cited 1×
“However, meta-analyses are compromised if the studies being synthesized do not reflect the true distribution of effect sizes [5,35–37].”
BackgroundPosterior-Hacking: Selective Reporting Invalidates Bayesian Results AlsoPreprint
Simonsohn, U. (2014) · SSRN Electronic Journal · doi:10.2139/ssrn.2374040 · Cited by 36
RIDLEY, J., KOLM, N., FRECKELTON, R. P., & GAGE, M. J. G. (2007) · Journal of Evolutionary Biology · doi:10.1111/j.1420-9101.2006.01291.x · Cited by 34
Star Wars: The Empirics Strike BackPreprint
Brodeur, A., Lé, M., Sangnier, M., & Zylberberg, Y. (2012) · SSRN Electronic Journal · doi:10.2139/ssrn.2089580 · Cited by 29
P-Curve and Effect Size: Correcting for Publication Bias Using Only Significant ResultsPreprint
Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2014) · SSRN Electronic Journal · doi:10.2139/ssrn.2377290 · Cited by 22
Meta‐analysis can “fail”: reply to Kotiaho and Tomkins
Jennions, M. D., Møller, A. P., & Hunt, J. (2004) · Oikos · doi:10.1111/j.0030-1299.2004.12265.x · Cited by 16 · cited 1×
“However, meta-analyses are compromised if the studies being synthesized do not reflect the true distribution of effect sizes [5,35 – 37].”
BackgroundSelf-citationHo, D. E. (2013) · Journal of Empirical Legal Studies · doi:10.1111/jels.12031 · Cited by 2 · cited 1×
“Ideally, methods should be assessed independently of results [13,44].”
BackgroundData from: The extent and consequences of p-hacking in science
Head, M. L., Holman, L., Lanfear, R., Kahn, A. T., & Jennions, M. D. (2015) · Data Archiving and Networked Services (DANS) · doi:10.5061/dryad.79d43 · Cited by 2
Self-citation
Cited by
Reporting animal research: Explanation and elaboration for the ARRIVE guidelines 2.0
Percie du Sert, N., Ahluwalia, A., Alam, S., Avey, M. T., Baker, M., Browne, W. J., Clark, A., Cuthill, I. C., Dirnagl, U., Emerson, M., et al. (2020) · PLOS Biology · doi:10.1371/journal.pbio.3000411 · Cited by 2,677
“Analysing the data using different methods and selectively reporting those with statistically significant results constitutes p-hacking and introduces bias in the literature [90,94].”
MethodResultMethods for the detection of carelessly invalid responses in survey data
Curran, P. G. (2016) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2015.07.006 · Cited by 1,356
Experimental design and analysis and their reporting: new guidance for publication in BJP
Curtis, M. J., Bond, R. A., Spina, D., Ahluwalia, A., Alexander, S. P. A., Giembycz, M. A., Gilchrist, A., Hoyer, D., Insel, P. A., Izzo, A. A., et al. (2015) · British Journal of Pharmacology · doi:10.1111/bph.12856 · Cited by 1,033
“The first point to make is that one should not collect data then try out various statistical tests until one obtains the ‘desired’ result, known as ‘P hacking’ (Head et al ., 2015).”
BackgroundThe contact hypothesis re-evaluated
PALUCK, E. L., GREEN, S. A., & GREEN, D. P. (2019) · Behavioural Public Policy · doi:10.1017/bpp.2018.25 · Cited by 856
“The same may be said for our analysis of p-hacking (Simonsohn et al., 2014; Head et al., 2015), which is symptomatic of research discretion that favors significant relationships.”
O'Dea, R. E., Lagisz, M., Jennions, M. D., Koricheva, J., Noble, D. W. A., Parker, T. H., Gurevitch, J., Page, M. J., Stewart, G., Moher, D., et al. (2021) · Biological Reviews · doi:10.1111/brv.12721 · Cited by 697
“Forexample,givenmultiplechoices instatistical analyses, researchersmight favour paths leading to statistical significance (i.e. ‘P-hacking’; Simmons, Nelson & Simonsohn, 2011; Head et al., 2015).”
BackgroundSelf-citation
Metadata
- Authors
- Head, M. L., Holman, L., Lanfear, R., Kahn, A. T., & Jennions, M. D.
- Affiliations
- Australian National University (AU)
- Corresponding author
- Head, Megan L.
- Venue
- PLOS Biology
- Publisher
- Public Library of Science (PLoS)
- Published
- 2015-03-13
- Volume
- 13
- Issue
- 3
- Pages
- e1002106
- ISSN
- 1544-9173 (ISSN-L)
- 1545-7885
- PMC
- PMC4359000
- OpenAlex
- W2164005910
- Semantic Scholar
- a07a64ba110e0f9f7156f3bd1e376f0d2e1cddf1
- Cited by
- 1,388
- References
- 71
- Citations by year
- 2026: 66 · 2025: 124 · 2024: 104 · 2023: 157 · 2022: 151 · 2021: 167 · 2020: 178 · 2019: 141 · 2018: 121 · 2017: 91 · 2016: 64 · 2015: 24
- FWCI
- 91.09
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- PLoS Biology (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Minerva Access (University of Melbourne) (submittedVersion, cc-by)
- Edinburgh Napier Research Repository (Edinburgh Napier University) (submittedVersion, cc-by)
- ANU Open Research (Australian National University) (submittedVersion, cc-by)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Education › Academic integrity and plagiarism
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Keywords
- p-hacking, research bias, statistical significance, meta-analysis, text mining, publication bias, effect size, scientific consensus
- MeSH terms
- Humans, Science, Statistics as Topic, Meta-Analysis as Topic, Publication Bias
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (qwen3.8-omni-flash).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.