Meta-assessment of bias in science
Open accessThe DOI matches its Crossref record; 53 of the 53 references it cites were found in a registry or an index.
Abstract
Significance: Science is said to be suffering a reproducibility crisis caused by many biases. How common are these problems, across the wide diversity of research fields? We probed for multiple bias-related patterns in a large random sample of meta-analyses taken from all disciplines. The magnitude of these biases varied widely across fields and was on average relatively small. However, we consistently observed that small, early, highly cited studies published in peer-reviewed journals were likely to overestimate effects. We found little evidence that these biases were related to scientific productivity, and we found no difference between biases in male and female researchers. However, a scientist’s early-career status, isolation, and lack of scientific integrity might be significant risk factors for producing unreliable results.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“less likely to publish papers that are retracted”
From Is science really facing a reproducibility crisis, and do we need it to? (2018) · cites this paper as [19]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“later, finer-grained analyses offered contrary evidence, by showing that researchers that publish at higher frequency, in journals with higher impact factor, and in countries where pressures to publish are high, are equally or more likely to correct their work”
From Is science really facing a reproducibility crisis, and do we need it to? (2018) · cites this paper as [19]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Indeed, in a meta-assessment of bias across all disciplines, small-study effects and gray-literature bias (both possible symptoms of reporting biases) were highly heterogeneously distributed (19).”
From Is science really facing a reproducibility crisis, and do we need it to? (2018) · cites this paper as [19]
The cited source says what the claim attributes to it.
In the citing paper
“and less likely to author papers reporting overestimated effects (19, 40, 41).”
From Is science really facing a reproducibility crisis, and do we need it to? (2018) · cites this paper as [19]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“less likely to author papers that contain duplicated images”
From Is science really facing a reproducibility crisis, and do we need it to? (2018) · cites this paper as [19]
Experimental — accuracy not yet measured
3 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Fitting Linear Mixed-Effects Models Using lme4
Bates, D., Mächler, M., Bolker, B., & Walker, S. (2015) · Journal of Statistical Software · doi:10.18637/jss.v067.i01 · Cited by 86,835
Publication and related bias in meta-analysis
Sterne, J. A. C., Gavaghan, D., & Egger, M. (2000) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(00)00242-0 · Cited by 2,282 · cited 2×
“Moreover, small-study effects may result not necessarily from QRP but also from legitimate choices made in designing small studies (17, 36).”
BackgroundPharmaceutical industry sponsorship and research outcome and quality: systematic review
Lexchin, J., Bero, L. A., Djulbegovic, B., & Clark, O. (2003) · BMJ · doi:10.1136/bmj.326.7400.1167 · Cited by 2,147 · cited 1×
“Industry bias: Industry sponsorship may affect the direction and magnitude of effects reported by biomedical studies (21).”
BackgroundSystematic Review of the Empirical Evidence of Study Publication Bias and Outcome Reporting Bias
Dwan, K., Altman, D. G., Arnaiz, J. A., Bloom, J., Chan, A.-W., Cronin, E., Decullier, E., Easterbrook, P. J., Von Elm, E., Gamble, C., et al. (2008) · PLoS ONE · doi:10.1371/journal.pone.0003081 · Cited by 1,546 · cited 1×
“J Empir Res Hum Res Ethics 2(4):15–34.”
Self-citationA comparison of methods to detect publication bias in meta‐analysis
Macaskill, P., Walter, S. D., & Irwig, L. (2001) · Statistics in Medicine · doi:10.1002/sim.698 · Cited by 1,286 · cited 1×
“Nevertheless, meta-regression estimates of small-study effects on odds ratio may run a modest risk of type I error (35) and were measured by our study more completely and accurately than the other biases (SI Multilevel Meta-Regression Analysis, Reliability and Consistency of Collected Data).”
MethodNegative results are disappearing from most disciplines and countries
Fanelli, D. (2012) · Scientometrics · doi:10.1007/s11192-011-0494-7 · Cited by 1,248 · cited 3×
“Systematic differences in the risk of bias between physical, biological, and social sciences were observed, particularly for the most prominent biases, as was expected based on previous evidence (5, 11, 33).”
MethodResultSelf-citationDissemination and publication of research findings: an updated review of related biases
Song, F., Parekh, Hooper, L., Loke, Y., Ryder, J., Sutton, A., Hing, C., Kwok, C., Pang, C., & Harvey, I. (2010) · Health Technology Assessment · doi:10.3310/hta14080 · Cited by 1,039 · cited 5×
“Gray literature bias: Studies might be less likely to be published if they yielded smaller and/or statistically nonsignificant effects and might be therefore only available in PhD theses, conference proceedings, books, personal communications, and other forms of “gray” literature (1).”
BackgroundResultInfluentialExtension of Nakagawa & Schielzeth's R2GLMM to random slopes models
Johnson, P. C. D. (2014) · Methods in Ecology and Evolution · doi:10.1111/2041-210x.12225 · Cited by 948
Do Pressures to Publish Increase Scientists' Bias? An Empirical Support from US States Data
Fanelli, D. (2010) · PLoS ONE · doi:10.1371/journal.pone.0010271 · Cited by 805
Self-citation“Positive” results increase down the Hierarchy of the Sciences
Fanelli, D. (2010) · PLoS ONE · doi:10.1371/journal.pone.0010068 · Cited by 734 · cited 4×
“Systematic differences in the risk of bias between physical, biological, and social sciences were observed, particularly for the most prominent biases, as was expected based on previous evidence (5, 11, 33).”
BackgroundMethodInfluentialSelf-citation
Research works printed but not linked: 2
- D Fanelli, When East meets West…does bias increase? A preliminary study on South Korea, United States and other countries. 8th International Conference on Webometrics, Informetrics and Scientometrics and 13th COLLNET Meeting, eds C Ho-Nam, K Hye-Sun, N Kyung-Ran, L Seon-Hee, K Hye-Jin, H Kretschmer (KISTI, Seoul, Korea), pp. 47–48 (2012).
- 10.1002/(SICI)1097-0258(19991030)18:20<2693::AID-SIM235>3.0.CO;2-V · doi:10.1002/(sici)1097-0258(19991030)18:20<2693::aid-sim235>3.0.co;2-v
Cited by
Redefine statistical significance
Benjamin, D. J., Berger, J. O., Johannesson, M., Nosek, B. A., Wagenmakers, E.-J., Berk, R., Bollen, K. A., Brembs, B., Brown, L., Camerer, C., et al. (2017) · Nature Human Behaviour · doi:10.1038/s41562-017-0189-z · Cited by 2,772
Self-citationSong, P., Rudan, D., Zhu, Y., Fowkes, F. J. I., Rahimi, K., Fowkes, F. G. R., & Rudan, I. (2019) · The Lancet Global Health · doi:10.1016/s2214-109x(19)30255-4 · Cited by 1,317
Song, P., Fang, Z., Wang, H., Cai, Y., Rahimi, K., Zhu, Y., Fowkes, F. G. R., Fowkes, F. J. I., & Rudan, I. (2020) · The Lancet Global Health · doi:10.1016/s2214-109x(20)30117-0 · Cited by 866
Open science challenges, benefits and tips in early career and beyond
Allen, C., & Mehler, D. M. A. (2019) · PLOS Biology · doi:10.1371/journal.pbio.3000246 · Cited by 608
“ECRs are at a particularly high risk of committing such errors due to lack of experience [53].”
BackgroundCorrecting for Bias in Psychology: A Comparison of Meta-Analytic Methods
Carter, E. C., Schönbrodt, F. D., Gervais, W. M., & Hilgard, J. (2019) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245919847196 · Cited by 588
Metadata
- Authors
- Fanelli, D., Costas, R., & Ioannidis, J. P. A.
- Affiliations
- Stanford University (US)
- Leiden University (NL)
- Stanford Medicine (US)
- Venue
- Proceedings of the National Academy of Sciences
- Publisher
- National Academy of Sciences
- Published
- 2017-03-20
- Volume
- 114
- Issue
- 14
- Pages
- 3714-3719
- ISSN
- 0027-8424 (ISSN-L)
- 1091-6490
- PMC
- PMC5389310
- OpenAlex
- W2600954565
- Cited by
- 364
- References
- 52
- Citations by year
- 2026: 22 · 2025: 17 · 2024: 35 · 2023: 59 · 2022: 33 · 2021: 38 · 2020: 41 · 2019: 56 · 2018: 45 · 2017: 16
- FWCI
- 10.69
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Proceedings of the National Academy of Sciences (publishedVersion)
- Europe PMC (PubMed Central) (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- Leiden Repository (Leiden University) (submittedVersion)
- Data Archiving and Networked Services (DANS) (submittedVersion, other-oa)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Keywords
- research bias, meta-analysis, publication bias, small-study effects, effect size overestimation, early-career researchers, research misconduct, co-authorship, citation impact, peer review, disciplinary differences
- MeSH terms
- Female, Humans, Male, Publishing, Research Design, Research Personnel, Scientific Misconduct, Selection Bias
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Meta-assessment of bias in science. https://app.cochat.ai/papers/10.1073/pnas.1618569114
- The paper
- Fanelli, D., Costas, R., & Ioannidis, J. P. A. (2017). Meta-assessment of bias in science. Proceedings of the National Academy of Sciences. https://doi.org/10.1073/pnas.1618569114
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.