Meta-assessment of bias in science
Open accessThe DOI matches its Crossref record; 53 of the 53 references it cites were found in a registry or an index.
Abstract
Significance: Science is said to be suffering a reproducibility crisis caused by many biases. How common are these problems, across the wide diversity of research fields? We probed for multiple bias-related patterns in a large random sample of meta-analyses taken from all disciplines. The magnitude of these biases varied widely across fields and was on average relatively small. However, we consistently observed that small, early, highly cited studies published in peer-reviewed journals were likely to overestimate effects. We found little evidence that these biases were related to scientific productivity, and we found no difference between biases in male and female researchers. However, a scientist’s early-career status, isolation, and lack of scientific integrity might be significant risk factors for producing unreliable results.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“less likely to publish papers that are retracted”
From Is science really facing a reproducibility crisis, and do we need it to? (2018) · cites this paper as [19]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“later, finer-grained analyses offered contrary evidence, by showing that researchers that publish at higher frequency, in journals with higher impact factor, and in countries where pressures to publish are high, are equally or more likely to correct their work”
From Is science really facing a reproducibility crisis, and do we need it to? (2018) · cites this paper as [19]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Indeed, in a meta-assessment of bias across all disciplines, small-study effects and gray-literature bias (both possible symptoms of reporting biases) were highly heterogeneously distributed (19).”
From Is science really facing a reproducibility crisis, and do we need it to? (2018) · cites this paper as [19]
The cited source says what the claim attributes to it.
In the citing paper
“and less likely to author papers reporting overestimated effects (19, 40, 41).”
From Is science really facing a reproducibility crisis, and do we need it to? (2018) · cites this paper as [19]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“less likely to author papers that contain duplicated images”
From Is science really facing a reproducibility crisis, and do we need it to? (2018) · cites this paper as [19]
Experimental — accuracy not yet measured
3 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Fitting Linear Mixed-Effects Models Using lme4
Bates, D., Mächler, M., Bolker, B., & Walker, S. (2015) · Journal of Statistical Software · doi:10.18637/jss.v067.i01 · Cited by 86,835
Publication and related bias in meta-analysis
Sterne, J. A. C., Gavaghan, D., & Egger, M. (2000) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(00)00242-0 · Cited by 2,282 · cited 2×
“Moreover, small-study effects may result not necessarily from QRP but also from legitimate choices made in designing small studies (17, 36).”
BackgroundPharmaceutical industry sponsorship and research outcome and quality: systematic review
Lexchin, J., Bero, L. A., Djulbegovic, B., & Clark, O. (2003) · BMJ · doi:10.1136/bmj.326.7400.1167 · Cited by 2,147 · cited 1×
“Industry bias: Industry sponsorship may affect the direction and magnitude of effects reported by biomedical studies (21).”
BackgroundSystematic Review of the Empirical Evidence of Study Publication Bias and Outcome Reporting Bias
Dwan, K., Altman, D. G., Arnaiz, J. A., Bloom, J., Chan, A.-W., Cronin, E., Decullier, E., Easterbrook, P. J., Von Elm, E., Gamble, C., et al. (2008) · PLoS ONE · doi:10.1371/journal.pone.0003081 · Cited by 1,546 · cited 1×
“J Empir Res Hum Res Ethics 2(4):15–34.”
Self-citationA comparison of methods to detect publication bias in meta‐analysis
Macaskill, P., Walter, S. D., & Irwig, L. (2001) · Statistics in Medicine · doi:10.1002/sim.698 · Cited by 1,286 · cited 1×
“Nevertheless, meta-regression estimates of small-study effects on odds ratio may run a modest risk of type I error (35) and were measured by our study more completely and accurately than the other biases (SI Multilevel Meta-Regression Analysis, Reliability and Consistency of Collected Data).”
MethodNegative results are disappearing from most disciplines and countries
Fanelli, D. (2012) · Scientometrics · doi:10.1007/s11192-011-0494-7 · Cited by 1,248 · cited 3×
“Systematic differences in the risk of bias between physical, biological, and social sciences were observed, particularly for the most prominent biases, as was expected based on previous evidence (5, 11, 33).”
MethodResultSelf-citationDissemination and publication of research findings: an updated review of related biases
Song, F., Parekh, Hooper, L., Loke, Y., Ryder, J., Sutton, A., Hing, C., Kwok, C., Pang, C., & Harvey, I. (2010) · Health Technology Assessment · doi:10.3310/hta14080 · Cited by 1,039 · cited 5×
“Gray literature bias: Studies might be less likely to be published if they yielded smaller and/or statistically nonsignificant effects and might be therefore only available in PhD theses, conference proceedings, books, personal communications, and other forms of “gray” literature (1).”
BackgroundResultInfluentialExtension of Nakagawa & Schielzeth's R2GLMM to random slopes models
Johnson, P. C. D. (2014) · Methods in Ecology and Evolution · doi:10.1111/2041-210x.12225 · Cited by 948
Do Pressures to Publish Increase Scientists' Bias? An Empirical Support from US States Data
Fanelli, D. (2010) · PLoS ONE · doi:10.1371/journal.pone.0010271 · Cited by 805
Self-citation“Positive” results increase down the Hierarchy of the Sciences
Fanelli, D. (2010) · PLoS ONE · doi:10.1371/journal.pone.0010068 · Cited by 734 · cited 4×
“Systematic differences in the risk of bias between physical, biological, and social sciences were observed, particularly for the most prominent biases, as was expected based on previous evidence (5, 11, 33).”
BackgroundMethodInfluentialSelf-citationInterpretation of tests of heterogeneity and bias in meta-analysis
Ioannidis, J. P. A. (2008) · Journal of Evaluation in Clinical Practice · doi:10.1111/j.1365-2753.2008.00986.x · Cited by 696 · cited 1×
“Moreover, small-study effects may result not necessarily from QRP but also from legitimate choices made in designing small studies (17, 36).”
BackgroundSelf-citationWhy Current Publication Practices May Distort Science
Young, N. S., Ioannidis, J. P. A., & Al-Ubaydli, O. (2008) · PLoS Medicine · doi:10.1371/journal.pmed.0050201 · Cited by 502
Self-citationMeta-research: Evaluation and Improvement of Research Methods and Practices
Ioannidis, J. P. A., Fanelli, D., Dunne, D. D., & Goodman, S. N. (2015) · PLOS Biology · doi:10.1371/journal.pbio.1002264 · Cited by 495 · cited 1×
“Nonetheless, concerns that papers reporting false or exaggerated findings might be widespread and growing have inspired an expanding literature of research on research (aka meta-research), which points to a postulated core set of bias patterns and factors that might increase the risk for researchers to engage in biasg…”
BackgroundSelf-citationSterling, T. D., Rosenbaum, W. L., & Weinkam, J. J. (1995) · The American Statistician · doi:10.1080/00031305.1995.10476125 · Cited by 453 · cited 3×
“(2013) Pressures to publish: Catalysts for the loss of scientific writing integrity? Fisheries 38(8):352–355.”
BackgroundTowards a new crown indicator: Some theoretical considerations
Waltman, L., van Eck, N. J., van Leeuwen, T. N., Visser, M. S., & van Raan, A. F. J. (2011) · Journal of Informetrics · doi:10.1016/j.joi.2010.08.001 · Cited by 424
Contextual sensitivity in scientific reproducibility
Van Bavel, J. J., Mende-Siedlecki, P., Brady, W. J., & Reinero, D. A. (2016) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1521897113 · Cited by 398
Intended and unintended consequences of a publish‐or‐perish culture: A worldwide survey
van Dalen, H. P., & Henkens, K. (2012) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.22636 · Cited by 355
Unpublished results hide the decline effect
Schooler, J. (2011) · Nature · doi:10.1038/470437a · Cited by 333
Ioannidis, J. P. A., & Trikalinos, T. A. (2005) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2004.10.019 · Cited by 325 · cited 1×
“Early-extreme: An alternative scenario to the decline effect might see earlier studies reporting extreme effects in any direction, because extreme and controversial findings have an early window of opportunity for publication (19).”
BackgroundSelf-citationReproducible Research Practices and Transparency across the Biomedical Literature
Iqbal, S. A., Wallach, J. D., Khoury, M. J., Schully, S. D., & Ioannidis, J. P. A. (2016) · PLOS Biology · doi:10.1371/journal.pbio.1002333 · Cited by 310
Self-citationFanelli, D., Costas, R., & Larivière, V. (2015) · PLOS ONE · doi:10.1371/journal.pone.0127556 · Cited by 268 · cited 4×
“Nonetheless, concerns that papers reporting false or exaggerated findings might be widespread and growing have inspired an expanding literature of research on research (aka meta-research), which points to a postulated core set of bias patterns and factors that might increase the risk for researchers to engage in biasg…”
BackgroundResultInfluentialSelf-citationNormal Misbehavior: Scientists Talk about the Ethics of Research
De Vries, R., Anderson, M. S., & Martinson, B. C. (2006) · Journal of Empirical Research on Human Research Ethics · doi:10.1525/jer.2006.1.1.43 · Cited by 254
Publication Pressure and Scientific Misconduct in Medical Scientists
Tijdink, J. K., Verbeke, R., & Smulders, Y. M. (2014) · Journal of Empirical Research on Human Research Ethics · doi:10.1177/1556264614552421 · Cited by 221
Accelerating scientific publication in biology
Vale, R. D. (2015) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1511912112 · Cited by 208 · cited 1×
“A link between pressures to publish and questionable research practices cannot be excluded, but is likely to be modulated by characteristics of study and authors, including the complexity of methodologies, the career stage of individuals, and the size and distance of collaborations (14, 39, 43).”
BackgroundSix domains of research ethics: A heuristic framework for the responsible conduct of research
Pimple, K. D. (2002) · Science and Engineering Ethics · doi:10.1007/s11948-002-0018-1 · Cited by 205
Researchers’ Individual Publication Rate Has Not Increased in a Century
Fanelli, D., & Larivière, V. (2016) · PLOS ONE · doi:10.1371/journal.pone.0149504 · Cited by 205 · cited 2×
“A link between pressures to publish and questionable research practices cannot be excluded, but is likely to be modulated by characteristics of study and authors, including the complexity of methodologies, the career stage of individuals, and the size and distance of collaborations (14, 39, 43).”
BackgroundResultSelf-citationBibliometric Evidence for a Hierarchy of the Sciences
Fanelli, D., & Glänzel, W. (2013) · PLoS ONE · doi:10.1371/journal.pone.0066938 · Cited by 197 · cited 2×
“Systematic differences in the risk of bias between physical, biological, and social sciences were observed, particularly for the most prominent biases, as was expected based on previous evidence (5, 11, 33).”
MethodSelf-citationCitation bias favoring statistically significant studies was present in medical research
Jannot, A.-S., Agoritsas, T., Gayet-Ageron, A., & Perneger, T. V. (2013) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2012.09.015 · Cited by 193 · cited 1×
“Citation bias: The number of citations received by a study might be correlated to the magnitude of effects reported (20).”
BackgroundQiu, J. (2010) · Nature · doi:10.1038/463142a · Cited by 182 · cited 2×
“Pressures to publish: Scientists subjected to direct or indirect pressures to publish might be more likely to exaggerate the magnitude and importance of their results to secure many high-impact publications and new grants (22, 23).”
BackgroundSystematic review of publication bias in studies on publication bias
Dubben, H.-H., & Beck-Bornholdt, H.-P. (2005) · BMJ · doi:10.1136/bmj.38478.497164.f7 · Cited by 153
Worsening file-drawer problem in the abstracts of natural, medical and social science databases
Pautasso, M. (2010) · Scientometrics · doi:10.1007/s11192-010-0233-5 · Cited by 136
US studies may overestimate effect sizes in softer research
Fanelli, D., & Ioannidis, J. P. A. (2013) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1302997110 · Cited by 130 · cited 5×
“Each dataset was standardized following previously established protocols (14).”
BackgroundMethodInfluentialSelf-citationPersonality and Ethical Decision-Making in Research: The Role of Perceptions of Self and others
Antes, A. L., Brown, R. P., Murphy, S. T., Waples, E. P., Mumford, M. D., Connelly, S., & Devenport, L. D. (2007) · Journal of Empirical Research on Human Research Ethics · doi:10.1525/jer.2007.2.4.15 · Cited by 112
Tijssen, R., van Leeuwen, T. N., & van Wijk, E. (2009) · Research Evaluation · doi:10.3152/095820209x393145 · Cited by 109
Science mapping analysis characterizes 235 biases in biomedical research
Chavalarias, D., & Ioannidis, J. P. A. (2010) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2009.12.011 · Cited by 105 · cited 2×
“Johnson PCD (2014) Extension of Nakagawa & Schielzeth’s R(2)GLMM to random slopes models.”
Self-citationLarge scale author name disambiguation using rule-based scoring and clustering
Caron, E., van Eck, N. J., & Noyons., E. (2014) · EUR Research Repository (Erasmus University Rotterdam) · Cited by 99
Males Are Overrepresented among Life Science Researchers Committing Scientific Misconduct
Fang, F. C., Bennett, J. W., & Casadevall, A. (2013) · mBio · doi:10.1128/mbio.00640-12 · Cited by 92
Horton, J., Vandermeer, B., Hartling, L., Tjosvold, L., Klassen, T. P., & Buscemi, N. (2010) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2009.04.007 · Cited by 81
de Winter, J. C., & Dodou, D. (2015) · PeerJ · doi:10.7717/peerj.733 · Cited by 78
When Should Potentially False Research Findings Be Considered Acceptable?
Djulbegovic, B., & Hozo, I. (2007) · PLoS Medicine · doi:10.1371/journal.pmed.0040026 · Cited by 77
Martinson, B. C., Crain, A. L., Anderson, M. S., & De Vries, R. (2009) · Academic Medicine · doi:10.1097/acm.0b013e3181bb2ca6 · Cited by 74 · cited 1×
“Pressures to publish: Scientists subjected to direct or indirect pressures to publish might be more likely to exaggerate the magnitude and importance of their results to secure many high-impact publications and new grants (22, 23).”
BackgroundBailey, C. D. (2012) · The Accounting Review · doi:10.2308/accr-50970 · Cited by 73
Editorial: Journal of Negative Results in Biomedicine
Pfeffer, C., & Olsen, B. R. (2002) · Journal of Negative Results in BioMedicine · doi:10.1186/1477-5751-1-2 · Cited by 62
Cancer Patients' Attitudes toward Future Research Uses of Stored Human Biological Materials
Helft, P. R., Champion, V. L., Eckles, R., Johnson, C. S., & Meslin, E. M. (2007) · Journal of Empirical Research on Human Research Ethics · doi:10.1525/jer.2007.2.3.15 · Cited by 50
Are Men More Likely than Women To Commit Scientific Misconduct? Maybe, Maybe Not
Kaatz, A., Vogelman, P. N., & Carnes, M. (2013) · mBio · doi:10.1128/mbio.00156-13 · Cited by 36 · cited 1×
“This hypothesis was supported by statistics of the US Office of Research Integrity (27), which, however, may have multiple alternative explanations (28).”
MethodLee, C.-S., & Schrank, A. M. (2010) · Social Forces · doi:10.1353/sof.0.0282 · Cited by 33 · cited 5×
“Our results also suggest that there is a connection between bias and retractions, offering some support to a responsible conduct of research program that assumes negligence, QRP, and misconduct to be connected phenomena that may be addressed by common interventions (38).”
BackgroundInfluentialNarcissism, Entitlement, and Questionable Research Practices in Counseling: A Pilot Study
Davis, M. S., Wester, K. L., & King, B. A. (2008) · Journal of Counseling & Development · doi:10.1002/j.1556-6678.2008.tb00498.x · Cited by 33 · cited 1×
“We also analyzed data using weighted multilevel regression, a method that yields similar but more conservative estimates (30).”
MethodPressures to Publish: Catalysts for the Loss of Scientific Writing Integrity?
Hayer, C., Kaemingk, M. A., Breeggemann, J. J., Dembkowski, D. J., Deslauriers, D., & Rapp, T. (2013) · Fisheries · doi:10.1080/03632415.2013.813845 · Cited by 21
Streamlined chemical tests rebuffed
Gilbert, N. (2010) · Nature · doi:10.1038/463142b · Cited by 17
de Winter, J., & Dodou, D. (2014) · doi:10.7287/peerj.preprints.447 · Cited by 17
Fanelli, D., & Ioannidis, J. P. A. (2014) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1322565111 · Cited by 5 · cited 1×
“The prevalence of these phenomena across multiple meta-analyses can be analyzed with multilevel weighted regression analysis (14) or, more straightforwardly, by conducting a second-order meta-analysis on regression estimates obtained within each meta-analysis (32).”
MethodSelf-citationThe Economics of Scientific Misconduct
Lacetera, N., & Zirulia, L. (2011) · Journal of Law, Economics, and Organization · doi:10.1093/jleo/ewp031 · Cited by 5
Explaining heterogeneity in meta-analysis: a comparison of methods
Thompson, S. G., & Sharp, S. J. (1999) · Statistics in Medicine · doi:10.1002/(sici)1097-0258(19991030)18:20
Research works printed but not linked: 1
- Fanelli D (2012) When East meets West. . .does bias increase? A preliminary study on South Korea, United States and other countries. 8th International Conference on Webometrics, Informetrics and Scientometrics and 13th COLLNET Meeting, eds Ho-Nam C, Hye-Sun K, Kyung-Ran N, Seon-Hee L, Hye-Jin K, Kretschmer H (KISTI, Seoul, Korea), pp 47-48.
Cited by
Redefine statistical significance
Benjamin, D. J., Berger, J. O., Johannesson, M., Nosek, B. A., Wagenmakers, E.-J., Berk, R., Bollen, K. A., Brembs, B., Brown, L., Camerer, C., et al. (2017) · Nature Human Behaviour · doi:10.1038/s41562-017-0189-z · Cited by 2,772
Self-citationSong, P., Rudan, D., Zhu, Y., Fowkes, F. J. I., Rahimi, K., Fowkes, F. G. R., & Rudan, I. (2019) · The Lancet Global Health · doi:10.1016/s2214-109x(19)30255-4 · Cited by 1,317
Song, P., Fang, Z., Wang, H., Cai, Y., Rahimi, K., Zhu, Y., Fowkes, F. G. R., Fowkes, F. J. I., & Rudan, I. (2020) · The Lancet Global Health · doi:10.1016/s2214-109x(20)30117-0 · Cited by 866
Open science challenges, benefits and tips in early career and beyond
Allen, C., & Mehler, D. M. A. (2019) · PLOS Biology · doi:10.1371/journal.pbio.3000246 · Cited by 608
“ECRs are at a particularly high risk of committing such errors due to lack of experience [53].”
BackgroundCorrecting for Bias in Psychology: A Comparison of Meta-Analytic Methods
Carter, E. C., Schönbrodt, F. D., Gervais, W. M., & Hilgard, J. (2019) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245919847196 · Cited by 588
Metadata
- Authors
- Fanelli, D., Costas, R., & Ioannidis, J. P. A.
- Affiliations
- Stanford University (US)
- Leiden University (NL)
- Stanford Medicine (US)
- Venue
- Proceedings of the National Academy of Sciences
- Publisher
- National Academy of Sciences
- Published
- 2017-03-20
- Volume
- 114
- Issue
- 14
- Pages
- 3714-3719
- ISSN
- 0027-8424 (ISSN-L)
- 1091-6490
- PMC
- PMC5389310
- OpenAlex
- W2600954565
- Cited by
- 364
- References
- 52
- Citations by year
- 2026: 22 · 2025: 17 · 2024: 35 · 2023: 59 · 2022: 33 · 2021: 38 · 2020: 41 · 2019: 56 · 2018: 45 · 2017: 16
- FWCI
- 10.69
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Proceedings of the National Academy of Sciences (publishedVersion)
- Europe PMC (PubMed Central) (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- Leiden Repository (Leiden University) (submittedVersion)
- Data Archiving and Networked Services (DANS) (submittedVersion, other-oa)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Keywords
- research bias, meta-analysis, publication bias, small-study effects, effect size overestimation, early-career researchers, research misconduct, co-authorship, citation impact, peer review, disciplinary differences
- MeSH terms
- Female, Humans, Male, Publishing, Research Design, Research Personnel, Scientific Misconduct, Selection Bias
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.