What difference do retractions make? An estimate of the epistemic impact of retractions on recent meta-analyses
Open accessThe DOI matches its Crossref record; 26 of the 26 references it cites were found in a registry or an index.
Abstract
Every year, several hundred publications are retracted due to fabrication and falsification of data or plagiarism and other breeches of research integrity and ethics. Despite considerable research on this phenomenon, the extent to which a retraction requires revising previous scientific estimates and beliefs – which we define as the epistemic impact - is unknown. We collected a representative sample of recently retracted studies that had been included in recent meta-analyses, and compared the summary effect size of these meta analyses with and without the refracted studies. On average, the retractions had occurred about six years prior to the publication of the corresponding meta-analyses. Our results suggest that retractions have varying impacts depending on their causes. In particular, removing from an analysis a study retracted because of issues with data, methods or results, led to a statistically significant reduction of the estimated effect size. Assuming that the results of these retracted studies are completely false, then the meta-analyses that had included them had overestimated the summary effect sizes by, averaging across effect size metrics, 30% (median, 13%). However, retractions due to plagiarism or other issues not related to data, methods or results had no impact on the conclusions of meta-analyses. Since retractions due to plagiarism or other non-data related issues typically constitute over 75% of total retractions, our results suggest that the epistemic impact of most retractions is likely to be null. However, our results also suggest that retractions due to issues with data, methods or results should be accompanied by a revision of relevant meta-analyses, and by extension a downwards revision of prior scientific beliefs.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures


Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In fact, research misconduct in published RCTs has already been found to pollute the overall evidence base, with investigations from the UK indicating that results from retracted research can significantly affect results of meta-analyses or systematic reviews in which they are included, further contributing to potential adverse clinical impacts [7, 8].”
From Checklist to assess Trustworthiness in RAndomised Controlled Trials (TRACT checklist): concept proposal and pilot (2023) · cites this paper as [8]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“In a recent paper, analyses by Fanelli and Moher92 suggested that meta-analyses may overestimate their summary effect sizes when they include studies later retracted for issues with data, methods or results.”
From An investigation into the impact and implications of published papers from retracted research: systematic search of affected literature (2019) · cites this paper as [92]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702
Misconduct accounts for the majority of retracted scientific publications
Fang, F. C., Steen, R. G., & Casadevall, A. (2012) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1212247109 · Cited by 1,232 · cited 1×
“These are known to be an imperfect tool because they may depict the causes of a retraction inaccurately, in particular by under-reporting the occurrence of scientific misconduct [23].”
BackgroundWhy most published research findings are false
Brooks, J. (2008) · Urologic Oncology: Seminars and Original Investigations · doi:10.1016/j.urolonc.2008.07.015 · Cited by 423
Meta-assessment of bias in science
Fanelli, D., Costas, R., & Ioannidis, J. P. A. (2017) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1618569114 · Cited by 364 · cited 2×
“This hypothesis was also supported by a previous meta-meta-analytical study, which found that primary studies whose first authors had articles retracted reported significantly more extreme effect sizes [14].”
BackgroundResultSelf-citationWhy Growing Retractions Are (Mostly) a Good Sign
Fanelli, D. (2013) · PLoS Medicine · doi:10.1371/journal.pmed.1001563 · Cited by 272
Self-citationMoylan, E. C., & Kowalczuk, M. K. (2016) · BMJ Open · doi:10.1136/bmjopen-2016-012047 · Cited by 212
The Retraction Penalty: Evidence from the Web of Science
Lu, S. F., Jin, G. Z., Uzzi, B., & Jones, B. (2013) · Scientific Reports · doi:10.1038/srep03146 · Cited by 191
Post retraction citations in context: a case study
Bar-Ilan, J., & Halevi, G. (2017) · Scientometrics · doi:10.1007/s11192-017-2242-0 · Cited by 191
Financial costs and personal consequences of research misconduct resulting in retracted publications
Stern, A. M., Casadevall, A., Steen, R. G., & Fang, F. C. (2014) · eLife · doi:10.7554/elife.02956 · Cited by 189 · cited 1×
“05% of the NIH budget, a figure that was deemed low in the authors’ own assessment [10].”
BackgroundThe career effects of scandal: Evidence from scientific retractions
Azoulay, P., Bonatti, A., & Krieger, J. L. (2017) · Research Policy · doi:10.1016/j.respol.2017.07.003 · Cited by 118 · cited 1×
“Authors of articles retracted for honest error appeared to suffer no negative consequence [4] and collaborators of authors of retracted articles suffer a loss of citations mainly when the retraction was due to misconduct [7–9].”
BackgroundSeife, C. (2015) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2014.7774 · Cited by 101
Costly collaborations: The impact of scientific fraud on co‐authors' careers
Mongeon, P., & Larivière, V. (2016) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.23421 · Cited by 83
Improving the integrity of published science: An expanded taxonomy of retractions and corrections
Fanelli, D., Ioannidis, J. P. A., & Goodman, S. N. (2018) · European Journal of Clinical Investigation · doi:10.1111/eci.12898 · Cited by 71
Self-citationRetraction statements and research malpractice in economics
Cox, A., Craig, R., & Tourish, D. (2018) · Research Policy · doi:10.1016/j.respol.2018.02.016 · Cited by 57
Set up a ‘self-retraction’ system for honest errors
Fanelli, D. (2016) · Nature · doi:10.1038/531415a · Cited by 53
Self-citationRetraction of publications in nursing and midwifery research: A systematic review
Al-Ghareeb, A., Hillel, S., McKenna, L., Cleary, M., Visentin, D., Jones, M., Bressington, D., & Gray, R. (2018) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2018.01.013 · Cited by 50
Guilt by association: How scientific misconduct harms prior collaborators
Hussinger, K., & Pellens, M. (2019) · Research Policy · doi:10.1016/j.respol.2018.01.012 · Cited by 49 · cited 1×
“Authors of articles retracted for honest error appeared to suffer no negative consequence [4] and collaborators of authors of retracted articles suffer a loss of citations mainly when the retraction was due to misconduct [7–9].”
BackgroundA Multidimensional Investigation of the Effects of Publication Retraction on Scholarly Impact
Shuai, X., Rollins, J., Moulinier, I., Custis, T., Edmunds, M., & Schilder, F. (2017) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.23826 · Cited by 48
An analysis of the validity of retraction annotation in pubmed and the web of science
Schmidt, M. (2018) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.23913 · Cited by 47
A theory and methodology to quantify knowledge
Fanelli, D. (2019) · Royal Society Open Science · doi:10.1098/rsos.181055 · Cited by 44
Self-citationRetracted Publications in Orthopaedics: Prevalence, Characteristics, and Trends
Rai, R., & Sabharwal, S. (2017) · Journal of Bone and Joint Surgery · doi:10.2106/jbjs.16.01116 · Cited by 40
Retractions in the Engineering Field: A Study on the Web of Science Database
Rubbo, P., Helmann, C. L., Bilynkievycz dos Santos, C., & Pilatti, L. A. (2019) · Ethics & Behavior · doi:10.1080/10508422.2017.1390667 · Cited by 37
Mott, A., Fairhurst, C., & Torgerson, D. (2019) · Journal of Health Services Research & Policy · doi:10.1177/1355819618797965 · Cited by 36 · cited 1×
“Several studies have documented the fact that retracted articles keep being cited after their retraction, although typically at much lower rates [5,6].”
BackgroundGarmendia, C. A., Nassar Gorra, L., Rodriguez, A. L., Trepka, M. J., Veledar, E., & Madhivanan, P. (2019) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2018.7661 · Cited by 35
Huh, S., Kim, S. Y., & Cho, H.-M. (2016) · PLOS ONE · doi:10.1371/journal.pone.0163588 · Cited by 29
Artemisinin Combination Therapy: A Good Antimalarial, but Is the Dose Right?
Garner, P. (2013) · PLoS Medicine · doi:10.1371/journal.pmed.1001565 · Cited by 18
Cited by
Mol, B. W., Lai, S., Rahim, A., Bordewijk, E. M., Wang, R., van Eekelen, R., Gurrin, L. C., Thornton, J. G., van Wely, M., & Li, W. (2023) · Research Integrity and Peer Review · doi:10.1186/s41073-023-00130-8 · Cited by 154
Avenell, A., Stewart, F., Grey, A., Gamble, G., & Bolland, M. (2019) · BMJ Open · doi:10.1136/bmjopen-2019-031909 · Cited by 59
Media and social media attention to retracted articles according to Altmetric
Serghiou, S., Marton, R. M., & Ioannidis, J. P. A. (2021) · PLOS ONE · doi:10.1371/journal.pone.0248625 · Cited by 55
“Such perpetuated misinformation is not inconsequential: guidelines and meta-analyses seem to be very rarely updated to remove retracted articles [11] and a recent preprint suggests that doing so would lead to a median reduction in estimated effect size of 13% and an average reduction of 30% [12].”
BackgroundReducing the Inadvertent Spread of Retracted Science: recommendations from the RISRS report
Schneider, J., Woods, N. D., Proescholdt, R., the RISRS Team, Burns, H., Howell, K., Campbell, M. T., Hsiao, T., Yip, Y. Y. ‘., Fu, Y., et al. (2022) · Research Integrity and Peer Review · doi:10.1186/s41073-022-00125-x · Cited by 54
A bibliometric study of article retractions in technology fields in developing economies countries
Eldakar, M. A. M., & Shehata, A. M. K. (2023) · Scientometrics · doi:10.1007/s11192-023-04823-1 · Cited by 27
Metadata
- Authors
- Fanelli, D., & Moher, D.
- Affiliations
- London School of Economics and Political Science (GB)
- University of Ottawa (CA)
- Corresponding author
- Fanelli, Daniele
- Venue
- bioRxiv (Cold Spring Harbor Laboratory)
- Publisher
- openRxiv
- Published
- 2019-08-13
- OpenAlex
- W2967929375
- Cited by
- 10
- References
- 26
- Citations by year
- 2023: 2 · 2022: 1 · 2021: 4 · 2020: 1 · 2019: 2
- Type
- posted content
- Language
- en
- Open access
- green
- doi.org
- Licence
- cc-by
- Locations
- bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by)
- Topics
- Social Sciences › Education › Academic integrity and plagiarism
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Keywords
- retracted study, meta-analysis, research integrity, data fabrication, research misconduct, plagiarism, effect size overestimation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.