CoChat

What difference do retractions make? An estimate of the epistemic impact of retractions on recent meta-analyses

Open access
Fanelli, Daniele, Moher, David · 2019 · bioRxiv (Cold Spring Harbor Laboratory)

The DOI matches its Crossref record; 26 of the 26 references it cites were found in a registry or an index.

Abstract

Every year, several hundred publications are retracted due to fabrication and falsification of data or plagiarism and other breeches of research integrity and ethics. Despite considerable research on this phenomenon, the extent to which a retraction requires revising previous scientific estimates and beliefs – which we define as the epistemic impact - is unknown. We collected a representative sample of recently retracted studies that had been included in recent meta-analyses, and compared the summary effect size of these meta analyses with and without the refracted studies. On average, the retractions had occurred about six years prior to the publication of the corresponding meta-analyses. Our results suggest that retractions have varying impacts depending on their causes. In particular, removing from an analysis a study retracted because of issues with data, methods or results, led to a statistically significant reduction of the estimated effect size. Assuming that the results of these retracted studies are completely false, then the meta-analyses that had included them had overestimated the summary effect sizes by, averaging across effect size metrics, 30% (median, 13%). However, retractions due to plagiarism or other issues not related to data, methods or results had no impact on the conclusions of meta-analyses. Since retractions due to plagiarism or other non-data related issues typically constitute over 75% of total retractions, our results suggest that the epistemic impact of most retractions is likely to be null. However, our results also suggest that retractions due to issues with data, methods or results should be accompanied by a revision of relevant meta-analyses, and by extension a downwards revision of prior scientific beliefs.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1101/734137

Cites

26 of 26 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1 Flow chart of the sample retrieval process.
Figure 1 Flow chart of the sample retrieval process.CC-BY · From the PDF
Figure 2 Relative size, in percentage, of meta-analytical pooled effect sizes calculated including and excluding the retracted study. Values of 100% (dashed line) indicate identical effect sizes, whereas values above 100% indicate that the meta-analysis with the retracted study yielded a larger effect size than without it. Empty squares indicate meta-analyses that included the same retracted study as another meta-analysis.
Figure 2 Relative size, in percentage, of meta-analytical pooled effect sizes calculated including and excluding the retracted study. Values of 100% (dashed line) indicate identical effect sizes, whereas values above 100% indicate that the meta-analysis with the retracted study yielded a larger effect size than without it. Empty squares indicate meta-analyses that included the same retracted study as another meta-analysis.CC-BY · From the PDF

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Cited by

See all 10

Metadata

Authors
Fanelli, D., & Moher, D.
Affiliations
  • London School of Economics and Political Science (GB)
  • University of Ottawa (CA)
Corresponding author
Fanelli, Daniele
Venue
bioRxiv (Cold Spring Harbor Laboratory)
Publisher
openRxiv
Published
2019-08-13
OpenAlex
W2967929375
Cited by
10
References
26
Citations by year
2023: 2 · 2022: 1 · 2021: 4 · 2020: 1 · 2019: 2
Type
posted content
Language
en
Open access
Licence
cc-by
Locations
bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by)
Topics
  • Social Sciences › Education › Academic integrity and plagiarism
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
Keywords
retracted study, meta-analysis, research integrity, data fabrication, research misconduct, plagiarism, effect size overestimation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.