CoChat

The DOI matches its Crossref record; 67 of the 73 references it cites were found in a registry or an index; 27 of the 53 sources its claims cite were checked for support.

Continued use of retracted papers: Temporal trends in citations and (lack of) awareness of retractions shown in citation contexts in biomedicine

Open access
Hsiao, Tzu-Kun, Schneider, Jodi · 2021 · Quantitative Science Studies

Abstract

We present the first database-wide study on the citation contexts of retracted papers, which covers 7,813 retracted papers indexed in PubMed. Our temporal analyses show that retracted papers continued to be cited. Furthermore, among the 13,252 postretraction citation contexts studied, only 722 (5.4%) citation contexts acknowledged the retraction, meaning 94.6% of post-retraction citations in biomedicine do not document the retraction and continue to cite retracted research as valid.

Overview

CoChat Summary

TL;DR

This database-wide study of 7,813 PubMed retracted papers, 169,434 iCite citations, and 48,134 PMC open-access citation contexts finds that retracted papers continued to be cited, but old retracted papers stopped being cited as time progressed. Among 13,252 postretraction citation contexts, only 722 (5.4%) acknowledged the retraction, and retraction did not change how the papers were cited.

Key findings

  • Retracted papers continued to be cited: 6,704 (85.81%) retracted papers were cited 169,434 times, while 1,109 (14.19%) were never cited; 75% of retracted papers were cited no more than six times after retraction. The proportion of active retracted papers decreased from 80.23% to 25.36% since 2013, though up to 2018 more than 50% were still active.
  • Among 13,252 postretraction citation contexts, only 722 (5.4%) acknowledged the retraction; 94.6% did not. In citing–cited pairs, 422 of 9,405 postretraction pairs (4.49%) acknowledged retractions, while 8,983 (95.51%) did not.
  • Retraction did not change citation patterns: pre- and postretraction citation contexts showed similar text progression, with postretraction contexts slightly more concentrated at the beginning and end of papers and more often in introduction/background and discussion/conclusion sections.
  • For the 722 intentional postretraction citation contexts, the most common purpose was related work (453, 62.74%); the second most prevalent was example of problematic science. In systematic reviews and meta-analyses, exclusion rationale was most common; in articles about retraction or scientific misconduct, subject of study was most common.

Methods

  • Data sources: retracted papers from PubMed using “retracted publication”[PT] on August 20, 2020 (7,813 results); citations from iCite (169,434 citations after exclusions); citation contexts from the PMCOA Citation Context Dataset built from 2,049,871 PMC open-access articles (48,134 contexts analyzed).
  • Postretraction citation was operationalized as a citation made after the calendar year of retraction. Temporal analyses categorized retracted papers as active, inactive, or uncited over time.
  • Intentional postretraction citations were identified using cue words in the citation context or a five-sentence acknowledgment window, or co-citation with the retraction notice, followed by manual review; purposes were manually classified into 11 categories, with intercoder agreement Kappa = .63 on a 100-context sample.

Limitations

  • Stated: findings cannot be projected into the future because journals and authors can reach back in time to retract articles; the data are right-censored because retractions typically happen after publication, and uncited or inactive retracted papers could become active in the future.
  • Stated: annotation was done primarily by the first author, and intercoder agreement was fair (Kappa = .63); the cue-word approach might falsely capture some citations, so each identified context was manually reviewed.
  • Evident from the text: citation contexts come only from the PMC open-access subset, and citations come from iCite/PubMed, so the study does not cover all postretraction citations; defining postretraction as after the calendar year of retraction may exclude same-year citations.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1162/qss_a_00155

Cites

67 of 67 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Figure 1. Boxplot of time lag between publication and retraction arranged by retraction year and publication year. The box areas show interquartile range (IQR, from 25% to 75%) of the time lag between publication and retraction. The upper whisker of each box is the longest time lag smaller than 1.5 IQR above the 75th percentile; the lower whisker is the shortest time lag greater than 1.5 IQR below the 25th percentile. The grey dotted line in the lower panel shows the maximum possible time lag (i.e., up to 2020, the year of data collection) for papers to be retracted in each publication year.
Figure 1. Boxplot of time lag between publication and retraction arranged by retraction year and publication year. The box areas show interquartile range (IQR, from 25% to 75%) of the time lag between publication and retraction. The upper whisker of each box is the longest time lag smaller than 1.5 IQR above the 75th percentile; the lower whisker is the shortest time lag greater than 1.5 IQR below the 25th percentile. The grey dotted line in the lower panel shows the maximum possible time lag (i.e., up to 2020, the year of data collection) for papers to be retracted in each publication year.CC BY · PubMed Central
Figure 2. Distribution of active, inactive, and uncited papers.
Figure 2. Distribution of active, inactive, and uncited papers.CC BY · PubMed Central
Figure 3. The percentage of active retracted papers with a given retraction year over time. Note that the citation data from 2020 are incomplete because the data were collected in August 2020. The shades denote the percentage of active retracted papers in a given citation year. Red shades denote that more than 50% of the papers retracted in a given year were active in a given citation year. Blue shades denote that less than 50% of the papers retracted in a given year were active in a given citation year. The grey area is the preretraction phase. The yellow box shows the percentages of papers retracted before 1995 that are active in citation year 2013. The green box shows the percentages of papers retracted before 2014 that are active in citation year 2018.
Figure 3. The percentage of active retracted papers with a given retraction year over time. Note that the citation data from 2020 are incomplete because the data were collected in August 2020. The shades denote the percentage of active retracted papers in a given citation year. Red shades denote that more than 50% of the papers retracted in a given year were active in a given citation year. Blue shades denote that less than 50% of the papers retracted in a given year were active in a given citation year. The grey area is the preretraction phase. The yellow box shows the percentages of papers retracted before 1995 that are active in citation year 2013. The green box shows the percentages of papers retracted before 2014 that are active in citation year 2018.CC BY · PubMed Central
Figure 4. Locations of citations to retracted papers. Text progression indicates the location of a citation context in the main text on a percentage scale. The IMRaD sections were identified from the section titles (Hsiao & Torvik, 2020). IMRaD not identified refers to the sections where the IMRaD section types could not be identified from the section titles. The y-axis scales do not range from 0–100% because each part of the text only has a few citation contexts. For clarity, to show the trends of citation contexts’ locations the scales of the y-axes were set from 0–3% for all citation contexts as well as for citation contexts in introduction/background sections and discussion/conclusion sections. For citation contexts in method sections, in result sections, and in sections for which the IMRaD section types were unidentifiable, the scales of the y-axes were set from 0–0.5%.
Figure 4. Locations of citations to retracted papers. Text progression indicates the location of a citation context in the main text on a percentage scale. The IMRaD sections were identified from the section titles (Hsiao & Torvik, 2020). IMRaD not identified refers to the sections where the IMRaD section types could not be identified from the section titles. The y-axis scales do not range from 0–100% because each part of the text only has a few citation contexts. For clarity, to show the trends of citation contexts’ locations the scales of the y-axes were set from 0–3% for all citation contexts as well as for citation contexts in introduction/background sections and discussion/conclusion sections. For citation contexts in method sections, in result sections, and in sections for which the IMRaD section types were unidentifiable, the scales of the y-axes were set from 0–0.5%.CC BY · PubMed Central
Figure 5. Distribution of text progression by the age of citation relative to retraction year. Age represents the number of years preceding (negative) or following (positive) retraction. Markers in the cells denote the percentage of citation context in each group of citation contexts having the same age relative to retraction year.
Figure 5. Distribution of text progression by the age of citation relative to retraction year. Age represents the number of years preceding (negative) or following (positive) retraction. Markers in the cells denote the percentage of citation context in each group of citation contexts having the same age relative to retraction year.CC BY · PubMed Central
Figure 6. Location of the retracted papers that were mentioned multiple times.
Figure 6. Location of the retracted papers that were mentioned multiple times.CC BY · PubMed Central
Figure 7. Citation purposes observed in different types of articles. The signs (−, +, ±) represent the tone identified in comparison citations. In research articles and review articles, most of the tones of comparison citations were negative (−), followed by positive (+) and neutral (±). In systematic reviews and meta-analyses, only positive comparison citations were observed. For articles about retraction, scientific misconduct, or peer review, the citation purposes observed in each article subtype are presented with different color keys.
Figure 7. Citation purposes observed in different types of articles. The signs (−, +, ±) represent the tone identified in comparison citations. In research articles and review articles, most of the tones of comparison citations were negative (−), followed by positive (+) and neutral (±). In systematic reviews and meta-analyses, only positive comparison citations were observed. For articles about retraction, scientific misconduct, or peer review, the citation purposes observed in each article subtype are presented with different color keys.CC BY · PubMed Central

Verification

Claims the paper makes

19 Valid · 6 Not valid · 12 Not checked · 96 Own results and interpretations

60 more claims · 3 with their cited sources checked

Create a free account to see all 133 claims and every source verdict.

See all claims — free account
  • It may state two separate claims as one. 37
  • Probably not a substantial claim. 38
  • Borderline: one of the tests for a substantial claim is unclear. 2
  • The quote only partly states this claim. 1

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Self-check results appear here once an independent check has confirmed them.

Cited works that were retracted

Works this paper cites that were later retracted, from the retraction notices publishers deposit. Major when the notice came out before this paper was published.

2 of 2

Citations

Cites

2 retracted

Research works printed but not linked: 8

  1. American Psychological Association, 2020, Publication manual of the American Psychological Association: The official guide to APA style
  2. Budd, 2011, Retracted publications in biomedicine: Cause for concern, 390
  3. Garfield, 1965, Can citation indexing be automated?, Statistical Association Methods for Mechanized Documentation, Symposium Proceedings, 269, 189
  4. Nane, 2015, Time to first citation estimation in the presence of additional information, 249
  5. PubMed Central, .., Open Access Subset
  6. Retraction Watch database user guide, 2018, Retraction Watch
  7. scite, 2020, Reference Check: An easy way to check the reliability of your references
  8. Zotero, 2019, Retracted item notifications with Retraction Watch integration

Cited by

See all 98

Metadata

Authors
Hsiao, T.-K., & Schneider, J.
Affiliations
University of Illinois Urbana-Champaign (US)
Corresponding author
Schneider, Jodi
Venue
Quantitative Science Studies
Publisher
MIT Press
Published
2021
Volume
2
Issue
4
Pages
1144-1169
ISSN
2641-3337 (ISSN-L)
OpenAlex
W4200394820
Cited by
98
References
73
Citations by year
2026: 27 · 2025: 22 · 2024: 17 · 2023: 19 · 2022: 9 · 2021: 4
FWCI
7.10
Type
journal article
Language
en
Open access
Locations
  • Quantitative Science Studies (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
Funders
  • U.S. National Library of Medicine (R01LM010817)
  • Alfred P. Sloan Foundation (G-2020-12623)
Topics
  • Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
  • Social Sciences › Education › Academic integrity and plagiarism
  • Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
Keywords
retracted article, citation practices, citation context, biomedical literature, research integrity

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-09-28, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-09-28 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.