CoChat
Sign Up
Cited by 5Academic integrity and plagiarismArtificial Intelligence in Healthcare and EducationMeta-analysis and systematic reviews

Performance of AI Tools in Citing Retracted Literature : Content Analysis

Open access
Labenbacher, Sebastian, Niederer, Maximilian, Hammer, Sascha, Bader, Matthias, Schreiber, Nikolaus, Bornemann-Cimenti, Helmar·2026·Journal of Medical Internet Research
DOI 10.2196/88766PMC PMC13134821
Open this paper in CoChat — free account

Abstract

Background: Generative artificial intelligence (GenAI) tools are increasingly used in scientific research to support literature searches, evidence synthesis, and manuscript preparation. While these systems promise substantial efficiency gains, concerns have emerged regarding their reliability, particularly their tendency to cite inaccurate, fabricated, or retracted literature. The unrecognized inclusion of retracted studies poses a serious risk to research integrity and evidence-based decision-making. Whether commonly used GenAI tools can reliably detect, exclude, or transparently communicate the retraction status of scientific publications remains unclear. Objective: This study aimed to evaluate the ability of freely available GenAI tools to correctly handle retracted scientific articles during literature searches. Primary and secondary outcomes focused on accuracy, reliability, and consistency in recognizing retracted literature. Methods: In this pragmatic trial, nine widely used free-access GenAI tools (ChatGPT 4, ChatGPT 5, Claude, Gemini, Perplexity, Microsoft Copilot, SciSpace, ScienceOS, and Consensus) were evaluated. Each tool was asked five predefined, standardized questions addressing topic overview, article identification, article summarization, and explicit assessment of retraction status. Overall, 15 retracted articles (the 10 most cited and 5 most recently retracted as of May 23, 2025) were selected from the Retraction Watch database. All questions were repeated twice to assess intratool consistency. Responses were independently rated as correct or incorrect by 2 researchers. Descriptive statistics summarized performance, and comparisons between general-purpose and research-focused AI tools were conducted using descriptive statistics. Interreviewer agreement was assessed using Cohen kappa coefficient. Results: None of the evaluated AI tools consistently handled retracted articles correctly. No model achieved perfect accuracy across all question sets. ChatGPT 5 performed best, defined by the primary outcome of achieving fully correct responses to all five predefined tasks (5/5) for the highest number of retracted articles, correctly answering all five questions for 8 of 15 articles (53.3%). Research-focused tools (SciSpace, ScienceOS, and Consensus) failed to produce a single fully correct response set. Retracted articles were frequently included in topic overviews without warning, with error rates exceeding 40% in several tools. When specifically asked about retraction status, most systems failed to provide correct or complete information. OpenEvidence only reported data for a subset of our retracted articles as it is only used in health care literature. It demonstrated strong performance in topic overviews but low accuracy in identifying retracted articles. Conclusions: Freely available GenAI tools are currently not able to detect, exclude, or appropriately flag retracted scientific literature. The widespread and confident reproduction of retracted studies represents a substantial threat to research integrity, particularly in medical and evidence-based fields. Until retraction-aware verification mechanisms are systematically integrated, independent source checking remains essential when using AI-assisted literature tools.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.2196/88766

Cites

3 of 3 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Generative AI chatbots may use and surface references to scientific research without disclosing their retracted status, including references to Riordan et al. and Patil et al.

    “While these chatbots can be trained to identify and warn users about retracted articles, they may also use and surface references to scientific research without disclosing their retracted status.9,30,31 This includes references to Riordan et al. and Patil et al. without disclosing that they have been retracted (see Figures 1-3).”

    From Why retracted articles can have an influential afterlife—and what to do about it (2026) · cites this paper as [31]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

1 retracted

Flagged first, then by citations

Research works printed but not linked: 28

  1. [3]Dwivedi YK Kshetri N Hughes L et al Opinion Paper: “So what if ChatGPT wrote it?” Multidisciplinary perspectives on opportunities, challenges and implications of generative conversational AI for research, practice and policy Int J Inf Manage Aug 2023 71 102642 doi 10.1016/j.ijinfomgt.2023.102642 · DOI · Paper page
  2. [4]Dergaa I Chamari K Zmijewski P Ben Saad H From human writing to artificial intelligence generated text: examining the prospects and potential threats of ChatGPT in academic writing Biol Sport Apr 2023 40 2 615 622 doi 10.5114/biolsport.2023.125623 Medline 37077800 PMC10108763 · DOI · Paper page
  3. [5]de la Torre-López J Ramírez A Romero JR Artificial intelligence to automate the systematic review of scientific literature Computing Oct 2023 105 10 2171 2194 doi 10.1007/s00607-023-01181-x · DOI · Paper page
  4. [6]Ge L Agrawal R Singer M et al Leveraging artificial intelligence to enhance systematic reviews in health research: advanced tools and challenges Syst Rev Oct 25 2024 13 1 269 doi 10.1186/s13643-024-02682-2 Medline 39456077 PMC11504244 · DOI · Paper page
  5. [7]Khlaif ZN Mousa A Hattab MK et al The potential and concerns of using AI in scientific research: ChatGPT performance evaluation JMIR Med Educ Sep 14 2023 9 e47049 doi 10.2196/47049 Medline 37707884 PMC10636627 · DOI · Paper page
  6. [8]Májovský M Černý M Kasal M Komarc M Netuka D Artificial intelligence can generate fraudulent but authentic-looking scientific medical articles: pandora’s box has been opened J Med Internet Res 05 31 2023 25 e46924 doi 10.2196/46924 Medline 37256685 PMC10267787 · DOI · Paper page
  7. [9]Montalbano P Ethical and legal considerations of medical artificial intelligence Prim Care Dec 2025 52 4 769 779 doi 10.1016/j.pop.2025.07.009 Medline 41110915 · DOI
  8. [10]Jan R Examining the reliability of ChatGPT: identifying retracted scientific literature and ensuring accurate citations and references Impacts of Generative AI on the Future of Research and Education IGI Global Scientific Publishing 2024 367 392 doi 10.4018/979-8-3693-0884-4.ch014 · DOI · Paper page
  9. [11]Gu T Feng H Li M Gu W Wang G Alarm: retracted articles on cancer imaging are not only continuously cited by publications but also used by ChatGPT to answer questions J Adv Res 05 2025 71 1 3 doi 10.1016/j.jare.2025.03.020 Medline 40086628 PMC12126723 · DOI
  10. [12]Kocyigit BF Okyay RA Seiil B Qumar AB Sumbul HE Analysis of retracted publications on artificial intelligence: trends, ethical concerns, and scientific integrity J Korean Med Sci Nov 17 2025 40 44 e280 doi 10.3346/jkms.2025.40.e280 Medline 41250649 PMC12624210 · DOI
  11. [13]Bakker C Boughton S Faggion CM Fanelli D Kaiser K Schneider J Reducing the residue of retractions in evidence synthesis: ways to minimise inappropriate citation and use of retracted data BMJ Evid Based Med Mar 21 2024 29 2 121 126 doi 10.1136/bmjebm-2022-111921 Medline 37463764 PMC10982619 · DOI · Paper page
  12. [14]AI-assisted knowledge gathering and its pitfalls with retracted articles Open Science Framework (OSF) URL 10.17605/OSF.IO/B6J2W Accessed 13-04-2026 · DOI
  13. [15]Bohsas H Alibrahim H Swed S et al Prevalence and knowledge of polycystic ovary syndrome (PCOS) and health-related practices among women of Syria: a cross-sectional study J Psychosom Obstet Gynaecol Dec 2024 45 1 2318194 doi 10.1080/0167482X.2024.2318194 Medline 38635351 · DOI
  14. [16]Estruch R Ros E Salas-Salvadó J et al Primary prevention of cardiovascular disease with a Mediterranean diet N Engl J Med Apr 4 2013 368 14 1279 1290 doi 10.1056/NEJMoa1200303 Medline 23432189 · DOI · Paper page
  15. [17]Fortuna F Gonzalez D Fritzler A et al Burnout components, perceived stress and hair cortisol in healthcare professionals during the second wave of COVID 19 pandemic Sci Rep Nov 21 2024 14 1 28828 doi 10.1038/s41598-024-79925-8 Medline 39572609 PMC11582650 · DOI
  16. [18]Fukuhara A Matsuda M Nishizawa M et al RETRACTED: Visfatin: a protein secreted by visceral fat that mimics the effects of insulin Science 01 21 2005 307 5708 426 430 doi 10.1126/science.1097243 Medline 15604363 · DOI · Paper page
  17. [19]Gautret P Lagier JC Parola P et al RETRACTED: Hydroxychloroquine and azithromycin as a treatment of COVID-19: results of an open-label non-randomized clinical trial Int J Antimicrob Agents 07 2020 56 1 105949 doi 10.1016/j.ijantimicag.2020.105949 Medline 32205204 PMC7102549 · DOI
  18. [20]Huang C Huang L Wang Y et al 6-month consequences of COVID-19 in patients discharged from hospital: a cohort study Lancet 01 16 2021 397 10270 220 232 doi 10.1016/S0140-6736(20)32656-8 Medline 33428867 PMC7833295 · DOI · Paper page
  19. [21]Jiang Y Jahagirdar BN Reinhardt RL et al Pluripotency of mesenchymal stem cells derived from adult marrow Nature New Biol 07 4 2002 418 6893 41 49 doi 10.1038/nature00870 Medline 12077603 · DOI
  20. [23]Mohammadniaei M Zhang M Ashley J et al A non-enzymatic, isothermal strand displacement and amplification assay for rapid detection of SARS-CoV-2 RNA Nat Commun Aug 24 2021 12 1 5089 doi 10.1038/s41467-021-25387-9 Medline 34429424 PMC8385016 · DOI
  21. [24]Morisky DE Ang A Krousel-Wood M Ward HJ Predictive validity of a medication adherence measure in an outpatient setting J Clin Hypertens (Greenwich) 05 2008 10 5 348 354 doi 10.1111/j.1751-7176.2008.07572.x Medline 18453793 PMC2562622 · DOI · Paper page
  22. [25]Papoutsis A Borody T Dolai S et al Detection of SARS-CoV-2 from patient fecal samples by whole genome sequencing Gut Pathog 01 30 2021 13 1 7 doi 10.1186/s13099-021-00398-5 Medline 33516247 PMC7846909 · DOI
  23. [26]Pearton SJ Norton DP Ip K Heo YW Steiner T RETRACTED: recent progress in processing and properties of ZnO Prog Mater Sci Mar 2005 50 3 293 340 doi 10.1016/j.pmatsci.2004.04.001 · DOI
  24. [27]Shi Y Ye P Wang C Yang K Guo J RETRACTED: surface hydrophilic modification for chip of centrifugal microfluidic immunoassay system Micromachines (Basel) 05 26 2022 13 6 831 doi 10.3390/mi13060831 Medline 35744444 PMC9229454 · DOI
  25. [28]Taylor DD Gercel-Taylor C MicroRNA signatures of tumor-derived exosomes as diagnostic biomarkers of ovarian cancer Gynecol Oncol 07 2008 110 1 13 21 doi 10.1016/j.ygyno.2008.04.033 Medline 18589210 · DOI
  26. [29]Wakefield A Murch S Anthony A et al RETRACTED: Ileal-lymphoid-nodular hyperplasia, non-specific colitis, and pervasive developmental disorder in children The Lancet Feb 1998 351 9103 637 641 doi 10.1016/S0140-6736(97)11096-0 9500320 · DOI · Paper page
  27. [32]Teixeira da Silva JA Santos‐d’Amorim K Bornemann‐Cimenti H The citation of retracted papers and impact on the integrity of the scientific biomedical literature Learn Publ Apr 2025 38 2 e1667 doi 10.1002/leap.1667 · DOI
  28. [33]AllahRakha N UNESCO’s AI ethics principles: challenges and opportunities Irshad J Law and Policy 2024 2 9 24 36 doi 10.59022/ijlp.225 · DOI

Cited by

Sorted by citations

Authors
Labenbacher, S., Niederer, M., Hammer, S., Bader, M., Schreiber, N., & Bornemann-Cimenti, H.
Affiliations
Medical University of Graz (AT)
Corresponding author
Hammer, Sascha
ORCID
Venue
Journal of Medical Internet Research
Publisher
JMIR Publications Inc.
Published
2026-05-01
Volume
28
Pages
e88766-e88766
ISSN
1438-8871 (ISSN-L)
DOI
10.2196/88766
PMC
PMC13134821
OpenAlex
W7142031956
Cited by
5
References
3
Citations by year
2026: 5
FWCI
28.37
Type
journal article
Language
en
Open access
Licence
cc-by
Locations
  • Journal of Medical Internet Research (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
Topics
  • Social Sciences › Safety Research › Academic integrity and plagiarism
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
Keywords
generative AI, AI-assisted literature searching, research integrity, ChatGPT, Claude, Gemini, OpenEvidence, AI reliability, Retraction Watch
MeSH terms
Generative Artificial Intelligence, Artificial Intelligence, Humans, Retraction of Publication as Topic, Reproducibility of Results

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review