Hallucination Rates and Reference Accuracy of ChatGPT and Bard for Systematic Reviews: Comparative Analysis
Open accessPublished version of a preprint
The DOI matches its Crossref record; 35 of the 35 references it cites were found in a registry or an index; 2 of the 2 sources its claims cite were checked for support.
Abstract
Background Large language models (LLMs) have raised both interest and concern in the academic community. They offer the potential for automating literature search and synthesis for systematic reviews but raise concerns regarding their reliability, as the tendency to generate unsupported (hallucinated) content persist. Objective The aim of the study is to assess the performance of LLMs such as ChatGPT and Bard (subsequently rebranded Gemini) to produce references in the context of scientific writing. Methods The performance of ChatGPT and Bard in replicating the results of human-conducted systematic reviews was assessed. Using systematic reviews pertaining to shoulder rotator cuff pathology, these LLMs were tested by providing the same inclusion criteria and comparing the results with original systematic review references, serving as gold standards. The study used 3 key performance metrics: recall, precision, and F1-score, alongside the hallucination rate. Papers were considered “hallucinated” if any 2 of the following information were wrong: title, first author, or year of publication. Results In total, 11 systematic reviews across 4 fields yielded 33 prompts to LLMs (3 LLMs×11 reviews), with 471 references analyzed. Precision rates for GPT-3.5, GPT-4, and Bard were 9.4% (13/139), 13.4% (16/119), and 0% (0/104) respectively (P<.001). Recall rates were 11.9% (13/109) for GPT-3.5 and 13.7% (15/109) for GPT-4, with Bard failing to retrieve any relevant papers (P<.001). Hallucination rates stood at 39.6% (55/139) for GPT-3.5, 28.6% (34/119) for GPT-4, and 91.4% (95/104) for Bard (P<.001). Further analysis of nonhallucinated papers retrieved by GPT models revealed significant differences in identifying various criteria, such as randomized studies, participant criteria, and intervention criteria. The study also noted the geographical and open-access biases in the papers retrieved by the LLMs. Conclusions Given their current performance, it is not recommended for LLMs to be deployed as the primary or exclusive tool for conducting systematic reviews. Any references generated by such models warrant thorough validation by researchers. The high occurrence of hallucinations in LLMs highlights the necessity for refining their training and functionality before confidently using them for rigorous academic purposes.
Overview
CoChat Summary
## TL;DR
This comparative analysis tested ChatGPT and Bard on 11 systematic reviews of shoulder rotator cuff pathology and found low retrieval accuracy plus frequent hallucinated references. GPT-4 performed best of the models tested, but the authors conclude that LLM-generated citations still require rigorous human validation and should not be used alone for systematic reviews.
## Key findings
- Across 11 systematic reviews in 4 fields, the study analyzed 471 references from 33 LLM prompts (3 models × 11 reviews).
- Precision was 9.4% (13/139) for GPT-3.5, 13.4% (16/119) for GPT-4, and 0% (0/104) for Bard (P<.001); recall was 11.9% (13/109) for GPT-3.5, 13.7% (15/109) for GPT-4, and 0 for Bard (P<.001).
- Hallucination rates were 39.6% (55/139) for GPT-3.5, 28.6% (34/119) for GPT-4, and 91.4% (95/104) for Bard (P<.001).
- Among nonhallucinated GPT-retrieved papers, title accuracy was very high (83/84, 99% for GPT-3.5 and 84/85, 99% for GPT-4), but DOI accuracy was low (13/82, 16% and 17/84, 20%); American authors were overrepresented in ChatGPT results (44% vs 16.5% in the original reviews for GPT-3.5, and 33% vs 16.5% for GPT-4; P<.001).
## Methods
- Sequential comparative design: PubMed search on July 27, 2023 identified English-language 2020 systematic reviews/meta-analyses on randomized clinical trials in shoulder rotator cuff pathology; 11 systematic reviews were selected after title/abstract/full-text screening and validation with PRISMA/PROSPERO.
- The LLMs tested were GPT-3.5, GPT-4, and Bard (PaLM 2.0); each fresh chat session was prompted with the same inclusion criteria, and two prompt versions were tried, keeping the one that produced more results.
- Outputs were compared with the original systematic review references using recall, precision, F1-score, and hallucination rate; papers were labeled hallucinated if any 2 of title, first author, or year were wrong.
## Limitations
- The authors state the study was restricted to shoulder rotator cuff pathology and to 3 LLMs, so the findings may not generalize to other specialties or models.
- The paper also notes that no established guidelines exist for using LLMs to optimize accuracy, and different prompts might yield different results.
- Evident from the text: the study focused on a narrow, single-topic set of reviews from one year (2020) and English-language publications only, which limits breadth and may constrain generalizability.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures






Verification
Claims the paper makes
1 Valid · 0 Not valid · 0 Not checked · 40 Own results and interpretations
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Papers from American authors were selected in 16.5% (18/109) of original systematic reviews, 44% (37/84) of GPT-3.5 papers, and 33% (28/85) of GPT-4 papers (P<.001).”
§ Hallucination Rates and Reference Accuracy of ChatGPT and Bard for Systematic Reviews: Comparative Analysis > Results
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In most cases, the number of references returned by LLMs was greater or equal to that of the original papers (Table 2).”
§ Hallucination Rates and Reference Accuracy of ChatGPT and Bard for Systematic Reviews: Comparative Analysis > Results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Papers identified by LLMs were present in the original systematic reviews (precision) in 9.4% (13/139), 13.4% (16/119), and 0% (0/104) of cases for GPT-3.5, GPT-4, and Bard (P<.001), respectively.”
§ Hallucination Rates and Reference Accuracy of ChatGPT and Bard for Systematic Reviews: Comparative Analysis > Results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“authors list (73/84, 87% vs 74/85, 87%; P>.99)”
§ Hallucination Rates and Reference Accuracy of ChatGPT and Bard for Systematic Reviews: Comparative Analysis > Results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“randomized studies (33/84, 39% vs 42/85, 49%; P=.24)”
- Also stated in: § Principal Findings · plain
“For instance, the criterion of “randomized study” was adhered to in only 39% (33/84) to 49% (42/85) of nonhallucinated papers generated by ChatGPT, even when the term “randomized” appeared in the title or abstract of the papers from the original systematic reviews.”
§ Hallucination Rates and Reference Accuracy of ChatGPT and Bard for Systematic Reviews: Comparative Analysis > Results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
- Also stated in: § Principal Findings · plain
18 more claims
Create a free account to see all 41 claims and every source verdict.
- It may state two separate claims as one. 12
- Probably not a substantial claim. 9
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“Hallucination rates have been documented to vary between 3% – 28% in current literature [30, 31].”
From Artificial Intelligence (AI) – Powered Documentation Systems in Healthcare: A Systematic Review (2025) · cites this paper as [30]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Studies in radiology, psychiatry, and clinical rehabilitation have reported fabrication rates ranging from 20 to 70%”
From How LLMs Cite and Why It Matters: A Cross-Model Audit of Reference Fabrication in AI-Assisted Academic Writing and Methods to Detect Phantom Citations (2026) · cites this paper as [12]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Experimental studies from 2023 to 2025 have found that the rate of hallucination in LLMgenerated citations ranged from 19.9% to 91.4%, varying widely across models and time [16], [17], [18], [19], [24].”
From LLM hallucinations in the wild: Large-scale evidence from non-existent citations (2026) · cites this paper as [18]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“When citations are missing, inaccurate, or fabricated, the resulting break in the evidence chain can obscure logical dependencies, weaken argumentation, and jeopardize academic integrity at the level of both individual papers and the broader literature [10, 22, 26].”
From CiteAudit: You Cited It, But Did You Read It? A Benchmark for Verifying Scientific References in the LLM Era (2026) · cites this paper as [10]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Chelli et al. found that Bard produced incorrect references in 91.4% of systematic review prompts, whereas GPT-4 had a lower but still notable error rate of 28.6%18.”
From Multi-model assurance analysis showing large language models are highly vulnerable to adversarial hallucination attacks during clinical decision support (2025) · cites this paper as [18]
Experimental — accuracy not yet measured
16 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gotzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · BMJ · doi:10.1136/bmj.b2700 · Cited by 17,648
Artificial hallucinations in ChatGPT: implications in scientific writing
Alkaissi, H., & McFarlane, S. I. (2023) · Cureus · doi:10.7759/cureus.35179 · Cited by 1,540 · cited 1×
“This concern has been broached in previous literature [26-29], but our study provides an experimental design to probe the matter more deeply.”
BackgroundThe nuts and bolts of PROSPERO: an international prospective register of systematic reviews
Booth, A., Clarke, M., Dooley, G., Ghersi, D., Moher, D., Petticrew, M., & Stewart, L. (2012) · Systematic Reviews · doi:10.1186/2046-4053-1-2 · Cited by 1,433 · cited 1×
“Additionally, the registration status of these reviews was verified in the PROSPERO database [14].”
BackgroundChatting about ChatGPT: how may AI and GPT impact academia and libraries?
Lund, B. D., & Wang, T. (2023) · Library Hi Tech News · doi:10.1108/lhtn-01-2023-0009 · Cited by 1,355
ChatGPT listed as author on research papers: many scientists disapprove
Stokel-Walker, C. (2023) · Nature · doi:10.1038/d41586-023-00107-z · Cited by 868 · cited 1×
“While it is known that ChatGPT can help researchers write papers [3-5], controversy exists about whether it should be used at all, whether its use should be disclosed, and whether it should be listed as an author or not [6].”
BackgroundCan artificial intelligence help for scientific writing?
Salvagno, M., Taccone, F. S., & Gerli, A. G. (2023) · Critical Care · doi:10.1186/s13054-023-04380-2 · Cited by 827
Abstracts written by ChatGPT fool scientists
Else, H. (2023) · Nature · doi:10.1038/d41586-023-00056-7 · Cited by 611
Boileau, P., Baqué, F., Valerio, L., Ahrens, P., Chuinard, C., & Trojani, C. (2007) · The Journal of Bone and Joint Surgery-American Volume · doi:10.2106/00004623-200704000-00008 · Cited by 603
Self-citationChatGPT and the future of medical writing
Biswas, S. (2023) · Radiology · doi:10.1148/radiol.223312 · Cited by 571
Learning to summarize from human feedback
Stiennon, N., Ouyang, L., Wu, J., Ziegler, D. M., Lowe, R., Voss, C., Radford, A., Amodei, D., & Christiano, P. (2020) · arXiv · doi:10.48550/arxiv.2009.01325 · Cited by 550
Cited by
Promises and challenges of generative artificial intelligence for human learning
Yan, L., Greiff, S., Teuber, Z., & Gašević, D. (2024) · Nature Human Behaviour · doi:10.1038/s41562-024-02004-5 · Cited by 499
“GenAI may perform reasonably well with generic questions (for example, what are Newton’s laws of motion?) but is more prone to errors with nuanced, contextspecific, timesensitive or highly technical information 73 .”
Omar, M., Sorin, V., Collins, J. D., Reich, D., Freeman, R., Gavin, N., Charney, A., Stump, L., Bragazzi, N. L., Nadkarni, G. N., et al. (2025) · Communications Medicine · doi:10.1038/s43856-025-01021-3 · Cited by 204
Scherbakov, D., Hubig, N., Jansari, V., Bakumenko, A., & Lenert, L. A. (2025) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaf063 · Cited by 154
Artificial Intelligence (AI) – Powered Documentation Systems in Healthcare: A Systematic Review
Bracken, A., Reilly, C., Feeley, A., Sheehan, E., Merghani, K., & Feeley, I. (2025) · Journal of Medical Systems · doi:10.1007/s10916-025-02157-4 · Cited by 113
“Hallucination rates have been documented to vary between 3% – 28% in current literature [30, 31].”
AI hallucinations can’t be stopped — but these techniques can limit their damage
Jones, N. (2025) · Nature · doi:10.1038/d41586-025-00068-5 · Cited by 117
Accelerating clinical evidence synthesis with large language models
Wang, Z., Cao, L., Danek, B., Jin, Q., Lu, Z., & Sun, J. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01840-7 · Cited by 69
AI Scribes in Health Care: Balancing Transformative Potential With Responsible Integration
Leung, T. I., Coristine, A. J., & Benis, A. (2025) · JMIR Medical Informatics · doi:10.2196/80898 · Cited by 86
Retrieval-augmented generation elevates local LLM quality in radiology contrast media consultation
Wada, A., Tanaka, Y., Nishizawa, M., Yamamoto, A., Akashi, T., Hagiwara, A., Hayakawa, Y. K., Kikuta, J., Shimoji, K., Sano, K., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01802-z · Cited by 70
Low, Y. S., Jackson, M. L., Hyde, R. J., Brown, R. E., Sanghavi, N. M., Baldwin, J. D., Pike, C. W., Muralidharan, J., Hui, G., Alexander, N., et al. (2025) · DIGITAL HEALTH · doi:10.1177/20552076251348850 · Cited by 51
“However, they are prone to hallucinating reference materials or treatment guidelines 17,18 and may produce “ recommendations ” that are satir-ical, inappropriate, 19 or misaligned with evidence-based guidelines.”
The Current Challenges of Software Engineering in the Era of Large Language Models
Gao, C., Hu, X., Gao, S., Xia, X., & Jin, Z. G. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3712005 · Cited by 123
Digital Doppelgängers and Lifespan Extension: What Matters?
Iglesias, S., Earp, B. D., Voinea, C., Mann, S. P., Zahiu, A., Jecker, N. S., & Savulescu, J. (2024) · The American Journal of Bioethics · doi:10.1080/15265161.2024.2416133 · Cited by 59
Peykani, P., Ramezanlou, F., Tănăsescu, C. R., & Ghanidel, S. (2025) · Applied Sciences · doi:10.3390/app15148103 · Cited by 83
Urban, M., Brom, C., Lukavský, J., Děchtěrenko, F., Hein, V., Svacha, F., Kmoníčková, P., & Urban, K. (2025) · British Journal of Educational Technology · doi:10.1111/bjet.13591 · Cited by 47
Mansoor, I., Abdullah, M., Rizwan, M. D., & Fraz, M. M. (2025) · Health Information Science and Systems · doi:10.1007/s13755-025-00403-0 · Cited by 51
When and how to disclose AI use in academic publishing: AMEE Guide No.192
Cleland, J., Driessen, E. W., Masters, K., Lingard, L., & Maggio, L. A. (2025) · Medical Teacher · doi:10.1080/0142159x.2025.2607513 · Cited by 58
Data Quality in the Age of AI: A Review of Governance, Ethics, and the FAIR Principles
Guillen-Aguinaga, M., Aguinaga-Ontoso, E., Guillen-Aguinaga, L., Guillen-Grima, F., & Aguinaga-Ontoso, I. (2025) · Data · doi:10.3390/data10120201 · Cited by 37
“Hallucinations—confident but factually incorrect outputs—arise from inadequate governance during training and evaluation, often resulting from misuse, weak security, or poor-quality inputs [262–267].”
The chat-chamber effect: Trusting the AI hallucination
Jacob, C., Kerrigan, P., & Bastos, M. (2025) · Big Data & Society · doi:10.1177/20539517241306345 · Cited by 22
Generative artificial intelligence and academic writing: friend or foe?
Cohen, J. F., & Moher, D. (2024) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2024.111646 · Cited by 36
Ivanisenko, T., Сергеевич, Д. П., & Иванисенко, В. А. (2024) · International Journal of Molecular Sciences · doi:10.3390/ijms252111811 · Cited by 31
Bayani, A., Ewane, L. P. E., dos Anjos, D. S. O., Mac-Seing, M., & Nikiema, J. N. (2025) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2025.106048 · Cited by 23
Çelik, S. U. (2025) · The American Journal of Surgery · doi:10.1016/j.amjsurg.2025.116657 · Cited by 34
Jagged competencies: Measuring the reliability of generative AI in academic research
Thomas, L. D. W., Romasanta, A. K. S., & Priego, L. P. (2025) · Journal of Business Research · doi:10.1016/j.jbusres.2025.115804 · Cited by 28
AI Tools in Academia: Evaluating NotebookLM as a Tool for Conducting Literature Reviews
Shor, R., Greene, E. A., Sumberg, L. L., & Weingrad, A. B. (2025) · Psychiatry · doi:10.1080/00332747.2025.2541531 · Cited by 30
Adam, G. P., DeYoung, J., Paul, A., Saldanha, I. J., Balk, E. M., Trikalinos, T. A., & Wallace, B. (2024) · JAMIA Open · doi:10.1093/jamiaopen/ooae098 · Cited by 27
Tomczyk, P., Brüggemann, P., Mergner, N., & Petrescu, M. (2026) · Journal of Librarianship and Information Science · doi:10.1177/09610006241295802 · Cited by 17
Li, H., Huang, J., Liu, K., Liu, J., Liu, Q., Zhou, Z., Zong, Z., & Mao, S. (2025) · European Journal of Surgical Oncology · doi:10.1016/j.ejso.2025.110096 · Cited by 19
Güneşli, I., Aksun, S., Fathelbab, J., & Yıldız, B. O. (2024) · Endocrine · doi:10.1007/s12020-024-04121-7 · Cited by 21
Detecting and Correcting Reference Hallucinations in Commercial LLMs and Deep Research Agents
Rao, D., Wong, E., & Callison-Burch, C. (2026) · arXiv · doi:10.48550/arxiv.2604.03173 · Cited by 16
Rashid, M., Yi, C. S., Sathapanasiri, T., Udayachalerm, S., Boonpattharatthiti, K., Insuk, S., Veettil, S. K., Lai, N. M., Chaiyakunapruk, N., Dhippayom, T., et al. (2025) · Value in Health · doi:10.1016/j.jval.2025.07.001 · Cited by 24
Lee, Y., Oh, J. H., Lee, D., Kang, M., & Lee, S. (2025) · Scientific Reports · doi:10.1038/s41598-025-99423-9 · Cited by 18
Camlet, A., Kusiak, A., Ossowska, A., & Świetlik, D. (2025) · Diagnostics · doi:10.3390/diagnostics15151851 · Cited by 19
Boltaboyeva, A., Baigarayeva, Z., Иманбек, Б., Ozhikenov, К., Getahun, A. J., Aidarova, T., & Karymsakova, N. (2025) · Applied Sciences · doi:10.3390/app15126840 · Cited by 23
Prompting for creative problem-solving: A process-mining study
Urban, M., Lukavský, J., Brom, C., Hein, V., Svacha, F., Děchtěrenko, F., & Urban, K. (2025) · Learning and Instruction · doi:10.1016/j.learninstruc.2025.102156 · Cited by 21
Chen, J., Mokmin, N. A. M., & Qi, S. (2025) · The Journal of Educational Research · doi:10.1080/00220671.2025.2510395 · Cited by 15
Ng, J. Y. (2025) · Integrative Medicine Research · doi:10.1016/j.imr.2025.101222 · Cited by 21
Tres escenarios para la IA en educación: del apoyo responsable a la cocreación
García‐Peñalvo, F. J. (2025) · Education in the Knowledge Society (EKS) · doi:10.14201/eks.32932 · Cited by 19
Gilvaz, V. J., Sudheer, A., & Reginato, A. M. (2025) · Current Rheumatology Reports · doi:10.1007/s11926-025-01193-w · Cited by 15
Ostrovsky, A. M. (2025) · The American Journal of Emergency Medicine · doi:10.1016/j.ajem.2025.03.060 · Cited by 18
Mun, I. B., & Hwang, K.-H. (2025) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2025.2480823 · Cited by 21
Zhang, D., Ma, Z., Gong, R., Lian, L., Li, Y., He, Z., Han, Y., Hui, J., Huang, J., Jiang, J., et al. (2024) · Journal of Medical Internet Research · doi:10.2196/58741 · Cited by 19
Omar, M., Sorin, V., Collins, J. D., Reich, D., Freeman, R., Gavin, N. P., Charney, A. W., Stump, L. S., Bragazzi, N. L., Nadkarni, G. N., et al. (2025) · medRxiv · doi:10.1101/2025.03.18.25324184 · Cited by 13
When LLMs Hallucinate: Examining the Effects of Erroneous Feedback in Math Tutoring Systems
Steinbach, M., Bhandari, S., Meyer, J., & Pardos, Z. A. (2025) · Proceedings of the Twelfth ACM Conference on Learning @ Scale · doi:10.1145/3698205.3729555 · Cited by 12
LITERAS: Biomedical literature review and citation retrieval agents
Gorenshtein, A., Shihada, K., Sorka, M., Aran, D., & Shelly, S. (2025) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2025.110363 · Cited by 16
Middle school students’ perspectives on adopting generative AI in K-12 education
Adisa, I. O., & Adefisayo, A. O. (2025) · The Journal of Educational Research · doi:10.1080/00220671.2025.2510393 · Cited by 13
KULA, B. İ., Kula, A., Bağcıer, F., & Alyanak, B. (2024) · The Korean Journal of Orthodontics · doi:10.4041/kjod24.106 · Cited by 16
Liu, Y., Li, H., Ouyang, J., Xue, Z., Wang, M., He, H., Song, B., Zheng, X., & Gan, W. (2025) · JMIR Perioperative Medicine · doi:10.2196/70047 · Cited by 16
Using AI to generate formative feedback in doctoral education
Tensen, D., Grainger, P., & Graham, W. (2025) · Assessment & Evaluation in Higher Education · doi:10.1080/02602938.2025.2536558 · Cited by 19
Campos, V. M. S., Prudente, T. P., Leão, L. L., Costa, M., Oliva, H. N. P., & Monteiro-Júnior, R. S. (2025) · Health Information Science and Systems · doi:10.1007/s13755-025-00368-0 · Cited by 14
Hack, S., Karni, R. J., Maniaci, A., Fundakowski, C. E., Castellani, L., Incandela, F., Accorona, R., Mayo-Yanez, M., Violati, M., Giannini, L., et al. (2026) · Oral Oncology · doi:10.1016/j.oraloncology.2026.107877 · Cited by 10
Pharmacometrics in the Age of Large Language Models: A Vision of the Future
Tosca, E. M., Aiello, L., Carlo, A. D., & Magni, P. (2025) · Pharmaceutics · doi:10.3390/pharmaceutics17101274 · Cited by 12
Dong, Y., Zhang, Z., Zhi, Y., Li, X., Guo, T., He, L., Zhao, S., Yang, X., Tang, J. T., Zhong, W., et al. (2025) · JHEP Reports · doi:10.1016/j.jhepr.2025.101579 · Cited by 13
Hallucination in AI-generated financial literature reviews: evaluating bibliographic accuracy
Erdem, O., Hassett, K., & Egriboyun, F. (2025) · International Journal of Data Science and Analytics · doi:10.1007/s41060-025-00731-0 · Cited by 10
Linardon, J., Messer, M., Anderson, C., Liu, C., McClure, Z., Jarman, H. K., Goldberg, S. B., & Torous, J. (2025) · BMJ Mental Health · doi:10.1136/bmjment-2025-301787 · Cited by 10
“20 These findings also raise concerns about whether less experienced researchers, who appear to use LLMs more frequently, are adequately equipped to detect and correct errors generated by these tools.”
Cinquin, O. (2024) · Briefings in Bioinformatics · doi:10.1093/bib/bbaf045 · Cited by 12
Large Language Models for Transforming Healthcare: A Perspective on DeepSeek‐R1
Zhou, J., Cheng, Y., He, S., Chen, Y.-C., & Chen, H. (2025) · MedComm – Future Medicine · doi:10.1002/mef2.70021 · Cited by 16
Performance of ChatGPT in Pediatric Audiology as Rated by Students and Experts
Ratuszniak, A., Gos, E., Lorens, A., Skarzynski, P. H., Skarżyńśki, H., & Jędrzejczak, W. W. (2025) · Journal of Clinical Medicine · doi:10.3390/jcm14030875 · Cited by 11
Mija, D., Kehlet, H., Rosero, E. B., & Joshi, G. P. (2024) · British Journal of Anaesthesia · doi:10.1016/j.bja.2024.09.010 · Cited by 9
Jain, A., Nimonkar, P., & Jadhav, P. (2025) · Journal of Cranio-Maxillofacial Surgery · doi:10.1016/j.jcms.2025.08.004 · Cited by 10
How Well Do ChatGPT and Claude Perform in Study Selection for Systematic Review in Obstetrics
Insuk, S., Boonpattharatthiti, K., Booncharoen, C., Chaipitak, P., Rashid, M., Veettil, S. K., Lai, N. M., Chaiyakunapruk, N., & Dhippayom, T. (2025) · Journal of Medical Systems · doi:10.1007/s10916-025-02246-4 · Cited by 15
Is your curriculum GenAI-proof? A method for GenAI impact assessment and a case study
Jongkind, R., Elings, E., Joukes, E., Broens, T., Leopold, H., Wiesman, F., & Meinema, J. G. (2025) · MedEdPublish · doi:10.12688/mep.20815.1 · Cited by 9
Šuto, J., Marušić, A., & Buljan, I. (2025) · JMIR Cancer · doi:10.2196/63347 · Cited by 14
Madsen, D. Ø., & Toston, D. M. (2025) · Digital · doi:10.3390/digital5030024 · Cited by 18
Ratnam, K. K. Y. (2025) · Digital Health · doi:10.1177/20552076251362070 · Cited by 21
Tekin, S., Oğuz, S. H., & Dağdelen, S. (2025) · Endocrine · doi:10.1007/s12020-025-04400-x · Cited by 11
Basil, M., Ahmed, W., Hajeomar, R., Strawbridge, J., Lynch, M. R., & Mukhalalati, B. (2026) · BMC Medical Education · doi:10.1186/s12909-025-08527-3 · Cited by 19
Kiyomiya, K., Aomori, T., & Ohtani, H. (2025) · Digital Health · doi:10.1177/20552076251323810 · Cited by 7
The AI Annotator: Large Language Models’ Potential in Scoring Sustainability Reports
Wu, Y., Hu, P., & Wang, D. D. (2025) · Systems · doi:10.3390/systems13100899 · Cited by 12
Exploring the use of AI authors and reviewers at Agents4Science
Bianchi, F., Queen, O., Thakkar, N., Sun, E., & Zou, J. (2025) · Nature Biotechnology · doi:10.1038/s41587-025-02963-8 · Cited by 25
Hasnain, M., Aurangzeb, K., Alhussein, M. A., Ghani, I., & Mahmood, M. H. (2025) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2025.1579375 · Cited by 6
He, Z., Zhao, L., Li, G., Wang, J., Cai, S., Tu, P., Chen, J., Wu, J., Zhang, J., Chen, R., et al. (2025) · Frontiers in Digital Health · doi:10.3389/fdgth.2025.1670510 · Cited by 8
Li, C., Jia, W.-W., Chu, Y., Menge, F., Speer, T., Reißfelder, C., Hohenberger, P., Jakob, J., & Yang, C. (2025) · Oncology Research and Treatment · doi:10.1159/000544978 · Cited by 7
Khairil, L., Benny, K. H. S., Jerry, J., Khatib, F. M., Ramli, M. D. C., & Kumar, S. (2026) · Pharmaceuticals · doi:10.3390/ph19060916 · Cited by 14
Cutting-edge AI tools revolutionizing scientific research in life sciences
Lorenc-Kukuła, K. (2025) · Biotechnologia · doi:10.5114/bta/200803 · Cited by 14
Singh, S., & Chandrasekhar, P. (2026) · Cureus · doi:10.7759/cureus.100845 · Cited by 6
Learning with, rather than through, AI: co-designing science education for critical AI literacy
Küçükuncular, A. (2025) · Frontiers in Education · doi:10.3389/feduc.2025.1716353 · Cited by 13
Xu, S., Zhao, Z., Liu, X., & Meng, X.-L. (2025) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-025-03138-w · Cited by 8
Sun, R., Tang, M., Zhou, J., Loan, N. T. T., & Wang, C.-Y. (2025) · Frontiers in Education · doi:10.3389/feduc.2025.1551721 · Cited by 17
Daraio, C. C., Leo, S. D., & Ferazzoli, F. (2025) · International Transactions in Operational Research · doi:10.1111/itor.70054 · Cited by 9
Application of Large Language Models in Complex Clinical Cases: Cross-Sectional Evaluation Study
Huang, Y., Yang, G., Shen, Y., Chen, H. D., Wu, W., Li, X., Wu, Y., Zhang, K., Xu, J., & Zhang, J. (2025) · JMIR Medical Informatics · doi:10.2196/73941 · Cited by 7
Giuliani, C., Benadi, G., Engel, F., Werner, J., Watter, M., Schwarzer, G., Groß, O., Zeiser, R., Binder, H., & Kaier, K. (2025) · JMIR Formative Research · doi:10.2196/73822 · Cited by 6
Information Extraction from Clinical Texts with Generative Pre-trained Transformer Models
Kim, M., Chung, P., Aghaeepour, N., & Kim, N. (2025) · International Journal of Medical Sciences · doi:10.7150/ijms.103332 · Cited by 8
AI, Digital Platforms, and the New Systemic Risk
Hacker, P., Edwards, L., & Kasirzadeh, A. (2026) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3805689.3806506 · Cited by 18
Evidence-Based Medicine: Past, Present, Future
Triposkiadis, F. K., & Brutsaert, D. L. (2025) · Journal of Clinical Medicine · doi:10.3390/jcm14145094 · Cited by 25
ChatGPT discovery of green image damaging information for large production companies
Thimm, H. H., & Rasmussen, K. B. (2024) · Journal of Cleaner Production · doi:10.1016/j.jclepro.2024.143978 · Cited by 6
Hooshiar, M. H. (2026) · The Journal of Prosthetic Dentistry · doi:10.1016/j.prosdent.2025.11.005 · Cited by 5
When trust collides: Exploring human-LLM cooperation intention through the prisoner’s dilemma
Jiang, G., Yang, S., Wang, Y., & Hui, P. (2026) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2026.103740 · Cited by 22
Editorial policies for use and acknowledgment of artificial intelligence in dental journals
de Queiroz, A. B. L., Sartori, L. R. M., Lima, G. D. S., & Moraes, R. R. (2025) · Journal of Dentistry · doi:10.1016/j.jdent.2025.105923 · Cited by 8
Ren, X., & Wei, Q. (2024) · Software · doi:10.3390/software3030018 · Cited by 9
Chigwada, J., & Ngulube, P. (2026) · Frontiers in Research Metrics and Analytics · doi:10.3389/frma.2026.1740510 · Cited by 5
Chérrez-Ojeda, I., Zuberbier, T., Rodas‐Valero, G., Sanchez, J., Rudenko, M., Dramburg, S., Demoly, P. M., Caimmi, D. P., Gómez, R. M., Ramon, G. D., et al. (2025) · Clinical and Translational Allergy · doi:10.1002/clt2.70130 · Cited by 11
Srikandabala, K., Prabagar, K., Jayatilleke, S., Zhang, P., Ellerbrock, R., Rinas, S., Silva, D. D., & Alahakoon, D. (2025) · IEEE Access · doi:10.1109/access.2025.3593523 · Cited by 13
Co-intelligent Design of Catalysis Research with Large Language Models: Hype or Reality?
Balcells, D. (2025) · ACS Catalysis · doi:10.1021/acscatal.5c04238 · Cited by 6
C-PATH: Conversational Patient Assistance and Triage in Healthcare System
Shi, Q. F., Han, Q., & Soares, C. (2025) · IEEE International Conference on Digital Health (ICDH) · doi:10.1109/icdh67620.2025.00016 · Cited by 7
Trewren, T., Fitzgerald, N. M., Jaensch, S., Nguyen, O., Tsymbal, A., Gao, C., Stretton, B., Anderson, S., Lin, D., Winterton, D., et al. (2025) · Journal of Perioperative Practice · doi:10.1177/17504589251346634 · Cited by 5
Evolving Consultation: Enhancing Ophthalmic Diagnostic Performance Using Large Language Model
Inooka, T., Ota, H., Taki, Y., Yasuda, S., Sajiki, A. F., Suzumura, A., Shimizu, H., Takeuchi, J., Tomita, R., Kominami, T., et al. (2025) · Ophthalmology Science · doi:10.1016/j.xops.2025.101004 · Cited by 5
Jiménez, C. A. R., Lokker, C., Ibhawoh, B., & Schwartz, L. J. (2026) · Discover Public Health · doi:10.1186/s12982-026-01568-z · Cited by 7
Rai, M. E., Ngaw, M., & Nannas, N. J. (2025) · Education Sciences · doi:10.3390/educsci15101400 · Cited by 10
Birsin, Z., Jeral, S., Cebeci, S., Çerme, E., Aliyev, V., Günaltılı, M., Abbasov, H., Çiçek, E., Demirci, N. S., & Alan, Ö. (2025) · Future Oncology · doi:10.1080/14796694.2025.2610463 · Cited by 6
Metadata
- Authors
- Chelli, M., Descamps, J., Lavoué, V., Trojani, C., Azar, M., Deckert, M., Raynier, J.-L., Clowez, G., Boileau, P., & Ruetsch-Chelli, C.
- Affiliations
- Assistance Publique – Hôpitaux de Paris (FR)
- Inserm (FR)
- Corresponding author
- Chelli, Mikaël
- ORCID
- Venue
- Journal of Medical Internet Research
- Publisher
- JMIR Publications Inc.
- Published
- 2024-05-22
- Volume
- 26
- Pages
- e53164
- ISSN
- 1438-8871 (ISSN-L)
- PMC
- PMC11153973
- OpenAlex
- W4398203672
- Semantic Scholar
- 9a0e8c101b72e9094ea554194698daa3aeb3bc6f
- Cited by
- 602
- References
- 30
- Citations by year
- 2026: 378 · 2025: 202 · 2024: 22
- FWCI
- 11.51
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- Journal of Medical Internet Research (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- HAL (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- hallucinations, large language models, ChatGPT, Google Bard, Gemini, systematic reviews, reference retrieval, literature search automation, precision and recall, citation bias, GPT-3.5, GPT-4, AI in scientific writing
- MeSH terms
- Systematic Reviews as Topic, Artificial Intelligence
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-09-28, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-09-28 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.