Hallucination Rates and Reference Accuracy of ChatGPT and Bard for Systematic Reviews: Comparative Analysis
Open accessPublished version of a preprint
The DOI matches its Crossref record; 35 of the 35 references it cites were found in a registry or an index; 2 of the 2 sources its claims cite were checked for support.
Abstract
Background Large language models (LLMs) have raised both interest and concern in the academic community. They offer the potential for automating literature search and synthesis for systematic reviews but raise concerns regarding their reliability, as the tendency to generate unsupported (hallucinated) content persist. Objective The aim of the study is to assess the performance of LLMs such as ChatGPT and Bard (subsequently rebranded Gemini) to produce references in the context of scientific writing. Methods The performance of ChatGPT and Bard in replicating the results of human-conducted systematic reviews was assessed. Using systematic reviews pertaining to shoulder rotator cuff pathology, these LLMs were tested by providing the same inclusion criteria and comparing the results with original systematic review references, serving as gold standards. The study used 3 key performance metrics: recall, precision, and F1-score, alongside the hallucination rate. Papers were considered “hallucinated” if any 2 of the following information were wrong: title, first author, or year of publication. Results In total, 11 systematic reviews across 4 fields yielded 33 prompts to LLMs (3 LLMs×11 reviews), with 471 references analyzed. Precision rates for GPT-3.5, GPT-4, and Bard were 9.4% (13/139), 13.4% (16/119), and 0% (0/104) respectively (P<.001). Recall rates were 11.9% (13/109) for GPT-3.5 and 13.7% (15/109) for GPT-4, with Bard failing to retrieve any relevant papers (P<.001). Hallucination rates stood at 39.6% (55/139) for GPT-3.5, 28.6% (34/119) for GPT-4, and 91.4% (95/104) for Bard (P<.001). Further analysis of nonhallucinated papers retrieved by GPT models revealed significant differences in identifying various criteria, such as randomized studies, participant criteria, and intervention criteria. The study also noted the geographical and open-access biases in the papers retrieved by the LLMs. Conclusions Given their current performance, it is not recommended for LLMs to be deployed as the primary or exclusive tool for conducting systematic reviews. Any references generated by such models warrant thorough validation by researchers. The high occurrence of hallucinations in LLMs highlights the necessity for refining their training and functionality before confidently using them for rigorous academic purposes.
Overview
CoChat Summary
## TL;DR
This comparative analysis tested ChatGPT and Bard on 11 systematic reviews of shoulder rotator cuff pathology and found low retrieval accuracy plus frequent hallucinated references. GPT-4 performed best of the models tested, but the authors conclude that LLM-generated citations still require rigorous human validation and should not be used alone for systematic reviews.
## Key findings
- Across 11 systematic reviews in 4 fields, the study analyzed 471 references from 33 LLM prompts (3 models × 11 reviews).
- Precision was 9.4% (13/139) for GPT-3.5, 13.4% (16/119) for GPT-4, and 0% (0/104) for Bard (P<.001); recall was 11.9% (13/109) for GPT-3.5, 13.7% (15/109) for GPT-4, and 0 for Bard (P<.001).
- Hallucination rates were 39.6% (55/139) for GPT-3.5, 28.6% (34/119) for GPT-4, and 91.4% (95/104) for Bard (P<.001).
- Among nonhallucinated GPT-retrieved papers, title accuracy was very high (83/84, 99% for GPT-3.5 and 84/85, 99% for GPT-4), but DOI accuracy was low (13/82, 16% and 17/84, 20%); American authors were overrepresented in ChatGPT results (44% vs 16.5% in the original reviews for GPT-3.5, and 33% vs 16.5% for GPT-4; P<.001).
## Methods
- Sequential comparative design: PubMed search on July 27, 2023 identified English-language 2020 systematic reviews/meta-analyses on randomized clinical trials in shoulder rotator cuff pathology; 11 systematic reviews were selected after title/abstract/full-text screening and validation with PRISMA/PROSPERO.
- The LLMs tested were GPT-3.5, GPT-4, and Bard (PaLM 2.0); each fresh chat session was prompted with the same inclusion criteria, and two prompt versions were tried, keeping the one that produced more results.
- Outputs were compared with the original systematic review references using recall, precision, F1-score, and hallucination rate; papers were labeled hallucinated if any 2 of title, first author, or year were wrong.
## Limitations
- The authors state the study was restricted to shoulder rotator cuff pathology and to 3 LLMs, so the findings may not generalize to other specialties or models.
- The paper also notes that no established guidelines exist for using LLMs to optimize accuracy, and different prompts might yield different results.
- Evident from the text: the study focused on a narrow, single-topic set of reviews from one year (2020) and English-language publications only, which limits breadth and may constrain generalizability.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures






Verification
Claims the paper makes
1 Valid · 0 Not valid · 0 Not checked · 40 Own results and interpretations
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Papers from American authors were selected in 16.5% (18/109) of original systematic reviews, 44% (37/84) of GPT-3.5 papers, and 33% (28/85) of GPT-4 papers (P<.001).”
§ Hallucination Rates and Reference Accuracy of ChatGPT and Bard for Systematic Reviews: Comparative Analysis > Results
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In most cases, the number of references returned by LLMs was greater or equal to that of the original papers (Table 2).”
§ Hallucination Rates and Reference Accuracy of ChatGPT and Bard for Systematic Reviews: Comparative Analysis > Results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Papers identified by LLMs were present in the original systematic reviews (precision) in 9.4% (13/139), 13.4% (16/119), and 0% (0/104) of cases for GPT-3.5, GPT-4, and Bard (P<.001), respectively.”
§ Hallucination Rates and Reference Accuracy of ChatGPT and Bard for Systematic Reviews: Comparative Analysis > Results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“authors list (73/84, 87% vs 74/85, 87%; P>.99)”
§ Hallucination Rates and Reference Accuracy of ChatGPT and Bard for Systematic Reviews: Comparative Analysis > Results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“randomized studies (33/84, 39% vs 42/85, 49%; P=.24)”
- Also stated in: § Principal Findings · plain
“For instance, the criterion of “randomized study” was adhered to in only 39% (33/84) to 49% (42/85) of nonhallucinated papers generated by ChatGPT, even when the term “randomized” appeared in the title or abstract of the papers from the original systematic reviews.”
§ Hallucination Rates and Reference Accuracy of ChatGPT and Bard for Systematic Reviews: Comparative Analysis > Results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
- Also stated in: § Principal Findings · plain
18 more claims
Create a free account to see all 41 claims and every source verdict.
- It may state two separate claims as one. 12
- Probably not a substantial claim. 9
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“Hallucination rates have been documented to vary between 3% – 28% in current literature [30, 31].”
From Artificial Intelligence (AI) – Powered Documentation Systems in Healthcare: A Systematic Review (2025) · cites this paper as [30]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Studies in radiology, psychiatry, and clinical rehabilitation have reported fabrication rates ranging from 20 to 70%”
From How LLMs Cite and Why It Matters: A Cross-Model Audit of Reference Fabrication in AI-Assisted Academic Writing and Methods to Detect Phantom Citations (2026) · cites this paper as [12]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Experimental studies from 2023 to 2025 have found that the rate of hallucination in LLMgenerated citations ranged from 19.9% to 91.4%, varying widely across models and time [16], [17], [18], [19], [24].”
From LLM hallucinations in the wild: Large-scale evidence from non-existent citations (2026) · cites this paper as [18]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“When citations are missing, inaccurate, or fabricated, the resulting break in the evidence chain can obscure logical dependencies, weaken argumentation, and jeopardize academic integrity at the level of both individual papers and the broader literature [10, 22, 26].”
From CiteAudit: You Cited It, But Did You Read It? A Benchmark for Verifying Scientific References in the LLM Era (2026) · cites this paper as [10]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Chelli et al. found that Bard produced incorrect references in 91.4% of systematic review prompts, whereas GPT-4 had a lower but still notable error rate of 28.6%18.”
From Multi-model assurance analysis showing large language models are highly vulnerable to adversarial hallucination attacks during clinical decision support (2025) · cites this paper as [18]
Experimental — accuracy not yet measured
16 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gotzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · BMJ · doi:10.1136/bmj.b2700 · Cited by 17,648
Artificial hallucinations in ChatGPT: implications in scientific writing
Alkaissi, H., & McFarlane, S. I. (2023) · Cureus · doi:10.7759/cureus.35179 · Cited by 1,540 · cited 1×
“This concern has been broached in previous literature [26-29], but our study provides an experimental design to probe the matter more deeply.”
BackgroundThe nuts and bolts of PROSPERO: an international prospective register of systematic reviews
Booth, A., Clarke, M., Dooley, G., Ghersi, D., Moher, D., Petticrew, M., & Stewart, L. (2012) · Systematic Reviews · doi:10.1186/2046-4053-1-2 · Cited by 1,433 · cited 1×
“Additionally, the registration status of these reviews was verified in the PROSPERO database [14].”
BackgroundChatting about ChatGPT: how may AI and GPT impact academia and libraries?
Lund, B. D., & Wang, T. (2023) · Library Hi Tech News · doi:10.1108/lhtn-01-2023-0009 · Cited by 1,355
ChatGPT listed as author on research papers: many scientists disapprove
Stokel-Walker, C. (2023) · Nature · doi:10.1038/d41586-023-00107-z · Cited by 868 · cited 1×
“While it is known that ChatGPT can help researchers write papers [3-5], controversy exists about whether it should be used at all, whether its use should be disclosed, and whether it should be listed as an author or not [6].”
BackgroundCan artificial intelligence help for scientific writing?
Salvagno, M., Taccone, F. S., & Gerli, A. G. (2023) · Critical Care · doi:10.1186/s13054-023-04380-2 · Cited by 827 · cited 1×
“While it is known that ChatGPT can help researchers write papers [3-5], controversy exists about whether it should be used at all, whether its use should be disclosed, and whether it should be listed as an author or not [6].”
BackgroundAbstracts written by ChatGPT fool scientists
Else, H. (2023) · Nature · doi:10.1038/d41586-023-00056-7 · Cited by 611
Boileau, P., Baqué, F., Valerio, L., Ahrens, P., Chuinard, C., & Trojani, C. (2007) · The Journal of Bone and Joint Surgery-American Volume · doi:10.2106/00004623-200704000-00008 · Cited by 603
Self-citationChatGPT and the future of medical writing
Biswas, S. (2023) · Radiology · doi:10.1148/radiol.223312 · Cited by 571
Learning to summarize from human feedback
Stiennon, N., Ouyang, L., Wu, J., Ziegler, D. M., Lowe, R., Voss, C., Radford, A., Amodei, D., & Christiano, P. (2020) · arXiv · doi:10.48550/arxiv.2009.01325 · Cited by 550
Cited by
Items 101–200 of 423
Rai, M. E., Ngaw, M., & Nannas, N. J. (2025) · Education Sciences · doi:10.3390/educsci15101400 · Cited by 10
Birsin, Z., Jeral, S., Cebeci, S., Çerme, E., Aliyev, V., Günaltılı, M., Abbasov, H., Çiçek, E., Demirci, N. S., & Alan, Ö. (2025) · Future Oncology · doi:10.1080/14796694.2025.2610463 · Cited by 6
Wang, X., Yin, C., He, H., Guo, J., Fu, X., & 白飞虎 (2026) · Frontiers in Public Health · doi:10.3389/fpubh.2026.1810358 · Cited by 10
Comparison of ChatGPT and Gemini as sources of references in otorhinolaryngologyPreprint
Jędrzejczak, W. W., Pastucha, M., Skarżyńśki, H., & Kochanek, K. (2024) · medRxiv · doi:10.1101/2024.08.12.24311896 · Cited by 3
Benchmarking large multimodal models for ophthalmic visual question answering with OphthalWeChat
Xu, P., Gong, X., Chen, X., Zhang, W., Yang, J., Yan, B., Yuan, M., Yalin, Z., He, M., & Shi, D. (2025) · Advances in Ophthalmology Practice and Research · doi:10.1016/j.aopr.2025.10.006 · Cited by 12
Evaluating the potential of ChatGPT-reformulated essays as written feedback in L2 writing
Chen, Y. (2025) · Computers and Education Artificial Intelligence · doi:10.1016/j.caeai.2025.100500 · Cited by 9
Sanduleanu, S., Ersahin, K., Bremm, J., Talibova, N., Damer, T., Erdogan, M., Kottlors, J., Goertz, L., Bruns, C. J., Maintz, D. C., et al. (2024) · AI · doi:10.3390/ai5040096 · Cited by 6
King, C., & Lopour, B. A. (2025) · Biomedical Engineering Education · doi:10.1007/s43683-025-00192-8 · Cited by 11
AI chatbots in higher education: Comparing expectations to evidence
Thoeni, A., & Fryer, L. K. (2026) · Computers in Human Behavior Reports · doi:10.1016/j.chbr.2026.101061 · Cited by 4
Penny‐Dimri, J. C., Bachmann, M., Cooke, W. R., Mathewlynn, S., Dockree, S., Tolladay, J., Kossen, J., Li, L., Gal, Y., & Jones, G. D. (2025) · The Lancet Obstetrics Gynaecology & Women s Health · doi:10.1016/j.lanogw.2025.100005 · Cited by 13
Can ChatGPT AI Replace or Contribute to Experts’ Diagnosis for Renovation Measures Identification?
Hidalgo-Betanzos, J. M., Prol-Godoy, I., Terés-Zubiaga, J., Briones-Llorente, R., & Martín-Garín, A. (2025) · Buildings · doi:10.3390/buildings15030421 · Cited by 7
Shen, Z., & Yu, C. (2025) · Brain Sciences · doi:10.3390/brainsci15080890 · Cited by 9
AI, authorship teams and agency: Enacting fission and fusion
Maggio, L. A., & Konopasky, A. (2025) · Medical Education · doi:10.1111/medu.70143 · Cited by 3
Large language models: unlocking new potential in patient education for thyroid eye disease
Gao, Y., Xu, Q., Zhang, O., Wang, H., Wang, Y., Wang, J., & Chen, X. (2025) · Endocrine · doi:10.1007/s12020-025-04339-z · Cited by 9
Seth, I., Marcaccini, G., Lim, B., Novo, J., Bacchi, S., Cuomo, R., Ross, R. J., & Rozen, W. M. (2025) · Informatics · doi:10.3390/informatics12030086 · Cited by 5
Goto, H., Shiraishi, Y., & Okada, S. (2024) · Cureus · doi:10.7759/cureus.74262 · Cited by 5
Fabi, A., Egli, C., Wendelspiess, S. R., Griewing, S., Haas, Y., Pellegrin, L. D., Schaefer, D. J., Qiu, S. S., Harder, Y., & Kappos, E. A. (2026) · JMIR Medical Informatics · doi:10.2196/80553 · Cited by 3
Gupta, A. K., Basha, A., Sontam, T. R., Hlavinka, W. J., Croen, B. J., Abdou, C., Abdullah, M. S., & Hamilton, R. G. (2025) · Baylor University Medical Center Proceedings · doi:10.1080/08998280.2025.2470033 · Cited by 5
Current Perspectives on the Artificial Intelligence in Critical Care Medicine
Lee, J., & Yoon, J. H. (2025) · Anesthesiology Clinics · doi:10.1016/j.anclin.2025.05.005 · Cited by 5
Zhang, Z., Scroggins, J. K., Harkins, S., Hulchafo, I. I., Moen, H., Tadiello, M., Barcelona, V., & Topaz, M. (2025) · Nursing Outlook · doi:10.1016/j.outlook.2025.102472 · Cited by 4
Ülkir, M., & Paslı, B. (2026) · Clinical Anatomy · doi:10.1002/ca.70187 · Cited by 7
Boonrit, N., Thaweechai, A., Kessarin, B., & Ruanglertboon, W. (2025) · JACCP JOURNAL OF THE AMERICAN COLLEGE OF CLINICAL PHARMACY · doi:10.1002/jac5.70115 · Cited by 3
Russinovich, M., Salem, A., Zanella-Béguelin, S., & Zunger, Y. (2025) · Communications of the ACM · doi:10.1145/3749447 · Cited by 10
Tsai, C., Lin, Y.-C., Hou, J.-U., Tsai, S., Yeh, P.-C., & Kao, C. (2025) · Digital Health · doi:10.1177/20552076251357468 · Cited by 4
A human in the loop approach to applying large language models for farm management insight
Mourtzinis, S., Silva, T. S., Lo, J. C. M., Smith, D. L., & Conley, S. P. (2025) · Scientific Reports · doi:10.1038/s41598-025-30991-6 · Cited by 7
Guo, A., Canagasingham, A., Rasiah, K., Chalasani, V., Mundy, J. A., & Chung, A. S. J. (2025) · ANZ Journal of Surgery · doi:10.1111/ans.70186 · Cited by 5
Gu, C., & Zhang, J. (2025) · Current Psychology · doi:10.1007/s12144-025-08899-1 · Cited by 5
Recent Advances and Future Directions of Artificial Intelligence in Glaucoma Management
Tang, W., Hang, Y., Zhang, J., & Dang, Y. (2025) · The Open Ophthalmology Journal · doi:10.2174/0118743641430437251130165439 · Cited by 4
Shamsi, A., Wang, T., Amraei, M., & Raju, N. V. (2025) · Acta Informatica Pragensia · doi:10.18267/j.aip.293 · Cited by 5
Evaluating the Reliability of GPT ‐4o in Histological Image Interpretation
Mavrych, V., Yousef, E. M., Yaqinuddin, A., Shaikh, A. A., & Bolgova, O. (2025) · Clinical Anatomy · doi:10.1002/ca.70065 · Cited by 5
ChatGPT: how to use it and the pitfalls/cautions in academia
Lee, J. (2025) · Annals of Pediatric Endocrinology & Metabolism · doi:10.6065/apem.2550028.014 · Cited by 5
Tensen, D., Carey, M. D., & Grainger, P. (2026) · Assessment & Evaluation in Higher Education · doi:10.1080/02602938.2026.2619902 · Cited by 5
When AI Gives Bad Advice: Critical thinking in human-AI collaborationsPreprint
Bilen, E., & Hervé, J. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5040466 · Cited by 5
Eskandar, K. (2026) · Société Internationale d’Urologie Journal · doi:10.3390/siuj7010011 · Cited by 5
LLM hallucinations in the wild: Large-scale evidence from non-existent citations
Zhao, Z., Wang, Y., Stuart, T., De Vaan, M., Ginsparg, P., & Yin, Y. (2026) · arXiv · doi:10.48550/arxiv.2605.07723 · Cited by 3
“First, experiments prompting LLMs to suggest references have documented hallucinated citations [16], [17], [18], [19].”
InfluentialSTRATEGIES FOR AVOIDING GPT HALLUCINATIONS
(2024) · International Research Journal of Modernization in Engineering Technology and Science · doi:10.56726/irjmets60718 · Cited by 3
Machine-learning-enabled modeling of pharmacokinetics and pharmacodynamics
Psarellis, Y. M., Pillai, N., Dhakal, S., & Mavroudis, P. D. (2026) · Drug Discovery Today · doi:10.1016/j.drudis.2026.104645 · Cited by 6
Ang, M. Y., Chen, L., Song, L., Lipovich, L., & Choo, S. W. (2026) · Life · doi:10.3390/life16061032 · Cited by 2
Hilkenmeier, F., Stoltenberg, M., & Stierle, C. M. G. (2026) · Systematic Reviews · doi:10.1186/s13643-026-03228-4 · Cited by 8
ChatGPT and other large language models for childhood asthma
Drummond, D., Girault, A., & Gonsard, A. (2025) · Paediatric Respiratory Reviews · doi:10.1016/j.prrv.2025.09.002 · Cited by 5
Leong, A. G., & Ormsby, K. (2025) · Technical Innovations & Patient Support in Radiation Oncology · doi:10.1016/j.tipsro.2025.100373 · Cited by 4
Ros, T. J., & Samuel, A. (2025) · New Directions for Adult and Continuing Education · doi:10.1002/ace.70014 · Cited by 4
Post-Public AI: Research in Groups and Teams
Bezrukova, K., & Griffith, T. L. (2025) · Small Group Research · doi:10.1177/10464964251363458 · Cited by 5
Li, K., Peng, Y., Li, L., Liu, B., & Huang, Z. (2025) · JMIR Formative Research · doi:10.2196/76458 · Cited by 2
Albaloul, O., Alajmi, A., Oster, N. J., & Killian, C. M. (2026) · Discover Artificial Intelligence · doi:10.1007/s44163-026-01344-7 · Cited by 6
Yutong, G. U. O., & Yang, C. (2026) · Metals · doi:10.3390/met16020162 · Cited by 5
Chisale, P. B. (2026) · Frontiers in Education · doi:10.3389/feduc.2025.1694819 · Cited by 2
Wang, X., & Li, X. (2026) · PLoS ONE · doi:10.1371/journal.pone.0345027 · Cited by 8
Cabezas-Clavijo, Á., & Sidorenko-Bautista, P. (2026) · Journal of Data and Information Science · doi:10.1515/jdis-2025-0326 · Cited by 22
Intelligent Decentralized Governance: A Case Study of KlimaDAO Decision-Making
Chen, J., Hsu, C., & Tsai, Y. (2025) · Electronics · doi:10.3390/electronics14122462 · Cited by 5
Automatically Engineering Trusted Software: A Research Roadmap
Brun, Y., Chakraborty, S., Goues, C. L., Păsăreanu, C. S., & Singla, A. K. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3779132 · Cited by 5
Dialogic triad: Balancing human-AI collaboration in academic writing for scholarly publication
Cao, L., & Lin, A. M. Y. (2025) · Journal of English for Research Publication Purposes · doi:10.1075/jerpp.00038.cao · Cited by 4
Özgen, A. N. K., Karbancioğlu, E. Ç., & Tezel, N. (2026) · Neurourology and Urodynamics · doi:10.1002/nau.70385 · Cited by 2
Leiva-Araos, A., Kalasapudi, V. S., Jiang, A., & Kaushal, H. (2025) · IEEE Access · doi:10.1109/access.2025.3613246 · Cited by 5
Li, C., Chu, Y., Liu, D.-N., Ghanad, E., Abdelhadi, S., Șandra-Petrescu, F., Reissfelder, C., Vassilev, G., & Yang, C. (2025) · medRxiv · doi:10.1101/2025.07.26.25332239 · Cited by 3
Aziz, M., & Brookhart, M. A. (2025) · Clinical Epidemiology · doi:10.2147/clep.s550565 · Cited by 3
Retrieval-Augmented Generation in Oncology: Promises, Pitfalls, and Early Applications
Thaker, N. G., Liu, W., Waddle, M., Showalter, T. N., Mastroleo, F., Luh, J. Y., Levitt, C., Ning, M., Loaiza‐Bonilla, A., & Hong, J. C. (2026) · AI in Precision Oncology · doi:10.1177/2993091x261446348 · Cited by 3
The ethics of using artificial intelligence in writing medical research papers
Yu, S., & Hwang, H. (2025) · Kosin Medical Journal · doi:10.7180/kmj.25.141 · Cited by 2
Zade, F. A., & Ebrahimkhanlou, A. (2025) · Journal of Computing in Civil Engineering · doi:10.1061/jccee5.cpeng-7001 · Cited by 13
Generative AI and multimodal pedagogy: implications for teaching, learning, and assessment
Istrate, O. (2026) · Frontiers in Education · doi:10.3389/feduc.2026.1828953 · Cited by 5
Ademolu, E. (2026) · AI & Society · doi:10.1007/s00146-026-02956-4 · Cited by 3
Kanbakan, A., Berikol, G. B., İlhan, B., Altıntaş, E., & Doğanay, F. (2026) · BMC Medical Education · doi:10.1186/s12909-025-08539-z · Cited by 2
Artificial intelligence and plastic surgery: Between innovation and responsibility
Lupon, E. L. (2025) · Annales de Chirurgie Plastique Esthétique · doi:10.1016/j.anplas.2025.11.003 · Cited by 2
Park, S., Lee, T. Y., & Lee, W. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3790271
Aydin, M. O., Vatansever, A., & Kafa, S. E. (2026) · Uludağ Üniversitesi Tıp Fakültesi Dergisi · doi:10.32708/uutfd.1870116 · Cited by 5
Han, C., Manoharan, S., Ye, X., & Speidel, U. (2026) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.70104 · Cited by 3
Che, X., & Liu, Y. (2026) · Frontiers in Medicine · doi:10.3389/fmed.2026.1910543 · Cited by 4
Velasco, A. M., Coelho, E. W. F., Paes, V. L. S., Castro, R. C., & de Oliveira, R. D. C. S. (2025) · Caderno Pedagógico · doi:10.54033/cadpedv22n5-216 · Cited by 5
Civelekler, M., & Çıtırık, M. (2026) · Journal of Glaucoma · doi:10.1097/ijg.0000000000002716 · Cited by 1
Calibrated Trust in Dealing with LLM Hallucinations: A Qualitative Study
Ryser, A., Allwein, F., & Schlippe, T. (2025) · International Conference on Foundation and Large Language Models (FLLM) · doi:10.1109/fllm67465.2025.11391250 · Cited by 7
When ChatGPT becomes a shortcut: psychological predictors of student plagiarism threshold
Cihodaru-Ștefanache, Ș., Popovici, M., Dumitru, M.-V., & Podina, I. R. (2026) · Interactive Learning Environments · doi:10.1080/10494820.2026.2645324 · Cited by 1
Yadav, V., Trushna, T., Mandal, U. K., Singh, M., Goel, A. D., Bairwa, M. L., Dabar, D., & Sabde, Y. (2025) · Medical Reference Services Quarterly · doi:10.1080/02763869.2025.2595567 · Cited by 3
Zhong, W., Zheng, G., Li, Q., Huang, Y., & Zhao, J. (2026) · Frontiers in Public Health · doi:10.3389/fpubh.2026.1818821 · Cited by 3
Gumilar, K. E., Ibrahim, I. H., Omran, K., Rajanagara, A. S., Yu, Z.-Y., Hsu, Y.-C., Lin, C.-H., Hung, T.-Y., Kousar, I., Tahir, N., et al. (2025) · Research Square · doi:10.21203/rs.3.rs-6676676/v1 · Cited by 3
Hamid, A. A., Alshahrani, R. S. S., Alghareeb, M., Hawsawi, H. S., Alghareeb, R. A., & Alsultan, M. A. (2026) · Journal of Clinical Medicine · doi:10.3390/jcm15124419 · Cited by 2
Harman, A. (2025) · Sport Management Education Journal · doi:10.1123/smej.2024-0018 · Cited by 1
Anand, E., Ghersin, I., Lingam, G., Devlin, K., Pelly, T., Singer, D., Tomlinson, C., Munro, R. E. J., Capstick, R., Antoniou, A., et al. (2025) · Diagnostics · doi:10.3390/diagnostics16010072 · Cited by 5
Fleurence, R., Qureshi, R., Aggarwal, R., Bian, J., Cheyne, S., Dawoud, D., Faden, R., & Chhatwal, J. (2026) · Value in Health · doi:10.1016/j.jval.2026.06.012 · Cited by 7
Artificial Intelligence in Applied Cognitive Psychology: A Commentary
Sharpe, B. T., Rod, M., & Horne, G. (2026) · International Journal of Cognitive Research in Science Engineering and Education · doi:10.23947/2334-8496-2026-14-1-115-124 · Cited by 1
Data Mining Trauma: AI-Assisted Qualitative Study of Cyber Victimization on Reddit
Antisdel, J., Miller, W. R., & Groves, D. J. (2025) · JMIR Infodemiology · doi:10.2196/75493 · Cited by 2
Carras, M. C., Qureshi, R., Naaman, K., Aldayel, F., Date, M., Aljuboori, D., & Thrul, J. (2025) · JMIR Formative Research · doi:10.2196/69707 · Cited by 2
McLaughlin, N. D., Srinivas, A. N., Lowe, Z. F., Botterbush, K. S., Patel, M. S., & Avila, M. J. (2026) · Neurosurgery Practice · doi:10.1227/neuprac.0000000000000244 · Cited by 1
“8 High rates of citation fabrication have been quanti fi ed in other fi elds, such as a study on systematic reviews in the fi eld of orthopedic surgery conducted by Chelli et al, 12 which found high hallucination rates in ChatGPT.”
Salzmann, M., Ramadanov, N., Prill, R., Hable, R., & Becker, R. (2026) · Knee Surgery Sports Traumatology Arthroscopy · doi:10.1002/ksa.70379 · Cited by 1
Critical ignoring reimagined: insights from STEM digital curation on Wikimedia platforms
Sigalov, S. E., & Tsybulsky, D. (2026) · Smart Learning Environments · doi:10.1186/s40561-026-00432-6 · Cited by 3
Madsen, D. Ø., & Silva, E. S. (2026) · The Journal of Academic Librarianship · doi:10.1016/j.acalib.2026.103267 · Cited by 5
Soujah, C., Bejjani, C., Adra, N., & Blackburn, L. (2025) · Hospital Pharmacy · doi:10.1177/00185787251372424 · Cited by 3
Informing patients with otologic balance disorders: A performance review of ChatGPT-4
Güngör, V., & Yaslikaya, S. (2026) · Medicine · doi:10.1097/md.0000000000047127 · Cited by 3
Chat GPT vs an experienced ophthalmologist: evaluating chatbot writing performance in ophthalmology
Katz, G., Zloto, O., Hostovsky, A., Huna-Baron, R., Mizrachi, I. B.-B., Burgansky, Z., Skaat, A., Vishnevskia-Dai, V., Fabian, I. D., Sagiv, O., et al. (2025) · Eye · doi:10.1038/s41433-025-03779-1 · Cited by 2
Assessing the Reliability of ChatGPT and Gemini in Identifying Relevant Orthodontic Literature
Asiri, S. N. (2025) · European Journal of General Dentistry · doi:10.1055/s-0045-1809617 · Cited by 3
Hallucinations in large language models within academic contexts: A systematic review.
Lai, K. E., Mustaffa, N., Preece, C. N., & Wong, F. Y. (2026) · Journal of Science and Technology Policy Management · doi:10.1108/jstpm-01-2026-0015 · Cited by 1
Jeon, K. (2026) · Journal of Korean Medical Association · doi:10.5124/jkma.25.0163 · Cited by 3
Rios-García, W., Silva-Jiménez, S., Galvez-Rodriguez, E. B., Alberca-Naira, Y., Via-y-Rada-Torres, A. D., & Rios-Garcia, A. A. (2026) · Journal of Medical Systems · doi:10.1007/s10916-025-02334-5 · Cited by 2
Alfabetización crítica en IA: Recursos educativos para una pedagogía de la descajanegrización
Artopoulos, A., & Lliteras, A. B. (2024) · Trayectorias Universitarias · doi:10.24215/24690090e168 · Cited by 9
SustainTima: a task-oriented text-based conversational AI agent for sustainable fast fashion brands
Campuzano, E. P., & Chakraborty, S. (2025) · International Journal of Fashion Design Technology and Education · doi:10.1080/17543266.2025.2584821 · Cited by 1
Artificial Intelligence in Spiritual Care: Modified Delphi Study
Winiger, F., Smith, C. E., Nourriz, S., Szilagyi, C., & Haußmann, A. D. (2026) · Journal of Medical Internet Research · doi:10.2196/97094 · Cited by 3
Enhancing Readability of Surgical Patient Education Resources Using ChatGPT
Zhu, L., Zamore, Z. H., Azad, T. D., Amlani, L. M., Chen, J., Harris, C., Khalifeh, J. M., Murali, S., Sylvester, S. A., Mundy, L. R., et al. (2026) · Journal of Surgical Research · doi:10.1016/j.jss.2026.03.030 · Cited by 1
Karni, J. E., Simon, C., & Hack, S. (2026) · Digital Health · doi:10.1177/20552076261450749 · Cited by 3
A Benchmark Comparison of a Domain-Focused Pipeline with ChatGPT
Zhang, C., Zhao, T., Saraoğlu, H., & Louton, D. A. (2025) · Journal of Computer Information Systems · doi:10.1080/08874417.2024.2446629 · Cited by 1
Metadata
- Authors
- Chelli, M., Descamps, J., Lavoué, V., Trojani, C., Azar, M., Deckert, M., Raynier, J.-L., Clowez, G., Boileau, P., & Ruetsch-Chelli, C.
- Affiliations
- Assistance Publique – Hôpitaux de Paris (FR)
- Inserm (FR)
- Corresponding author
- Chelli, Mikaël
- ORCID
- Venue
- Journal of Medical Internet Research
- Publisher
- JMIR Publications Inc.
- Published
- 2024-05-22
- Volume
- 26
- Pages
- e53164
- ISSN
- 1438-8871 (ISSN-L)
- PMC
- PMC11153973
- OpenAlex
- W4398203672
- Semantic Scholar
- 9a0e8c101b72e9094ea554194698daa3aeb3bc6f
- Cited by
- 602
- References
- 30
- Citations by year
- 2026: 378 · 2025: 202 · 2024: 22
- FWCI
- 11.51
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- Journal of Medical Internet Research (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- HAL (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- hallucinations, large language models, ChatGPT, Google Bard, Gemini, systematic reviews, reference retrieval, literature search automation, precision and recall, citation bias, GPT-3.5, GPT-4, AI in scientific writing
- MeSH terms
- Systematic Reviews as Topic, Artificial Intelligence
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-09-28, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-09-28 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.