Hallucination Rates and Reference Accuracy of ChatGPT and Bard for Systematic Reviews: Comparative Analysis
Open accessPublished version of a preprint
The DOI matches its Crossref record; 35 of the 35 references it cites were found in a registry or an index; 2 of the 2 sources its claims cite were checked for support.
Abstract
Background Large language models (LLMs) have raised both interest and concern in the academic community. They offer the potential for automating literature search and synthesis for systematic reviews but raise concerns regarding their reliability, as the tendency to generate unsupported (hallucinated) content persist. Objective The aim of the study is to assess the performance of LLMs such as ChatGPT and Bard (subsequently rebranded Gemini) to produce references in the context of scientific writing. Methods The performance of ChatGPT and Bard in replicating the results of human-conducted systematic reviews was assessed. Using systematic reviews pertaining to shoulder rotator cuff pathology, these LLMs were tested by providing the same inclusion criteria and comparing the results with original systematic review references, serving as gold standards. The study used 3 key performance metrics: recall, precision, and F1-score, alongside the hallucination rate. Papers were considered “hallucinated” if any 2 of the following information were wrong: title, first author, or year of publication. Results In total, 11 systematic reviews across 4 fields yielded 33 prompts to LLMs (3 LLMs×11 reviews), with 471 references analyzed. Precision rates for GPT-3.5, GPT-4, and Bard were 9.4% (13/139), 13.4% (16/119), and 0% (0/104) respectively (P<.001). Recall rates were 11.9% (13/109) for GPT-3.5 and 13.7% (15/109) for GPT-4, with Bard failing to retrieve any relevant papers (P<.001). Hallucination rates stood at 39.6% (55/139) for GPT-3.5, 28.6% (34/119) for GPT-4, and 91.4% (95/104) for Bard (P<.001). Further analysis of nonhallucinated papers retrieved by GPT models revealed significant differences in identifying various criteria, such as randomized studies, participant criteria, and intervention criteria. The study also noted the geographical and open-access biases in the papers retrieved by the LLMs. Conclusions Given their current performance, it is not recommended for LLMs to be deployed as the primary or exclusive tool for conducting systematic reviews. Any references generated by such models warrant thorough validation by researchers. The high occurrence of hallucinations in LLMs highlights the necessity for refining their training and functionality before confidently using them for rigorous academic purposes.
Overview
CoChat Summary
## TL;DR
This comparative analysis tested ChatGPT and Bard on 11 systematic reviews of shoulder rotator cuff pathology and found low retrieval accuracy plus frequent hallucinated references. GPT-4 performed best of the models tested, but the authors conclude that LLM-generated citations still require rigorous human validation and should not be used alone for systematic reviews.
## Key findings
- Across 11 systematic reviews in 4 fields, the study analyzed 471 references from 33 LLM prompts (3 models × 11 reviews).
- Precision was 9.4% (13/139) for GPT-3.5, 13.4% (16/119) for GPT-4, and 0% (0/104) for Bard (P<.001); recall was 11.9% (13/109) for GPT-3.5, 13.7% (15/109) for GPT-4, and 0 for Bard (P<.001).
- Hallucination rates were 39.6% (55/139) for GPT-3.5, 28.6% (34/119) for GPT-4, and 91.4% (95/104) for Bard (P<.001).
- Among nonhallucinated GPT-retrieved papers, title accuracy was very high (83/84, 99% for GPT-3.5 and 84/85, 99% for GPT-4), but DOI accuracy was low (13/82, 16% and 17/84, 20%); American authors were overrepresented in ChatGPT results (44% vs 16.5% in the original reviews for GPT-3.5, and 33% vs 16.5% for GPT-4; P<.001).
## Methods
- Sequential comparative design: PubMed search on July 27, 2023 identified English-language 2020 systematic reviews/meta-analyses on randomized clinical trials in shoulder rotator cuff pathology; 11 systematic reviews were selected after title/abstract/full-text screening and validation with PRISMA/PROSPERO.
- The LLMs tested were GPT-3.5, GPT-4, and Bard (PaLM 2.0); each fresh chat session was prompted with the same inclusion criteria, and two prompt versions were tried, keeping the one that produced more results.
- Outputs were compared with the original systematic review references using recall, precision, F1-score, and hallucination rate; papers were labeled hallucinated if any 2 of title, first author, or year were wrong.
## Limitations
- The authors state the study was restricted to shoulder rotator cuff pathology and to 3 LLMs, so the findings may not generalize to other specialties or models.
- The paper also notes that no established guidelines exist for using LLMs to optimize accuracy, and different prompts might yield different results.
- Evident from the text: the study focused on a narrow, single-topic set of reviews from one year (2020) and English-language publications only, which limits breadth and may constrain generalizability.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures






Verification
Claims the paper makes
1 Valid · 0 Not valid · 0 Not checked · 40 Own results and interpretations
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Papers from American authors were selected in 16.5% (18/109) of original systematic reviews, 44% (37/84) of GPT-3.5 papers, and 33% (28/85) of GPT-4 papers (P<.001).”
§ Hallucination Rates and Reference Accuracy of ChatGPT and Bard for Systematic Reviews: Comparative Analysis > Results
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In most cases, the number of references returned by LLMs was greater or equal to that of the original papers (Table 2).”
§ Hallucination Rates and Reference Accuracy of ChatGPT and Bard for Systematic Reviews: Comparative Analysis > Results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Papers identified by LLMs were present in the original systematic reviews (precision) in 9.4% (13/139), 13.4% (16/119), and 0% (0/104) of cases for GPT-3.5, GPT-4, and Bard (P<.001), respectively.”
§ Hallucination Rates and Reference Accuracy of ChatGPT and Bard for Systematic Reviews: Comparative Analysis > Results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“authors list (73/84, 87% vs 74/85, 87%; P>.99)”
§ Hallucination Rates and Reference Accuracy of ChatGPT and Bard for Systematic Reviews: Comparative Analysis > Results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“randomized studies (33/84, 39% vs 42/85, 49%; P=.24)”
- Also stated in: § Principal Findings · plain
“For instance, the criterion of “randomized study” was adhered to in only 39% (33/84) to 49% (42/85) of nonhallucinated papers generated by ChatGPT, even when the term “randomized” appeared in the title or abstract of the papers from the original systematic reviews.”
§ Hallucination Rates and Reference Accuracy of ChatGPT and Bard for Systematic Reviews: Comparative Analysis > Results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
- Also stated in: § Principal Findings · plain
18 more claims
Create a free account to see all 41 claims and every source verdict.
- It may state two separate claims as one. 12
- Probably not a substantial claim. 9
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“Hallucination rates have been documented to vary between 3% – 28% in current literature [30, 31].”
From Artificial Intelligence (AI) – Powered Documentation Systems in Healthcare: A Systematic Review (2025) · cites this paper as [30]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Studies in radiology, psychiatry, and clinical rehabilitation have reported fabrication rates ranging from 20 to 70%”
From How LLMs Cite and Why It Matters: A Cross-Model Audit of Reference Fabrication in AI-Assisted Academic Writing and Methods to Detect Phantom Citations (2026) · cites this paper as [12]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Experimental studies from 2023 to 2025 have found that the rate of hallucination in LLMgenerated citations ranged from 19.9% to 91.4%, varying widely across models and time [16], [17], [18], [19], [24].”
From LLM hallucinations in the wild: Large-scale evidence from non-existent citations (2026) · cites this paper as [18]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“When citations are missing, inaccurate, or fabricated, the resulting break in the evidence chain can obscure logical dependencies, weaken argumentation, and jeopardize academic integrity at the level of both individual papers and the broader literature [10, 22, 26].”
From CiteAudit: You Cited It, But Did You Read It? A Benchmark for Verifying Scientific References in the LLM Era (2026) · cites this paper as [10]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Chelli et al. found that Bard produced incorrect references in 91.4% of systematic review prompts, whereas GPT-4 had a lower but still notable error rate of 28.6%18.”
From Multi-model assurance analysis showing large language models are highly vulnerable to adversarial hallucination attacks during clinical decision support (2025) · cites this paper as [18]
Experimental — accuracy not yet measured
16 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gotzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · BMJ · doi:10.1136/bmj.b2700 · Cited by 17,648
Artificial hallucinations in ChatGPT: implications in scientific writing
Alkaissi, H., & McFarlane, S. I. (2023) · Cureus · doi:10.7759/cureus.35179 · Cited by 1,540 · cited 1×
“This concern has been broached in previous literature [26-29], but our study provides an experimental design to probe the matter more deeply.”
BackgroundThe nuts and bolts of PROSPERO: an international prospective register of systematic reviews
Booth, A., Clarke, M., Dooley, G., Ghersi, D., Moher, D., Petticrew, M., & Stewart, L. (2012) · Systematic Reviews · doi:10.1186/2046-4053-1-2 · Cited by 1,433 · cited 1×
“Additionally, the registration status of these reviews was verified in the PROSPERO database [14].”
BackgroundChatting about ChatGPT: how may AI and GPT impact academia and libraries?
Lund, B. D., & Wang, T. (2023) · Library Hi Tech News · doi:10.1108/lhtn-01-2023-0009 · Cited by 1,355
ChatGPT listed as author on research papers: many scientists disapprove
Stokel-Walker, C. (2023) · Nature · doi:10.1038/d41586-023-00107-z · Cited by 868 · cited 1×
“While it is known that ChatGPT can help researchers write papers [3-5], controversy exists about whether it should be used at all, whether its use should be disclosed, and whether it should be listed as an author or not [6].”
BackgroundCan artificial intelligence help for scientific writing?
Salvagno, M., Taccone, F. S., & Gerli, A. G. (2023) · Critical Care · doi:10.1186/s13054-023-04380-2 · Cited by 827 · cited 1×
“While it is known that ChatGPT can help researchers write papers [3-5], controversy exists about whether it should be used at all, whether its use should be disclosed, and whether it should be listed as an author or not [6].”
BackgroundAbstracts written by ChatGPT fool scientists
Else, H. (2023) · Nature · doi:10.1038/d41586-023-00056-7 · Cited by 611
Boileau, P., Baqué, F., Valerio, L., Ahrens, P., Chuinard, C., & Trojani, C. (2007) · The Journal of Bone and Joint Surgery-American Volume · doi:10.2106/00004623-200704000-00008 · Cited by 603
Self-citationChatGPT and the future of medical writing
Biswas, S. (2023) · Radiology · doi:10.1148/radiol.223312 · Cited by 571
Learning to summarize from human feedback
Stiennon, N., Ouyang, L., Wu, J., Ziegler, D. M., Lowe, R., Voss, C., Radford, A., Amodei, D., & Christiano, P. (2020) · arXiv · doi:10.48550/arxiv.2009.01325 · Cited by 550
Cited by
Items 201–300 of 423
Joranger, P., Lafontan, S. R., & Brevik, A. (2025) · JMIR Formative Research · doi:10.2196/73248 · Cited by 3
Matsutomo, N., Fukami, M., & Yamamoto, T. (2025) · Annals of Nuclear Medicine · doi:10.1007/s12149-025-02047-2 · Cited by 4
Ren, K., Weng, Q., Chen, Q., Li, H., Xie, D., Zeng, C., Wei, J., Lei, G., & Wang, Y. (2026) · Journal of Orthopaedic Surgery and Research · doi:10.1186/s13018-026-06844-x · Cited by 1
Wagg, A. (2026) · Société Internationale d’Urologie Journal · doi:10.3390/siuj7010012 · Cited by 1
Liao, Z., Huang, J., Liu, Y., Li, F., Tang, L., Cong, L., Wang, H., & Luo, S. (2025) · Aesthetic Plastic Surgery · doi:10.1007/s00266-025-05314-9 · Cited by 2
Human-machine interaction in the age of generative AI
Niraula, D., Shotande, M. O., & El Naqa, I. (2025) · The Cancer Journal · doi:10.1097/ppo.0000000000000797 · Cited by 1
Baldwin, J. D., Dinh, C., Mukerji, A., Sanghavi, N., & Gombar, S. (2025) · medRxiv · doi:10.1101/2025.07.01.25330655 · Cited by 1
Norbu, P., & Wangdi, S. (2025) · Information Discovery and Delivery · doi:10.1108/idd-01-2024-0015 · Cited by 1
Alvin, S. (2026) · International Journal of Strategic Communication · doi:10.1080/1553118x.2026.2638835 · Cited by 3
Seifi, A., & Seyfi, A. (2026) · Critical Care Explorations · doi:10.1097/cce.0000000000001474 · Cited by 1
Singh, R., Venugopal, V. K., Daga, R., et al. (2026) · European Journal of Radiology Artificial Intelligence · doi:10.1016/j.ejrai.2026.100088 · Cited by 1
Ayravainen, L. E. M., Hinds, J. M., & Davidson, B. I. (2025) · PLoS ONE · doi:10.1371/journal.pone.0336269 · Cited by 4
Jain, A., & Merchant, Y. P. (2026) · Oral and Maxillofacial Surgery · doi:10.1007/s10006-026-01550-8 · Cited by 1
Scheinkman, R., Kraft, G., Kasheri, E., & Nouri, K. (2025) · International Journal of Dermatology · doi:10.1111/ijd.17716 · Cited by 1
Hristov, H., Bekirski, K., Somova, E., Ignatov, A., Stavrev, S., & Poptolev, Z. (2025) · International Conference on Electronics, Engineering Physics and Earth Science (EEPES) · doi:10.3390/engproc2025104047 · Cited by 3
The Silent Author in Urology: Quantifying Large Language Model Influence at the Corpus LevelPreprint
Arezki, A., Tsui, J. M. G., & Kassouf, W. (2025) · medRxiv · doi:10.1101/2025.07.21.25331681 · Cited by 1
Park, Y., Zhang, H.-S., & Bai, S. W. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-02431-w · Cited by 3
Chen, C. K., Liang, S.-H., & Wang, T. (2026) · Scientometrics · doi:10.1007/s11192-026-05727-6 · Cited by 1
Naser, M. (2026) · arXiv · doi:10.48550/arxiv.2603.03299 · Cited by 1
“Studies in radiology, psychiatry, and clinical rehabilitation have reported fabrication rates ranging from 20 to 70%, depending on the model version, prompt specificity, and the availability of relevant literature in the model's training corpus [12 – 14].”
Why Large Language Models Cannot Be Certified for Safety-Critical Systems
Sayakov, A. (2026) · American Impact Review · doi:10.66308/air.e2026066
Citation Inaccuracies and the Need for Multi-Level Oversight in AI-Assisted Medical Writing
Rajaratnam, V., Omar, U. F., Kee, K., & Kaliya-Perumal, A.-K. (2026) · Standards · doi:10.3390/standards6010010 · Cited by 3
Altunışık, N., Toplu, S. A., & Türkmen, D. (2026) · Cutaneous and Ocular Toxicology · doi:10.1080/15569527.2026.2656177 · Cited by 1
Evaluating the Effectiveness of LLMs Against Manual Searches for Scientific Literature Retrieval
Acosta-Madrigal, C., Mena-Rodríguez, F., Pereira-Cerdas, L., Gonzalez-Villanueva, R., & Hidalgo‐Céspedes, J. (2026) · IEEE Jornadas Costarricenses de Investigación en Computación e Informática (JoCICI) · doi:10.1109/jocici69279.2026.11701711
Allaf, F., & Özcan, M. (2026) · Diagnostics · doi:10.3390/diagnostics16193180
Tan, J. C. K., Masselos, K., Nguyen, D., Agar, A., & Coroneo, M. T. (2026) · AJO International · doi:10.1016/j.ajoint.2026.100302
Sommerfeldt, J. M., Pierre, J. J., Fischer, J. L., RILEY, C. A., & Tolisano, A. M. (2026) · American Journal of Otolaryngology · doi:10.1016/j.amjoto.2026.104948
Chow, J. C., Yeh, Y.-T., & Chou, W. (2024) · doi:10.2196/preprints.64656
Civelekler, M., & Çıtırık, M. (2025) · Journal of Hospital Librarianship · doi:10.1080/15323269.2025.2576895
Wang, Z., Hu, M., Li, X., Zhang, Z., & Muwei, L. I. (2026) · Research Square · doi:10.21203/rs.3.rs-9358054/v1
Rathomi, A., Marjuni, & Afifudin (2026) · SIWAYANG JOURNAL PUBLIKASI ILMIAH BIDANG PARIWISATA KEBUDAYAAN DAN ANTROPOLOGI · doi:10.47353/siwayang.v5i1.5233
Semantic degradation of scientific text in AI editing
Semiletova, A. N. (2026) · Language and Text · doi:10.17759/langt.2026130216
Sharma, D., Singh, A., & Mahgoub, A. (2026) · Cureus · doi:10.7759/cureus.116639
Shimazaki, T., & Tachikawa, M. (2026) · Physical Chemistry Chemical Physics · doi:10.1039/d6cp01432a
Sajjad, M., Mirza, S. B., Sajjad, A., & Amer, R. (2026) · Cureus · doi:10.7759/cureus.117238
Liang, Z., Sheffield, C., Butera, G., & Antani, S. (2026) · JMIR AI · doi:10.2196/93761
Naser, M. Z. (2026) · Language Resources and Evaluation · doi:10.1007/s10579-026-09938-4 · Cited by 2
Rezaeian, M. (2026) · مجله علمی دانشگاه علوم پزشکی رفسنجان · doi:10.66224/jrums.24.10.887
Shiukashvili, N., Rochikashvili, M., Kupradze, V., Gonjilashvili, N., Gvajaia, N., Kutchava, L., Janikashvili, N., Tevzadze, N., Undilashvili, A., & Ekaladze, E. (2026) · Journal of Medical Education and Curricular Development · doi:10.1177/23821205261442106 · Cited by 3
Human Shadows in Machine Minds: Quantitative Study Interpreting AI Responses to the Rorschach Test
Csigó, K., & Cserey, G. (2026) · JMIR Mental Health · doi:10.2196/88186
Kurulkar, G., Ingale, S., Shinde, A., Jangale, Y. N., & Itkar, S. A. (2025) · Cureus Journal of Computer Science. · doi:10.7759/s44389-025-05957-4
Artifizielle Intelligenz in der Psychotherapie – werden Psychotherapeut*innen bald überflüssig?
Hohagen, F. (2025) · PSYCH up2date · doi:10.1055/a-2649-3274
AI-Powered Competency Modeling and Adaptive Content Generation for Language Proficiency Development
Sense, F., Dye, I., Collins, M., Swan, G. S., Krusmark, M., Smith, S. A., & Myers, T. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-30015-7_17
Hems, N. (2026) · Essentials · doi:10.1007/978-3-662-73363-9_2
Vanmala, & Dubey, A. K. (2025) · International Conference on Research in Computational Intelligence and Communication Networks (ICRCICN) · doi:10.1109/icrcicn68210.2025.11364831
Çalışır, A., Şahin, M., Türkoğlu, F., Şener, S., Duyan, A. G., & Alptekin, H. (2026) · Updates in Surgery · doi:10.1007/s13304-026-02802-8
Chérrez-Ojeda, I., Zuberbier, T., Rodas‐Valero, G., Sanchez, J. M., Rudenko, M., Dramburg, S., Demoly, P. M., Caimmi, D. P., Gómez, R. M., Ramon, G. D., et al. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.5042342
Ethics, Authorship, and Transparency: Responsible AI Use in Research and Writing
(2026) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3373-9340-7.ch004
Elmas, R., Ulutaş, M. A., & Yılmaz, M. (2026) · Journal of Biological Education · doi:10.1080/00219266.2026.2737444
Boyd, A., Srinivasan, M., Wijesinha, R., Coates, M., & Affandi, S. (2026) · International Journal of Cardiology Innovations · doi:10.1016/j.ijci.2026.100013
Mahmoudzadeh, R., Zaichik, M., Selvan, K., Islam, T., Salabati, M., & Leffler, C. T. (2025) · medRxiv · doi:10.1101/2025.06.09.25329271
Sökmen, D., Albayrak, A. T., Sertkaya, Z., Başağa, Y., & Şerefoğlu, E. C. (2025) · The Journal of Sexual Medicine · doi:10.1038/s41443-025-01200-9 · Cited by 1
Lad, H., Kwon, E., Chadha, A., Haimowitz, S. Z., Gold, B. S., Kaye, R. J., & Hsueh, W. D. (2026) · Journal of Otorhinolaryngology Hearing and Balance Medicine · doi:10.3390/ohbm7020028
Scallan, R. M., Atwood, B. I., Moser, C. H., Eberhardt, G. L., Stucky, C. H., & Kessinger, S. K. (2026) · Military Medicine · doi:10.1093/milmed/usag312
Drašković, N., Rešetar, M., & Vejzagić, V. (2026) · International Convention on Information and Communication Technology, Electronics and Microelectronics · doi:10.1109/mipro70003.2026.11591963 · Cited by 1
A Formal Approach to LLM Hallucination Detection and Mitigation
Wang, Z., Liu, Y., & Song, W. W. (2026) · International Conference on Cloud Computing and Big Data Analytics (ICCCBDA) · doi:10.1109/icccbda69573.2026.11676510
Will AI write the next ‘Chapter’ in literature reviews?
Blanc‐Durand, F., Koopman, M., Patel, S. P., Aldea, M. D., & Kather, J. N. (2025) · Annals of Oncology · doi:10.1016/j.annonc.2025.11.015
Spiță, M.-E., & Schipor, O.-A. (2026) · International Conference on Development and Application Systems (DAS) · doi:10.1109/das69882.2026.11553334 · Cited by 1
Sutcu, Y. B., Özcan, Ş. G., Dinçer, M. T., Atlı, Z., Trabulus, S., & Seyahi, N. (2026) · Kidney and Dialysis · doi:10.3390/kidneydial6010009 · Cited by 1
Is your curriculum GenAI-proof? A method for GenAI impact assessment and a case study
Jongkind, R., Elings, E., Joukes, E., Broens, T., Leopold, H., Wiesman, F., & Meinema, J. G. (2026) · MedEdPublish · doi:10.12688/mep.20815.2 · Cited by 1
Alhalawany, R. M., Khatatbeh, Y. M., & Jawkhab, A. A. (2026) · Healthcare · doi:10.3390/healthcare14172721
Bayala, Y. L. T., Tinni, I. A., Yaméogo, T. B. W.-Y., Porgo, A. N., Nacanabo, W. M., Traoré, A. D., Aziz, A., Kaboré, F., Tiendrébeogo, W. J. S. Z., & Ouedraogo, D. (2026) · Discover Artificial Intelligence · doi:10.1007/s44163-026-02394-7
OnkoRAG: Modelling Cross-Lingual German-Arabic Medical Question Answering
Jennings, N. G., Petukhova, V. V., & Klakow, D. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37249-9_54
Governing Artificial Intelligence in the Evolving Academic Publishing Ecosystem
Karaarslan, E., & Aydın, Ö. (2026) · Journal of Metaverse · doi:10.57019/jmv.1899774 · Cited by 1
Generating Synthetic Medical Dialogues in European Portuguese: Preliminary Results with GPT Models
Montenegro, L., Gomes, L. M., & Machado, J. M. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-05176-9_28
Impact of Generative AI on Operative Planning in Otolaryngology Residents
Hack, S., Hayu, D., Karni, J. E., Glikson, E., Remer, E., Alon, E. E., Ahmed, O. G., & Takashima, M. (2026) · Surgical Innovation · doi:10.1177/15533506261446833 · Cited by 2
Descamps, J., Bouguennec, N., Porcher, R., Resche‐Rigon, M., & Bouché, P.-A. (2026) · Orthopaedics & Traumatology Surgery & Research · doi:10.1016/j.otsr.2026.104793 · Cited by 2
Self-citationМинимизация ошибок нейросетевых моделей в библиотечно-издательской сфере: подходы и их эффективность
Рзянкин, И. С. (2025) · Sovremennye informacionnye tehnologii i IT-obrazovanie. · doi:10.25559/sitito.021.202502.272-286
Trewren, T., Gheihman, G., Bratkovic, K., Lin, D., & Stretton, B. (2026) · Perioperative Care and Operating Room Management · doi:10.1016/j.pcorm.2026.100732
Hallucinations in Scholarly LLMs: A Conceptual Overview and Practical Implications
Lamba, N., Tiwari, S., & Gaur, M. (2025) · Open Conference Proceedings · doi:10.52825/ocp.v8i.3175 · Cited by 3
Farhad, M., Masud, M. M., Rasheed, A. S. A., & Musthfa, N. (2026) · doi:10.1145/3793638.3793643
AI the Teacher? A Study on the Use of Artificial Intelligence Tools in Learning
Kobierski, M. (2025) · Linguistics Beyond and Within (LingBaW) · doi:10.31743/lingbaw.18219
Surabhi, A. S., & Martha, S. (2026) · International Journal of Engineering and Technology Innovation · doi:10.46604/ijeti.2026.16326
Using AI to Broaden the Reach and Inclusivity of Psychological PublicationsPreprint
Wingen, T., Lecuona, O. Z., & Dorrough, A. R. (2025) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/f4wrk_v1
Erümit, S. F. (2026) · European Journal of Education · doi:10.1111/ejed.70616
Nowacki, L., & Wrochna, A. E. (2026) · Communication Research and Practice · doi:10.1080/22041451.2026.2672876
Descamps, J., Bouguennec, N., Porcher, R., Resche‐Rigon, M., & Bouché, P.-A. (2026) · Revue de Chirurgie Orthopédique et Traumatologique · doi:10.1016/j.rcot.2026.07.013
Self-citationPre-service english teachers’ tendency to integrate ChatGPT into exam routines: a metaphor analysis
Çepni, S. B., & Çetinkaya, Ş. E. (2026) · SN Social Sciences · doi:10.1007/s43545-026-01500-x
Jung, E., Yoo, J., Kim, S. G., & Kim, Y. S. (2026) · Gastroenterology report · doi:10.1093/gastro/goag098
Peterson, G. M., Oermann, M. H., Owens, J. K., Templeton, G. F., Bailey, H. E., & Carter‐Templeton, H. (2026) · Research Square · doi:10.21203/rs.3.rs-10346244/v1
Boceck, D., Laugwitz, L., Sturm, M., Bezdan, D., Gschwind, A., Haack, T. B., & Ossowski, S. (2026) · npj Genomic Medicine · doi:10.1038/s41525-026-00611-x · Cited by 3
Herausforderungen beim Einsatz von KI in der qualitativen Forschung
Kempny, C., Annaç, K., Yılmaz-Aslan, Y., & Brzoska, P. (2026) · Springer eBooks · doi:10.1007/978-3-662-73089-8_2
Egbueri, J. C., Agbasi, J. C., Usman, A. G., Uwajingba, H. C., & Abba, S. I. (2026) · Springer hydrogeology · doi:10.1007/978-3-032-18853-3_6
Lee, K. S., Jankowitz, B. T., Hong, C., Cordeiro, J. G., Vilela-Filho, O., Prevedello, D. M., & Yang, I. (2026) · Neurosurgical Review · doi:10.1007/s10143-026-04462-z
Nutzung von künstlicher Intelligenz in Klinik und Praxis
Borgmann, H. (2026) · Die Urologie · doi:10.1007/s00120-026-02885-6
Ding, S. R., Ahmed, M., Malik, T., Somagani, R., & Vohra, F. F. (2025) · Cureus · doi:10.7759/cureus.98901 · Cited by 1
Civelekler, M., & Çıtırık, M. (2026) · Ocular Immunology and Inflammation · doi:10.1080/09273948.2026.2615858 · Cited by 1
Iarovoi, D., & Teh, P. L. (2026) · Lecture notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering · doi:10.1007/978-3-032-34792-3_11
Evaluating AI Performance in Systematic Literature Reviews for HEOR: A Case Study
Wang-Silvanto, J., Jajoo, M., Guo, H., Ohri, R., Nelissen, J., & Comabella, C. C. I. (2026) · Journal of health economics and outcomes research · doi:10.36469/001c.165173 · Cited by 1
Jay, M., Abi‐Jaoude, J., Straus, S. E., Eskander, A., Lipscombe, L. L., Dharma, C., & Yu, C. (2026) · Global Epidemiology · doi:10.1016/j.gloepi.2026.100282
Artificial Intelligence in Surgical Care: A Point-Counterpoint Analysis
Qadri, S., Kewalramani, D., Aboutanos, M. B., Narayan, M., & Quintana, M. T. (2026) · Trauma Surgery & Acute Care Open · doi:10.1136/tsaco-2025-002093 · Cited by 1
Li, J., Huang, S., Qiu, Y., Chen, W., Zhang, Y., Zhao, X., Qu, G., Bu, Y., Feng, X., Xu, J., et al. (2026) · JMIR Metascience and Research Integrity · doi:10.2196/preprints.105947
AI in the Age of Scientific Saturation: A Survey of How LLMs Rediscover the Obvious
Thang, H. V., & Vi, T. L. (2025) · International Conference on Circuit, Systems and Communication (ICCSC) · doi:10.1109/iccsc66714.2025.11135215
Metadata
- Authors
- Chelli, M., Descamps, J., Lavoué, V., Trojani, C., Azar, M., Deckert, M., Raynier, J.-L., Clowez, G., Boileau, P., & Ruetsch-Chelli, C.
- Affiliations
- Assistance Publique – Hôpitaux de Paris (FR)
- Inserm (FR)
- Corresponding author
- Chelli, Mikaël
- ORCID
- Venue
- Journal of Medical Internet Research
- Publisher
- JMIR Publications Inc.
- Published
- 2024-05-22
- Volume
- 26
- Pages
- e53164
- ISSN
- 1438-8871 (ISSN-L)
- PMC
- PMC11153973
- OpenAlex
- W4398203672
- Semantic Scholar
- 9a0e8c101b72e9094ea554194698daa3aeb3bc6f
- Cited by
- 602
- References
- 30
- Citations by year
- 2026: 378 · 2025: 202 · 2024: 22
- FWCI
- 11.51
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- Journal of Medical Internet Research (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- HAL (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- hallucinations, large language models, ChatGPT, Google Bard, Gemini, systematic reviews, reference retrieval, literature search automation, precision and recall, citation bias, GPT-3.5, GPT-4, AI in scientific writing
- MeSH terms
- Systematic Reviews as Topic, Artificial Intelligence
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-09-28, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-09-28 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.