Diagnostic accuracy and citation integrity of four large language models on otolaryngology vignettes
Abstract
Objectives This study aimed to compare the diagnostic accuracy and citation integrity of four large language models (LLMs) including one general (ChatGPT-4) and three intended for clinical and research use (OpenEvidence, Perplexity, and Pathway), using standardized otolaryngology clinical vignettes. Methods One hundred validated otolaryngology clinical vignettes were presented to each LLM with a prompt requesting both a diagnosis and supporting citations. Diagnostic accuracy was determined against reference answers, and errors were categorized as logical, informational, or explicit. Citation number, source type, hallucination rate, and journal CiteScores were also compared. Results All models demonstrated high diagnostic accuracy (82.0%-91.0%), with ChatGPT-4 achieving the highest numerical accuracy (91.0%), though differences between models were not statistically significant (p = 0.057). Logical errors were most frequent across all models. OpenEvidence and Perplexity generated the most citations per response, while ChatGPT-4 produced the fewest and had the highest hallucination rate (23.0%). Source preferences varied, with OpenEvidence and Pathway favoring narrative reviews and Perplexity favoring government/public health websites. OpenEvidence had the highest mean CiteScore for journal citations. Conclusion This is the first study to assess both diagnostic accuracy and citation integrity of LLMs in otolaryngology. While ChatGPT-4 was most accurate, it had the highest rate of citation hallucinations, suggesting a trade-off between accuracy and source reliability. OpenEvidence, though slightly less accurate, provided more consistent and verifiable references, demonstrating the ability to prioritize citation integrity alongside diagnostic performance for clinical integration.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Cites
Survey of Hallucination in Natural Language Generation
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023) · ACM Computing Surveys · doi:10.1145/3571730 · Cited by 5,025 · cited 1× · see citing works
“This aligns with recent findings showing that even high-performing LLMs, including those with RAG capabilities, frequently cite unverifiable or misattributed references [13, 27–30 ].”
The imperative for regulatory oversight of large language models (or generative AI) in healthcare
Meskó, B., & Topol, E. J. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00873-0 · Cited by 1,047 · see citing works
Improving the accuracy of medical diagnosis with causal machine learning
Richens, J. G., Lee, C. M., & Johri, S. (2020) · Nature Communications · doi:10.1038/s41467-020-17419-7 · Cited by 598 · see citing works
Regulating ChatGPT and other Large Generative AI Models
Hacker, P., Engel, A., & Mauer, M. (2023) · 2023 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3593013.3594067 · Cited by 466 · cited 1× · see citing works
“This concern is mirrored in ongoing regulatory conversations; the FDA and other international bodies have emphasized the importance of transparency, explainability, and auditability in clinical decision support tools, particularly those driven by machine learning [39, 40 ].”
Trust and medical AI: the challenges we face and the expertise needed to overcome them
Quinn, T. P., Senadeera, M., Jacobs, S., Coghlan, S., & Le, V. (2021) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaa268 · Cited by 287 · see citing works
Ullah, E., Parwani, A., Baig, M. M., & Singh, R. (2024) · Diagnostic Pathology · doi:10.1186/s13000-024-01464-7 · Cited by 281 · see citing works
Retrieval augmented generation for large language models in healthcare: A systematic review
Amugongo, L. M., Mascheroni, P., Brooks, S., Doering, S., & Seidel, J. (2025) · PLOS Digital Health · doi:10.1371/journal.pdig.0000877 · Cited by 267 · see citing works
Savage, T., Nayak, A., Gallo, R., Rangan, E., & Chen, J. H. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01010-1 · Cited by 265 · see citing works
Artificial intelligence and increasing misinformation
Monteith, S., Glenn, T., Geddes, J. R., Whybrow, P. C., Achtyes, E., & Bauer, M. (2024) · The British Journal of Psychiatry · doi:10.1192/bjp.2023.136 · Cited by 208 · see citing works
Enhancing diagnostic capability with multi-agents conversational large language models
Chen, X., Yi, H., You, M., Liu, W., Wang, L., Li, H., Zhang, X., Guo, Y., Fan, L., Chen, G., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01550-0 · Cited by 117 · see citing works
Enhancing medical AI with retrieval-augmented generation: A mini narrative review
Gargari, O. K., & Habibi, G. (2025) · Digital Health · doi:10.1177/20552076251337177 · Cited by 105 · see citing works
Abouammoh, N. A., Alhasan, K. A., Aljamaan, F., Raina, R., Malki, K. H., Altamimi, I., Muaygil, R. A., Wahabi, H., Jamal, A. A., Alhaboob, A. A. N., et al. (2025) · JMIR Medical Education · doi:10.2196/63400 · Cited by 101 · see citing works
“Doctor ChatGPT, Can You Help Me?” The Patient’s Perspective: Cross-Sectional Study
Armbruster, J., Bussmann, F., Rothhaas, C., Titze, N., Grützner, P. A., & Freischmidt, H. (2024) · Journal of Medical Internet Research · doi:10.2196/58831 · Cited by 86 · see citing works
Masters, K. (2023) · Medical Teacher · doi:10.1080/0142159x.2023.2208731 · Cited by 76 · see citing works
Ganjavi, C., Eppler, M., O’Brien, D., Ramacciotti, L. S., Ghauri, M. S., Anderson, I., Choi, J., Dwyer, D., Stephens, C., Shi, V., et al. (2024) · PLOS Digital Health · doi:10.1371/journal.pdig.0000596 · Cited by 73 · see citing works
An automated framework for assessing how well LLMs cite relevant medical references
Wu, K., Wu, E., Wei, K., Zhang, A., Casasola, A., Nguyen, T., Riantawan, S., Shi, P., Ho, D., & Zou, J. (2025) · Nature Communications · doi:10.1038/s41467-025-58551-6 · Cited by 72 · cited 1× · see citing works
“This aligns with recent findings showing that even high-performing LLMs, including those with RAG capabilities, frequently cite unverifiable or misattributed references [13, 27–30 ].”
Shan, G., Chen, X., Wang, C., Liu, L., Gu, Y., Jiang, H., & Shi, T. (2025) · JMIR Medical Informatics · doi:10.2196/64963 · Cited by 71 · see citing works
Application of large language models in disease diagnosis and treatment
Yang, X., Li, T., Su, Q., Liu, Y., Kang, C., Lyu, Y., Zhao, L., Nie, Y., & Pan, Y. (2024) · Chinese Medical Journal · doi:10.1097/cm9.0000000000003456 · Cited by 68 · see citing works
Diagnostic and Management Applications of ChatGPT in Structured Otolaryngology Clinical Scenarios
Qu, R. W., Qureshi, U., Petersen, G., & Lee, S. C. (2023) · OTO Open · doi:10.1002/oto2.67 · Cited by 62 · see citing works
Ma, R., Cheng, Q., Yao, J., Peng, Z., Yan, M., Lu, J., Liao, J., Tian, L., Shu, W., Zhang, Y., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01461-0 · Cited by 60 · see citing works
ChatGPT and Clinical Training: Perception, Concerns, and Practice of Pharm-D Students
Zawiah, M., Al-Ashwal, F., Gharaibeh, L., Abu Farha, R., Alzoubi, K., Abu Hammour, K., Qasim, Q. A., & Abrah, F. (2023) · Journal of Multidisciplinary Healthcare · doi:10.2147/jmdh.s439223 · Cited by 51 · see citing works
Goisauf, M., Abadía, M. C., Akyüz, K., Bobowicz, M., Buyx, A., Colussi, I. A., Fritzsche, M.-C., Lekadir, K., Marttinen, P., Mayrhofer, M. T., et al. (2025) · Journal of Medical Internet Research · doi:10.2196/71236 · Cited by 47 · see citing works
Long, C., Subburam, D., Lowe, K., dos Santos, A., Zhang, J. J., Hwang, S. W., Saduka, N., Horev, Y., Su, T., Côté, D. W. J., et al. (2024) · Otolaryngology · doi:10.1002/ohn.864 · Cited by 46 · see citing works
ChatGPT Versus Consultants: Blinded Evaluation on Answering Otorhinolaryngology Case–Based Questions
Buhr, C. R., Smith, H., Huppertz, T., Bahr‐Hamm, K., Matthias, C., Blaikie, A. J., Kelsey, T., Kühn, S., & Eckrich, J. (2023) · JMIR Medical Education · doi:10.2196/49183 · Cited by 45 · see citing works
Perceptions of ChatGPT in healthcare: usefulness, trust, and risk
Chen, S., Kuo, H.-Y., & Chang, S.-H. (2024) · Frontiers in Public Health · doi:10.3389/fpubh.2024.1457131 · Cited by 45 · see citing works
Influence of a Large Language Model on Diagnostic Reasoning: A Randomized Clinical Vignette Study
Goh, E., Gallo, R., Hom, J., Strong, E., Weng, Y., Kerman, H., Cool, J., Kanjee, Z., Parsons, A. S., Ahuja, N., et al. (2024) · doi:10.1101/2024.03.12.24303785 · Cited by 38 · see citing works
Evaluation of GPT-3.5 and GPT-4 for supporting real-world information needs in healthcare deliveryPreprint
Dash, D., Thapa, R., Banda, J. M., Swaminathan, A., Cheatham, M., Kashyap, M., Kotecha, N., Chen, J. H., Gombar, S., Downing, N. L., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2304.13714 · Cited by 33 · see citing works
Citation-Enhanced Generation for LLM-based Chatbots
Li, W., Li, J., Ma, W., & Liu, Y. (2024) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2024.acl-long.79 · Cited by 31 · see citing works
The Comparative Diagnostic Capability of Large Language Models in Otolaryngology
Warrier, A. K., Singh, R., Haleem, A., Zaki, H., & Eloy, J. A. (2024) · The Laryngoscope · doi:10.1002/lary.31434 · Cited by 26 · see citing works
Self-citationA comparison of the diagnostic ability of large language models in challenging clinical cases
Khan, M. P., & O'Sullivan, E. D. (2024) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2024.1379297 · Cited by 14 · see citing works
Huang, R. S., Benour, A., Kemppainen, J., & Leung, F. (2024) · BMC Medical Education · doi:10.1186/s12909-024-06115-5 · Cited by 13 · see citing works
New FDA Policies Could Limit the Full Value of AI in Medicine
Gottlieb, S. L. (2025) · JAMA Health Forum · doi:10.1001/jamahealthforum.2025.0289 · Cited by 10 · see citing works
Ahn, J., Kang, B. G., Chang, M. Y., & Yoon, S. (2025) · Clinical and Experimental Otorhinolaryngology · doi:10.21053/ceo.2025-00121 · Cited by 10 · see citing works
Mayo‐Yáñez, M., Rameau, A., Vaira, L. A., Louvrier, M., Barrueco, Á. S., Rueda, I. A., García‐Curdi, F., Mejuto-Torreiro, L., Klein‐Rodríguez, A., Herranz-Larrañeta, J., et al. (2025) · Ear Nose & Throat Journal · doi:10.1177/01455613251351774 · Cited by 8 · see citing works
ChatGPT 4.0 and algor in generating concept maps: an observational study
Maniaci, A., Gagliano, C., Salerno, V. M., et al. (2025) · European Archives of Oto-Rhino-Laryngology · doi:10.1007/s00405-025-09255-6 · Cited by 5 · see citing works
Domain‐Specific Customization for Language Models in Otolaryngology: The ENT GPT Assistant
Bicknell, B. T., Rivers, N. J., Skelton, A., Sheehan, D. H., Hodges, C. C., Fairburn, S. C., Greene, B., & Panuganti, B. A. (2025) · OTO Open · doi:10.1002/oto2.70125 · Cited by 4 · see citing works
Detecting Reference Errors in Scientific Literature with Large Language Models
Zhang, T. M., & Abernethy, N. F. (2024) · arXiv · doi:10.48550/arxiv.2411.06101 · Cited by 3 · see citing works
Gupta, G., Singh, A., Manikandan, S. V., & Ehtesham, A. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2405.06712 · Cited by 2 · see citing works
Flagged first, then by citations
Research works printed but not linked: 2
- Pensak ML, Hart CK, Patil YJ (2018) Otolaryngology Cases: The University of Cincinnati Clinical Portfolio, Second edition. Thieme
- Scopus CiteScore | Elsevier. Accessed August 10, 2025. https://www.elsevier.com/products/scopus/metrics/citescore
Cited by
Still being checked.
- Authors
- Pennington-FitzGerald, W., Warrier, A., Durant, S., Sharaf, I., Carlino, F., Pamula, S. D., Eloy, J. A., & Levi, J.
- Affiliations
- Boston University (US)
- Rutgers New Jersey Medical School (US)
- Corresponding author
- Pennington-FitzGerald, William
- Venue
- European Archives of Oto-Rhino-Laryngology
- Published
- 2026-05-13
- Volume
- 283
- Issue
- 7
- Pages
- 4707-4714
- ISSN
- 0937-4477 (ISSN-L)
- 1434-4726
- OpenAlex
- W7160917631
- Semantic Scholar
- f9812db7f3170ffea402d1f5f97a0219d1eb18c1
- Cited by
- 0
- References
- 38
- FWCI
- 0.00
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- Locations
- European Archives of Oto-Rhino-Laryngology (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Health Sciences › Medicine › Family Practice › Clinical Reasoning and Diagnostic Skills
- Keywords
- large language models, otolaryngology, diagnostic accuracy, citation integrity, hallucinations, ChatGPT-4, OpenEvidence, perplexity, pathways, clinical vignettes, source reliability, CiteScore, clinical decision support
- MeSH terms
- Large Language Models, Humans, Otolaryngology
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
