Human-AI collaboration in large language model-assisted brain MRI differential diagnosis: a usability study
Open accessThe DOI matches its Crossref record; 9 of the 9 references it cites were found in a registry or an index.
Abstract
Objectives: This study investigated the impact of human-large language model (LLM) collaboration on the accuracy and efficiency of brain MRI differential diagnosis. Materials and methods: In this retrospective study, forty brain MRI cases with a challenging but definitive diagnosis were randomized into two groups of twenty cases each. Six radiology residents with an average experience of 6.3 months in reading brain MRI exams evaluated one set of cases supported by conventional internet search (Conventional) and the other set utilizing an LLM-based search engine and hybrid chatbot. A cross-over design ensured that each case was examined with both workflows in equal frequency. For each case, readers were instructed to determine the three most likely differential diagnoses. LLM responses were analyzed by a panel of radiologists. Benefits and challenges in human-LLM interaction were derived from observations and participant feedback. Results: LLM-assisted brain MRI differential diagnosis yielded superior accuracy (70/114; 61.4% (LLM-assisted) vs 53/114; 46.5% (conventional) correct diagnoses, p = 0.033, chi-square test). No difference in interpretation time or level of confidence was observed. An analysis of LLM responses revealed that correct LLM suggestions translated into correct reader responses in 82.1% of cases (60/73). Inaccurate case descriptions by readers (9.2% of cases), LLM hallucinations (11.5% of cases), and insufficient contextualization of LLM responses were identified as challenges related to human-LLM interaction. Conclusion: Human-LLM collaboration has the potential to improve brain MRI differential diagnosis. Yet, several challenges must be addressed to ensure effective adoption and user acceptance. Key Points: Question While large language models (LLM) have the potential to support radiological differential diagnosis, the role of human-LLM collaboration in this context remains underexplored. Findings LLM-assisted brain MRI differential diagnosis yielded superior accuracy over conventional internet search. Inaccurate case descriptions, LLM hallucinations, and insufficient contextualization were identified as potential challenges. Clinical relevance Our results highlight the potential of an LLM-assisted workflow to increase diagnostic accuracy but underline the necessity to study collaborative efforts between humans and LLMs over LLMs in isolation. Graphical Abstract:
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Toward expert-level medical question answering with large language models
Singhal, K., Tu, T., Gottweis, J., Sayres, R., Wulczyn, E., Amin, M., Hou, L., Clark, K., Pfohl, S. R., Cole-Lewis, H., et al. (2025) · Nature Medicine · doi:10.1038/s41591-024-03423-7 · Cited by 1,734
Li, Y., Li, Z., Zhang, K., Dan, R., Jiang, S., & Zhang, Y. (2023) · arXiv · doi:10.48550/arxiv.2303.14070 · Cited by 512
Creation and Adoption of Large Language Models in Medicine
Shah, N. H., Entwistle, D., & Pfeffer, M. A. (2023) · JAMA · doi:10.1001/jama.2023.14217 · Cited by 435 · cited 1×
“In real-world practice, it is plausible that radiologists or radiology residents would use LLMs as an adjunct tool to support diagnostic reasoning rather than for fully autonomous differential diagnosis [11].”
Dratsch, T., Chen, X., Rezazade Mehrizi, M., Kloeckner, R., Mähringer-Kunz, A., Püsken, M., Baeßler, B., Sauer, S., Maintz, D., & Pinto dos Santos, D. (2023) · Radiology · doi:10.1148/radiol.222176 · Cited by 369 · cited 2×
“One study employing an AI-based mammogram classi fi cation system demonstrated that inexperienced and experienced readers alike are susceptible to automation bias, which describes the inclination of human users to adhere to incorrect recommendations from automated decision-making systems [8].”
Towards Expert-Level Medical Question Answering with Large Language Models
Singhal, K., Tu, T., Gottweis, J., Sayres, R., Wulczyn, E., Hou, L., Clark, K., Pfohl, S., Cole-Lewis, H., Neal, D., et al. (2023) · arXiv · doi:10.48550/arxiv.2305.09617 · Cited by 338
Med-HALT: Medical Domain Hallucination Test for Large Language Models
Pal, A., Umapathi, L. K., & Sankarasubbu, M. (2023) · Proceedings of the 27th Conference on Computational Natural Language Learning (CoNLL) · doi:10.18653/v1/2023.conll-1.21 · Cited by 152
Med-HALT: Medical Domain Hallucination Test for Large Language Models
Pal, A., Umapathi, L. K., & Sankarasubbu, M. (2023) · arXiv · doi:10.48550/arxiv.2307.15343 · Cited by 152
Bernstein, M. H., Atalay, M. K., Dibble, E. H., Maxwell, A. W. P., Karam, A. R., Agarwal, S., Ward, R. C., Healey, T. T., & Baird, G. L. (2023) · European Radiology · doi:10.1007/s00330-023-09747-1 · Cited by 140 · cited 1×
“Similarly, incorrect AI results were shown to negatively impact radiologist performance in lung cancer detection based on chest radiography [9].”
Sarangi, P. K., Irodi, A., Panda, S., Nayak, D. S. K., & Mondal, H. (2024) · Indian Journal of Radiology and Imaging · doi:10.1055/s-0043-1777289 · Cited by 28
Cited by
Next-generation agentic AI for transforming healthcare
Karunanayake, N. (2025) · Informatics and Health · doi:10.1016/j.infoh.2025.03.001 · Cited by 165
Clinical insights: A comprehensive review of language models in medicine
Neveditsin, N., Lingras, P., & Mago, V. (2025) · PLOS Digital Health · doi:10.1371/journal.pdig.0000800 · Cited by 37
“…[173,179] Conversation Mental Health Bots [182–184], Medical Chatbots and Health Assis-tants [80,185,186], Triaging [187], Differential Diagnosis [188] MotiVAte [189], Depression_Reddit [190], CLPsych [191], Dreaddit [192], Clinical Vignettes [187] MedPaLM, DRG-LLaMA [193], openCHA [80]…”
Agentic AI and large language models in radiology: opportunities and hallucination challenges
Salehi, S., Singh, Y., Horst, K. K., Hathaway, Q. A., & Erickson, B. J. (2025) · Bioengineering · doi:10.3390/bioengineering12121303 · Cited by 32
“…across current systems —defined as AI-generated outputs that appear plausible but contain factually incorrect or fabricated medical information, generating plausible but incorrect anatomical descriptions, phantom lesions, or mischaracterized pathologies that directly threaten patient safety [4,5].”
Metadata
- Authors
- Kim, S. H., Wihl, J., Schramm, S., Berberich, C., Rosenkranz, E., Schmitzer, L., Serguen, K., Klenk, C., Lenhart, N., Zimmer, C., Wiestler, B., & Hedderich, D. M.
- Venue
- European Radiology
- Published
- 2025-03-07
- Volume
- 35
- Issue
- 9
- Pages
- 5252-5263
- ISSN
- 0938-7994 (ISSN-L)
- 1432-1084
- OpenAlex
- W4408215286
- Cited by
- 38
- References
- 9
- FWCI
- 6.96
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Human-AI collaboration in large language model-assisted brain MRI differential diagnosis: a usability study. https://app.cochat.ai/papers/10.1007/s00330-025-11484-6
- The paper
- Kim, S. H., Wihl, J., Schramm, S., Berberich, C., Rosenkranz, E., Schmitzer, L., Serguen, K., Klenk, C., Lenhart, N., Zimmer, C., Wiestler, B., & Hedderich, D. M. (2025). Human-AI collaboration in large language model-assisted brain MRI differential diagnosis: a usability study. European Radiology. https://doi.org/10.1007/s00330-025-11484-6
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.