Pattern Recognition or Medical Knowledge? The Problem with Multiple-Choice Questions in Medicine
Open accessPublished version of a preprint
Abstract
Large Language Models (LLMs) such as Chat-GPT demonstrate significant potential in the medical domain and are often evaluated using multiple-choice questions (MCQs) modeled on exams like the USMLE.However, such benchmarks may overestimate true clinical understanding by rewarding pattern recognition and test-taking heuristics.To investigate this, we created a fictional medical benchmark centered on an imaginary organ, the Glianorex, allowing us to separate memorized knowledge from reasoning ability.We generated textbooks and MCQs in English and French using leading LLMs, then evaluated proprietary, open-source, and domain-specific models in a zero-shot setting.Despite the fictional content, models achieved an average score of 64%, while physicians scored only 27%.Fine-tuned medical models outperformed base models in English but not in French.Ablation and interpretability analyses revealed that models frequently relied on shallow cues, test-taking strategies, and hallucinated reasoning to identify the correct choice.These results suggest that standard MCQ-based evaluations may not effectively measure clinical reasoning and highlight the need for more robust, clinically meaningful assessment methods for LLMs.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Real-world deployments of clinical LLM assistants in electronic health records contexts emphasize summarization, information retrieval, and documentation support rather than autonomous clinical decision-making.
“This positioning also aligns with observed real-world deployments of clinical LLM assistants in electronic health records contexts, where primary use cases emphasize summarization, information retrieval, and documentation support rather than autonomous clinical decision-making22.”
From Leveraging computerized adaptive testing for cost-effective evaluation of large language models in medical benchmarking (2026) · cites this paper as [22]
The Medical Implications section discusses risks of clinical LLM use and clinical decision support, but does not describe observed deployments in electronic health records or their primary use cases. Searches for EHR summarization, information retrieval, and documentation support found no passage addressing that comparison. (LLM, conf=0.76)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 6
- Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring Massive Multitask Language Un- derstanding. Proceedings of the International Con- ference on Learning Representations (ICLR).
- Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. 2020. What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams. arXiv preprint. ArXiv:2009.13081 [cs] version: 1. · arXiv
- Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Ef- ficient Memory Management for Large Language Model Serving with PagedAttention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles.
- Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. 2022. MedMCQA: A Large-scale Multi-Subject Multi-Choice Dataset for Medical do- main Question Answering. In Proceedings of the Conference on Health, Inference, and Learning, vol- ume 174 of Proceedings of Machine Learning Re- search, pages 248-260.
- PMLR.
- Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging LLM-as-a-judge with MT-bench and Chatbot Arena. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS '23, Red Hook, NY, USA.
Cited by
Sorted by citations
Large Language Models lack essential metacognition for reliable medical reasoning
Griot, M., Hemptinne, C., Vanderdonckt, J., & Yuksel, D. (2025) · Nature Communications · doi:10.1038/s41467-024-55628-6 · Cited by 113 · see citing works
Self-citationImplementation of large language models in electronic health records
Griot, M., Vanderdonckt, J., & Yuksel, D. (2025) · PLOS Digital Health · doi:10.1371/journal.pdig.0001141 · Cited by 23 · see citing works
Self-citationMedMobile: a mobile-sized language model with clinical capabilities
Vishwanath, K., Stryker, J., Alyakin, A., Alber, D. A., & Oermann, E. K. (2025) · BMJ Digital Health & AI · doi:10.1136/bmjdhai-2025-000068 · Cited by 3 · see citing works
Zheng, T., Jiang, Z., Liu, J., & Feng, S. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-02671-w · Cited by 1 · see citing works
Do Language Models Think Like Doctors?Preprint
McCoy, L. G., Swamy, R., Sagar, N., Wang, M., Cao, J., Bacchi, S., Fong, J. M. N., Tan, N. C., Tan, K. C., Buckley, T. A., et al. (2025) · medRxiv · doi:10.1101/2025.02.11.25321822 · Cited by 1 · see citing works
Athlete-LLMs: A Specialized Large Language Model for Athlete Psychology
Yang, T., Hu, D., & Yang, B. (2024) · IEEE International Conference on Computer and Communications (ICCC) · doi:10.1109/iccc62609.2024.10941926
A hybrid deployment model for generative artificial intelligence in hospitals
Griot, M., Hemptinne, C., Vanderdonckt, J. M., & Yüksel, D. (2025) · Machine Learning Health · doi:10.1088/3049-477x/addb51
Self-citationPerformance of reasoning large language models on nephrology multiple-choice questionsPreprint
Kitano, F., Masaki, M., Ichikawa, D., Shibagaki, Y., & Noda, R. (2025) · medRxiv · doi:10.64898/2025.12.03.25341427
Shi, J., Chen, H., Shi, C., Su, C., Yue, S., Li, W., Liu, X., Zhou, Y., Zhong, Y., Chen, B., et al. (2026) · BMC Nursing · doi:10.1186/s12912-026-04647-3
Performance of Large Language Models in Neurology Multiple‐Choice Questions
Habibi, G., Gargari, O. K., Hosseini, M., & Afchangi, K. (2026) · Acta Neurologica Scandinavica · doi:10.1155/ane/5623086
Evaluating Open-Weight Large Language Models for a Local Clinical Assistant
Pires, M. J., & Véstias, M. (2026) · International Young Engineers Forum (YEF-ECE) · doi:10.1109/yef-ece70590.2026.11614644
A Comprehensive Evaluation of the Phi Family of Small Language Models in Healthcare
Awasthi, R., Shree, S., Bhattad, A., Mishra, S., Singh, N., Khare, A., Edara, S. R., Ramachandran, S. P., Mahapatra, D., Cywinski, J. B., et al. (2026) · IEEE International Conference on Healthcare Informatics (ICHI) · doi:10.1109/ichi69079.2026.00108
Ran, X., Yin, H., Yang, Y., Duan, S., Luo, X., Liu, Y., & Liao, M. (2026) · doi:10.2196/preprints.111146
Ran, X., Yin, H., Yang, Y., Duan, S., Luo, X., Liu, Y., & Liao, M. (2026) · JMIR Medical Education · doi:10.2196/111146
- Authors
- Griot, M., Vanderdonckt, J., Yuksel, D., & Hemptinne, C.
- Affiliations
- UCLouvain (BE)
- Cliniques Universitaires Saint-Luc (BE)
- Venue
- Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
- Published
- 2025
- Pages
- 5321-5341
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W4412945732
- Cited by
- 13
- References
- 0
- Citations by year
- 2026: 6 · 2025: 6 · 2024: 1
- FWCI
- 12.48
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, cc-by-nc-sa)
- Digital Access to Libraries (submittedVersion, other-oa)
- arXiv (Cornell University)
- Funders
- Fédération Wallonie-Bruxelles
- Fondation Saint Luc
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Medicine › Family Practice › Clinical Reasoning and Diagnostic Skills
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Keywords
- large language models, multiple-choice questions, medical knowledge assessment, clinical reasoning, pattern recognition, hallucinations, zero-shot evaluation, multilingual language models, interpretability, LLM benchmark
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
