The Illusion of Readiness in Health AI
Open accessAbstract
Large language models have demonstrated remarkable performance in a wide range of medical benchmarks. Yet underneath the seemingly promising results lie salient growth areas, especially in cutting-edge frontiers such as multimodal reasoning. In this paper, we introduce a series of adversarial stress tests to systematically assess the robustness of flagship models and medical benchmarks. Our study reveals prevalent brittleness in the presence of simple adversarial transformations: leading systems can guess the right answer even with key inputs removed, yet may get confused by the slightest prompt alterations, while fabricating convincing yet flawed reasoning traces. Using clinician-guided rubrics, we demonstrate that popular medical benchmarks vary widely in what they truly measure. Our study reveals significant competency gaps of frontier AI in attaining real-world readiness for health applications. If we want AI to earn trust in healthcare, we must demand more than leaderboard wins and must hold AI systems accountable to ensure robustness, sound reasoning, and alignment with real medical demands.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
CAT-based θ is not interpreted as clinical ability; instead, it is an operational proficiency index on a calibrated, secure knowledge bank under a fixed evaluation protocol.
“At the same time, consistent with this literature, we do not interpret CAT-based as clinical ability; instead, is an operational proficiency index on a calibrated, secure knowledge bank under a fixed evaluation protocol21.”
From Leveraging computerized adaptive testing for cost-effective evaluation of large language models in medical benchmarking (2026) · cites this paper as [21]
The Discussion (p. 12) cautions that benchmark success does not establish medical viability, supporting the decision not to equate a test score with clinical ability. But the paper does not address CAT-based θ or define it as an operational proficiency index on a calibrated, secure knowledge bank under a fixed protocol; searches for those terms found no matching passage. (LLM, conf=0.85)
Experimental — accuracy not yet measuredRecent studies caution that medical MCQ performance can be inflated by test-taking heuristics and may not reliably indicate clinically meaningful understanding or readiness.
“Recent studies caution that medical MCQ performance can be inflated by test-taking heuristics and may not reliably indicate clinically meaningful understanding or readiness20,21.”
From Leveraging computerized adaptive testing for cost-effective evaluation of large language models in medical benchmarking (2026) · cites this paper as [21]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
The citation lists haven’t been fetched yet
Cited by 1
Research works printed but not linked: 13
- [1][1] Singhal, K., Azizi, S., Tu, T., Mahdavi, S.S., Wei, J., Chung, H.W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. : Large language models encode clinical knowledge. Nature 620 (7972), 172-180 (2023) -
- [7][7] Arora, R.K., Wei, J., Hicks, R.S., Bowman, P., Qui˜ nonero-Candela, J., Tsimpourlas, F., Sharman, M., Shah, M., Vallone, A., Beutel, A., et al.: Healthbench: Evaluating large language models towards improved human health. arXiv preprint -
- [9][9] Farquhar, S., Kossen, J., Kuhn, L., Gal, Y.: Detecting hallucinations in large language models using semantic entropy. Nature 630 (8017), 625-630 (2024) -
- [10][10] Jin, Q., Chen, F., Zhou, Y., Xu, Z., Cheung, J.M., Chen, R., Summers, R.M., Rousseau, J.F., Ni, P., Landsman, M.J., et al. : Hidden flaws behind expert-level accuracy of multimodal gpt-4 vision in medicine. NPJ Digital Medicine 7 (1), 190 (2024) -
- [11][11] Wang, S., Zhao, F., Bu, D., Lu, Y., Gong, M., Liu, H., Yang, Z., Zeng, X., Yuan, Z., Wan, B., et al. : Lins: A general medical q&a framework for enhancing the quality and credibility of llm-generated responses. Nature Communications 16 (1), 9076 (2025) -
- [12][12] Acosta, J.N., Falcone, G.J., Rajpurkar, P., Topol, E.J.: Multimodal biomedical ai. Nature medicine 28 (9), 1773-1784 (2022) -
- [15][15] Medicine., T.N.E.J.: NEJM Image Challenge. https://www.nejm.org/ image-challenge. accessed: Jan 01, 2024 -
- [16][16] American Medical Association, J.: JAMA Challenge. https://jamanetwork.com/. accessed: Jan 01, 2024 -
- [20][20] Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q.V., Zhou, D., et al. : Chain-of-thought prompting elicits reasoning in large language models. In: Advances in Neural Information Processing Systems (NeurIPS) (2022) -
- [21][21] Lau, J.J., Gayen, S., Ben Abacha, A., Demner-Fushman, D.: A dataset of clinically generated visual questions and answers about radiology images. Scientific data 5 (1), 1-10 (2018) -
- [23][23] Johnson, A.E., Pollard, T.J., Berkowitz, S.J., Greenbaum, N.R., Lungren, M.P., Deng, C.-y., Mark, R.G., Horng, S.: Mimic-cxr, a de-identified publicly available database of chest radiographs with free-text reports. Scientific data 6 (1), 1-8 (2019) -
- [27][27] Yue, X., Ni, Y., Zhang, K., Zheng, T., Liu, R., Zhang, G., Stevens, S., Jiang, D., Ren, W., Sun, Y., Wei, C., Yu, B., Yuan, R., Sun, R., Yin, M., Zheng, B., Yang, Z., Liu, Y., Huang, W., Sun, H., Su, Y., Chen, W.: Mmmu: A massive multidiscipline multimodal understanding and reasoning benchmark for expert agi. In: Proceedings of CVPR (2024) -
- [29][29] Wu, Z., Chen, X., Pan, Z., Liu, X., Liu, W., Dai, D., Gao, H., Ma, Y., Wu, C., Wang, B., Xie, Z., Wu, Y., Hu, K., Wang, J., Sun, Y., Li, Y., Piao, Y., Guan, K., Liu, A., Xie, X., You, Y., Dong, K., Yu, X., Zhang, H., Zhao, L., Wang, Y., Ruan, C.: DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understanding (2024). https://arxiv.org/abs/2412.10302 - · arXiv
- Authors
- Gu, Y., Fu, J., Liu, X., Valanarasu, J. M. J., Codella, N. C., Tan, R., Liu, Q., Jin, Y., Zhang, S., Wang, J., Wang, R., Song, L., Qin, G., Usuyama, N., Wong, C., Cheng, H., Lee, H., Sanapathi, P., Hilado, S., Naumann, T., Alvarez-Valle, J., Bian, J., Wei, M., Malik, K., Zhou, L., Gao, J., Horvitz, E., Lungren, M. P., Burger, D., Topol, E., Poon, H., & Vozila, P.
- ORCID
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2025
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2509.18234
- OpenAlex
- W4414803090
- Cited by
- 1
- References
- 0
- Citations by year
- 2026: 1
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Keywords
- large language models, health AI, adversarial robustness, multimodal reasoning, AI accountability
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
