MedAgentBench: a virtual EHR environment to benchmark medical LLM agents
Open accessAbstract
BACKGROUND Recent large language models (LLMs) have demonstrated significant advancements, particularly in their ability to serve as agents, thereby surpassing their traditional role as chatbots.These agents can leverage their planning and tool utilization capabilities to address tasks specified at a high level.This suggests new potential to reduce the burden of administrative tasks and address current health care staff shortages.However, a standardized dataset to benchmark the agent capabilities of LLMs in medical applications is currently lacking, making it difficult to evaluate their performance on complex tasks in interactive health care environments. METHODSTo address this gap in the deployment of agentic artificial intelligence (AI) in health care, we introduce MedAgentBench, a broad evaluation suite designed to assess the agent capabilities of LLMs within medical records contexts.MedAgentBench encompasses 300 patient-specific clinically derived tasks from 10 categories written by human physicians, realistic profiles of 100 patients with over 700,000 data elements, a Fast Healthcare Interoperability Resources-compliant interactive environment, and an accompanying codebase.The environment uses standard application programming interfaces and communication infrastructure used in modern electronic health record (EHR) systems so that it can be easily migrated into live EHR systems. RESULTSMedAgentBench presents an unsaturated agent-oriented benchmark at which current state-of-the-art LLMs exhibit some ability to succeed.The best model (Claude 3.5 Sonnet v2) achieves a success rate of 69.67%.However, there is still substantial room for improvement, which gives the community a clear direction for future optimization efforts.Furthermore, there is significant variation in performance across task categories.CONCLUSIONS Agent-based task frameworks and benchmarks are the necessary next step to advance the potential and capabilities for effectively improving and integrating AI systems into clinical workflows.MedAgentBench establishes this and is publicly available at https://github .com /stanfordmlgroup /MedAgentBench, offering a valuable framework for model developers to track progress and drive continuous improvements in the agent capabilities of LLMs within the medical domain.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Current benchmarks primarily evaluate isolated predictions on static datasets and fail to capture the performance of agents in multistep clinical workflows where intermediate decisions influence subsequent actions.
“Current benchmarks primarily evaluate isolated predictions on static datasets and fail to capture the performance of agents in multistep clinical workflows where intermediate decisions influence subsequent actions [34].”
From Medical AI Agents for Clinical Decision Support: Viewpoint Using the Planning, Action, Reflection, and Memory (PARM) Analytical Lens (2026) · cites this paper as [34]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Allocation of Physician Time in Ambulatory Practice: A Time and Motion Study in 4 Specialties
Sinsky, C., Colligan, L., Li, L., Prgomet, M., Reynolds, S., Goeders, L., Westbrook, J., Tutty, M., & Blike, G. (2016) · Annals of Internal Medicine · doi:10.7326/m16-0961 · Cited by 1,494 · see citing works
Ethical and regulatory challenges of AI technologies in healthcare: A narrative review
Mennella, C., Maniscalco, U., De Pietro, G., & Esposito, M. (2024) · Heliyon · doi:10.1016/j.heliyon.2024.e26297 · Cited by 1,083 · see citing works
Trust and medical AI: the challenges we face and the expertise needed to overcome them
Quinn, T. P., Senadeera, M., Jacobs, S., Coghlan, S., & Le, V. (2021) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaa268 · Cited by 287 · see citing works
Gorilla: Large Language Model Connected with Massive APIs
Patil, S. G., Zhang, T., Wang, X., & Gonzalez, J. E. (2023) · arXiv · doi:10.48550/arxiv.2305.15334 · Cited by 226 · see citing works
Ellahham, S., Ellahham, N., & Simsekler, M. C. E. (2020) · American Journal of Medical Quality · doi:10.1177/1062860619878515 · Cited by 177 · see citing works
Artificial Intelligence in U.S. Health Care Delivery
Sahni, N. R., & Carrus, B. (2023) · New England Journal of Medicine · doi:10.1056/nejmra2204673 · Cited by 175 · see citing works
Survey on large language model-enhanced reinforcement learning: Concept, taxonomy, and methods
Cao, Y., Zhao, H., Cheng, Y., Shu, T., Chen, Y., Liu, G., Liang, G., Zhao, J., Yan, J., & Li, Y. (2025) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2024.3497992 · Cited by 174 · see citing works
LLM-based agentic systems in medicine and healthcare
Qiu, J., Lam, K., Li, G., Acharya, A., Wong, T. Y., Darzi, A., Yuan, W., & Topol, E. J. (2024) · Nature Machine Intelligence · doi:10.1038/s42256-024-00944-1 · Cited by 168 · see citing works
Meta-Prompting: Enhancing Language Models with Task-Agnostic ScaffoldingPreprint
Suzgun, M., & Kalai, A. (2024) · arXiv.org · doi:10.48550/arxiv.2401.12954 · Cited by 159 · see citing works
The rise of agentic AI teammates in medicine
Zou, J., & Topol, E. J. (2025) · The Lancet · doi:10.1016/s0140-6736(25)00202-8 · Cited by 138 · see citing works
Self-citationTripathi, S., Sukumaran, R., & Cook, T. S. (2024) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocad258 · Cited by 137 · see citing works
Pavuluri, S., Sangal, R., Sather, J., & Taylor, R. A. (2024) · BMJ Health & Care Informatics · doi:10.1136/bmjhci-2024-101120 · Cited by 129 · see citing works
Large Language Models lack essential metacognition for reliable medical reasoning
Griot, M., Hemptinne, C., Vanderdonckt, J., & Yuksel, D. (2025) · Nature Communications · doi:10.1038/s41467-024-55628-6 · Cited by 113 · see citing works
AgentBench: Evaluating LLMs as Agents
Liu, X., Yu, H., Zhang, H., Xu, Y., Lei, X., Lai, H., Gu, Y., Ding, H., Men, K., Yang, K., et al. (2023) · arXiv · doi:10.48550/arxiv.2308.03688 · Cited by 79 · see citing works
How artificial intelligence could transform emergency care
Kachman, M. M., Brennan, I., Oskvarek, J. J., Waseem, T. C., & Pines, J. M. (2024) · The American Journal of Emergency Medicine · doi:10.1016/j.ajem.2024.04.024 · Cited by 62 · see citing works
MMedAgent: Learning to Use Medical Tools with Multi-modal Agent
Li, B., Yan, T., Pan, Y., Luo, J., Ji, R., Ding, J., Xu, Z., Liu, S., Dong, H., Lin, Z., et al. (2024) · Findings of the Association for Computational Linguistics: EMNLP 2024 · doi:10.18653/v1/2024.findings-emnlp.510 · Cited by 54 · see citing works
Shi, W., Xu, R., Zhuang, Y., Yu, Y., Zhang, J., Wu, H., Zhu, Y., Ho, J. C., Yang, C., & Wang, M. D. (2024) · Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2024.emnlp-main.1245 · Cited by 54 · see citing works
Implications of Large Language Models for Quality and Efficiency of Neurologic Care
Moura, L. M. V. R., Jones, D. T., Sheikh, I., Murphy, S. N., Kalfin, M., Kummer, B. R., Weathers, A. L., Grinspan, Z. M., Silsbee, H. M., Jones, L. K., et al. (2024) · Neurology · doi:10.1212/wnl.0000000000209497 · Cited by 53 · see citing works
Abi‐Rafeh, J., Xu, H. H., Kazan, R., Tevlin, R., & Furnas, H. J. (2023) · Aesthetic Surgery Journal · doi:10.1093/asj/sjad260 · Cited by 52 · see citing works
A new paradigm for accelerating clinical data science at Stanford MedicinePreprint
Datta, S., Posada, J., Olson, G., Li, W., O’Reilly, C., Balraj, D., Mesterhazy, J., Pallas, J. I., Desai, P., & Shah, N. H. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.10534 · Cited by 50 · see citing works
AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents
Ma, C., Zhang, J., Zhu, Z., Yang, C., Yang, Y., Jin, Y., Lan, Z., Kong, L., & He, J. (2024) · arXiv · doi:10.48550/arxiv.2401.13178 · Cited by 26 · see citing works
Superhuman performance of a large language model on the reasoning tasks of a physicianPreprint
Brodeur, P. G., Buckley, T. A., Kanjee, Z., Goh, E., Ling, E., Jain, P., Cabral, S., Abdulnour, R.-E. E., Haimovich, A. D., Freed, J. A., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2412.10849 · Cited by 26 · see citing works
Self-citationAgentClinic: a multimodal agent benchmark to evaluate AI in simulated clinical environments
Schmidgall, S., Ziaei, R., Harris, C., Reis, E., Jopling, J., & Moor, M. (2024) · arXiv · doi:10.48550/arxiv.2405.07960 · Cited by 24 · see citing works
$τ$-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains
Yao, S., Shinn, N., Razavi, P., & Narasimhan, K. (2024) · arXiv · doi:10.48550/arxiv.2406.12045 · Cited by 9 · see citing works
Uptegraft, C. C., Black, K. C., Gale, J. J., Marshall, A. D. A., & He, S. (2025) · JMIR AI · doi:10.2196/66741 · Cited by 6 · see citing works
Self-citationYu, H., Zhou, J., Li, L., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2409.18924 · Cited by 6 · see citing works
Many-Shot In-Context Learning in Multimodal Foundation ModelsPreprint
Jiang, Y., Irvin, J., Wang, J. H., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2405.09798 · Cited by 5 · see citing works
Self-citationShi, W., Xu, R., Zhuang, Y., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2401.07128 · Cited by 2 · see citing works
MMedAgent: Learning to Use Medical Tools with Multi-modal Agent
Li, B., Yan, T., Pan, Y., Luo, J., Ji, R., Ding, J., Xu, Z., Liu, S., Dong, H., Lin, Z., et al. (2024) · arXiv · doi:10.48550/arxiv.2407.02483
Flagged first, then by citations
Research works printed but not linked: 4
- [7]7. Sahni NR, Carrus B. Artificial intelligence in US health care delivery. N Engl J Med 2023;389:348-358. DOI: 10 .1056 / NEJMra2204673.
- [18]18. Pal A, Umapathi LK, Sankarasubbu M. Medmcqa: a large-scale multi-subject multi-choice dataset for medical domain question answering. In: Proceedings of the Conference on Health, Inference, and Learning. PMLR, 2022:248-260. (https://proceedings.mlr. press/v174/pal22a.html).
- [20]20. Johri S, Jeong J, Tran BA, et al. CRAFT-MD: a conversational eval- uation framework for comprehensive assessment of clinical LLMs. In: Proceedings of the AAAI 2024 Spring Symposium on Clinical Foundation Models. Washington, DC: AAAI Press, 2024 (https:// openreview.net/forum?id=Bk2nbTDtm8).
- [26]26. Li XL, Liu EZ, Liang P, Hashimoto T. Autobencher: creating salient, novel, difficult datasets for language models. 2024 (https:// arxiv.org/html/2407.08351v1). Preprint.
Cited by
Sorted by citations
Large language models for clinical decision support in gastroenterology and hepatology
Wiest, I. C., Bhat, M., Clusmann, J., Schneider, C. V., Jiang, X., & Kather, J. N. (2025) · Nature Reviews Gastroenterology & Hepatology · doi:10.1038/s41575-025-01108-1 · Cited by 33 · see citing works
A comprehensive survey of AI agents in healthcare
Xu, G., Li, X., Chen, Y., Duan, Y., Wu, S., Yu, H., Chiu, C.-H., Ni, J., Tang, N., Li, T. J.-J., et al. (2026) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2026.105045 · Cited by 32 · see citing works
Artificial intelligence agents in cancer research and oncology
Truhn, D., Azizi, S., Zou, J., Cerdá-Alberich, L., Mahmood, F., & Kather, J. N. (2026) · Nature reviews. Cancer · doi:10.1038/s41568-025-00900-0 · Cited by 25 · see citing works
Self-citationBenchmarking large language model-based agent systems for clinical decision tasks
Liu, Y., Carrero, Z. I., Jiang, X., Ferber, D., Wölflein, G., Zhang, L., Jayabalan, S., Lenz, T., Hui, Z., & Kather, J. N. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-02443-6 · Cited by 31 · see citing works
From Agents to Governance: Essential AI Skills for Clinicians in the Large Language Model Era
Cao, W., Zhang, Q., Liu, J., & Liu, S. (2026) · Journal of Medical Internet Research · doi:10.2196/86550 · Cited by 24 · see citing works
Multi-step retrieval and reasoning improves radiology question answering with large language models
Wind, S., Sopa, J., Truhn, D., Lotfinia, M., Nguyen, T.-T., Bressem, K. K., Adams, L. C., Rusu, M., Koestler, H., Wellein, G., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-02250-5 · Cited by 25 · see citing works
Scaling medical AI across clinical contexts
Li, M. M., Reis, B. Y., Rodman, A. M., Cai, T., Dagan, N., Balicer, R. D., Loscalzo, J., Kohane, I. S., & Žitnik, M. (2026) · Nature Medicine · doi:10.1038/s41591-025-04184-7 · Cited by 17 · see citing works
Tripathi, S., Cook, T. S., & Kim, W. (2026) · Radiology · doi:10.1148/radiol.252730 · Cited by 14 · see citing works
Enhancing Clinical Decision-Making: Integrating Multi-Agent Systems with Ethical AI Governance
Chen, Y.-J., Albarqawi, A., & Chen, C.-S. (2025) · IEEE Conference on Computational Intelligence in Bioinformatics and Computational Biology (CIBCB) · doi:10.1109/cibcb66090.2025.11177136 · Cited by 11 · see citing works
A clinical environment simulator for dynamic AI evaluation
Luo, L., Kim, S. E., Zhang, X., Kernbach, J. M., Kenia, R., Acosta, J. N., Nathanson, L. A., Haimovich, A. D., Rodman, A. M., Goh, E., et al. (2026) · Nature Medicine · doi:10.1038/s41591-026-04252-6 · Cited by 11 · see citing works
Self-citationA Novel Approach to Overseeing the Clinical Application of Generative AI
Patel, B., & Blumenthal, D. (2026) · JAMA Health Forum · doi:10.1001/jamahealthforum.2025.6947 · Cited by 7 · see citing works
Zeng, Y., Xie, H., Gu, F., Chen, J., Zhang, G., Jiang, X., & Lau, S. T. (2026) · European Journal of Cardiovascular Nursing · doi:10.1093/eurjcn/zvag029 · Cited by 4 · see citing works
Self-citationDinc, R., & Ardic, N. (2026) · JMIR Medical Informatics · doi:10.2196/92584 · Cited by 2 · see citing works
Mine, Y., Taji, T., Okazaki, S., Takeda, S., Shimoe, S., Kaku, M., Nikawa, H., Kakimoto, N., & Murayama, T. (2026) · Journal of Dentistry · doi:10.1016/j.jdent.2026.106675 · Cited by 2 · see citing works
Doubleday, A. F., Cheverko, C. M., Bolgova, O., Mavrych, V., Mohamed, F. R. R., Westrick, J. C., Juarez, L., Rush, E., Solka, K. A., Byram, J. N., et al. (2026) · Journal of Dentistry · doi:10.1016/j.jdent.2026.106724 · Cited by 2 · see citing works
When Chatbots Become Agents: The Next Phase of Healthcare AI
Karunanayake, N. (2026) · Journal of Medical Systems · doi:10.1007/s10916-026-02402-4 · Cited by 2 · see citing works
Governing Clinical Readiness Claims Derived from Medical AI Benchmark Results
Dong, Y., Cheng, J., Ding, C., & Lu, R. (2026) · Journal of Medical Systems · doi:10.1007/s10916-026-02445-7 · Cited by 2 · see citing works
Beyond the Algorithm: A Field Guide to Deploying AI Agents in Clinical PracticePreprint
Gallifant, J., Kellogg, K. C., Butler, M., Centi, A., Chen, S., Doyle, P., Dutta, S., Guo, J., Hadfield, M., Kim, E., et al. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5732262 · Cited by 2 · see citing works
Cafferty, O. B., Jeffries, S. D., Pelletier, E. D., Tu, Z., Sinha, A., Hemmerling, T. M., & collaborators (2026) · Anesthesia & Analgesia · doi:10.1213/ane.0000000000007991 · Cited by 1 · see citing works
Venerito, V., Lopalco, G., Iannone, F., & Gupta, L. (2026) · The Lancet Rheumatology · doi:10.1016/s2665-9913(26)00076-7 · Cited by 1 · see citing works
A roadmap for medical large language models: a review of foundations, applications, and challenges
Sha, Y.-Y., Yu, L., Lin, Z.-H., Kaur, A., Lou, Y., Gornale, S. S., Zhang, T., Wong, L. S., Wang, Z., Yan, Y., et al. (2026) · Military Medical Research · doi:10.1016/j.mmr.2026.100050 · Cited by 2 · see citing works
The Doctor Will (Still) See You Now: On the Structural Limits of Agentic AI in Healthcare
Aránguiz-Dias, G., Meimandi, K. J., Griffith, A., Dias, C. A., Kim, G. R., Saadeddin, L., & Kochenderfer, M. J. (2026) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3805689.3812224 · Cited by 2 · see citing works
Toward a test of medical AI superintelligence
Goh, E., Wu, D., Walton, C., McCoy, L. G., Perez, A., Wegner, L. S., Haredasht, F. N., Luo, L., Lacar, K., Buckley, T. A., et al. (2026) · Nature Medicine · doi:10.1038/s41591-026-04539-8 · Cited by 1 · see citing works
Self-citationPalermi, S., Saglietto, A., D’Ascenzo, F., Ferrari, G. M. D., Bruno, F., Tse, G., Kizilkilic, S. E., Tjong, F. V. Y., Ruipérez-Campillo, S., Keser, N., et al. (2026) · Progress in Cardiovascular Diseases · doi:10.1016/j.pcad.2026.07.005 · Cited by 1 · see citing works
Corso, F., Zec, A., Favali, M., Ligero, M., Hilgers, L., Invernizzi, L. M., Mazzeo, L., Morales, A. M., Müller‐Franzes, G. A., Machado, I. P., et al. (2026) · Cell Reports Medicine · doi:10.1016/j.xcrm.2026.102966 · Cited by 2 · see citing works
Kang, X., Ding, W., Matsumoto, K., Wang, L., Yu, H.-T., & Shi, X. (2026) · Information Fusion · doi:10.1016/j.inffus.2026.104702 · Cited by 1 · see citing works
Tolsgaard, M. G., Grierson, L., Pusic, M. V., & Turner, L. (2026) · Medical Education · doi:10.1111/medu.70309 · Cited by 2 · see citing works
Hassoon, A., Peng, X., Irimia, R. M., Lianjie, A., Leo, H., Bandeira, A., Woo, H., Dredze, M., Abdulnour, R.-E. E., McDonald, K. M., et al. (2026) · medRxiv · doi:10.64898/2026.02.22.26346832 · Cited by 1 · see citing works
Agentic Chart Review from Longitudinal Clinical Notes: a Lung Cancer Guideline Concordance Use CasePreprint
Jiang, Y., He, X., Ai, X., Majji, R. K., Maniar, R., Jalal, S. I., Fedele, D. A., Hollenbach, J. P., Liu, J., Zhuang, Y., et al. (2026) · medRxiv · doi:10.64898/2026.06.02.26354727 · Cited by 1 · see citing works
Self-citationPerformance assessment strategies for language model applications in healthcare
García, V., Sidulova, M., & Badano, A. G. (2026) · Artificial Intelligence in the Life Sciences · doi:10.1016/j.ailsci.2026.100162
A framework for using AI to drive care model transformation: building cars rather than faster horses
Li, R., Rosengaus, L., Gohil, L., Sharp, C., Pfeffer, M. A., Sehgal, N. L., Entwistle, D., & Hofmann, L. V. (2026) · npj Health Systems · doi:10.1038/s44401-026-00071-6
Haider, S. A., Prabha, S., Gomez-Cabello, C. A., Genovese, A., Collaço, B. G., Wood, N. G., Lifson, M. A., Tao, C., Bagaria, S. P., & Forte, A. J. (2026) · doi:10.2196/preprints.93344
Castano-Villegas, N., Barrientos, K. M., Villa, M. C., Zea, J., Velásquez, L., & Gómez, O. I. Q. (2026) · doi:10.2196/preprints.93699
Externalized Living Memory: Structuring Clinical Knowledge for the Age of AI Agents (Preprint)Preprint
Lee, W.-J., & Wu, C.-F. (2026) · doi:10.2196/preprints.96809
Systematic Evaluation of Large Language Models for Post-Discharge Clinical Action Extraction
Dalmia, S., Mantravadi, A., & Desikan, P. (2026) · IEEE International Conference on Healthcare Informatics (ICHI) · doi:10.1109/ichi69079.2026.00110
Chandrasekaran, M. (2026) · IEEE Middle East and North Africa Communications Conference (MENACOMM) · doi:10.1109/menacomm69507.2026.11532940
Overview of Medical NLP Code Generation with FHIR for Clinical Trial Screening
Tao, L., Mi, L., Wu, C., Wen, D., Tang, B., Zhang, X., Zong, H., & Li, Z. (2026) · Communications in computer and information science · doi:10.1007/978-981-95-7299-1_34
Don’t stop the heart: a performance analysis of large language models and potassium dosingPreprint
Blotske, K., Zhao, X., Henry, K., Murray, B., Gao, Y., Smith, S. E., Wayne, N. B., Ku, P., Smith, B., Moua, S., et al. (2026) · medRxiv · doi:10.64898/2026.06.02.26354762
Bian, J., Afshar, M., Scifres, C., Webber, E. C., Burton, D., Vawdrey, D., Wang, F., Melton, G. B., Shah, N. H., Patzer, R. E., et al. (2026) · npj Health Systems · doi:10.1038/s44401-026-00107-x
(2026) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2026.105062
Reis, T. C. (2026) · Artificial Intelligence in Emergency Medicine · doi:10.1016/j.aiemed.2026.100028
Hard to Halt: Automation Bias in Agent-Driven Sequencing Prior Authorization WorkflowsPreprint
Nie, M., Chung, W. K., Waxler, J. L., Lee, M., Weng, C., Lewis, R., Ahimaz, P. R., Wang, K., & Liu, C. (2026) · medRxiv · doi:10.64898/2026.06.16.26355782
From agentic artificial intelligence to accountable military healthcare: A readiness imperative
Vijayasimha, M., Srikanth, M., & Jha, R. K. (2026) · Medical Journal Armed Forces India · doi:10.1016/j.mjafi.2026.05.005
Zazueta, L., Covarrubias, B. L. L., Cota, C. X. N., Vázquez-Briseño, M., Hipólito, J. I. N., Romo-Cárdenas, G. S., & Avilés-Rodríguez, G. J. (2026) · Healthcare · doi:10.3390/healthcare14131865
PATHFinder Agent for Tailored Prenatal Care
Balloli, V., Samuel, C., Abdelnabi, S., Peahl, A. F., & Bondi, E. (2026) · ACM Interactive Health Conference · doi:10.1145/3786579.3804996
Q-SAFe: Quantum-Safe Agentic Federated Learning Scheme for Telemedicine Edge Networks
Khan, N. M., Bhattacharya, P., Roy, S., Shetty, S. S., Gadekallu, T. R., Bvuma, S., & Jhaveri, R. H. (2026) · International Conference on Communications · doi:10.1109/icc59461.2026.11587570
He, C., Li, Y., Yang, J., & Zhao, S. (2026) · Applied Sciences · doi:10.3390/app16147213
Toward Trustworthy Large Language Model Agents in Healthcare
Hasan, H., Salman, S., Dargham, A. T. A., Mohanna, A., & Chehab, A. (2026) · International Conference on Computer Networks, Big Data and IoT (ICCBI) · doi:10.1109/iccbi68589.2026.11619647
Gao, G., Gao, Q., Pajić, M., & Brunskill, E. (2026) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3770855.3817568
Black, K. C., Bunning, B., Jiang, Y., & Chen, J. H. (2026) · The Lancet Digital Health · doi:10.1016/j.landig.2026.101057
Self-citationArtificial intelligence agents and agentic artificial intelligence applied to precision medicine.
Costa, F. F. (2026) · Drug Discovery Today · doi:10.1016/j.drudis.2026.104781
Can a General-Purpose Coding Agent Analyze a Production Hospital Data Warehouse?Preprint
Marshall, N. P., Haberkorn, W. J., Faulkenberry, J., Palakkattil, G., Haredasht, F. N., Schwenk, H. T., Chen, J. H., & Morse, K. (2026) · medRxiv · doi:10.64898/2026.09.02.26362008
Self-citationBellucci, G., Gu, W., Rose, P. W., & Baranzini, S. E. (2026) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2026.1883853
Agentic AI workflows in anatomic pathology: Opportunities and challenges
Ngo, A., Tian, R., Mohammad, A., Weon, J. L., Arvisais‐Anhalt, S., Minor, J., & Youens, K. (2026) · Journal of Pathology Informatics · doi:10.1016/j.jpi.2026.100718
Security-Augmented Use Case Flow Refinement for LLM-Based Agentic Systems
Wang, G., Li, B., Wu, J., & Shao, Z. (2026) · Electronics · doi:10.3390/electronics15184135
Sorka, M., Gorenshtein, A., Abramovitch, H., Soontrapa, P., Shelly, S., & Aran, D. (2026) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocag143
Waisberg, E., & Guarnieri, J. W. (2026) · Annals of Biomedical Engineering · doi:10.1007/s10439-026-04381-6
Doan, N. N., Elen, B., Sun, C., Dumontier, M., & Ertaylan, G. (2026) · doi:10.2196/preprints.112384
Wright, E., Bennett, C., & Turner, J. (2026) · Journal of Health Informatics and Digital Systems · doi:10.68159/y283179347
Worth, J. E., Perez, A., Wu, D. J., Brodeur, P. G., Tat, E., Manrai, A. K., Wu, D., Walton, C., McCoy, L. G., Jain, P., et al. (2026) · BMJ Digital Health & AI · doi:10.1136/bmjdh-2026-000113 · Cited by 1 · see citing works
Self-citationAgentInspect: Diagnosing Behavioral Failures in Artificial Intelligence Agents
Manke, R., Wardat, M., Khomh, F., & Rajan, H. (2026) · Proceedings of the ACM on software engineering. · doi:10.1145/3832174
Yu, S., Gu, X., Gao, Y., Zhao, F., & Zhang, J. (2026) · Integrated Diagnostics · doi:10.1016/j.intdia.2026.100006
A survey of LLM reasoning in healthcare and medicine: Individual modeling and collaborative agents
Lin, Q., Xiao, L., Pu, B., Shi, K., Du, Y., Xu, J., He, K., Zhao, S., Zhang, J., Cambria, E., et al. (2026) · Information Fusion · doi:10.1016/j.inffus.2026.104827 · Cited by 1 · see citing works
Moll, J., Lübberstedt, J., Nuernbergk, C., Stroh, J., Mertens, L., Purcarea, A., Zirn, C., Benchaaben, Z., Drexel, F., Häntze, H., et al. (2026) · npj Precision Oncology · doi:10.1038/s41698-026-01733-2 · Cited by 1 · see citing works
Clinical AI in 2025–2026 — Six Themes from the SAIL Year in Review
Kim, E., Elia, M. V., Li, M. M., Healey, E., Erdman, L., Chandak, P., Argaw, P., Li, B., Black, K. C., Marwaha, J. S., et al. (2026) · NEJM AI · doi:10.1056/aip2600817
Self-citationAgentic AI Systems in Radiology: Design, Applications, Evaluation, and Challenges
Blüthgen, C., Veen, D. V., Truhn, D., Kather, J. N., Moor, M., Polacin, M., Chaudhari, A. S., Frauenfelder, T., Langlotz, C. P., Krauthammer, M. O., et al. (2026) · Radiology · doi:10.1148/radiol.253226
- Authors
- Jiang, Y., Black, K. C., Geng, G., Park, D., Zou, J., Ng, A. Y., & Chen, J. H.
- Affiliations
- Stanford University (US)
- Stanford Medicine (US)
- Venue
- NEJM AI
- Publisher
- Massachusetts Medical Society
- Published
- 2025-08-28
- Volume
- 2
- Issue
- 9
- Pages
- AIdbp2500144-AIdbp2500144
- ISSN
- 2836-9386 (ISSN-L)
- OpenAlex
- W4413801638
- Cited by
- 66
- References
- 17
- Citations by year
- 2026: 62 · 2025: 4
- FWCI
- 15.89
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- NEJM AI (publishedVersion)
- PubMed (publishedVersion)
- Funders
- National Institutes of Health
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Keywords
- LLM agents, benchmarking, electronic health records, agentic AI, medical records, FHIR, clinical workflow, administrative burden, patient information, Claude 3.5 Sonnet
- MeSH terms
- Large Language Models, Artificial Intelligence, Humans, Benchmarking, Electronic Health Records
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
