CoChat
Sign Up
Cited by 48Artificial Intelligence in Healthcare and EducationMachine Learning in HealthcareArtificial Intelligence in Healthcare

Large language model agents can use tools to perform clinical calculations

Open access
Goodell, Alex J., Chu, Simon N., Rouholiman, Dara, Chu, Larry F.·2025·npj Digital Medicine
DOI 10.1038/s41746-025-01475-8PMC PMC11914283
Open this paper in CoChat — free account

Abstract

Large language models (LLMs) can answer expert-level questions in medicine but are prone to hallucinations and arithmetic errors. Early evidence suggests LLMs cannot reliably perform clinical calculations, limiting their potential integration into clinical workflows. We evaluated ChatGPT's performance across 48 medical calculation tasks, finding incorrect responses in one-third of trials (n = 212). We then assessed three forms of agentic augmentation: retrieval-augmented generation, a code interpreter tool, and a set of task-specific calculation tools (OpenMedCalc) across 10,000 trials. Models with access to task-specific tools showed the greatest improvement, with LLaMa and GPT-based models demonstrating a 5.5-fold (88% vs 16%) and 13-fold (64% vs 4.8%) reduction in incorrect responses, respectively, compared to the unimproved models. Our findings suggest that integration of machine-readable, task-specific tools may help overcome LLMs' limitations in medical calculations.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1038/s41746-025-01475-8

Cites

8 of 8 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • GenAI tools can significantly improve clinical accuracy, workflow efficiency, and decision-support capacity when implemented under structured supervision.

    “Studies27,29,30,42 provide empirical evidence that GenAI tools can significantly improve clinical accuracy, workflow efficiency, and decision-support capacity when implemented under structured supervision.”

    From Generative Artificial Intelligence in Healthcare: Automation Bias, Deskilling, and Cognitive Implications – A Systematic Review (2026) · cites this paper as [42]

    The Results (Focused analysis, page unavailable) show significantly higher accuracy on clinical calculation vignettes with tool augmentation, including GPT accuracy rising from 36.1% to 95.2% with OpenMedCalc. The Discussion describes workflow benefits and decision-support uses as future possibilities, not measured improvements; the study also does not establish that the tools were implemented under structured clinical supervision. (LLM, conf=0.87)

    Experimental — accuracy not yet measured

  • Clinician performance is optimized when AI outputs are analyzed rather than uncritically accepted.

    “Empirical evidence27,29,42 indicated that clinician performance is optimized when AI outputs are analyzed rather than uncritically accepted.”

    From Generative Artificial Intelligence in Healthcare: Automation Bias, Deskilling, and Cognitive Implications – A Systematic Review (2026) · cites this paper as [42]

    The Results evaluate LLM accuracy on clinical calculations, and the Discussion notes that even the best configuration made errors. Neither section examines clinician performance when clinicians analyze AI outputs versus accept them uncritically. Searches for clinician review, verification, and reliance found no such comparison in our copy. (LLM, conf=0.74)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 30

  1. [2]Kung, T. H. et al. Performance of ChatGPT on USMLE: potential for ai-assisted medical education using large language models. PLoS Digit. Health 2, e0000198 (2023). 36812645 PMC9931230 · DOI · Paper page
  2. [3]Singhal, K. et al. Large language models encode clinical knowledge. Nature 620, 172–180 (2023). 37438534 PMC10396962 · DOI · Paper page
  3. [4]OpenAI. Gpt-4 technical report. http://arxiv.org/abs/2303.08774 (2023). · arXiv
  4. [6]Patel, D. et al. Evaluating prompt engineering on GPT-3.5’s performance in USMLE-style medical calculations and clinical scenarios generated by GPT-4. Sci. Rep. 14, 17341 (2024). 39069520 PMC11284198 · DOI
  5. [8]Yao, S. et. al. ReAct: Synergizing Reasoning and Acting in Language Models. The Eleventh International Conference on Learning Representations. https://openreview.net/forum?id=WE_vluYUL-X (2023).
  6. [9]Xi, Z. H. et al. The rise and potential of large language model based agents: a survey. Sci China Inf Sci. 68, 121101 (2025). · DOI · Paper page
  7. [10]Yubo, W., Xueguang, M. & Wenhu, C. Augmenting Black-box LLMs with Medical Textbooks for Biomedical Question Answering. 10.48550/arXiv.2309.02233 (2024). · DOI
  8. [11]Jin, Q., Yang, Y., Chen, Q. & Lu, Z. Genegpt: augmenting large language models with domain tools for improved access to biomedical information. Bioinformatics 40, btae075 (2024). 38341654 PMC10904143 · DOI · Paper page
  9. [12]Imani, S., Du, L. & Shrivastava, H. MathPrompter: Mathematical Reasoning using Large Language Models. In Proc. 61st Annual Meeting of the Association for Computational Linguistics (eds Sitaram, S., Beigman K. B. & Williams, J. D.) Volume 5: Industry Track, 37–42 (Association for Computational Linguistics, Toronto, Canada, 2023).
  10. [14]Walker, G. Emergentology: the ups and downs of developing the MDCalc app. Emerg. Med. News 38, 18 (2016). · DOI
  11. [15]Bilimoria, K. Y. et al. Development and evaluation of the universal ACS NSQIP surgical risk calculator: a decision aid and informed consent tool for patients and surgeons. J. Am. Coll. Surg. 217, 833 (2013). 24055383 PMC3805776 · DOI
  12. [16]Shahian, D. M. et al. The society of thoracic surgeons 2018 adult cardiac surgery risk models: part 1—background, design considerations, and model development. Ann. Thorac. Surg. 105, 1411–1418 (2018). 29577925 · DOI · Paper page
  13. [19]Bumgardner, V. K. C. et al. Institutional platform for secure self-service large language model exploration. arXiv [cs.CR]. http://arxiv.org/abs/2402.00913 (2024). PMC12150735 40502230 · arXiv
  14. [21]Hong, H. J., Schmiesing, C. A. & Goodell, A. J. Enhancing the readability of preoperative patient instructions using large language models. Anesthesiology 141, 608–610 (2024). 39136480 · DOI
  15. [22]Chung, P. et al. Large language model capabilities in perioperative risk prediction and prognostication. JAMA Surg. 159, 928–937 (2024). 38837145 PMC11154375 · DOI
  16. [23]Kwan, B., Bell, J. F., Longhurst, C. A., Goldhaber, N. H. & Clay, B. Implementation of an electronic health record-integrated instant messaging system in an academic health system. J. Am. Med. Inform. Assoc. 31, 997–1000 (2024). 38287641 PMC10990552 · DOI
  17. [24]Schick, T. D. et al. Toolformer: Language models can teach themselves to use tools. Adv. Neural Inf. Process. Syst. 36, 68539–68551 (2023).
  18. [25]Goodell, A. J., Chu, S. N., Rouholiman, D. & Chu, L. F. Augmentation of ChatGPT with clinician-informed tools improves performance on medical calculation tasks. medRxiv 2023-12. 10.1101/2023.12.13.23299881 (2023). · DOI
  19. [26]Khandekar, N. et al. Medcalc-bench: Evaluating large language models for medical calculations. arXiv [cs.CL]. http://arxiv.org/abs/2406.12036 (2024). · arXiv
  20. [27]Jin, Q. et al. Agentmd: empowering language agents for risk prediction with large-scale clinical tool learning. arXiv [cs.CL]. http://arxiv.org/abs/2402.13225 (2024). PMC12549800 41130954 · arXiv
  21. [28]Zhu, Y. et al. Menti: bridging medical calculator and llm agent with nested tool calling. arXiv [cs.AI]. http://arxiv.org/abs/2410.13610 (2024). · arXiv
  22. [29]Wan, N. et al. Humans continue to outperform large language models in complex clinical decision-making: a study with medical calculators. arXiv [cs.CL]. http://arxiv.org/abs/2411.05897 (2024). PMC12919463 41726433 · arXiv
  23. [30]Fleming, S. L. et al. MedAlign: A Clinician-Generated Dataset for Instruction Following with Electronic Medical Records. In Proc. AAAI Conference on Artificial Intelligence, Vol. 38, 22021–22030 (2024). PMC12826664 41584261
  24. [31]Pal, A. et al. Med-HALT: Medical Domain Hallucination Test for Large Language Models. In Proc. 27th Conference on Computational Natural Language Learning (CoNLL) (eds Jiang, J., Reitter, D. & Deng, S) 314–334 (Association for Computational Linguistics, Singapore, 2023) https://aclanthology.org/2023.conll-1.21/
  25. [33]Qiao, S. et al. Reasoning with Language Model Prompting: A Survey. In Proc. 61st Annual Meeting of the Association for Computational Linguistics, Vol. 1 (eds Rogers, A., Boyd-Graber, J. & Okazaki, N) 5368–5393 (Association for Computational Linguistics, Toronto, Canada, 2023).
  26. [34]Shashikumar, S. P., Wardi, G., Malhotra, A. & Nemati, S. Artificial intelligence sepsis prediction algorithm learns to say “I don’t know”. npj Digit. Med. 4, 1–9 (2021). 34504260 PMC8429719 · DOI
  27. [35]Tang, X. et al. MedAgents: Large Language Models as Collaborators for Zero-shot Medical Reasoning. 10.48550/arXiv.2311.10537 (2024). · DOI
  28. [36]Liao, Y., Meng, Y., Liu, H., Wang, Y. & Wang, Y. An automatic evaluation framework for multi-turn medical consultations capabilities of large language models. 10.48550/arXiv.2309.02077 (2023). · DOI
  29. [37]Schmidgall, S. et al. Agentclinic: a multimodal agent benchmark to evaluate ai in simulated clinical environments. arXiv [cs.HC] http://arxiv.org/abs/2405.07960 (2024). · arXiv
  30. [39]Willard, B. T. & Louf, R. Efficient guided generation for large language models. arXiv [cs.CL]. http://arxiv.org/abs/2307.09702 (2023). · arXiv

Cited by

Sorted by citations

Authors
Goodell, A. J., Chu, S. N., Rouholiman, D., & Chu, L. F.
Affiliations
  • Stanford Medicine (US)
  • University of California, San Francisco (US)
Corresponding author
Goodell, Alex J.
ORCID
Venue
npj Digital Medicine
Publisher
Springer Science and Business Media LLC
Published
2025-03-17
Volume
8
Issue
1
Pages
163-163
ISSN
2398-6352 (ISSN-L)
DOI
10.1038/s41746-025-01475-8
PMC
PMC11914283
OpenAlex
W4408529625
Cited by
48
References
8
Citations by year
2026: 38 · 2025: 10
FWCI
11.55
Type
journal article
Language
en
Open access
Licence
Locations
  • npj Digital Medicine (publishedVersion, cc-by-nc-nd)
  • PubMed Central (publishedVersion, cc-by-nc-nd)
  • PubMed (publishedVersion)
  • eScholarship (California Digital Library) (submittedVersion, cc-by-nc-nd)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
  • National Institutes of Health
  • University of California, San Francisco
  • National Institute of General Medical Sciences (T32GM089626)
  • National Institute of Allergy and Infectious Diseases (T32AI125222)
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
  • Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
Keywords
LLM agents, hallucinations, ChatGPT, retrieval-augmented generation, code interpreter, llamas, clinical workflow integration

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review