General-purpose large language models outperform specialized clinical AI tools on medical benchmarks
Open accessAbstract
Specialized clinical artificial intelligence (AI) tools are entering medical practice despite scarce independent evaluation. We quantitatively evaluate two clinical AI tools, OpenEvidence and UpToDate Expert AI, built on large language models (LLMs) against three frontier LLMs: GPT-5.2, Gemini 3.1 Pro and Claude Opus 4.6. Our evaluation has three stages: (1) 500 MedQA questions testing medical knowledge, (2) 500 HealthBench items measuring alignment with clinicians and (3) the real clinical queries (RCQ) benchmark, built from 100 de-identified queries from physicians to a general-purpose language model in a live clinical environment. For the RCQ benchmark, 12 US clinicians performed randomized, blinded review of model outputs, producing 1,800 model-question annotations. Frontier LLMs outperformed clinical AI tools in all three evaluations. Clinical AI tools performed comparably to auto-enabled Google Search AI Overview on the RCQ. These findings highlight the need for independent, real-world evaluation of AI tools before they enter clinical settings.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Specialized proprietary LLMs (e.g., OpenEvidence and UpToDate Expert AI) are designed to provide responses grounded in medical literature and are seeing increasing adoption in medical practice among health professionals and medical students.
“Also, specialized proprietary LLMs (e.g., OpenEvidence and UpToDate Expert AI) are designed to provide responses grounded in medical literature, 115 and are seeing increasing adoption in medical practice, 9 among health professionals and medical students.115,116”
From Large Language Models in Pharmacist-Led Direct Patient Care: A Rapid Review of Published Evaluations (2026) · cites this paper as [9]
Split into 2 atoms — 0 confirmed, 2 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.
Experimental — accuracy not yet measuredMore specialized LLM tools, such as OpenEvidence, have reported high accuracy and performance on the real clinical questions submitted by physicians.
“More specialized LLM tools, such as OpenEvidence, have reported high accuracy and performance on the real clinical questions submitted by physicians.14”
From Large Language Model–Generated Electronic Consultations: A Blinded Comparative Study of OpenEvidence and Gastroenterologist Electronic Consultations (2026) · cites this paper as [14]
The paper evaluates OpenEvidence on 100 real physician queries using blinded clinician ratings (RCQ evaluation; Fig. 2), so the setting and performance assessment match. But it does not report an accuracy percentage for those queries, and it places OpenEvidence below the general-purpose LLMs in clinician-rated performance. Its 89.6% accuracy figure comes from MedQA questions, not the real clinical queries. (LLM, conf=0.84)
Experimental — accuracy not yet measuredAnother study using blinded clinician reviewers to evaluate LLM performance across real clinical queries did not demonstrate a performance advantage of the specialized clinical LLM-based tools OpenEvidence and UpToDate Expert AI over the tested general-purpose LLMs.
“Likewise, another study using blinded clinician reviewers to evaluate LLM performance across real clinical queries did not demonstrate a performance advantage of the specialized clinical LLM-based tools -OpenEvidence and UpToDate Expert AI -over the tested general-purpose LLMs (Gemini 3.1 Pro, Claude Opus 4.6, and GPT-5.2). 9”
From Large Language Models in Pharmacist-Led Direct Patient Care: A Rapid Review of Published Evaluations (2026) · cites this paper as [9]
Prior studies have not evaluated a large set of paired, real-world specialist eConsults using an eConsult-specific quality instrument.
“These studies have reported that LLM-generated responses may be perceived favorably, but they have not evaluated a large set of paired, real-world specialist eConsults using an eConsult-specific quality instrument.”
From Large Language Model–Generated Electronic Consultations: A Blinded Comparative Study of OpenEvidence and Gastroenterologist Electronic Consultations (2026) · cites this paper as [14]
The Abstract describes an evaluation of 100 real clinician queries, and the RCQ statistical analysis describes paired comparisons of model responses. Neither section addresses prior studies of specialist eConsults or whether those studies used an eConsult-specific quality instrument. Searches for eConsults, specialist consultations, and favorable perceptions found no passage establishing the claimed research gap. (LLM, conf=0.76)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
The citation lists haven’t been fetched yet
References 14 · Cited by 49
Research works printed but not linked: 2
- HLTH 2025: Wolters Kluwer showcases UpToDate Expert AI and workflow innovations. Wolterskluwer https://www.wolterskluwer. com/en/news/uptodate-expert-ai-workflow-hlth-2025 (2025).
- Vishwanath, K. et al. Medical large language models are easily distracted. Preprint at https://doi.org/10.48550/ arXiv.2504.01201 (2025). · DOI
- Authors
- Vishwanath, K., Alyakin, A., Ghosh, M., Hage, A., Neifert, S. N., Orillac, C., Mandelberg, N. J., Khan, H. A., Lee, J. V., Yao, J. J., Small, W. R., Varma, A., Hewitt, D. B., Aphinyanaphongs, Y., Alber, D. A., & Oermann, E. K.
- Affiliations
- NYU Langone Health (US)
- Washington University in St. Louis (US)
- The University of Texas at Austin (US)
- New York University (US)
- Corresponding author
- Vishwanath, Krithik
- Oermann, Eric Karl
- ORCID
- Venue
- Nature Medicine
- Published
- 2026-06-12
- Volume
- 32
- Issue
- 7
- Pages
- 2405-2409
- ISSN
- 1078-8956 (ISSN-L)
- 1546-170X
- OpenAlex
- W7164566809
- Cited by
- 49
- References
- 14
- Citations by year
- 2026: 49
- FWCI
- 96.36
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Nature Medicine (publishedVersion, cc-by)
- PubMed (publishedVersion)
- PubMed Central (publishedVersion, cc-by)
- Funders
- National Cancer Institute (P30CA016087)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Keywords
- large language models, medical question answering, MedQA, OpenEvidence, GPT-5.2, Gemini, Claude Opus, clinical decision support
- MeSH terms
- Large Language Models, Generative Artificial Intelligence, Artificial Intelligence, Humans, Benchmarking
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
