CoChat
Sign Up
Cited by 13Artificial Intelligence in Healthcare and EducationMachine Learning in HealthcareMultimodal Machine Learning Applications

AgentClinic: a multimodal benchmark for tool-using clinical AI agents

Open access
Schmidgall, Samuel, Ziaei, Rojin, Harris, Carl, Kim, Ji Woong, Reis, Eduardo Pontes, Jopling, Jeffrey, Moor, Michael·2026·npj Digital Medicine
DOI 10.1038/s41746-026-02674-7PMC PMC13324710
Open this paper in CoChat — free account

Abstract

Evaluating large language models (LLM) in clinical scenarios is crucial to assessing their potential clinical utility. Existing benchmarks rely heavily on static question-answering, which does not accurately depict the complex, sequential nature of clinical decision-making. Here, we introduce AgentClinic, a multimodal agent benchmark for evaluating LLMs in simulated clinical environments that include patient interactions, multimodal data collection under incomplete information, and the usage of various tools, resulting in an in-depth evaluation across nine medical specialties and seven languages. We find that solving MedQA problems in the sequential decision-making format of AgentClinic is considerably more challenging, resulting in diagnostic accuracies that can drop to below a tenth of the original accuracy. Overall, we observe that agents sourced from Claude-3.5 outperform other LLM backbones in most settings. Nevertheless, we see stark differences in the LLMs’ ability to make use of tools, such as experiential learning, adaptive retrieval, and reflection cycles. Strikingly, Llama-3 shows up to 92% relative improvements with the notebook tool that allows for writing and editing notes that persist across cases. To further scrutinize our clinical simulations, we leverage real-world electronic health records, perform a clinical reader study, perturb agents with biases, and explore patient-centric metrics that this interactive environment enables.

AI snapshot

Written from the paper’s full text.

TL;DR

AgentClinic is a multimodal, tool-using clinical agent benchmark that replaces static medical QA with interactive patient–doctor–measurement–moderator simulations across specialties, languages, biases, and image-based cases. It finds that sequential diagnostic accuracy can fall far below static MedQA performance, that Claude-3.5 generally outperforms other backbones, and that tool use and bias robustness vary sharply across models.

Key findings

  • On AgentClinic-MedQA, Claude-3.5 reached 62.1% ± 3.3, OpenBioLLM-70B 58.3% ± 4.2, human physicians 54% ± 28.5, GPT-4 51.6% ± 3.3, GPT-3.5 36.6%, GPT-4o 34.2% ± 3.4, Llama 3 70B 19% ± 2.5, and Llama 2 70B-chat 4.5% ± 1.3; on AgentClinic-MIMIC-IV, Claude-3.5 reached 42.9% ± 3.3 and GPT-4 34.0% ± 3.1. The paper reports that solving MedQA in AgentClinic’s sequential format can drop diagnostic accuracy to below a tenth of the original accuracy.
  • Tool effects were model-dependent: Llama-3 showed up to 92% relative improvement with the notebook tool in the abstract, while Llama3-70b averaged a 9.4% increase across all tools and reached 41.1% with Notebook and Reflection CoT; GPT-3.5 decreased across all tools, including a 27.1% drop with Adaptive RAG (Book).
  • Bias affected models differently: GPT-4 had unbiased accuracy 52%, with normalized accuracy 92% for patient cognitive bias, 96.7% for doctor cognitive bias, 98.6% for patient implicit bias, and 97.1% for doctor implicit bias; Mixtral-8 × 7B had unbiased accuracy 37%, with normalized accuracy 83.7% for doctor biases, 89% for patient biases, and 88.3% for implicit bias. Patient-agent perceptions showed self-diagnosis bias drops of 4.7 points in confidence, 2 points in consultation, and 1 point in compliance, while implicit biases had profound effects across all three perception categories.
  • In multimodal AgentClinic-NEJM, when images were provided initially, Claude 3.5 Sonnet reached 37.2% ± 2.2, GPT-4 27.7% ± 2.0, GPT-4o 21.4% ± 1.7, and GPT-4o-mini 8.0% ± 1.2; when images had to be requested, accuracies were 35.4% ± 2.4, 25.4% ± 2.1, 19.1% ± 1.4, and 6.1% ± 1.2, respectively. Average information coverage in AgentClinic-MedQA was 67%, higher for correct diagnoses (72%) than incorrect ones (63%).

Methods

  • AgentClinic uses four language agents: a patient agent, a doctor agent being evaluated, a measurement agent for simulated exams/readings, and a moderator that compares the doctor’s final diagnosis to ground truth. Most experiments allowed N = 20 doctor–patient/measurement interactions before diagnosis.
  • Cases were sourced from USMLE/MedQA, de-identified MIMIC-IV electronic health records, NEJM case challenges, and MedMCQA specialist reports; the benchmark covers 9 medical specialties and 7 languages, with 120 multimodal cases, 215 USMLE-based language agents, 260 specialist cases, and 749 multilingual cases reported in the Discussion.
  • Evaluations compared 11 models, six Agent Toolbox tools (Zero-Shot CoT, One-Shot CoT, Reflection CoT, Adaptive RAG Book, Adaptive RAG Web, Notebook), 23 seeded cognitive/implicit biases, varying interaction limits, patient-agent models, and patient-centric ratings of confidence, compliance, and consultation willingness on 1–10 scales.

Limitations

  • The paper states that AgentClinic is a simplified clinical environment; the moderator uses an LLM to determine accuracy; the measurement agent may introduce errors or hallucinations; human baselining/rating used only N = 3 clinicians; proprietary model training data and possible MedQA data leakage are uncertain; moderator sensitivity was not explored; human evaluators’ medical specialties were not controlled; tool commands abstract real clinical workflows and limit ecological validity; and all interacting agents are simulated.
  • The text makes evident that patient confidence, compliance, and consultation ratings come from simulated patient agents rather than real patients, with no semantic descriptors for individual Likert levels and no inter-run reliability analysis for those scalar ratings.
  • The paper also notes that cross-model communication effects and multi-agent emergent dynamics remain areas for further research, and that the benchmark’s results may depend on the specific LLM used for each role.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1038/s41746-026-02674-7

Cites

45 of 45 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

32 of 42 cited sources checked · 15 supported · 5 flagged · accuracy not yet measured

  • Findingcitation

    This reduction is a much smaller drop than was observed in previous work (10.2% maximum reduction on BiasMedQA dataset15).

    “it is a much smaller drop than was observed in previous work (10.2% maximum reduction on BiasMedQA dataset15)”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [15] Schmidgall, S. et al. Evaluation and mitigation of cognitive biases in medical language models. npj Digit. Med. 7.1, 295 (2024). source

      The Results (Model evaluation; page unavailable) report a 24.9% average accuracy decrease for false-consensus bias and a 14.0% worst-case drop for GPT-4 alone. Both exceed the claimed 10.2% maximum reduction on BiasMedQA. (LLM, conf=0.88)

      Checked by a reasoning model — may be wrong.

      Experimental — accuracy not yet measured
  • Study design and conductcitation

    The agents are built using a random sample of diagnostic questions from USMLE, MIMIC-IV, and NEJM case challenges to ground them in medically relevant situations.

    “In order to build agents that are grounded in medically relevant situations, we use a random sample of diagnostic questions from the US Medical Licensing Exam (USMLE), from deidentified electronic health records (MIMIC-IV)16, and from the New England Journal of Medicine (NEJM) case challenges.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [16] Johnson, A. E. et al. Mimic-iv, a freely accessible electronic health record dataset. Sci. Data 10, 1 (2023). source
  • Interpretationcitation

    GPT-4 is relatively robust to bias compared with other language models.

    “Previous work studying cognitive bias in LLMs has shown that GPT-4 is relatively robust to bias compared with other language models15”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [15] Schmidgall, S. et al. Evaluation and mitigation of cognitive biases in medical language models. npj Digit. Med. 7.1, 295 (2024). source
  • Findingown data

    On AgentClinic-MedQA with GPT-4 patient and measurement agents, OpenBioLLM-70B achieved a diagnostic accuracy of 58.3% ± 4.2.

    “OpenBioLLM-70B 58.3 ± 4.2”
    Quote in the paperSubstantial claim
  • Findingown data

    MedQA accuracy was only weakly predictive of accuracy on AgentClinic-MedQA.

    “MedQA accuracy was only weakly predictive of accuracy on AgentClinic-MedQA”
    • Also stated in: § Comparison of models · plain
      “a model’s performance on the static MedQA benchmark is only weakly predictive of its performance in our interactive AgentClinic-MedQA setting”
    Quote in the paperSubstantial claim
  • Classified as probably not a substantial claim. 36
  • May combine two claims. 33
  • The quote only partly states this claim. 7

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    LLMs have shown the ability to retrieve relevant medical texts.

    “Recently, LLMs have shown the ability to encode clinical knowledge7,8, retrieve relevant medical texts9, and perform accurate single-turn medical question-answering3,10–12.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [9] Xiong, G., Jin, Q., Lu, Z., & Zhang, A. Benchmarking retrieval-augmented generation for medicine. In Findings of the Association for Computational Linguistics: ACL 2024 (pp. 6233–6251) (2024). source

      In “Question-Only Retrieval (QOR)” (page unavailable), the paper assigns retrieval to separate algorithms such as BM25 and MedCPT; it says the retrieved snippets are then prepended to the LLMs’ input. It therefore does not show LLMs themselves retrieving relevant medical texts, as the citing sentence claims. (LLM, conf=0.84)

      Checked by a reasoning model — may be wrong.

      Experimental — accuracy not yet measured

  • May combine two claims.

    Premisecitation

    MedAgents emphasizes QA improvement through agent collaboration but does not simulate patient interactions or decision-making processes.

    “MedAgents32 emphasizes QA improvement through agent collaboration but does not simulate patient interactions or decision-making processes.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [32] Tang, X. et al. Medagents: Large language models as collaborators for zero-shot medical reasoning. Findings of the Association for Computational Linguistics: ACL 2024. 2024. source

      The Conclusion describes MedAgents as a medical QA framework using collaborating, role-playing agents. But §2 Method explicitly lists decision making as its fifth stage, and §2.5 says an LLM acts as a medical decision maker to answer the clinical question. The paper therefore contradicts the claim that MedAgents does not simulate decision-making processes; these passages do not describe patient interactions. (LLM, conf=0.88)

      Checked by a reasoning model — may be wrong.

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The faithfulness of LLMs in establishing judgment is established in previous works.

    “The faithfulness of LLMs in establishing judgment is established in previous works33–35.”
    Quote in the paperNot a substantial claim3 of 3 sources checked
    • cites [33] Gu, J. et al. A survey on llm-as-a-judge. Preprint at https://doi.org/10.48550/arXiv.2411.15594 (2024). source

      The Introduction (p. 2) says that merely using an LLM as a judge does not ensure accurate evaluations. Section 7.3 (p. 47) says reliability still requires research and optimization and that challenges remain unresolved, contrary to the claim that faithfulness has been established. (LLM, conf=0.85)

      Checked by a reasoning model — may be wrong.

      Experimental — accuracy not yet measured
    • cites [34] Chen, J. & Mueller, J. Quantifying uncertainty in answers from any language model and enhancing their trustworthiness. In Proc. 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). (2024). source

      Section 5 (page unavailable) says LLMs are often used to judge answer correctness, but explicitly notes that the reliability of those judgments remains in question. Section 6.3 (page unavailable) reports more reliable average evaluations after low-confidence judgments are discarded. The paper therefore supports improving judgment reliability under a specific method, not the unqualified claim that LLM faithfulness in making judgments is established. (LLM, conf=0.83)

      Experimental — accuracy not yet measured
    • cites [35] Chen, G. H., Chen, S., Liu, Z., Jiang, F. & Wang, B. Humans or LLMs as the judge? a study on judgement bias. In Proc. 2024 Conference on Empirical Methods in Natural Language Processing, 8301–8327 https://aclanthology.org/2024.emnlp-main.474 (Association for Computational Linguistics, 2024). source

      The Abstract (page unavailable) says LLM judges have biases that question the reliability of their evaluations; it reports that even cutting-edge judges possess considerable biases. Section 2.2 (page unavailable) likewise says their annotation results require validation, rather than treating the faithfulness of their judgments as established. (LLM, conf=0.86)

      Checked by a reasoning model — may be wrong.

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The presence of prompts which induce cognitive biases can decrease the diagnostic accuracy of LLMs by as much as 26%.

    “Additionally, it has been found that the presence of prompts which induce cognitive biases can decrease the diagnostic accuracy of LLMs by as much as 26%15.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [15] Schmidgall, S. et al. Evaluation and mitigation of cognitive biases in medical language models. npj Digit. Med. 7.1, 295 (2024). source

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    LLMs have surpassed the average human score on the USMLE, rising from 38.1% in September 2021 to 90.2% in November 2023, where the human passing score is 60% and human expert score is 87%.

    “LLMs have quickly surpassed the average human score on the United States Medical Licensing Exam (USMLE) in a short amount of time, from 38.1% in September 20212 to 90.2% in November 20233 (human passing score is 60%, human expert score is 87%4).”
    Quote in the paperNot a substantial claim3 of 3 sources checked
    • cites [2] Gu, Y. et al. Domain-specific language model pretraining for biomedical natural language processing. ACM Trans. Comput. Healthc. (HEALTH) 3, 1–23 (2021). source

      The citation to Gu et al. is attached to the 38.1% score in September 2021, not the later score or human benchmarks. Searches for that figure and for the USMLE found no relevant passage. Section 2.3 (p. 7) describes biomedical NLP question-answering datasets, while Section 3.1 (p. 14) reports BLURB benchmark results, not licensing-exam scores. (LLM, conf=0.76)

      Experimental — accuracy not yet measured
    • cites [3] Nori, H. et al. Can generalist foundation models outcompete special-purpose tuning? case study in medicine. Preprint at https://doi.org/10.48550/arXiv.2311.16452 (2023). source

      Table 1 reports 90.2% for GPT-4 with Medprompt on the US four-option MedQA benchmark. Section 3.1 says those questions are in the style of the USMLE, so the paper does not establish a 90.2% score on the licensing exam itself. The earlier score and human benchmarks have separate citations in the citing sentence and are not judged here. (LLM, conf=0.86)

      Experimental — accuracy not yet measured
    • cites [4] Liévin, V., Hother, C. E., Motzfeldt, A. G. & Winther, O. Can large language models reason about medical questions? Patterns 5, (2023). source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    LLMs have shown the ability to encode clinical knowledge.

    “Recently, LLMs have shown the ability to encode clinical knowledge7,8, retrieve relevant medical texts9, and perform accurate single-turn medical question-answering3,10–12.”
    Quote in the paperNot a substantial claim2 of 2 sources checked
    • cites [7] Singhal, K. et al. Large language models encode clinical knowledge. Nature 620, 172–180 (2023). source
    • cites [8] Vaid, A., Landi, I., Nadkarni, G. & Nabeel, I. Using fine-tuned large language models to parse clinical notes in musculoskeletal pain disorders. Lancet Digital Health 5, e855–e858 (2023). source

      The Results passage under “Using fine-tuned large language models to parse clinical notes in musculoskeletal pain disorders” (page unavailable) reports that fine-tuned LLMs classified pain location and acuity from clinical notes. This demonstrates an ability to learn and use specific clinical information, but the paper does not establish the broader claim that LLMs encode clinical knowledge generally. (LLM, conf=0.78)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    LLMs can display racial biases.

    “Previous work has indicated that LLMs can display racial biases13 and might also lead to incorrect diagnoses due to inaccurate patient feedback14.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [13] Omiye, J. A., Lester, J. C., Spichak, S., Rotemberg, V. & Daneshjou, R. Large language models propagate race-based medicine. NPJ Digit. Med. 6, 195 (2023). source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    LLMs have shown the ability to perform accurate single-turn medical question-answering.

    “Recently, LLMs have shown the ability to encode clinical knowledge7,8, retrieve relevant medical texts9, and perform accurate single-turn medical question-answering3,10–12.”
    Quote in the paperNot a substantial claim4 of 4 sources checked
    • cites [3] Nori, H. et al. Can generalist foundation models outcompete special-purpose tuning? case study in medicine. Preprint at https://doi.org/10.48550/arXiv.2311.16452 (2023). source
    • cites [10] Liévin, V., Hother, C. E. & Winther, O. Can large language models reason about medical questions? Preprint at https://doi.org/10.48550/arXiv.2207.08143 (2022). source
    • cites [11] Wu, C. et al. Pmc-llama: towards building open-source language models for medicine. J. Am. Med. Inform. Assoc. 31, 1833–1843 (2023). source
    • cites [12] Chen, Z. et al. Meditron-70b: scaling medical pretraining for large language models. Preprint at https://doi.org/10.48550/arXiv.2311.16079 (2023). source

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    A study suggests the average family physician asks 3.2 questions and spends less than 2 minutes before arriving at a conclusion.

    “one study suggest that the average family physician asks 3.2 questions and spends less than 2 minutes before arriving at a conclusion24”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [24] Ely, J. W. et al. Analysis of questions asked by family doctors regarding patient care. Bmj 319, 358–361 (1999). source

      The Demographic data section reports 3.2 questions per 10 patients seen, not 3.2 questions per physician without a denominator. The Answers section reports a mean of 118 seconds pursuing an answer to a question; it does not measure time before arriving at a clinical conclusion, and most questions were not pursued. (LLM, conf=0.90)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Reported multiple choice evaluations typically show specialist QAs have lower performance.

    “reported multiple choice evaluations where specialist QAs typically have lower performance3”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [3] Nori, H. et al. Can generalist foundation models outcompete special-purpose tuning? case study in medicine. Preprint at https://doi.org/10.48550/arXiv.2311.16452 (2023). source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Hospitalized patients receive an average of 1.42 diagnostic images per patient stay.

    “hospitalized patients receiving an average of 1.42 diagnostic images per patient stay22”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [22] Smith-Bindman, R. et al. Use of diagnostic imaging studies and associated radiation exposure for patients enrolled in large integrated health care systems, 1996-2010. Jama 307, 2400–2409 (2012). source

      The Results (page unavailable) report 1,420 imaging examinations per 1,000 health-plan enrollees per year, equivalent to 1.42 per enrollee per year. They do not report 1.42 images per hospitalized patient per stay; the cited figure has a different population and denominator. (LLM, conf=0.88)

      Experimental — accuracy not yet measured

  • May combine two claims.

    Premisecitation

    Strong LLM judges like GPT-4 can match both controlled and crowd-sourced human preferences well, achieving over 80% agreement, the same level of agreement as between humans.

    “Recent research25 has shown that strong LLM judges like GPT-4 can match both controlled and crowd-sourced human preferences well, achieving over 80% agreement, which is the same level of agreement between humans”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [25] Zheng, L. et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Adv. Neural Inf. Process. Syst. 36, 46595–46623 (2023). source

  • May combine two claims.

    Premisecitation

    Previous experiments in this work and prior work provided measurement results through text and did not explore the model's ability to understand visual context.

    “the previous experiments in this work and prior work23 provided measurement results through text, and did not explore the ability of the model to understand visual context.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [23] Tu, T. et al. Towards conversational diagnostic artificial intelligence. Nature 642.8067, 442–450 (2025). source

      The Methods > Evaluation section (page not provided) says the consultations took place through a synchronous text-chat interface, supporting the point that this study did not test the model’s understanding of visual context. The copy does not specifically say that measurement results were provided through text, or establish that detail for other prior work. (LLM, conf=0.80)

      Experimental — accuracy not yet measured

  • May combine two claims.

    Premisecitation

    Previous benchmarks like AMIE, SAPS, and CRAFT-MD focus on dialogue-based evaluations but lack multimodal capabilities and do not simulate real-world biases, tool usage, multilingual, or specialist cases.

    “Previous benchmarks like AMIE23, SAPS30, and CRAFT-MD31 focus on dialogue-based evaluations but lack multimodal capabilities and do not simulate real-world biases, tool usage, multilingual, or specialist cases.”
    Quote in the paperSubstantial claim2 of 3 sources checked
    • cites [23] Tu, T. et al. Towards conversational diagnostic artificial intelligence. Nature 642.8067, 442–450 (2025). source

      The Main and Methods > Evaluation sections describe AMIE as a dialogue-based evaluation conducted through text chat, supporting the claim’s characterization of AMIE as non-multimodal. The paper does not assess SAPS or CRAFT-MD or establish the full list of alleged benchmark gaps. Moreover, Discussion > Diagnostic performance says AMIE used scenarios across a range of specialties, so the assertion that it did not simulate specialist cases is not supported for AMIE. (LLM, conf=0.84)

      Experimental — accuracy not yet measured
    • cites [30] Liao, Y. et al. Automatic interactive evaluation for large language models with state aware patient simulator. Preprint at https://doi.org/10.48550/arXiv.2403.08495 (2024). source

      Section 2.4 (page not provided) confirms that SAPS evaluates multi-round diagnostic dialogues, supporting the dialogue-based description for SAPS. The cited paper does not establish the claimed limitations for AMIE, SAPS, and CRAFT-MD as a group: the passages do not compare their multimodal capabilities or their handling of biases, tools, multilingual cases, and specialist cases. (LLM, conf=0.85)

      Experimental — accuracy not yet measured
    • cites [31] Johri, S. et al. An evaluation framework for clinical use of large language models in patient interaction tasks. Nat. Med. 31.1, 77–86 (2025). source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    GPT-4 shows superior performance compared to GPT-3.5 in knowledge assessments.

    “with GPT-4 showing superior performance compared to GPT-3.5 in knowledge assessments3,43.”
    Quote in the paperNot a substantial claim1 of 2 sources checked
    • cites [3] Nori, H. et al. Can generalist foundation models outcompete special-purpose tuning? case study in medicine. Preprint at https://doi.org/10.48550/arXiv.2311.16452 (2023). source
    • cites [43] OpenAI et al. Gpt-4 technical report 2303.08774 (2023).

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Premisecitation

    Mixtral-8 × 7B and Llama 2-70B-Chat do not report training on the MedQA test or train set.

    “Currently, Mixtral-8 × 7B28 and Llama 2-70B-Chat29 do not report training on the MedQA test or train set.”
    Quote in the paperNot a substantial claim2 of 2 sources checked
    • cites [28] Jiang, A. Q. et al. Mixtral of experts. Preprint at https://doi.org/10.48550/arXiv.2401.04088 (2024). source

      The Results section lists benchmarks used to evaluate Mixtral and Llama 2, but does not address whether either model was trained on MedQA. Searches for MedQA and medical question-answering training data found no relevant passage in our copy; the absence of a report does not establish the claim under this rubric. (LLM, conf=0.70)

      Experimental — accuracy not yet measured
    • cites [29] Touvron, H. et al. Llama: open and efficient foundation language models. Preprint at https://doi.org/10.48550/arXiv.2302.13971 (2023). source

      The cited paper reports results for the original LLaMA models, including LLaMA-65B in §3.2 (p. 5), not Llama 2-70B-Chat or Mixtral-8 × 7B. Searches for MedQA training data and those model names found no passage addressing whether either model reported training on the MedQA train or test set. (LLM, conf=0.76)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Mixtral 8 × 7B has demonstrated the ability to either surpass or equal the performance of other models like llama-2-70B and gpt-3.5 across a range of benchmarks.

    “which has demonstrated its ability to either surpass or equal the performance of other models like llama-2-70B and gpt-3.5 across a range of benchmarks.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [28] Jiang, A. Q. et al. Mixtral of experts. Preprint at https://doi.org/10.48550/arXiv.2401.04088 (2024). source

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    The 13B parameter PMC-LLaMA model demonstrates superior performance on various public medical QA benchmarks, even surpassing ChatGPT in certain evaluations.

    “The 13B parameter PMC-LLaMA model demonstrates superior performance on various public medical QA benchmarks, even surpassing ChatGPT in certain evaluations.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [11] Wu, C. et al. Pmc-llama: towards building open-source language models for medicine. J. Am. Med. Inform. Assoc. 31, 1833–1843 (2023). source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    MedLLaMA3 ranked second on the HuggingFace MedLLM Leaderboard in May 2024.

    “On May 2024, MedLLaMA3 ranked second on the HuggingFace MedLLM Leaderboard, showcasing its exceptional performance in medical-related natural language processing tasks”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [45] Xie, Q. et al. Me llama: foundation large language models for medical applications. Preprint at https://doi.org/10.48550/arXiv.2402.12749 (2024). source

      The Model Performance and Methods sections describe Me-LLaMA models built on LLaMA2, but do not address MedLLaMA3. Searches for MedLLaMA3 and the May 2024 HuggingFace MedLLM Leaderboard ranking found no passage supporting the claimed second-place rank. (LLM, conf=0.76)

      Experimental — accuracy not yet measured

  • May combine two claims.

    Findingown data

    GPT-4 reached its highest accuracy of 42.2% with Reflection CoT, surpassing Claude 3.5 in this specific tool.

    “GPT-4 reached its highest accuracy of 42.2% with Reflection CoT, surpassing Claude 3.5 in this specific tool”
    Quote in the paperNot a substantial claim

  • The quote only partly states this claim.

    Findingown data

    On AgentClinic-MIMIC-IV with GPT-4 patient and measurement agents, OpenBioLLM-70B achieved a diagnostic accuracy of 38.1% ± 3.2.

    “OpenBioLLM-70B 38.1 ± 3.2”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Claude 3.5 achieved the highest overall performance with an average accuracy of 51.3% across the six agent tools.

    “Claude 3.5 achieved the highest overall performance with an average accuracy of 51.3%”
    Quote in the paperNot a substantial claim

  • The quote only partly states this claim.

    Findingown data

    On AgentClinic-MIMIC-IV with GPT-4 patient and measurement agents, GPT-4o achieved a diagnostic accuracy of 24.0% ± 2.9.

    “GPT-4o 24.0% ± 2.9”
    Quote in the paperNot a substantial claim

  • The quote only partly states this claim.

    Findingown data

    On AgentClinic-MIMIC-IV with GPT-4 patient and measurement agents, MedLlama3-8B achieved a diagnostic accuracy of 29.7% ± 2.6.

    “MedLlama3-8B 29.7 ± 2.6”
    Quote in the paperNot a substantial claim

  • The quote only partly states this claim.

    Findingown data

    On AgentClinic-MIMIC-IV with GPT-4 patient and measurement agents, Mixtral-8 × 7B achieved a diagnostic accuracy of 29.5% ± 3.1.

    “Mixtral-8 × 7B 29.5% ± 3.1”
    Quote in the paperNot a substantial claim

  • The quote only partly states this claim.

    Findingown data

    On AgentClinic-MIMIC-IV with GPT-4 patient and measurement agents, Llama 3 70B achieved a diagnostic accuracy of 8.5% ± 1.9.

    “Llama 3 70B 8.5% ± 1.9”
    Quote in the paperNot a substantial claim

  • The quote only partly states this claim.

    Findingown data

    On AgentClinic-MIMIC-IV with GPT-4 patient and measurement agents, Claude-3.5 achieved a diagnostic accuracy of 42.9% ± 3.3.

    “Claude-3.5 42.9% ±3.3”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    GPT-3.5 experienced decreased performance across all tools, particularly with Adaptive RAG (Book), which led to a 27.1% drop.

    “GPT-3.5 experienced decreased performance across all tools, particularly with Adaptive RAG (Book), which led to a 27.1% drop”
    Quote in the paperNot a substantial claim

  • The quote only partly states this claim.

    Findingown data

    On AgentClinic-MIMIC-IV with GPT-4 patient and measurement agents, PMC-Llama 7B achieved a diagnostic accuracy of 34.3% ± 3.0.

    “PMC-Llama 7B 34.3 ± 3.0”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    GPT-4 and Mixtral-8 × 7B have an unbiased accuracy equal to 52% and 37% respectively on AgentClinic-MedQA.

    “GPT-4 and Mixtral-8 × 7B have an unbiased accuracy equal to 52% and 37% respectively”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    GPT-4o-mini had mixed results, benefiting from some tools like Reflection CoT and Adaptive RAG (Web) but showing slight decreases with others.

    “GPT-4o-mini had mixed results, benefiting from some tools like Reflection CoT and Adaptive RAG (Web) but showing slight decreases with others”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Claude 3.5 Sonnet maintained high and consistent performance across all languages, achieving an average accuracy of 48.4%.

    “Claude 3.5 Sonnet stood out by maintaining high and consistent performance across all languages, achieving an average accuracy of 48.4%”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Comments on the patient agent noted them being overly verbose and unnecessarily repeating the question back to the doctor agent.

    “comments were made on them being overly verbose and unnecessarily repeating the question back to the doctor agent.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    GPT-4 showed reduced accuracy in Emergency Medicine (32.3%) and Geriatrics (40%).

    “showed reduced accuracy in Emergency Medicine (32.3%) and Geriatrics (40%)”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The doctor agent adopts a neutral tone and does not open the dialogue with an inviting question, instead focusing on the patient's current symptoms and medical history.

    “the doctor agent adopts a neutral tone and does not open the dialogue with an inviting question. Instead, it cuts right to the chase, immediately focusing on the patient’s current symptoms and medical history.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Implicit biases had a profound effect on all three categories of patient perception, with education bias consistently reducing patient perception across all three categories.

    “implicit biases had a profound effect on all three categories of patient perception, with education bias consistently reducing patient perception across all three categories”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The lower rating for the doctor agent stems from issues such as providing a bad opening statement, making basic errors, overly focusing on a particular diagnostic, or not being diligent enough.

    “the lower rating for the doctor agent stems from several points such as providing a bad opening statement, making basic errors, overly focusing on a particular diagnostic, or not being diligent enough.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    All models performed best in English, with performance varying significantly across other languages.

    “Overall, all models performed best in English, with performance varying significantly across other languages.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Claude 3.5 excelled in Internal Medicine (78.3%), Otolaryngology (76.7%), and Gynecology (74.3%).

    “excelling in Internal Medicine (78.3%), Otolaryngology (76.7%), and Gynecology (74.3%)”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    GPT-4 demonstrated strong performance in Gynecology (68.5%) and Ophthalmology (65.2%).

    “GPT-4 demonstrated strong performance in Gynecology (68.5%) and Ophthalmology (65.2%)”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    GPT-4 maintained high diagnostic accuracy under cognitive and implicit biases, while Mixtral-8 × 7B experienced significant performance degradation.

    “Notably, models like GPT-4 demonstrated resilience to cognitive and implicit biases, maintaining high diagnostic accuracy, while others like Mixtral-8 × 7B experienced significant performance degradation.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The LLM-simulated agents (patient, doctor, measurement, moderator) adequately represent the corresponding real clinical actors and interactions.
    • Rests on an assumption: The bias prompts induce effects in the LLM agents comparable to the corresponding human biases in clinical settings.

  • May combine two claims.

    Findingown data

    A GPT-3.5 doctor agent achieves 38% diagnostic accuracy when interacting with a GPT-4 patient agent and 37% when interacting with a GPT-3.5 patient agent.

    “When a GPT-3.5 doctor agent interacts with a GPT-4 patient agent, the accuracy comes out to 38%, but when a GPT-3.5 doctor interacts with a GPT-3.5 patient agent the accuracy comes out to a very similar value of 37% which would be expected to be much lower.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Between the implicit biases, sexual orientation bias had the lowest effect on patient perceptions, followed by racial bias and gender bias.

    “between the implicit biases, sexual orientation bias had the lowest effect on patient perceptions, followed by racial bias and gender bias”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Most patient cognitive biases did not have a strong effect on any of the patient perceptions when compared to an unbiased patient agent except for in the case of self-diagnosis, which had sizeable drops in confidence (4.7 points) and consultation (2 points), and a minor drop in compliance (1 point).

    “most patient cognitive biases did not have a strong effect on any of the patient perceptions when compared to an unbiased patient agent except for in the case of self-diagnosis, which had sizeable drops in confidence (4.7 points) and consultation (2 points), and a minor drop in compliance (1 point)”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    Overall, the findings indicate a hierarchy in model performance, with Claude 3.5 consistently outperforming other models across most tools, except in the case of Reflection CoT where GPT-4 excels.

    “Overall, the findings indicate a hierarchy in model performance, with Claude 3.5 consistently outperforming other models across most tools, except in the case of Reflection CoT where GPT-4 excels”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationreasoning

    The biases presented in prior work were quite simple and did not allow for the bias to present in a realistic way, which is typically subtle and through interaction.

    “However, these biases were quite simple, presenting a cognitive bias snippet at the beginning of each question (e.g., “Recently, there was a patient with similar symptoms that you diagnosed with permanent loss of smell”). This form of presentation did not allow for the bias to present in a realistic way, which is typically subtle and through interaction.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    LLMs might lead to incorrect diagnoses due to inaccurate patient feedback.

    “Previous work has indicated that LLMs can display racial biases13 and might also lead to incorrect diagnoses due to inaccurate patient feedback14.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [14] Ziaei, R. & Schmidgall, S. Language models are susceptible to incorrect patient self-diagnosis in medical applications. In Deep Generative Models for Health Workshop NeurIPS 2023 (2023).

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Classified as probably not a substantial claim.

    Study design and conductown data

    Each case scenario was manually validated after being populated by GPT-4.

    “We initially use an LLM (GPT-4) to populate the structured JSON, and then manually validate each of the case scenarios.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    Multilingual patient cases were converted from AgentClinic-MedQA to the target language using GPT-4 and then manually corrected by native speakers.

    “Multilingual patient cases are converted from AgentClinic-MedQA to the the target language using GPT-4 and then manually corrected by native speakers.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    The slight decrease in accuracy at larger N is likely due to the growing input size, which can be difficult for language models.

    “This is likely due to the growing input size, which can be difficult for language models.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    The generated cases were manually inspected by native speakers to ensure linguistic accuracy.

    “The generated cases were manually inspected by native speakers to ensure linguistic accuracy.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S1 (asserted), S3 (load-bearing).

    Interpretationreasoning

    Cross-communication between different language models provides an additional challenge in the doctor-patient agent interaction.

    “We suspect that cross-communication between different language models provides an additional challenge.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Premisecitation

    Existing medical QA literature identifies Psychiatry and Otolaryngology as the least challenging specialties.

    “existing medical QA literature that identifies Psychiatry and Otolaryngology as the least challenging specialties17”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [17] Pal, A., Umapathi, L. K. & Sankarasubbu, M. Medmcqa: a large-scale multi-subject multi-choice dataset for medical domain question answering. In Conference on health, inference, and learning, 248–260 (PMLR, 2022).

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductown data

    The human evaluation used three MD evaluators who rated 20 randomly selected English-language AgentClinic-MedQA dialogues.

    “the three human clinician evaluators (individuals with MDs) rated a total of N = 20 complete dialogue simulations, randomly selected from the English-language AgentClinic-MedQA dataset.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    The measurement agent's reliance on an LLM to provide results may introduce errors or hallucinations.

    “its reliance on using an LLM to provide results may introduce errors or hallucinations, which could be mitigated through a database or SQL tool.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    The discrepancy between this study's findings and existing literature may indicate inherent difficulties in diagnosing diseases through dialogue-based interactions as opposed to multiple-choice question formats.

    “This discrepancy may indicate inherent difficulties in diagnosing diseases through dialogue-based interactions as opposed to multiple-choice question formats.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    The large drop in accuracy is partially because the doctor agent did not provide a diagnosis at all, perhaps due to not having enough information.

    “This large drop in accuracy is partially because of the doctor agent not providing a diagnosis at all, perhaps due to not having enough information.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Premisecitation

    There is a linear relationship between a language model's self-recognition capability and the strength of its self-preference bias.

    “Recent work supports this hypothesis by demonstrating a linear relationship between self-recognition capability and the strength of self-preference bias18.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [18] Panickssery, A., Bowman, S. R. & Feng, S. Llm evaluators recognize and favor their own generations. Adv. Neural Inf. Process. Syst. 37, 68772–68802 (2024). source

      Checked before — its earlier verdict is held back until a new check runs.

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation

    Language models can recognize their own text with high accuracy and display disproportionate preference to that text.

    “This work shows that language models can recognize their own text with high accuracy, and display disproportionate preference to that text, which may suggest there is an advantage for doctor models which have the same LLM acting as the patient agent.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [18] Panickssery, A., Bowman, S. R. & Feng, S. Llm evaluators recognize and favor their own generations. Adv. Neural Inf. Process. Syst. 37, 68772–68802 (2024). source

      Checked before — its earlier verdict is held back until a new check runs.

  • May combine two claims.

    Interpretationreasoning

    AgentClinic advances the field by providing an interactive, multimodal environment with bias simulation and tool integration, offering a more comprehensive evaluation platform for medical AI systems.

    “AgentClinic advances the field by providing an interactive, multimodal environment with bias simulation and tool integration, offering a more comprehensive evaluation platform for medical AI systems.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The LLM-simulated agents (patient, doctor, measurement, moderator) adequately represent the corresponding real clinical actors and interactions.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductown data

    The three human dialogue evaluators were all certified with MDs, but their specific medical specialties were not controlled for.

    “Additionally, for the human dialogue ratings, the three evaluators were all certified with MDs (ER, JJ, MM), but their specific medical specialties were not controlled for, representing a potential area for more targeted evaluation in future studies.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    No additional inter-run reliability analysis was performed for the scalar patient ratings.

    “No additional inter-run reliability analysis was performed for these scalar ratings.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductreasoning

    The tool-use commands in AgentClinic are abstractions of real-world clinical workflows, and this simplification limits ecological validity.

    “We recognize that the tool-use commands in AgentClinic are abstractions of real-world clinical workflows.”
    • Also stated in: § Discussion · plain
      “Nevertheless, we acknowledge that this simplification limits ecological validity.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationcitation

    There may be an advantage for LLMs that act as both the patient and the doctor agent, because LLMs recognize their own text with high accuracy and display disproportionate preference to that text.

    “Another limitation for the experiments on varying the patient LLM suggest that there may be an advantage for LLMs which act as both the patient and the doctor agent, because LLMs are able to recognize their own text with high accuracy, and display disproportionate preference to that text18.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [18] Panickssery, A., Bowman, S. R. & Feng, S. Llm evaluators recognize and favor their own generations. Adv. Neural Inf. Process. Syst. 37, 68772–68802 (2024). source

      Checked before — its earlier verdict is held back until a new check runs.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The sensitivity of the benchmark results to variations in the moderator agent's own model or system prompt was not explored.

    “While our moderator agent is based on an LLM judge shown to have high agreement with humans, the sensitivity of our benchmark results to variations in the moderator agent’s own model or system prompt was not explored.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    OpenBioLLM-70B exhibits superior performance across multiple biomedical benchmarks, outperforming other open-source models and even larger proprietary models like GPT-4 and Med-PaLM-2.

    “The model exhibits superior performance across multiple biomedical benchmarks, outperforming other open-source models and even larger proprietary models like GPT-4 and Med-PaLM-2.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [44] Ankit Pal, M. S. Openbiollms: advancing open-source large language models for healthcare and life sciences. https://huggingface.co/aaditya/OpenBioLLM-Llama3-70B (2024).

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Classified as probably not a substantial claim.

    Premisecitation

    Adaptive retrieval enables the LLM to actively determine when and what information to retrieve.

    “we employ adaptive retrieval50,51, which enables the LLM to actively determine when and what information to retrieve.”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [50] Zhengbao, J. et al. Active retrieval augmented generation. Proceedings of the 2023 conference on empirical methods in natural language processing. (2023).

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [51] Asai, A. et al. Self-rag: Learning to retrieve, generate, and critique through self-reflection. The Twelfth International Conference on Learning Representations. (2023).

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    Analyses emphasize relative differences between experimental conditions rather than interpretation of absolute rating values.

    “Analyses emphasize relative differences between experimental conditions (e.g., biased vs. unbiased) rather than interpretation of absolute rating values.”
    Quote in the paperSubstantial claim
68 of 68

  • The paper relies on this without support.

    Clinical work is a multiplexed task involving sequential decision making, requiring handling uncertainty with limited information and finite resources while obtaining relevant information from patients.

    “However, clinical work is a multiplexed task that involves sequential decision making, requiring the doctor to handle uncertainty with limited information and finite resources while compassionately taking care of patients and obtaining relevant information from them.”

  • The paper relies on this without support.

    Static multiple choice evaluations dominate the recent literature and present all necessary information in case vignettes, not reflecting sequential decision making.

    “This capability is not currently reflected in the static multiple choice evaluations (that dominate the recent literature) where all the necessary information is presented in a case vignettes and where the LLM is tasked to answer a question, or to just select the most plausible answer choice for a given question.”

  • The paper relies on this without support.

    23 different biases are known to be present in clinical environments.

    “We also support the ability for agents to exhibit 23 different biases that are known to be present in clinical environments.”

  • The paper relies on this without support.

    Adding context into the agent’s system prompt instructing them to play out a bias causes the agent to exhibit that bias.

    “These biases are introduced by adding context into the agent’s system prompt instructing them to play out that bias as part of their role.”

  • The paper relies on this without support.

    Patient agents' compliance and consultation ratings reflect real patient behavior.

    “our evaluation metrics go beyond diagnostic accuracy by giving emphasis to the patient agents with measures like patient compliance and consultation ratings.”

  • The paper relies on this without support.

    The realism of the actual dialogue in AgentClinic has not yet been evaluated.

    “the realism of the actual dialogue itself has yet to be evaluated.”

  • The paper relies on this without support.

    The NEJM case challenges have pathology-confirmed diagnoses.

    “have an associated pathology-confirmed diagnosis.”

  • The authors state this as an assumption.

    The patient agents simulated by language models may not fully capture the complexity and variability of real human patients.

    “the patient agents used in our study are simulated by language models, which may not fully capture the complexity and variability of real human patients.”

  • The authors state this as an assumption.

    The confidence, compliance, and consultation ratings provided by simulated agents may not perfectly reflect real-world patient perceptions.

    “the confidence, compliance, and consultation ratings provided by these agents may not perfectly reflect real-world patient perceptions”

  • The authors state this as an assumption.

    Metrics derived from simulated agents are not a direct proxy for real human patient experiences.

    “metrics derived from simulated agents are not a direct proxy for real human patient experiences”

  • The paper relies on this without support.

    The LLM-simulated agents (patient, doctor, measurement, moderator) adequately represent the corresponding real clinical actors and interactions.

    “Another consideration in our study is the use of LLMs to simulate all interacting agents. While this approach enables controlled and scalable experimentation, it introduces the possibility that model-specific behaviors or inherent architectural priors could influence the outcomes.”

  • The paper relies on this without support.

    The bias prompts induce effects in the LLM agents comparable to the corresponding human biases in clinical settings.

    “In this work, we introduce cognitive bias prompts in the LLM system prompt for both the patient and doctor agents.”

  • The paper relies on this without support.

    The relevant patient information identified in each MedQA case constitutes the complete set of information needed for diagnosis, so coverage measured against it is meaningful.

    “Coverage was calculated as the ratio of extracted information to the total relevant information available in the MedQA cases.”
13 of 13

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • GPT-4 is given two different highest tool-use accuracies

    Major

    The paper says GPT-4 benefited most from Adaptive RAG (Web) at 43.9%, then calls 42.2% with Reflection CoT its highest accuracy. Both cannot be its highest tool-use accuracy.

  • Specialty list omits a specialty evaluated later

    Minor

    The stated list of nine AgentClinic-Spec specialties names only eight and omits Gynecology, for which the results later report accuracy.

2 of 2

Cites

Flagged first, then by citations

Research works printed but not linked: 5

  1. Ziaei, R. & Schmidgall, S. Language models are susceptible to incorrect patient self-diagnosis in medical applications. In Deep Generative Models for Health Workshop NeurIPS 2023 (2023).
  2. Pal, A., Umapathi, L. K. & Sankarasubbu, M. Medmcqa: a large-scale multi-subject multi-choice dataset for medical domain question answering. In Conference on health, inference, and learning, 248–260 (PMLR, 2022).
  3. Christiano, P. F. et al. Deep reinforcement learning from human preferences. Adv. Neural Inform. Process. Syst. 30, (2017).
  4. Zhengbao, J. et al. Active retrieval augmented generation. Proceedings of the 2023 conference on empirical methods in natural language processing. (2023).
  5. Asai, A. et al. Self-rag: Learning to retrieve, generate, and critique through self-reflection. The Twelfth International Conference on Learning Representations. (2023).

Cited by

Sorted by citations

Authors
Schmidgall, S., Ziaei, R., Harris, C., Kim, J. W., Reis, E. P., Jopling, J., & Moor, M.
Affiliations
  • Johns Hopkins University (US)
  • Stanford University (US)
  • Hospital Israelita Albert Einstein (BR)
  • ETH Zurich (CH)
Corresponding author
Schmidgall, Samuel
ORCID
Venue
npj Digital Medicine
Publisher
Springer Science and Business Media LLC
Published
2026-04-27
Volume
9
Issue
1
ISSN
2398-6352 (ISSN-L)
DOI
10.1038/s41746-026-02674-7
PMC
PMC13324710
OpenAlex
W7156104865
Cited by
11
References
41
Citations by year
2026: 11
FWCI
22.23
Type
journal article
Language
en
Open access
Licence
Locations
  • npj Digital Medicine (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • Repository for Publications and Research Data (ETH Zurich) (publishedVersion, cc-by)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
  • PubMed Central (submittedVersion, cc-by)
  • ETH Zürich Research Collection (cc-by)
Funders
National Science Foundation,United States (DGE 2139757)
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
Keywords
large language models, multimodal benchmark, tool use, sequential decision making, MedQA, clinical simulation, electronic health records, LLM evaluation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-07.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-07.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review