CoChat
Sign Up
PreprintCited by 7Artificial Intelligence in Healthcare and EducationMachine Learning in HealthcareArtificial Intelligence in Healthcare

AI Hospital: Benchmarking Large Language Models in a Multi-agent Medical Interaction Simulator

Open access
Fan, Zhihao, Tang, Jialong, Chen, Wei, Wang, Siyuan, Wei, Zhongyu, Xi, Jun, Huang, Fei, Zhou, Jingren·2024·arXiv
DOI 10.48550/arxiv.2402.09742arXiv 2402.09742
Open this paper in CoChat — free accountOpen PDF

Abstract

Artificial intelligence has significantly advanced healthcare, particularly through large language models (LLMs) that excel in medical question answering benchmarks. However, their real-world clinical application remains limited due to the complexities of doctor-patient interactions. To address this, we introduce \textbf{AI Hospital}, a multi-agent framework simulating dynamic medical interactions between \emph{Doctor} as player and NPCs including \emph{Patient}, \emph{Examiner}, \emph{Chief Physician}. This setup allows for realistic assessments of LLMs in clinical scenarios. We develop the Multi-View Medical Evaluation (MVME) benchmark, utilizing high-quality Chinese medical records and NPCs to evaluate LLMs' performance in symptom collection, examination recommendations, and diagnoses. Additionally, a dispute resolution collaborative mechanism is proposed to enhance diagnostic accuracy through iterative discussions. Despite improvements, current LLMs exhibit significant performance gaps in multi-turn interactions compared to one-step approaches. Our findings highlight the need for further research to bridge these gaps and improve LLMs' clinical diagnostic capabilities. Our data, code, and experimental results are all open-sourced at \url{https://github.com/LibertFan/AI_Hospital}.

AI snapshot

Written from the paper’s full text.

TL;DR

The paper introduces AI Hospital, a multi-agent simulator with Doctor, Patient, Examiner, and Chief Physician agents, plus the MVME benchmark for evaluating LLMs in multi-turn Chinese clinical diagnosis. It finds that interactive LLM doctors substantially underperform a one-step GPT-4 upper bound, though a dispute-resolution collaboration mechanism improves diagnostic accuracy.

Key findings

  • Interactive LLMs scored far below the one-step GPT-4 upper bound on diagnostic results, diagnostic rationales, and treatment plans: GPT-4 interactive scored 29.36, 30.76, and 26.93, versus one-step GPT-4 scores of 58.89, 66.59, and 53.16, respectively.
  • Collaboration improved performance: “3 Doctors” reached 39.60 diagnostic results, 44.23 diagnostic rationales, and 37.26 treatment plans, exceeding “2 Doctors” and “2 Doctors w/o DR”; removing dispute resolution reduced effectiveness.
  • Agent behavior validation showed high reliability: Patient relevance/honesty, Examiner accuracy/consistency, and Doctor consistency were all over 95 across evaluated models.
  • In 219 GPT-4 cases rated 1 point for incorrect diagnosis, error types were omission of auxiliary examinations (99), exclusive focus on complications (52), and erroneous judgment (68).

Methods

  • AI Hospital simulates diagnostic dialogue with a Doctor player and NPCs: Patient, Examiner, and Chief Physician; GPT-3.5 powers Patient and Examiner, while GPT-4 powers Chief Physician.
  • MVME uses 506 Chinese medical records across departments, expert-reviewed with 94% deemed correct; evaluation covers symptom collection, examination recommendation, diagnostic results, diagnostic rationales, and treatment plans.
  • Chief Physician scores reports from 1–4, normalized to 0–100; disease entities are linked to ICD-10 for precision, recall, and F1; experiments evaluate GPT-3.5, GPT-4, Wenxin-4.0, Qwen-Max, Baichuan 13B, and HuatuoGPT-II 13B/34B, with 50 samples for agent behavior analysis.

Limitations

  • Paper-stated: primarily Chinese medical records limit generalizability; 506 cases may not capture rare diseases or full real-world complexity; simulated interactions may not perfectly replicate human nuances; treatment plan evaluation does not consider feasible alternatives; extensive OpenAI API use may increase environmental burden.
  • Evident from the text: entity-level metrics are calculated only for diseases because symptom and examination entity linking lacks readily available tools; among medical LLMs, only HuatuoGPT-II was selected because many others lost instruction-following ability and could not engage in the benchmark dialogue.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2402.09742

Cites

0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 9 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    An illustrative error involved failure to detect gallbladder stones due to the absence of a recommended abdominal ultrasound.

    “An illustrative case involved the failure to detect gallbladder stones, attributed to the absence of a recommended abdominal ultrasound.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The Chief Physician (GPT-4) can accurately evaluate Doctor performance by comparing summary report to medical record.
  • Findingown data

    Current LLMs show significant performance gaps in multi-turn interactions compared to one-step approaches.

    “Despite improvements, current LLMs exhibit significant performance gaps in multi-turn interactions compared to one-step approaches.”
    • Also stated in: § 1 Introduction · plain
      “Our results reveal a substantial performance gap between interactive LLMs and the one-step GPT-4, which serves as an upper bound by directly utilizing all patient information in a single interaction.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: GPT-4's one-step diagnosis performance is an upper bound of LLM performance.
    • Rests on an assumption: The Chief Physician (GPT-4) can accurately evaluate Doctor performance by comparing summary report to medical record.
  • Findingown data

    The dispute-resolution mechanism shows encouraging results in clinical diagnosis.

    “demonstrating promising results in clinical diagnosis”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The expert diagnoses in the collected medical records are correct and can serve as ground truth for evaluating LLM diagnostic accuracy.
  • Findingcitation

    In the AI Hospital doctor-patient interactions, the completeness of patient information collected by the Patient agent has an approximately linear relationship with the quality of the final diagnosis.

    “When communicating with the Doctor agent, the Patient agent there exists an approximately linear relationship between the completeness of collected patient information and the quality of final diagnosis, which is also observed in [10].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [10]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Findingreasoning

    In some cases, GPT-4 focuses only on patient-reported symptoms, such as soft tissue swelling in the feet, while ignoring underlying complications like diabetes.

    “In certain cases, GPT-4 focuses only the symptoms given by the patient, such as soft tissue swelling in the feet, while ignoring underlying complications, such as diabetes.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The Chief Physician (GPT-4) can accurately evaluate Doctor performance by comparing summary report to medical record.
  • May combine two claims. 20
  • The quote only partly states this claim. 2
  • Classified as probably not a substantial claim. 20

  • May combine two claims.

    Findingown data

    The agreement rates for the four questionnaire questions were 99.1%, 95.6%, 99.4%, and 100.0%.

    “The agreement rates for each question are 99.1%, 95.6%, 99.4%, and 100.0%.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The GPT-4-powered intern doctor's misdiagnoses included omission of auxiliary examinations (99 cases), exclusive focus on complications (52 cases), and erroneous judgment (68 cases).

    “Omission of auxiliary examinations 99”
    • Also stated in: § H Potential of AI Hospital Framework · plain
      “Exclusive focus on complications | 52”
    • Also stated in: § H Potential of AI Hospital Framework · plain
      “Erroneous judgment | 68”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The expert diagnoses in the collected medical records are correct and can serve as ground truth for evaluating LLM diagnostic accuracy.

  • May combine two claims. The quote only partly states this claim.

    Findingown data

    GPT-4's diagnostic errors are mainly categorized into three types: omission of auxiliary examinations, exclusive focus on complications, and erroneous judgment.

    “Through a systematic manual review, these errors are mainly categorized into three distinct types, which are detailed in Table 5.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The Chief Physician (GPT-4) can accurately evaluate Doctor performance by comparing summary report to medical record.

  • The quote only partly states this claim.

    Findingnone given

    Collaborative use of multiple models can exceed the performance of interactive GPT-4.

    “The collaborative use of models can exceed the performance of GPT-4, thereby validating the efficacy of the cooperative mechanism”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The Chief Physician (GPT-4) can accurately evaluate Doctor performance by comparing summary report to medical record.

  • May combine two claims.

    Findingown data

    The dispute resolution collaborative mechanism improves diagnostic performance to some extent but remains below the one-step GPT-4 upper bound.

    “The dispute resolution collaborative mechanism enhances performance to a certain degree but also falls short of the upper bound.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: GPT-4's one-step diagnosis performance is an upper bound of LLM performance.
    • Rests on an assumption: The Chief Physician (GPT-4) can accurately evaluate Doctor performance by comparing summary report to medical record.

  • May combine two claims.

    Findingown data

    GPT-4 produced incorrect diagnostic results in 219 cases, rated as 1 point by the Chief Physician.

    “We analyze an analysis on 219 cases where GPT-4 render incorrect diagnostic results, and rated as 1 point by the Chief Physician.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The Chief Physician (GPT-4) can accurately evaluate Doctor performance by comparing summary report to medical record.

  • May combine two claims.

    Findingown data

    For the Doctor role, overly complex prompts caused dialogue-flow problems, such as ignoring prompts or producing cognitive confusion, including the doctor sometimes outputting patient responses.

    “Particularly for the Doctor role, we discover that overly complex prompts could lead to issues in the dialogue flow, such as not adhering to the prompts or causing cognitive confusion (e.g., the doctor sometimes outputting the patient's responses).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The experiments show the performance of various LLMs and confirm the effectiveness of the dispute-resolution-centered collaborative approach.

    “In our experiment, the results not only demonstrate the performance of different LLMs but also confirm the efficacy of our dispute resolution-centered collaborative approach.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The expert diagnoses in the collected medical records are correct and can serve as ground truth for evaluating LLM diagnostic accuracy.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    After excluding records with deficiencies, 506 cases remained.

    “Subsequent to the exclusion of records with deficiencies, such as incomplete information, a total of 506 cases remained.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Large language models achieve strong results on static medical question answering benchmarks, even matching human experts.

    “These models demonstrate impressive performance on static medical question answering benchmarks like MedQA [5], PubMedQA [6], and MedMCQA [7], even rivaling human experts.”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites [5]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [6]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [7]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Premisecitation

    Patients often have insufficient medical knowledge and ambiguous understandings, which makes it hard for them to communicate their condition accurately in a single interaction.

    “In practice, patients frequently lack sufficient medical knowledge and may possess ambiguous understandings of various medical concepts, making it difficult for them to accurately and comprehensively communicate their physical condition to healthcare providers in a single interaction [8].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [8]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based). May combine two claims.

    Interpretationnone given

    Heavy reliance on OpenAI's LLM API may increase environmental burden, which could be reduced by using smaller, more efficient open-source models in future work.

    “Lastly, the extensive use of OpenAI's LLM API may increase the environmental burden, which could be mitigated by leveraging smaller, more efficient open-source models in future studies.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    The metrics reflect LLMs' dynamic and static medical decision-making abilities, including proactive inquiry, information gathering, clinical knowledge, and comprehensive judgment.

    “These metrics can reflect the LLMs' both dynamic and static medical decision-making abilities, including proactive inquiry, information gathering, clinical knowledge and comprehensive judgment.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Interpretationnone given

    AI Hospital advances medical interaction research by focusing on clinical diagnosis.

    “In AI Hospital, we take a step forward in the field of medical interactions by concentrating on clinical diagnosis.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    We randomly select 50 medical record samples and ask each agent to generate 50 multi-turn dialogue trajectories.

    “We randomly select 50 medical record samples and ask each agent generate 50 multi-turn dialogue trajectories within the AI Hospital framework.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationnone given

    Using mainly Chinese medical records may restrict how well the findings generalize to other languages and healthcare systems.

    “The use of primarily Chinese medical records may limit generalizability to other languages and healthcare systems.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationnone given

    The authors generated simulated patients and examiners from collected medical records and established a comprehensive engineering process.

    “We generate simulated patients and examiner using collected medical records and established a comprehensive engineering process.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The data sources were de-identified to remove sensitive information before collection.

    “The data sources have undergone a process of de-identification, removing sensitive information before our collection.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Interpretationnone given

    AI Hospital creates a real-time interactive consultation scenario.

    “We introduce AI Hospital to build a real-time interactive consultation scenario.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Study design and conductown data

    The AI Hospital framework simulates realistic medical interactions, with all metrics scoring above 95, indicating reliable and consistent agent behaviors.

    “Table 2 demonstrates the effectiveness of the AI Hospital framework in simulating realistic medical interactions, with high scores (all over 95) across all metrics indicating reliable and consistent agent behaviors.”
    • Also stated in: § Results and Analysis · plain
      “It validates the reliability and effectiveness of the proposed multi-agent system, laying a solid foundation for assessing LLMs' performance in clinical diagnosis.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    Existing LLMs struggle to make reasonable decisions in multi-turn interactions, leading to suboptimal diagnostic accuracy.

    “Despite dedicated prompting engineering, LLMs struggle to make reasonable decisions in multi-turn interactions, leading to suboptimal diagnostic accuracy.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: GPT-4's one-step diagnosis performance is an upper bound of LLM performance.
    • Rests on an assumption: The Chief Physician (GPT-4) can accurately evaluate Doctor performance by comparing summary report to medical record.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Interpretationnone given

    The current treatment plan evaluation system is inadequate because it ignores feasible alternative strategies, which may underestimate LLM performance.

    “The current treatment plan evaluation system is insufficient, as it does not consider feasible alternative strategies, potentially underestimating LLMs' performance.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    Current large language models also have difficulty recommending appropriate medical examinations, which worsens the limitation in medical interaction.

    “Moreover, their challenge in recommending appropriate medical examinations further exacerbates this limitation.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    The evaluation of the other parts reflects the Doctor's diagnostic and treatment capabilities.

    “The evaluation of the other parts can reflect the Doctor's diagnostic and treatment capabilities.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Interpretationnone given

    Simulated agent interactions may not perfectly capture human-to-human nuances and need further validation.

    “Simulated interactions between agents may not perfectly replicate human-to-human nuances, requiring further validation.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given

    The authors argue that agreement among doctors about a patient's physical condition is the foundation of collaborative diagnosis.

    “We contend that a consensus on the physical condition of patients among Doctors constitutes the cornerstone of collaborative diagnosis.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    We select samples from 10 secondary departments, randomly choosing five cases per department for review.

    “To verify the quality of the collected medical records, we select samples from 10 secondary departments, randomly choosing five cases per department for review.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductnone given

    We manually label all metrics and report average values.

    “We manually label all the metrics and report the average values.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    The validation steps ensure the high quality of the dataset and that the benchmark is based on audited medical cases.

    “These validation steps ensure the high quality of our dataset, ensuring that the benchmark is based on audited medical cases.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

    Interpretationreasoning

    The evaluation method is highly scalable and applicable.

    “Therefore, our evaluation method offers high scalability and applicability.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    The quantitative results highlight challenges for existing LLMs in posing pertinent questions, eliciting crucial symptoms, and recommending appropriate medical examinations.

    “Our quantitative results highlights the challenges existing LLMs face in posing pertinent questions, eliciting crucial symptoms, and recommending appropriate medical examinations.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: GPT-4's one-step diagnosis performance is an upper bound of LLM performance.
    • Rests on an assumption: The Chief Physician (GPT-4) can accurately evaluate Doctor performance by comparing summary report to medical record.

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationreasoning

    The final prompts work with most LLMs, allowing AI Hospital agents to perform their roles effectively.

    “These final prompts are adaptable to most LLMs, enabling the agents in AI Hospital to perform their respective duties effectively.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    They built the MVME benchmark to test how well different LLMs perform in interactive consultations.

    “Based on the platform, we build a benchmark MVME to explore the feasibility of different LLMs in interactive consultations.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The authors made efforts to ensure their medical record datasets are diverse and representative.

    “To mitigate this risk, we have made efforts to ensure the diversity and representativeness of our medical record datasets.”
    Quote in the paperNot a substantial claim
34 of 34

  • The paper relies on this without support.

    Real-world clinical application of LLMs is limited due to complexities of doctor-patient interactions.

    “However, their real-world clinical application remains limited due to the complexities of doctor-patient interactions.”

  • The paper relies on this without support.

    Patients often exhibit a strong dependence on doctors for guidance and clarification.

    “As a result, patients often exhibit a strong dependence on doctors for guidance and clarification.”

  • The paper relies on this without support.

    There is a scarcity of research assessing LLMs' ability to simulate doctor-patient interaction.

    “Despite the crucial importance of this dynamic, patientcentered diagnostic process, there is a scarcity of research assessing the ability of large language models (LLMs) to simulate this interaction.”

  • The paper relies on this without support.

    Medical records are a valuable resource for reconstructing hospital visit experience.

    “Medical records are a valuable resource for reconstructing the hospital visit experience and simulating real-world medical interactions.”

  • The paper relies on this without support.

    The presence of objective data in medical records indicates the patient underwent those tests at the doctor's recommendation.

    “The presence of these data in medical records indicates that the patient underwent these tests during the diagnostic process at the doctor's recommendation”

  • The authors state this as an assumption.

    GPT-4's one-step diagnosis performance is an upper bound of LLM performance.

    “We consider the performance of GPT-4 in this one-step setting as the upper bound of LLM performance.”

  • The paper relies on this without support.

    The Chief Physician (GPT-4) can accurately evaluate Doctor performance by comparing summary report to medical record.

    “Chief Physician first requires the Doctor to provide a comprehensive summary report for the patient, and then evaluates the summary report by comparing it with raw medical record, which serves as the gold standard.”

  • The paper relies on this without support.

    The expert diagnoses in the collected medical records are correct and can serve as ground truth for evaluating LLM diagnostic accuracy.

    “Table 6 : The translated template of questionnaire for expert verification of medical record.”

  • The paper relies on this without support.

    Real-world doctors do not make diagnoses before having sufficient information.

    “Real-world doctors do not make diagnoses before having sufficient information.”

  • The paper relies on this without support.

    Real-world doctors can actively inquire about various subjective information from patients, such as symptoms.

    “They possess the ability to actively inquire about various subjective information from patients (such as symptoms)”

  • The paper relies on this without support.

    Real-world doctors know what examinations are needed to obtain more quantitative and objective information.

    “and know what examinations are needed to obtain more quantitative and objective information.”
11 of 11

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Table 10 caption identifies the wrong agent

    Minor

    The caption calls Table 10 a prompt for the patient agent, but the prompt itself is explicitly for the Examiner.

  • Questionnaire is described as having four questions but contains five

    Minor

    Appendix F says the questionnaire has four questions, whereas Table 8 numbers five, including a question about the doctor's summary.

  • Figure 5 is assigned two different subjects

    Minor

    Appendix B identifies Figure 5 as an example collaboration flow, while the Figure 5 caption identifies it as a linear regression analysis.

3 of 3

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 4

  1. PA Trott. International classification of diseases for oncology. Journal of clinical pathology, 30(8):782, 1977.
  2. Bradley Efron. Bootstrap methods: another look at the jackknife. In Breakthroughs in statistics: Methodology and distribution, pages 569-593. Springer, 1992.
  3. Jing Li, Aixin Sun, Jianglei Han, and Chenliang Li. A survey on deep learning for named entity recognition. IEEE Transactions on Knowledge and Data Engineering, 34(1):50-70, 2020.
  4. Nigel Collier and Jin-Dong Kim. Introduction to the bio-entity recognition task at jnlpba. In Pro- ceedings of the International Joint Workshop on Natural Language Processing in Biomedicine and its Applications (NLPBA/BioNLP), pages 73-78, 2004.

Cited by

Sorted by citations

Related

Authors
Fan, Z., Tang, J., Chen, W., Wang, S., Wei, Z., Xi, J., Huang, F., & Zhou, J.
Affiliations
  • Alibaba Group (China) (CN)
  • Huazhong University of Science and Technology (CN)
  • Fudan University (CN)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2024
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2402.09742
arXiv
2402.09742
OpenAlex
W4391899415
Semantic Scholar
dc651710957209c99399d4d3ef1f0579dfcba11c
Cited by
6
References
0
Citations by year
2025: 4 · 2024: 2
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
  • Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
Keywords
large language models, multi-agent systems, medical question answering, diagnosis, multi-turn dialogue, diagnostic accuracy, dispute resolution mechanisms

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-07.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-07.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review