CoChat
Sign Up
Cited by 121Artificial Intelligence in Healthcare and EducationMachine Learning in HealthcareArtificial Intelligence in Healthcare

Enhancing diagnostic capability with multi-agents conversational large language models

Open access
Chen, Xi, Yi, Huahui, You, Mingke, Liu, WeiZhi, Wang, Li, Li, Hairui, Zhang, Xue, Guo, Yingman +6 more·2025·npj Digital Medicine
DOI 10.1038/s41746-025-01550-0PMC PMC11906805
Open this paper in CoChat — free account

Abstract

Large Language Models (LLMs) show promise in healthcare tasks but face challenges in complex medical scenarios. We developed a Multi-Agent Conversation (MAC) framework for disease diagnosis, inspired by clinical Multi-Disciplinary Team discussions. Using 302 rare disease cases, we evaluated GPT-3.5, GPT-4, and MAC on medical knowledge and clinical reasoning. MAC outperformed single models in both primary and follow-up consultations, achieving higher accuracy in diagnoses and suggested tests. Optimal performance was achieved with four doctor agents and a supervisor agent, using GPT-4 as the base model. MAC demonstrated high consistency across repeated runs. Further comparative analysis showed MAC also outperformed other methods including Chain of Thoughts (CoT), Self-Refine, and Self-Consistency with higher performance and more output tokens. This framework significantly enhanced LLMs' diagnostic capabilities, effectively bridging theoretical knowledge and practical clinical application. Our findings highlight the potential of multi-agent LLMs in healthcare and suggest further research into their clinical implementation.

AI snapshot

Written from the paper’s full text.

TL;DR

The authors developed a Multi-Agent Conversation (MAC) framework, inspired by clinical multidisciplinary team discussions, to improve LLM diagnosis of rare diseases. Across 302 rare disease cases, MAC outperformed single-agent GPT-3.5 and GPT-4 in diagnostic accuracy and recommended-test helpfulness, with best results using GPT-4 and four doctor agents plus a supervisor.

Key findings

  • In primary consultation, MAC with GPT-4 and four doctor agents achieved 34.11% most-likely-diagnosis accuracy, 48.12% possible-diagnosis accuracy, and 78.26% helpful further-diagnostic-test rate, compared with GPT-4 alone at 19.65%, 34.55%, and 58.17%, respectively.
  • In follow-up consultation, MAC with GPT-4 and four doctor agents achieved 53.86% most-likely-diagnosis accuracy and 67.88% possible-diagnosis accuracy, compared with GPT-4 alone at 37.86% and 59.71%.
  • Optimal MAC performance was observed with four doctor agents and a supervisor agent using GPT-4 as the base model; excluding the supervisor reduced GPT-4 primary-consultation most-likely-diagnosis accuracy from 34.11% to 32.67% and possible-diagnosis accuracy from 48.12% to 45.47%.
  • MAC also outperformed Chain of Thought, Self-Refine, and Self-Consistency across evaluation metrics, and reliability analysis showed moderate agreement (>0.4) in 23 of 28 evaluations and fair agreement (0.35–0.4) in the remaining five. Average cost per case with GPT-4 was $0.12 for primary consultations and $0.17 for follow-up consultations.

Methods

  • The MAC framework was built using AutoGen, with an admin agent presenting patient information, a supervisor agent guiding consensus, and doctor agents discussing diagnoses and tests; conversations ended at consensus or after a maximum of thirteen rounds.
  • The study included 302 rare diseases from 33 categories in Orphanet, selected by normalized weighted random sampling; clinical case reports published after January 2022 were identified from Medline and curated into primary and follow-up consultation scenarios.
  • GPT-3.5, GPT-4, and MAC were evaluated on disease-specific knowledge and diagnostic tasks; subgroup analyses varied base model, number of doctor agents (2–5), supervisor inclusion, and specialty assignment. Outputs were evaluated using GPT-4o with human validation on 302 cases, and analyses included ANOVA/Kruskal–Wallis, chi-square, and Fleiss’ kappa.

Limitations

  • The paper states the sample size was relatively small given the vast number of rare diseases and represents only a preliminary exploration; primary consultation data were manually extracted and may not always reflect actual consultation.
  • The paper notes that model output explanations do not fully capture inherent interpretability, and MAC should be seen as a supportive tool for clinicians rather than a replacement.
  • Evident from the text: the study used published case reports rather than actual human participants, and GPT-3.5 MAC configurations with more than two doctor agents in follow-up consultations were not feasible due to token output limitations.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1038/s41746-025-01550-0

Cites

50 of 50 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

30 of 35 cited sources checked · 18 supported · 0 flagged · accuracy not yet measured

  • Premisecitation

    Chain-of-thought prompting has been used to decompose complex diagnostic tasks into linear steps, which aids in integrating context and delivering comprehensive diagnostic reasoning.

    “Chain-of-thought (CoT) prompting has been used to decompose complex diagnostic tasks into linear steps, which aids in integrating context and delivering comprehensive diagnostic reasoning33.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [33] Wada, A. et al. Optimizing GPT-4 Turbo Diagnostic Accuracy in Neuroradiology through Prompt Engineering and Confidence Thresholds. Diagnostics 14, 1541 (2024). source
  • Premisecitation

    One study found that GPT-4's top diagnosis matched the final diagnosis in 39% of complex cases.

    “One study examined GPT-4’s performance in diagnosing complex cases using published case records, finding that the model’s top diagnosis matched the final diagnosis in 39% of cases2.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [2] Kanjee, Z., Crowe, B. & Rodman, A. Accuracy of a Generative Artificial Intelligence Model in a Complex Diagnostic Challenge. JAMA 330, 78–80 (2023). source
  • Premisecitation

    Mere output explanations do not fully capture the model's inherent interpretability.

    “However, it should be noted that mere output explanations do not fully capture the model’s inherent interpretability.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [38] Agarwal, C., Tanneru, S. H. & Lakkaraju, H. Faithfulness vs. plausibility: On the (un) reliability of explanations from large language models. Preprint arXiv:2402.04614 (2024). source
  • Premisecitation

    The multi-agent framework significantly enhances the capabilities of LLMs for managing complex tasks, including solving mathematical problems and performing retrieval-augmented code generation.

    “This technique significantly enhances the capabilities of LLMs for managing complex tasks, including solving mathematical problems and performing retrieval-augmented code generation11–14.”
    Quote in the paperSubstantial claim3 of 4 sources checked
    • cites [11] Wu, Q. et al. Autogen: Enabling next-gen llm applications via multi-agent conversation. Preprint arXiv:2308.08155 (2023). source
    • cites [12] Smit, A., Duckworth, P., Grinsztajn, N., Barrett, T. & Pretorius, A. Are we going MAD? Benchmarking Multi-Agent Debate between Language Models for Medical Q&A. In: Deep Generative Models for Health Workshop NeurIPS 2023) (NeurIPS, 2023).

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [13] Chan, C.-M. et al. Chateval: Towards better llm-based evaluators through multi-agent debate. Preprint arXiv:2308.07201 (2023). source

      The Introduction says multi-agent collaboration can improve completion of arduous tasks, and the Automatic NLG evaluation section reports better results on arithmetic problems. Our copy does not support the retrieval-augmented code-generation example; ChatEval’s own experiments concern evaluation of generated responses, not that task. (LLM, conf=0.87)

      Experimental — accuracy not yet measured
    • cites [14] Hong, S. et al. Metagpt: Meta programming for a multi-agent collaborative framework. URL https://arxiv.org/abs/2308.00352 (2023). source

      The Conclusion (p. 9) says MetaGPT enhances the problem-solving capabilities of LLM-based multi-agent systems and improves code-generation quality. The paper’s reported tasks are software engineering and programming; the searched passages do not support the more specific examples of solving mathematical problems or performing retrieval-augmented code generation. (LLM, conf=0.86)

      Experimental — accuracy not yet measured
  • Premisecitation

    A previous study demonstrated limitations of LLMs in the diagnosis of complex cases.

    “A previous study has demonstrated the limitations of LLMs in the diagnosis of complex cases2.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [2] Kanjee, Z., Crowe, B. & Rodman, A. Accuracy of a Generative Artificial Intelligence Model in a Complex Diagnostic Challenge. JAMA 330, 78–80 (2023). source
  • Classified as probably not a substantial claim. 49
  • May combine two claims. 29
  • The quote only partly states this claim. 1

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    MEDAGENTS is an innovative multi-disciplinary collaboration framework that enhances large language models' performance in zero-shot medical question answering.

    “For example, MEDAGENTS is an innovative multi-disciplinary collaboration framework that enhances large language models’ performance in zero-shot medical question answering25.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [25] Tang, X. et al. Medagents: Large language models as collaborators for zero-shot medical reasoning. Preprint arXiv:2311.10537 (2023). source

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Models may sometimes provide incorrect explanations alongside correct answers, or vice versa.

    “Previous study has indicated that models may sometimes provide incorrect explanations alongside correct answers, or vice versa38.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [38] Agarwal, C., Tanneru, S. H. & Lakkaraju, H. Faithfulness vs. plausibility: On the (un) reliability of explanations from large language models. Preprint arXiv:2402.04614 (2024). source

      The Introduction (page unavailable) says that plausible explanations may not accurately reflect a model’s reasoning, and the Security Concerns section gives an example of a refusal accompanied by a misleading explanation. Our copy does not establish the converse—an incorrect answer accompanied by a correct explanation. (LLM, conf=0.81)

      Experimental — accuracy not yet measured

  • May combine two claims.

    Premisecitation

    GPT-4 demonstrated superior diagnostic accuracy compared to GPT-3.5 across 75 consecutive real-world clinical cases, with its differential diagnosis lists more closely aligning with those generated by human experts.

    “The results indicated that GPT-4 demonstrated superior diagnostic accuracy compared to GPT-3.5, with its differential diagnosis lists more closely aligning with those generated by human experts31.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [31] Ríos-Hoyo, A. et al. Evaluation of large language models as a diagnostic aid for complex medical cases. Front. Med. 11, 1380148 (2024). source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Timely diagnosis of rare diseases has always been challenged by a lack of high-quality medical resources in many regions of the world.

    “Timely diagnosis of rare diseases has always been challenged by a lack of high-quality medical resources in many regions of the world”
    Quote in the paperNot a substantial claim2 of 2 sources checked
    • cites [7] Adachi, T. et al. Enhancing Equitable Access to Rare Disease Diagnosis and Treatment around the World: A Review of Evidence, Policies, and Challenges. Int. J. Environ. Res. Public Health 20, 4732 (2023). source
    • cites [8] Serrano, J. G. et al. Advancing Understanding of Inequities in Rare Disease Genomics. Clin. Therapeutics 45, 745–753 (2023). source

      The Introduction (page unavailable) says limited access to health care can prevent genetic evaluation and may lengthen time to diagnosis. The Abstract (page unavailable) also identifies proximity to a major medical center as an access barrier. The paper does not establish that a lack of high-quality medical resources has always impeded timely rare-disease diagnosis across many regions of the world. (LLM, conf=0.84)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Chain-of-Thought (CoT) techniques enable models to generate more extensive outputs detailing their reasoning processes.

    “Conversely, techniques such as Chain-of-Thought (CoT) enable models to generate more extensive outputs detailing their reasoning processes.”
    Quote in the paperNot a substantial claim2 of 2 sources checked
    • cites [39] Wei, J. et al. Chain-of-thought prompting elicits reasoning in large language models. 35, 24824–24837 (2022). source
    • cites [40] Xu, J. et al. Faithful logical reasoning via symbolic chain-of-thought. Preprint arXiv:2405.18357 (2024). source

  • Classified as probably not a substantial claim.

    Premisecitation

    Recent advancements in multi-agent systems have shown promising results in medical decision-making and diagnosis.

    “Recent advancements in multi-agent systems have shown promising results in medical decision-making and diagnosis.”
    Quote in the paperSubstantial claim3 of 3 sources checked
    • cites [25] Tang, X. et al. Medagents: Large language models as collaborators for zero-shot medical reasoning. Preprint arXiv:2311.10537 (2023). source

      The Conclusion reports that the multi-agent MedAgents framework surpassed zero-shot baselines on nine medical question-answering datasets, supporting promising results in medical reasoning. The Experiments section describes benchmark questions, not an evaluation of diagnostic performance or real-world clinical decision-making, so those parts of the claim go beyond what the paper establishes. (LLM, conf=0.83)

      Experimental — accuracy not yet measured
    • cites [28] Wang, H. et al. Beyond Direct Diagnosis: LLM-based Multi-Specialist Agent Consultation for Automatic Diagnosis. (2024). source
    • cites [29] Ke, Y. H. et al. Enhancing diagnostic accuracy through multi-agent conversations: Using large language models to mitigate cognitive bias. (2024). source

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Delays in diagnosis lead to delays in treatment and poor prognosis.

    “delays in diagnosis lead to delays in treatment and poor prognosis7,8.”
    Quote in the paperNot a substantial claim2 of 2 sources checked
    • cites [7] Adachi, T. et al. Enhancing Equitable Access to Rare Disease Diagnosis and Treatment around the World: A Review of Evidence, Policies, and Challenges. Int. J. Environ. Res. Public Health 20, 4732 (2023). source

      Section 3.2, “The Patient Journey” (page unavailable), says that delayed diagnosis leads to disease progression and worse health outcomes, supporting the poor-prognosis part of the claim. It discusses delays and barriers to treatment, but does not establish that diagnostic delays themselves cause treatment delays. (LLM, conf=0.84)

      Experimental — accuracy not yet measured
    • cites [8] Serrano, J. G. et al. Advancing Understanding of Inequities in Rare Disease Genomics. Clin. Therapeutics 45, 745–753 (2023). source

      The Introduction (page not available in our copy) says barriers to genetic evaluation may result in a longer time to diagnosis. The paper does not say that diagnostic delays cause treatment delays or poor prognosis; its survey lists “better treatment” as a potential benefit of diagnosis, not an observed consequence of delay. (LLM, conf=0.81)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    In most cases, CoT enhancement can improve model performance.

    “In most cases, this enhancement can improve model performance39,40.”
    Quote in the paperNot a substantial claim2 of 2 sources checked
    • cites [39] Wei, J. et al. Chain-of-thought prompting elicits reasoning in large language models. 35, 24824–24837 (2022). source

      The Results (p. 4) report performance gains from chain-of-thought prompting for models of approximately 100B parameters, especially on more complicated problems. They also say it does not positively affect small models, while Appendix A.1 (p. 16) says it hurts performance for most models below 10B parameters. The paper therefore supports improvement under particular conditions, not the unqualified claim that it improves performance in most cases. (LLM, conf=0.88)

      Experimental — accuracy not yet measured
    • cites [40] Xu, J. et al. Faithful logical reasoning via symbolic chain-of-thought. Preprint arXiv:2405.18357 (2024). source

      Section 4.2 (Main Result; page unavailable) reports that SymbCoT outperformed vanilla CoT on the paper’s logical-reasoning tests. This supports performance gains from the specific enhancement studied, but not the broader claim that CoT enhancements improve model performance in most cases generally. (LLM, conf=0.82)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims. The quote only partly states this claim.

    Study design and conductcitation

    For self-consistency, a framework with 10 distinct reasoning paths and a temperature parameter of 0.7 was implemented.

    “For self-consistency, following established literature, we implemented a framework with 10 distinct reasoning paths and set the temperature parameter to 0.723.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [23] Wang, X. et al. Self-consistency improves chain of thought reasoning in language models. Preprint arXiv:2203.11171 (2022). source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductcitation

    The sampling weights were adjusted based on the disease count in each type and moderated by natural logarithm transformation.

    “The sampling weights were adjusted based on the disease count in each type and moderated by natural logarithm transformation43,44.”
    Quote in the paperNot a substantial claim1 of 2 sources checked
    • cites [43] Efraimidis, P. & Spirakis, P. Weighted Random Sampling. In: Encyclopedia of Algorithms (ed Kao, M.-Y.) (Springer US, 2008). source
    • cites [44] West, R. M. Best practice in statistics: The use of log transformation. Ann. Clin. Biochem. 59, 162–165 (2022). source

      The Introduction’s ‘Natural logarithm or common logarithm’ section says natural logarithms can be used for log transformation. The paper illustrates transformation with urine albumin measurements; it does not describe adjusting sampling weights by disease counts or applying a logarithm to those weights. (LLM, conf=0.85)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductcitation

    The self-refine approach was configured with four iterative refinement rounds at a temperature of 0.7.

    “The self-refine approach was configured with four iterative refinement rounds at a temperature of 0.7, as adopted in prior research22.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [22] Madaan, A. et al. Self-refine: Iterative refinement with self-feedback. Adv. Neural Inf. Process. Syst. 36, 46534–46594 (2023). source

      Section 3.1 says SELF-REFINE continues for up to four feedback–refinement iterations, not necessarily four rounds in every case. Appendix L.2 uses a temperature of 0.7 for critique generation in a code-readability experiment, but that experiment runs five iterations and uses temperature 0.0 for the code editor; it does not establish 0.7 as the setting for the four-iteration configuration. (LLM, conf=0.88)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

    Interpretationcitation

    The limitations of explanations provided by large language models (LLMs) in their outputs should be noted, as the issue of explainability remains a persistent challenge that necessitates ongoing research.

    “In general, the limitations of explanations provided by large language models (LLMs) in their outputs should be noted, as the issue of explainability remains a persistent challenge that necessitates ongoing research41.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [41] Luo, H. & Specia, L. From understanding to utilization: A survey on explainability for large language models. Preprint arXiv:2401.12874 (2024). source

  • May combine two claims.

    Findingown data

    In primary consultations with four doctor agents, GPT-4 achieved higher accuracy for most likely diagnosis (34.11% vs 24.28%), possible diagnoses (48.12% vs 36.64%), and helpfulness of further diagnostic tests (78.26% vs 77.37%) compared to GPT-3.5.

    “In primary consultations with four doctor agents, GPT-4 achieved higher accuracy for the most likely diagnosis (34.11% vs 24.28%), possible diagnoses (48.12% vs 36.64%), and helpfulness of further diagnostic tests (78.26% vs 77.37%).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingnone given

    The optimal number of doctor agents was four, with three agents producing comparable results.

    “The optimal number of doctor agents was found to be four, with three agents producing comparable results.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    GPT-3.5 and GPT-4 are capable of diagnosing diseases based on apparent symptoms, such as identifying pericarditis and epilepsy through clinical presentation, but lack in-depth exploration of the underlying causes of these conditions.

    “In this study, GPT-3.5 and GPT-4 are capable of diagnosing diseases based on apparent symptoms, such as identifying pericarditis and epilepsy through clinical presentation. However, they lack in-depth exploration of the underlying causes of these conditions.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    MAC provided insights into how the models performed medical reasoning and handled different opinions during multi-agent conversations.

    “MAC provided insights into how the models performed medical reasoning and handled different opinions during multi-agent conversations, which may serve as interpretation on how LLMs reasoned through the task.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    For MAC using GPT-3.5 as the base model, settings with more than 2 doctor agents were not feasible due to token output limitation.

    “For MAC using GPT-3.5 as the base model, settings with more than 2 doctor agents were not feasible due to token output limitation.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    For Self-Refine, increasing refinement rounds from 2 to 4 enhanced performance, but further rounds (10) led to a slight decline.

    “For self-refine, increasing the number of refinement rounds from 2 to 4 enhanced performance, particularly in most likely diagnosis and helpfulness of further diagnostic test, although further rounds (10) led to a slight decline in performance.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Self-consistency showed improved performance when increasing reasoning paths from 5 to 10, but a small decline was observed when increasing reasoning paths to 20.

    “Self-consistency showed improved performance when increasing reasoning paths from 5 to 10, but a small decline was also observed when increasing reasoning paths to 20.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    For MAC, increasing conversation rounds to 13 and 25 raised output token counts but did not yield significant performance improvements.

    “For MAC, increasing the conversation rounds to 13 and 25 raised output token counts but did not yield significant performance improvements.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    GPT-4o-mini's performance is comparable to GPT-3.5-turbo but significantly lower than GPT-4.

    “The performance of GPT-4O-mini is comparable to that of GPT-3.5-turbo but significantly lower than that of GPT-4.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    All methods (CoT, Self-Consistency, Self-Refine, and MAC) showed improved accuracy for most likely diagnosis, possible diagnoses, and helpfulness of further diagnostic tests in both primary and follow-up consultations.

    “The results indicated that all methods showed improved accuracy for the most likely diagnosis, possible diagnoses, and helpfulness of further diagnostic tests in both the primary and follow-up consultations.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Self-Refine and Self-Consistency showed improvements but did not reach MAC's performance.

    “while Self-Refine and Self-Consistency showed improvements but did not reach MAC’s performance.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The number of doctor agents plays a crucial role in the framework's performance, with configurations of four agents consistently producing the best outcomes, closely followed by three agents.

    “The number of doctor agents plays a crucial role in the framework’s performance. Configurations with four agents consistently produced the best outcomes, closely followed by those with three agents.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    When using GPT-4 as the base model, the average cost per case was $0.12 USD for primary consultations and $0.17 USD for follow-up consultations.

    “when using GPT-4 as the base model, the average cost per case was $0.12 USD for primary consultations and $0.17 USD for follow-up consultations”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Findingnone given

    Single-agent models like GPT-3.5 and GPT-4 possess extensive knowledge but show limitations in applying this knowledge to real-world medical cases.

    “While single-agent models like GPT-3.5 and GPT-4 possess extensive knowledge, they show limitations in applying this knowledge to real-world medical cases.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    MAC may serve as a valuable tool that could help doctors analyze patient information and provide useful second opinions.

    “Therefore, MAC may serve as a valuable tool that could help doctors analyze patient information and provide useful second opinions.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    MAC may serve as a valuable second opinion tool when doctors face challenging cases.

    “It may serve as a valuable second opinion tool when doctors face challenging cases.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductown data

    The MAC framework showed minimal performance variation across three repeated testing rounds across six settings, with Fleiss' kappa moderate agreement (>0.4) in 23 of 28 evaluations and fair agreement (0.35-0.4) in the remaining five.

    “Results demonstrated minimal performance variation across testing rounds. Fleiss’ kappa analysis showed moderate agreement (>0.4) in 23 out of 28 evaluations, and fair agreement (0.35–0.4) in the remaining five.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Premisereasoning

    By facilitating an in-depth analysis that single-agent models may not achieve, multi-agent frameworks have the potential to improve the performance of LLMs in the diagnosis of rare diseases.

    “By facilitating an in-depth analysis that single-agent models may not achieve, they have the potential to improve the performance of LLMs in the diagnosis of rare diseases.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    One to nine kinds of rare disease were randomly selected for each disease category.

    “One to nine kinds of rare disease were randomly selected for each category.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

    Interpretationnone given

    The MAC framework holds the potential to be generalized to other medical challenges.

    “The framework also holds the potential to be generalized to other medical challenges, which warrants further exploration.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Premisenone given

    Even with technological advancements, LLMs still face significant challenges when diagnosing complex and rare diseases, and their effectiveness lacks validation in real clinical applications.

    “However, even with these technological advancements, LLMs still face significant challenges when diagnosing complex and rare diseases, and their effectiveness lacks validation in real clinical applications.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Interpretationnone given

    MAC provides valuable diagnosis suggestions and recommends further diagnostic tests during different stages of clinical consultation, and is applicable to all types of rare diseases.

    “MAC provides valuable diagnosis suggestions and recommend further diagnostic tests during different stages of clinical consultation, and is applicable to all types of rare diseases.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductown data

    The MAC framework's reliability was evaluated through three repeated testing rounds across six settings.

    “The MAC framework’s reliability was evaluated through three repeated testing rounds across six settings.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Premisereasoning

    The adoption of a multi-agent framework may facilitate dynamic and interactive diagnostic processes through multi-agent conversation, where agents discuss the same question and finally reach consensus on the output.

    “The adoption of multi-agent framework may facilitate dynamic and interactive diagnostic processes through multi-agent conversation (MAC), where agents would discuss the same question and finally reach consensus on the output.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    Normalized random sampling was employed to enhance the representativeness of the selected cases.

    “However, normalized random sampling was employed to enhance the representativeness of the selected cases.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    The increased output facilitated exploration of diverse reasoning paths and enabled reflection and revision of previous outputs, allowing more in-depth analysis that potentially uncovered underlying causes of diseases.

    “The increased output facilitated the exploration of diverse reasoning paths and enabled the reflection and revision of previous outputs. This allowed for more in-depth analysis, potentially uncovering underlying causes of diseases that may have been overlooked.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductreasoning

    The sample size in this study was relatively small, representing only a preliminary exploration.

    “Given the vast number of rare diseases, the sample size in this study was relatively small, representing only a preliminary exploration.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    The extraction of the primary consultation from patient information was subjective and may not always accurately reflect the actual consultation.

    “Although this was performed by medical professionals, the extraction was subjective and may not always accurately reflect the actual consultation.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    The multi-agent collaboration model will continue to offer significant advantages in solving complex medical tasks.

    “Therefore, even with the advent of more advanced models, it is reasonable to deduce that the multi-agent collaboration model will continue to offer significant advantages in solving complex medical tasks.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: LLMs are unlikely to surpass human capabilities in domains requiring extensive medical knowledge and sophisticated analytical skills in the short term.

  • Classified as probably not a substantial claim.

    Study design and conductcitation

    The most likely diagnosis and possible diagnoses were evaluated using a five-point scale described by Bond et al. 45.

    “For the rating scales, the most likely diagnosis and possible diagnoses were evaluated using a five-point scale described by Bond et al. 45, which is shown in Table 5.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [45] Amis, A. A. & Dawkins, G. P. Functional anatomy of the anterior cruciate ligament. Fibre bundle actions related to ligament replacements and injuries. J. Bone Jt. Surg. Br. Vol. 73, 260–267 (1991). source

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductnone given

    Human evaluation was conducted through panel discussions among three physicians who were blinded to the model conditions and reviewed the content in a randomized order.

    “Human evaluation was conducted through panel discussions among three physicians who were blinded to the model conditions and reviewed the content in a randomized order.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    Diagnostic performance was evaluated using accuracy and rating scales.

    “Diagnostic performance was evaluated using accuracy and rating scales.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    Two specialist doctors independently screened the search results using defined criteria.

    “Two specialist doctors independently screened the search results using defined criteria.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductnone given

    The physicians discussed each case and reached a consensus score.

    “The physicians discussed each case and reached a consensus score.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductnone given

    One investigator manually extracted data from each clinical case report, which was subsequently reviewed by a specialist doctor for extraction accuracy.

    “One investigator manually extracted data from each clinical case report, which was subsequently reviewed by a specialist doctor for extraction accuracy.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    The MAC framework substantially improved diagnostic outcomes, effectively bridging the gap between theoretical knowledge and practical clinical application.

    “The MAC framework substantially improved diagnostic outcomes, effectively bridging the gap between theoretical knowledge and practical clinical application.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning

    A normalized weighted random sampling method was used for selection to ensure a balanced representation.

    “Owing to the varied distribution of rare diseases among different types, a normalized weighted random sampling method was used for selection to ensure a balanced representation.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    Disease specific knowledge was evaluated using Likert Scale.

    “Disease specific knowledge was evaluated using Likert Scale.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    To quantify the inter-run agreement, Fleiss' kappa was employed.

    “To quantify the inter-run agreement, Fleiss’ kappa, a statistical measure for assessing reliability among multiple tests, was employed.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    More complex tasks, such as analyzing diagnostic dialogue, required human involvement due to the advanced medical reasoning needed.

    “However, more complex tasks, such as analyzing diagnostic dialogue, required human involvement due to the advanced medical reasoning needed.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    The reliability of the MAC framework was evaluated, which involved three repeated rounds of testing across six distinct settings.

    “The reliability of the MAC framework was evaluated, which involved three repeated rounds of testing across six distinct settings, assessing MAC’s performance consistency with different base models, with and without the supervisor agent, and with and without specialty assignments to doctor agents.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning

    For discontinuous data, chi-square test was used to test for differences among the models.

    “For these discontinuous data, chi-square test was used to test for differences among the models.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationown data

    These results underscore the value of multi-agent LLMs in healthcare.

    “These results underscore the value of multi-agent LLMs in healthcare and warrant further investigation into their implementation and refinement for clinical use.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    The search for clinical case reports was conducted by one investigator and reviewed by another investigator.

    “The search was conducted by one investigator and reviewed by another investigator.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The evaluation process relied on physicians to assess the reliability of LLM evaluations.

    “The evaluation process also relied on physicians to assess the reliability of LLM evaluations.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

    Interpretationreasoning

    The reasoning process can be considered an explanation, offering valuable insights into how the model arrives at its answers.

    “The reasoning process can thus be considered an explanation, offering valuable insights into how the model arrives at its answers.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationown data

    The dataset obtained in this study can serve as a benchmark for future research.

    “This dataset can also serve as a benchmark for future research.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    The further diagnostic test recommendations were evaluated using a Likert scale.

    “The further diagnostic test recommendations were evaluated using a Likert scale, which is shown in Table 6.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The Shapiro-Wilk test was used to check if the data followed a normal distribution.

    “The Shapiro–Wilk test was used to check if the data followed a normal distribution.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    Depending on the distribution, an ANOVA or Kruskal-Wallis test was applied to test for difference in performance among the models.

    “Depending on the distribution, an ANOVA, or Kruskal–Wallis test, was applied to test for difference in performance among the models.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    Fleiss-Kappa was used to assess the consistency of model outputs among multiple runs.

    “Flessi-Kappa was used to assess the consistency of model’ outputs among multiple runs.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    Any disagreements in screening were resolved through a group discussion.

    “Any disagreements were resolved through a group discussion.”
    Quote in the paperNot a substantial claim
64 of 64

  • The paper relies on this without support.

    The curated clinical cases simulate real-world clinical consultations and are representative of real clinical practice.

    “was tested on curated clinical cases simulating real-world clinical consultations (Fig. 3).”

  • The paper relies on this without support.

    The evaluation metrics (accuracy of most likely diagnosis, possible diagnoses, helpfulness of further diagnostic tests) validly measure diagnostic capability.

    “MAC significantly outperformed single-agent models GPT-3.5 and GPT-4 in terms of diagnostic accuracy and helpfulness of further recommended test in both primary and follow-up consultation.”

  • The paper relies on this without support.

    The rare disease cases sampled are representative of rare diseases generally.

    “This study included 302 kinds of rare disease from 33 different disease categories.”

  • The paper relies on this without support.

    The multi-agent conversation framework’s consensus output reflects a correct or clinically useful diagnostic process analogous to a real multi-disciplinary team discussion.

    “different agents can simulate the collaborative nature of a medical team, simulating a Multi-Disciplinary Team (MDT) discussion that is commonly adopted in clinical practice to solve complex medical tasks.”

  • The paper relies on this without support.

    GPT-3.5-turbo and GPT-4 are commonly tested for medical purposes.

    “GPT-3.5-turbo and GPT-4 are commonly tested for medical purposes and were selected as the base model for the Multi-Agent Conversation Framework (MAC).”

  • The paper relies on this without support.

    LLMs are unlikely to surpass human capabilities in domains requiring extensive medical knowledge and sophisticated analytical skills in the short term.

    “it is unlikely that LLMs will surpass human capabilities in domains requiring extensive medical knowledge and sophisticated analytical skills in the short term.”
6 of 6

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • MAC's primary most-likely diagnosis accuracy nearly doubled GPT-3.5 (16.23%, p < 0.001).

    “and GPT-3.5 (16.23%, p < 0.001)”

    From A Review of Multi-Agent AI Systems for Biological and Clinical Data Analysis (2026) · cites this paper as [35]

    Table 1 in the Results reports 16.23% most-likely-diagnosis accuracy for standalone GPT-3.5 and 34.11% for four-agent MAC using GPT-4 in primary consultations—approximately twice as high. The Results say MAC significantly outperformed standalone models, but our copy does not give p < 0.001 for this specific comparison. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • Helpfulness rate for recommended further diagnostic tests reached 78.26%, an improvement over the GPT-4 baseline of 58.17% (p < 0.001).

    “The helpfulness rate for recommended further diagnostic tests reached 78.26%, a marked improvement over the GPT-4 baseline of 58.17% (p < 0.001).”

    From A Review of Multi-Agent AI Systems for Biological and Clinical Data Analysis (2026) · cites this paper as [35]

    Table 1 in Results reports a 78.26% helpful rate for further diagnostic tests with the four-agent GPT-4 framework, versus 58.17% for standalone GPT-4, in primary consultations. The Results describe MAC’s improvement as significant, but our copy does not give p < 0.001 for this specific comparison; it also limits these figures to primary consultations. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

  • In Chen et al.'s system, four AI doctor agents, each with a specialized knowledge focus or reasoning style, discuss a patient case, while a fifth agent acts as a supervisor to coordinate the conversation.

    “four AI “doctor” agents, each with a specialized knowledge focus or reasoning style, discuss a patient case, while a fifth agent acts as a supervisor to coordinate the conversation”

    From A Review of Multi-Agent AI Systems for Biological and Clinical Data Analysis (2026) · cites this paper as [35]

    The Methods (Study design) and Results (Fig. 2 caption; pages unavailable) describe three doctor agents discussing a case with one coordinating supervisor, not four doctors and a fifth supervisor. The Methods (Subgroup analysis within MAC) describe assigning specialties as a tested variation; the Results say the standard configuration had no assigned specialties. The supervisor’s coordinating role is supported, but the claimed agent count and characterization of every doctor as specialized are not. (LLM, conf=0.90)

    Experimental — accuracy not yet measured

  • MAC follow-up most-likely diagnosis accuracy significantly exceeded GPT-3.5 (29.36%, p < 0.001).

    “and GPT-3.5 (29.36%, p < 0.001)”

    From A Review of Multi-Agent AI Systems for Biological and Clinical Data Analysis (2026) · cites this paper as [35]

    In Results, Table 2 reports 29.36% most-likely-diagnosis accuracy for standalone GPT-3.5 in follow-up consultations, below the MAC results. The Results text says MAC significantly outperformed single-agent GPT-3.5 in follow-up consultations, but the available passages do not give p < 0.001 for this comparison. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • MAC follow-up most-likely diagnosis accuracy significantly exceeded GPT-4 (37.86%, p = 0.010).

    “significantly exceeding GPT-4 (37.86%, p = 0.010)”

    From A Review of Multi-Agent AI Systems for Biological and Clinical Data Analysis (2026) · cites this paper as [35]

    In Results, Table 2 lists GPT-4’s follow-up most-likely diagnosis accuracy as 37.86% and higher accuracies for MAC. The Results text says MAC significantly outperformed GPT-4 in follow-up consultation, but the cited passages do not substantiate the specific p-value of 0.010. (LLM, conf=0.87)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Category limit cannot yield 302 diseases

    Minor

    Selecting at most nine diseases from each of 33 categories yields at most 297 diseases, not 302.

    Reported 302 · recomputed 297

  • GPT-3.5 test helpfulness differs from Table 1

    Minor

    The prose reports 77.37% for the four-agent GPT-3.5 configuration, whereas Table 1 reports 77.59%.

  • Two-agent GPT-4 accuracy differs from Table 1

    Minor

    The prose gives 31.31% most-likely-diagnosis accuracy for two GPT-4 doctor agents in primary consultations; Table 1 gives 31.13%.

  • Figure 7 caption rules out tested GPT-3.5 configurations

    Minor

    The caption says more than two GPT-3.5 doctor agents were infeasible, but Table 1 reports primary-consultation results for three, four, and five agents.

4 of 4

Cites

Flagged first, then by citations

Research works printed but not linked: 2

  1. Smit, A., Duckworth, P., Grinsztajn, N., Barrett, T. & Pretorius, A. Are we going MAD? Benchmarking Multi-Agent Debate between Language Models for Medical Q&A. In: Deep Generative Models for Health Workshop NeurIPS 2023) (NeurIPS, 2023).
  2. Pal, A., Umapathi, L. K. & Sankarasubbu, M. Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering. In: Conference on Health, Inference, and Learning) (PMLR, 2022).

Cited by

Sorted by citations

Related

Authors
Chen, X., Yi, H., You, M., Liu, W., Wang, L., Li, H., Zhang, X., Guo, Y., Fan, L., Chen, G., Lao, Q., Fu, W., Li, K., & Li, J.
Affiliations
  • Sichuan University (CN)
  • First Affiliated Hospital Zhejiang University (CN)
  • Beijing University of Posts and Telecommunications (CN)
Corresponding author
  • Fu, Weili
  • Li, Kang
  • Li, Jian
ORCID
Venue
npj Digital Medicine
Publisher
Springer Science and Business Media LLC
Published
2025-03-13
Volume
8
Issue
1
Pages
159-159
ISSN
2398-6352 (ISSN-L)
DOI
10.1038/s41746-025-01550-0
PMC
PMC11906805
OpenAlex
W4408411093
Cited by
117
References
40
Citations by year
2026: 84 · 2025: 32
FWCI
26.14
Type
journal article
Language
en
Open access
Licence
Locations
  • npj Digital Medicine (publishedVersion, cc-by-nc-nd)
  • PubMed (publishedVersion)
  • PubMed Central (submittedVersion, cc-by-nc-nd)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
  • Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
Keywords
multi-agent LLM systems, disease diagnosis, rare diseases, clinical reasoning, GPT-4, clinical decision support

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-07.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-07.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review