The DOI matches its Crossref record; 53 of the 53 references it cites were found in a registry or an index; 30 of the 35 sources its claims cite were checked for support.
Enhancing diagnostic capability with multi-agents conversational large language models
Open accessAbstract
Large Language Models (LLMs) show promise in healthcare tasks but face challenges in complex medical scenarios. We developed a Multi-Agent Conversation (MAC) framework for disease diagnosis, inspired by clinical Multi-Disciplinary Team discussions. Using 302 rare disease cases, we evaluated GPT-3.5, GPT-4, and MAC on medical knowledge and clinical reasoning. MAC outperformed single models in both primary and follow-up consultations, achieving higher accuracy in diagnoses and suggested tests. Optimal performance was achieved with four doctor agents and a supervisor agent, using GPT-4 as the base model. MAC demonstrated high consistency across repeated runs. Further comparative analysis showed MAC also outperformed other methods including Chain of Thoughts (CoT), Self-Refine, and Self-Consistency with higher performance and more output tokens. This framework significantly enhanced LLMs' diagnostic capabilities, effectively bridging theoretical knowledge and practical clinical application. Our findings highlight the potential of multi-agent LLMs in healthcare and suggest further research into their clinical implementation.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
The authors developed a Multi-Agent Conversation (MAC) framework, inspired by clinical multidisciplinary team discussions, to improve LLM diagnosis of rare diseases. Across 302 rare disease cases, MAC outperformed single-agent GPT-3.5 and GPT-4 in diagnostic accuracy and recommended-test helpfulness, with best results using GPT-4 and four doctor agents plus a supervisor.
Key findings
- In primary consultation, MAC with GPT-4 and four doctor agents achieved 34.11% most-likely-diagnosis accuracy, 48.12% possible-diagnosis accuracy, and 78.26% helpful further-diagnostic-test rate, compared with GPT-4 alone at 19.65%, 34.55%, and 58.17%, respectively.
- In follow-up consultation, MAC with GPT-4 and four doctor agents achieved 53.86% most-likely-diagnosis accuracy and 67.88% possible-diagnosis accuracy, compared with GPT-4 alone at 37.86% and 59.71%.
- Optimal MAC performance was observed with four doctor agents and a supervisor agent using GPT-4 as the base model; excluding the supervisor reduced GPT-4 primary-consultation most-likely-diagnosis accuracy from 34.11% to 32.67% and possible-diagnosis accuracy from 48.12% to 45.47%.
- MAC also outperformed Chain of Thought, Self-Refine, and Self-Consistency across evaluation metrics, and reliability analysis showed moderate agreement (>0.4) in 23 of 28 evaluations and fair agreement (0.35–0.4) in the remaining five. Average cost per case with GPT-4 was $0.12 for primary consultations and $0.17 for follow-up consultations.
Methods
- The MAC framework was built using AutoGen, with an admin agent presenting patient information, a supervisor agent guiding consensus, and doctor agents discussing diagnoses and tests; conversations ended at consensus or after a maximum of thirteen rounds.
- The study included 302 rare diseases from 33 categories in Orphanet, selected by normalized weighted random sampling; clinical case reports published after January 2022 were identified from Medline and curated into primary and follow-up consultation scenarios.
- GPT-3.5, GPT-4, and MAC were evaluated on disease-specific knowledge and diagnostic tasks; subgroup analyses varied base model, number of doctor agents (2–5), supervisor inclusion, and specialty assignment. Outputs were evaluated using GPT-4o with human validation on 302 cases, and analyses included ANOVA/Kruskal–Wallis, chi-square, and Fleiss’ kappa.
Limitations
- The paper states the sample size was relatively small given the vast number of rare diseases and represents only a preliminary exploration; primary consultation data were manually extracted and may not always reflect actual consultation.
- The paper notes that model output explanations do not fully capture inherent interpretability, and MAC should be seen as a supportive tool for clinicians rather than a replacement.
- Evident from the text: the study used published case reports rather than actual human participants, and GPT-3.5 MAC configurations with more than two doctor agents in follow-up consultations were not feasible due to token output limitations.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
26 Own result · 24 Own method · 13 Own interpretation · 12 Supported · 11 No basis given · 9 Partly supported · 2 No full text of the sources · accuracy not yet measured
Every source the paper cites for this claim was checked and supports it.
In the paper
“Chain-of-thought (CoT) prompting has been used to decompose complex diagnostic tasks into linear steps, which aids in integrating context and delivering comprehensive diagnostic reasoning33.”
§ Enhancing diagnostic capability with multi-agents conversational large language models > Discussion
What the sources say
- Supportedcites [33] Wada, A. et al. Optimizing GPT-4 Turbo Diagnostic Accuracy in Neuroradiology through Prompt Engineering and Confidence Thresholds. Diagnostics 14, 1541 (2024). source
Section 2.3.2 describes a sequential diagnostic prompt that starts with clinical information, incorporates imaging findings, and refines the diagnosis. It says this structure was designed to integrate data gradually and improve diagnostic reasoning, supporting the citing sentence’s paraphrase. (LLM, conf=0.90)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“One study examined GPT-4’s performance in diagnosing complex cases using published case records, finding that the model’s top diagnosis matched the final diagnosis in 39% of cases2.”
§ Enhancing diagnostic capability with multi-agents conversational large language models > Discussion
What the sources say
- Supportedcites [2] Kanjee, Z., Crowe, B. & Rodman, A. Accuracy of a Generative Artificial Intelligence Model in a Complex Diagnostic Challenge. JAMA 330, 78–80 (2023). source
The Results report that GPT-4’s top diagnosis agreed with the final diagnosis in 39% (27/70) of cases. The study describes these as diagnostically difficult cases, matching the citing sentence’s description of complex cases. (LLM, conf=0.95)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“However, it should be noted that mere output explanations do not fully capture the model’s inherent interpretability.”
§ Enhancing diagnostic capability with multi-agents conversational large language models > Discussion
What the sources say
- Supportedcites [38] Agarwal, C., Tanneru, S. H. & Lakkaraju, H. Faithfulness vs. plausibility: On the (un) reliability of explanations from large language models. Preprint arXiv:2402.04614 (2024). source
The Introduction says that LLM-generated explanations can seem logical and coherent without accurately reflecting the model’s actual reasoning process. This supports the claim that output explanations alone do not fully reveal the model’s internal decision-making. (LLM, conf=0.91)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“This technique significantly enhances the capabilities of LLMs for managing complex tasks, including solving mathematical problems and performing retrieval-augmented code generation11–14.”
§ Enhancing diagnostic capability with multi-agents conversational large language models > Introduction
What the sources say
- Supportedcites [11] Wu, Q. et al. Autogen: Enabling next-gen llm applications via multi-agent conversation. Preprint arXiv:2308.08155 (2023). source
The Introduction says AutoGen’s multi-agent applications support tasks of varying complexity and achieve strong performance. Section A1 reports better performance on math problems than the alternatives tested, while Section A2 describes using AutoGen for retrieval-augmented code generation. The copy does not provide page numbers for these passages. (LLM, conf=0.88)
- Not matchedcites [12] Smit, A., Duckworth, P., Grinsztajn, N., Barrett, T. & Pretorius, A. Are we going MAD? Benchmarking Multi-Agent Debate between Language Models for Medical Q&A. In: Deep Generative Models for Health Workshop NeurIPS 2023) (NeurIPS, 2023).
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Partly supportedcites [13] Chan, C.-M. et al. Chateval: Towards better llm-based evaluators through multi-agent debate. Preprint arXiv:2308.07201 (2023). sourceExperimental — accuracy not yet measured
The Introduction says multi-agent collaboration can improve completion of arduous tasks, and the Automatic NLG evaluation section reports better results on arithmetic problems. Our copy does not support the retrieval-augmented code-generation example; ChatEval’s own experiments concern evaluation of generated responses, not that task. (LLM, conf=0.87)
- Partly supportedcites [14] Hong, S. et al. Metagpt: Meta programming for a multi-agent collaborative framework. URL https://arxiv.org/abs/2308.00352 (2023). sourceExperimental — accuracy not yet measured
The Conclusion (p. 9) says MetaGPT enhances the problem-solving capabilities of LLM-based multi-agent systems and improves code-generation quality. The paper’s reported tasks are software engineering and programming; the searched passages do not support the more specific examples of solving mathematical problems or performing retrieval-augmented code generation. (LLM, conf=0.86)
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“A previous study has demonstrated the limitations of LLMs in the diagnosis of complex cases2.”
§ Enhancing diagnostic capability with multi-agents conversational large language models > Introduction
What the sources say
- Supportedcites [2] Kanjee, Z., Crowe, B. & Rodman, A. Accuracy of a Generative Artificial Intelligence Model in a Complex Diagnostic Challenge. JAMA 330, 78–80 (2023). source
The Results report that the model’s top diagnosis matched the final diagnosis in only 39% of challenging cases, while its differential included the final diagnosis in 64%. The Discussion reiterates these findings, supporting the claim that the study demonstrated limitations in diagnosing complex cases. (LLM, conf=0.90)
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Another study demonstrated that while LLMs show potential for clinical decision-making, they currently underperform compared to human clinicians in diagnosing common conditions such as appendicitis, cholecystitis, diverticulitis, and pancreatitis.”
§ Enhancing diagnostic capability with multi-agents conversational large language models > Discussion
What the sources say
- Supportedcites [30] Hager, P. et al. Evaluation and mitigation of the limitations of large language models in clinical decision-making. Nat. Med. 30, 2613–2622 (2024). source
The Results section reports that current LLMs performed significantly worse than clinicians on aggregate across the studied diseases, including appendicitis, cholecystitis, diverticulitis, and pancreatitis. It also notes an important exception: most models matched clinician performance on appendicitis alone. (LLM, conf=0.91)
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Additionally, one study showed that RLHF can improve model responses to be more accurate and aligned with human-like diagnostic reasoning36.”
§ Enhancing diagnostic capability with multi-agents conversational large language models > Discussion
What the sources say
- Partly supportedcites [36] Dou, C. et al. Integrating physician diagnostic logic into large language models: Preference learning from process feedback. Preprint arXiv:2401.05695 (2024). sourceExperimental — accuracy not yet measured
The Abstract and Conclusion report that PLPF integrates physicians’ diagnostic logic and improves diagnostic accuracy by 17.6% in standardized patient testing. But §2.4 says the study trains on preference data using direct preference optimization (DPO), not RLHF; the Introduction distinguishes PLPF from traditional approaches. The result supports the claimed improvement, but not its attribution to RLHF. (LLM, conf=0.88)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“This observation is consistent with prior research indicating that although agents can assume specific roles, their fundamental abilities may be insufficient in the domain-specific knowledge and skills require to function optimally as clinical specialists in complex clinical task26,27.”
§ Enhancing diagnostic capability with multi-agents conversational large language models > Discussion
What the sources say
- Supportedcites [26] Liu, L. et al. A survey on medical large language models: Technology, application, trustworthiness, and future directions. Preprint arXiv:2406.03712 (2024). source
Split into 2 atoms — 2 confirmed, 0 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.
- Supportedcites [27] Chen, X. et al. Evaluating and Enhancing Large Language Models’ Performance in Domain-Specific Medicine: Development and Usability Study With DocOA. J. Med. Internet Res. 26, e58158 (2024). source
Split into 2 atoms — 2 confirmed, 0 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“This study involved 302 rare diseases selected from a pool of over 7000 across 33 types in the Orphanet Database, a comprehensive rare disease database co-funded by the European Commission42.”
§ Enhancing diagnostic capability with multi-agents conversational large language models > Methods > Data acquirement
What the sources say
- Supportedcites [42] Nguengang Wakap, S. et al. Estimating cumulative point prevalence of rare diseases: analysis of the Orphanet database. Eur. J. Hum. Genet. EJHG 28, 165–173 (2020). source
The citation follows the description of Orphanet, rather than the citing authors’ account of their own study sample. The Introduction (page unavailable) describes Orphanet as a comprehensive rare-disease knowledge base and says its network is cofunded by the European Commission. (LLM, conf=0.92)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“MAC significantly outperformed single-agent models GPT-3.5 and GPT-4 in terms of diagnostic accuracy and helpfulness of further recommended test in both primary and follow-up consultation.”
§ Enhancing diagnostic capability with multi-agents conversational large language models > Results > Performance of MAC on diagnostic ability
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Probably not a substantial claim. 49
- It may state two separate claims as one. 29
- The quote only partly states this claim. 1
Every source the paper cites for this claim was checked and supports it.
In the paper
“For example, MEDAGENTS is an innovative multi-disciplinary collaboration framework that enhances large language models’ performance in zero-shot medical question answering25.”
§ Enhancing diagnostic capability with multi-agents conversational large language models > Discussion
What the sources say
- Supportedcites [25] Tang, X. et al. Medagents: Large language models as collaborators for zero-shot medical reasoning. Preprint arXiv:2311.10537 (2023). source
The Method section describes MEDAGENTS as a proposed multi-disciplinary collaboration framework. The Main Results section reports that it outperforms zero-shot baselines on nine medical question-answering datasets, supporting the claimed performance enhancement. (LLM, conf=0.93)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Previous study has indicated that models may sometimes provide incorrect explanations alongside correct answers, or vice versa38.”
§ Enhancing diagnostic capability with multi-agents conversational large language models > Discussion
What the sources say
- Partly supportedcites [38] Agarwal, C., Tanneru, S. H. & Lakkaraju, H. Faithfulness vs. plausibility: On the (un) reliability of explanations from large language models. Preprint arXiv:2402.04614 (2024). sourceExperimental — accuracy not yet measured
The Introduction (page unavailable) says that plausible explanations may not accurately reflect a model’s reasoning, and the Security Concerns section gives an example of a refusal accompanied by a misleading explanation. Our copy does not establish the converse—an incorrect answer accompanied by a correct explanation. (LLM, conf=0.81)
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“The results indicated that GPT-4 demonstrated superior diagnostic accuracy compared to GPT-3.5, with its differential diagnosis lists more closely aligning with those generated by human experts31.”
§ Enhancing diagnostic capability with multi-agents conversational large language models > Discussion
What the sources say
- Supportedcites [31] Ríos-Hoyo, A. et al. Evaluation of large language models as a diagnostic aid for complex medical cases. Front. Med. 11, 1380148 (2024). source
The Methods describe 75 sequential Massachusetts General Hospital case records. The Results report that GPT-4 included the correct diagnosis in its differential more often than GPT-3.5 (68% versus 48%) and that its lists overlapped more with the human case discussants’ lists, although that overlap was fair at best. (LLM, conf=0.93)
hedgedcorrelationaln = 75- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Timely diagnosis of rare diseases has always been challenged by a lack of high-quality medical resources in many regions of the world”
§ Enhancing diagnostic capability with multi-agents conversational large language models > Discussion
What the sources say
- Supportedcites [7] Adachi, T. et al. Enhancing Equitable Access to Rare Disease Diagnosis and Treatment around the World: A Review of Evidence, Policies, and Challenges. Int. J. Environ. Res. Public Health 20, 4732 (2023). source
Section 3.4, Disease Management (page not provided), identifies limited diagnostic services, laboratories, specialists, and professional training as challenges to rare-disease diagnosis. Section 3.2, The Patient Journey (page not provided), reports delayed diagnosis across multiple regions. Together these support the claim that limited medical resources hinder timely diagnosis in many parts of the world. (LLM, conf=0.88)
- Partly supportedcites [8] Serrano, J. G. et al. Advancing Understanding of Inequities in Rare Disease Genomics. Clin. Therapeutics 45, 745–753 (2023). sourceExperimental — accuracy not yet measured
The Introduction (page unavailable) says limited access to health care can prevent genetic evaluation and may lengthen time to diagnosis. The Abstract (page unavailable) also identifies proximity to a major medical center as an access barrier. The paper does not establish that a lack of high-quality medical resources has always impeded timely rare-disease diagnosis across many regions of the world. (LLM, conf=0.84)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Conversely, techniques such as Chain-of-Thought (CoT) enable models to generate more extensive outputs detailing their reasoning processes.”
§ Enhancing diagnostic capability with multi-agents conversational large language models > Discussion
What the sources say
- Supportedcites [39] Wei, J. et al. Chain-of-thought prompting elicits reasoning in large language models. 35, 24824–24837 (2022). source
Section 2 (pp. 2–3) says chain-of-thought prompting elicits a coherent series of intermediate reasoning steps before the final answer, providing an interpretable account of how a model might reach it. This supports the claim that CoT produces more extensive outputs describing the model’s reasoning, without establishing that those steps reveal its internal computations. (LLM, conf=0.90)
- Supportedcites [40] Xu, J. et al. Faithful logical reasoning via symbolic chain-of-thought. Preprint arXiv:2405.18357 (2024). source
The “Chain-of-Thought (CoT)” section says CoT encourages models to articulate step-by-step reasoning leading to an answer. The Related work section also describes CoT as including intermediate steps and their rationales, supporting the paraphrase that its outputs detail the reasoning process. (LLM, conf=0.91)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Recent advancements in multi-agent systems have shown promising results in medical decision-making and diagnosis.”
§ Enhancing diagnostic capability with multi-agents conversational large language models > Discussion
What the sources say
- Partly supportedcites [25] Tang, X. et al. Medagents: Large language models as collaborators for zero-shot medical reasoning. Preprint arXiv:2311.10537 (2023). sourceExperimental — accuracy not yet measured
The Conclusion reports that the multi-agent MedAgents framework surpassed zero-shot baselines on nine medical question-answering datasets, supporting promising results in medical reasoning. The Experiments section describes benchmark questions, not an evaluation of diagnostic performance or real-world clinical decision-making, so those parts of the claim go beyond what the paper establishes. (LLM, conf=0.83)
- Supportedcites [28] Wang, H. et al. Beyond Direct Diagnosis: LLM-based Multi-Specialist Agent Consultation for Automatic Diagnosis. (2024). source
The Conclusion reports that the paper’s multi-specialist agent approach surpassed existing benchmarks in diagnostic accuracy. The framework description explains that it combines specialist agents’ predictions to make diagnostic decisions, supporting the broad claim of promising results in medical decision-making and diagnosis. (LLM, conf=0.91)
- Supportedcites [29] Ke, Y. H. et al. Enhancing diagnostic accuracy through multi-agent conversations: Using large language models to mitigate cognitive bias. (2024). source
The Principal Results report improved diagnostic accuracy with the multi-agent framework, including a 71.3% correctness rate for the top differential diagnosis. The Conclusions describe its potential to enhance clinical decision-making, supporting the claim's broad description of promising results. (LLM, conf=0.92)
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“delays in diagnosis lead to delays in treatment and poor prognosis7,8.”
§ Enhancing diagnostic capability with multi-agents conversational large language models > Discussion
What the sources say
- Partly supportedcites [7] Adachi, T. et al. Enhancing Equitable Access to Rare Disease Diagnosis and Treatment around the World: A Review of Evidence, Policies, and Challenges. Int. J. Environ. Res. Public Health 20, 4732 (2023). sourceExperimental — accuracy not yet measured
Section 3.2, “The Patient Journey” (page unavailable), says that delayed diagnosis leads to disease progression and worse health outcomes, supporting the poor-prognosis part of the claim. It discusses delays and barriers to treatment, but does not establish that diagnostic delays themselves cause treatment delays. (LLM, conf=0.84)
- Partly supportedcites [8] Serrano, J. G. et al. Advancing Understanding of Inequities in Rare Disease Genomics. Clin. Therapeutics 45, 745–753 (2023). sourceExperimental — accuracy not yet measured
The Introduction (page not available in our copy) says barriers to genetic evaluation may result in a longer time to diagnosis. The paper does not say that diagnostic delays cause treatment delays or poor prognosis; its survey lists “better treatment” as a potential benefit of diagnosis, not an observed consequence of delay. (LLM, conf=0.81)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“In most cases, this enhancement can improve model performance39,40.”
§ Enhancing diagnostic capability with multi-agents conversational large language models > Discussion
What the sources say
- Partly supportedcites [39] Wei, J. et al. Chain-of-thought prompting elicits reasoning in large language models. 35, 24824–24837 (2022). sourceExperimental — accuracy not yet measured
The Results (p. 4) report performance gains from chain-of-thought prompting for models of approximately 100B parameters, especially on more complicated problems. They also say it does not positively affect small models, while Appendix A.1 (p. 16) says it hurts performance for most models below 10B parameters. The paper therefore supports improvement under particular conditions, not the unqualified claim that it improves performance in most cases. (LLM, conf=0.88)
- Partly supportedcites [40] Xu, J. et al. Faithful logical reasoning via symbolic chain-of-thought. Preprint arXiv:2405.18357 (2024). sourceExperimental — accuracy not yet measured
Section 4.2 (Main Result; page unavailable) reports that SymbCoT outperformed vanilla CoT on the paper’s logical-reasoning tests. This supports performance gains from the specific enhancement studied, but not the broader claim that CoT enhancements improve model performance in most cases generally. (LLM, conf=0.82)
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“For self-consistency, following established literature, we implemented a framework with 10 distinct reasoning paths and set the temperature parameter to 0.723.”
§ Enhancing diagnostic capability with multi-agents conversational large language models > Methods > Comparison with other methods to improve diagnostic performance
What the sources say
- Supportedcites [23] Wang, X. et al. Self-consistency improves chain of thought reasoning in language models. Preprint arXiv:2203.11171 (2022). source
The Experiment Setup (p. 4) specifies temperature 0.7 for PaLM-540B and GPT-3, and the Main Results (p. 5) include 10 among the tested numbers of sampled reasoning paths. The paper supports these as self-consistency settings, though it does not prescribe 10 paths or temperature 0.7 for every model. (LLM, conf=0.87)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one. The quote only partly states this claim.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“The sampling weights were adjusted based on the disease count in each type and moderated by natural logarithm transformation43,44.”
§ Enhancing diagnostic capability with multi-agents conversational large language models > Methods > Data acquirement
What the sources say
- Not checked — no full text of the sourcecites [43] Efraimidis, P. & Spirakis, P. Weighted Random Sampling. In: Encyclopedia of Algorithms (ed Kao, M.-Y.) (Springer US, 2008). source
- Partly supportedcites [44] West, R. M. Best practice in statistics: The use of log transformation. Ann. Clin. Biochem. 59, 162–165 (2022). sourceExperimental — accuracy not yet measured
The Introduction’s ‘Natural logarithm or common logarithm’ section says natural logarithms can be used for log transformation. The paper illustrates transformation with urine albumin measurements; it does not describe adjusting sampling weights by disease counts or applying a logarithm to those weights. (LLM, conf=0.85)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“The self-refine approach was configured with four iterative refinement rounds at a temperature of 0.7, as adopted in prior research22.”
§ Enhancing diagnostic capability with multi-agents conversational large language models > Methods > Comparison with other methods to improve diagnostic performance
What the sources say
- Partly supportedcites [22] Madaan, A. et al. Self-refine: Iterative refinement with self-feedback. Adv. Neural Inf. Process. Syst. 36, 46534–46594 (2023). sourceExperimental — accuracy not yet measured
Section 3.1 says SELF-REFINE continues for up to four feedback–refinement iterations, not necessarily four rounds in every case. Appendix L.2 uses a temperature of 0.7 for critique generation in a code-readability experiment, but that experiment runs five iterations and uses temperature 0.0 for the code editor; it does not establish 0.7 as the setting for the four-iteration configuration. (LLM, conf=0.88)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“In general, the limitations of explanations provided by large language models (LLMs) in their outputs should be noted, as the issue of explainability remains a persistent challenge that necessitates ongoing research41.”
§ Enhancing diagnostic capability with multi-agents conversational large language models > Discussion
What the sources say
- Supportedcites [41] Luo, H. & Specia, L. From understanding to utilization: A survey on explainability for large language models. Preprint arXiv:2401.12874 (2024). source
Section 5 (Evaluation) cautions that it is unclear whether LLM-generated explanations help people understand the model’s reasoning and calls for better evaluation methods. Sections 1 (Introduction) and 6 (Conclusion) describe explainability as a substantial challenge and identify directions for further research. (LLM, conf=0.91)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In primary consultations with four doctor agents, GPT-4 achieved higher accuracy for the most likely diagnosis (34.11% vs 24.28%), possible diagnoses (48.12% vs 36.64%), and helpfulness of further diagnostic tests (78.26% vs 77.37%).”
§ Enhancing diagnostic capability with multi-agents conversational large language models > Results > Performance of MAC on diagnostic ability
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The optimal number of doctor agents was found to be four, with three agents producing comparable results.”
§ Enhancing diagnostic capability with multi-agents conversational large language models > Discussion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In this study, GPT-3.5 and GPT-4 are capable of diagnosing diseases based on apparent symptoms, such as identifying pericarditis and epilepsy through clinical presentation. However, they lack in-depth exploration of the underlying causes of these conditions.”
§ Enhancing diagnostic capability with multi-agents conversational large language models > Discussion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“MAC provided insights into how the models performed medical reasoning and handled different opinions during multi-agent conversations, which may serve as interpretation on how LLMs reasoned through the task.”
§ Enhancing diagnostic capability with multi-agents conversational large language models > Discussion
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“For MAC using GPT-3.5 as the base model, settings with more than 2 doctor agents were not feasible due to token output limitation.”
§ Enhancing diagnostic capability with multi-agents conversational large language models > Results > Performance of MAC on diagnostic ability
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For self-refine, increasing the number of refinement rounds from 2 to 4 enhanced performance, particularly in most likely diagnosis and helpfulness of further diagnostic test, although further rounds (10) led to a slight decline in performance.”
§ Enhancing diagnostic capability with multi-agents conversational large language models > Results > Influence of output token on model performance
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Self-consistency showed improved performance when increasing reasoning paths from 5 to 10, but a small decline was also observed when increasing reasoning paths to 20.”
§ Enhancing diagnostic capability with multi-agents conversational large language models > Results > Influence of output token on model performance
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For MAC, increasing the conversation rounds to 13 and 25 raised output token counts but did not yield significant performance improvements.”
§ Enhancing diagnostic capability with multi-agents conversational large language models > Results > Influence of output token on model performance
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The performance of GPT-4O-mini is comparable to that of GPT-3.5-turbo but significantly lower than that of GPT-4.”
§ Enhancing diagnostic capability with multi-agents conversational large language models > Results > Generalizability of other LLM
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The results indicated that all methods showed improved accuracy for the most likely diagnosis, possible diagnoses, and helpfulness of further diagnostic tests in both the primary and follow-up consultations.”
§ Enhancing diagnostic capability with multi-agents conversational large language models > Results > Comparison with other methods to improve diagnostic performance
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“while Self-Refine and Self-Consistency showed improvements but did not reach MAC’s performance.”
§ Enhancing diagnostic capability with multi-agents conversational large language models > Results > Comparison with other methods to improve diagnostic performance
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The number of doctor agents plays a crucial role in the framework’s performance. Configurations with four agents consistently produced the best outcomes, closely followed by those with three agents.”
§ Enhancing diagnostic capability with multi-agents conversational large language models > Discussion
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“when using GPT-4 as the base model, the average cost per case was $0.12 USD for primary consultations and $0.17 USD for follow-up consultations”
§ Enhancing diagnostic capability with multi-agents conversational large language models > Results > Cost analysis
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“While single-agent models like GPT-3.5 and GPT-4 possess extensive knowledge, they show limitations in applying this knowledge to real-world medical cases.”
§ Enhancing diagnostic capability with multi-agents conversational large language models > Discussion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Therefore, MAC may serve as a valuable tool that could help doctors analyze patient information and provide useful second opinions.”
§ Enhancing diagnostic capability with multi-agents conversational large language models > Introduction
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“It may serve as a valuable second opinion tool when doctors face challenging cases.”
§ Enhancing diagnostic capability with multi-agents conversational large language models > Discussion
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Results demonstrated minimal performance variation across testing rounds. Fleiss’ kappa analysis showed moderate agreement (>0.4) in 23 out of 28 evaluations, and fair agreement (0.35–0.4) in the remaining five.”
§ Enhancing diagnostic capability with multi-agents conversational large language models > Results > Reliability analysis
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“By facilitating an in-depth analysis that single-agent models may not achieve, they have the potential to improve the performance of LLMs in the diagnosis of rare diseases.”
§ Enhancing diagnostic capability with multi-agents conversational large language models > Introduction
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“One to nine kinds of rare disease were randomly selected for each category.”
§ Enhancing diagnostic capability with multi-agents conversational large language models > Results > Study sample
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The framework also holds the potential to be generalized to other medical challenges, which warrants further exploration.”
§ Enhancing diagnostic capability with multi-agents conversational large language models > Discussion
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“However, even with these technological advancements, LLMs still face significant challenges when diagnosing complex and rare diseases, and their effectiveness lacks validation in real clinical applications.”
§ Enhancing diagnostic capability with multi-agents conversational large language models > Discussion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“MAC provides valuable diagnosis suggestions and recommend further diagnostic tests during different stages of clinical consultation, and is applicable to all types of rare diseases.”
§ Enhancing diagnostic capability with multi-agents conversational large language models > Discussion
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“The MAC framework’s reliability was evaluated through three repeated testing rounds across six settings.”
§ Enhancing diagnostic capability with multi-agents conversational large language models > Results > Reliability analysis
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The adoption of multi-agent framework may facilitate dynamic and interactive diagnostic processes through multi-agent conversation (MAC), where agents would discuss the same question and finally reach consensus on the output.”
§ Enhancing diagnostic capability with multi-agents conversational large language models > Introduction
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“However, normalized random sampling was employed to enhance the representativeness of the selected cases.”
§ Enhancing diagnostic capability with multi-agents conversational large language models > Discussion
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The increased output facilitated the exploration of diverse reasoning paths and enabled the reflection and revision of previous outputs. This allowed for more in-depth analysis, potentially uncovering underlying causes of diseases that may have been overlooked.”
§ Enhancing diagnostic capability with multi-agents conversational large language models > Discussion
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Given the vast number of rare diseases, the sample size in this study was relatively small, representing only a preliminary exploration.”
§ Enhancing diagnostic capability with multi-agents conversational large language models > Discussion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Although this was performed by medical professionals, the extraction was subjective and may not always accurately reflect the actual consultation.”
§ Enhancing diagnostic capability with multi-agents conversational large language models > Discussion
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Therefore, even with the advent of more advanced models, it is reasonable to deduce that the multi-agent collaboration model will continue to offer significant advantages in solving complex medical tasks.”
§ Enhancing diagnostic capability with multi-agents conversational large language models > Discussion
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
- Rests on an assumption: LLMs are unlikely to surpass human capabilities in domains requiring extensive medical knowledge and sophisticated analytical skills in the short term.
In the paper
“For the rating scales, the most likely diagnosis and possible diagnoses were evaluated using a five-point scale described by Bond et al. 45, which is shown in Table 5.”
§ Enhancing diagnostic capability with multi-agents conversational large language models > Methods > Performance evaluation
What the sources say
- Not checked — no full text of the sourcecites [45] Amis, A. A. & Dawkins, G. P. Functional anatomy of the anterior cruciate ligament. Fibre bundle actions related to ligament replacements and injuries. J. Bone Jt. Surg. Br. Vol. 73, 260–267 (1991). source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Human evaluation was conducted through panel discussions among three physicians who were blinded to the model conditions and reviewed the content in a randomized order.”
§ Enhancing diagnostic capability with multi-agents conversational large language models > Methods > Performance evaluation
plainn = 302- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Diagnostic performance was evaluated using accuracy and rating scales.”
§ Enhancing diagnostic capability with multi-agents conversational large language models > Methods > Performance evaluation
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Two specialist doctors independently screened the search results using defined criteria.”
§ Enhancing diagnostic capability with multi-agents conversational large language models > Methods > Data acquirement
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“The physicians discussed each case and reached a consensus score.”
§ Enhancing diagnostic capability with multi-agents conversational large language models > Methods > Performance evaluation
plainn = 302- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“One investigator manually extracted data from each clinical case report, which was subsequently reviewed by a specialist doctor for extraction accuracy.”
§ Enhancing diagnostic capability with multi-agents conversational large language models > Methods > Data extraction
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The MAC framework substantially improved diagnostic outcomes, effectively bridging the gap between theoretical knowledge and practical clinical application.”
§ Enhancing diagnostic capability with multi-agents conversational large language models > Discussion
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Owing to the varied distribution of rare diseases among different types, a normalized weighted random sampling method was used for selection to ensure a balanced representation.”
§ Enhancing diagnostic capability with multi-agents conversational large language models > Methods > Data acquirement
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Disease specific knowledge was evaluated using Likert Scale.”
§ Enhancing diagnostic capability with multi-agents conversational large language models > Methods > Performance evaluation
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“To quantify the inter-run agreement, Fleiss’ kappa, a statistical measure for assessing reliability among multiple tests, was employed.”
§ Enhancing diagnostic capability with multi-agents conversational large language models > Methods > Reliability analysis
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, more complex tasks, such as analyzing diagnostic dialogue, required human involvement due to the advanced medical reasoning needed.”
§ Enhancing diagnostic capability with multi-agents conversational large language models > Discussion
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“The reliability of the MAC framework was evaluated, which involved three repeated rounds of testing across six distinct settings, assessing MAC’s performance consistency with different base models, with and without the supervisor agent, and with and without specialty assignments to doctor agents.”
§ Enhancing diagnostic capability with multi-agents conversational large language models > Methods > Reliability analysis
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“For these discontinuous data, chi-square test was used to test for differences among the models.”
§ Enhancing diagnostic capability with multi-agents conversational large language models > Methods > Statistical analysis
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“These results underscore the value of multi-agent LLMs in healthcare and warrant further investigation into their implementation and refinement for clinical use.”
§ Enhancing diagnostic capability with multi-agents conversational large language models > Discussion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“The search was conducted by one investigator and reviewed by another investigator.”
§ Enhancing diagnostic capability with multi-agents conversational large language models > Methods > Data acquirement
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“The evaluation process also relied on physicians to assess the reliability of LLM evaluations.”
§ Enhancing diagnostic capability with multi-agents conversational large language models > Discussion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The reasoning process can thus be considered an explanation, offering valuable insights into how the model arrives at its answers.”
§ Enhancing diagnostic capability with multi-agents conversational large language models > Discussion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This dataset can also serve as a benchmark for future research.”
§ Enhancing diagnostic capability with multi-agents conversational large language models > Discussion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“The further diagnostic test recommendations were evaluated using a Likert scale, which is shown in Table 6.”
§ Enhancing diagnostic capability with multi-agents conversational large language models > Methods > Performance evaluation
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“The Shapiro–Wilk test was used to check if the data followed a normal distribution.”
§ Enhancing diagnostic capability with multi-agents conversational large language models > Methods > Statistical analysis
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Depending on the distribution, an ANOVA, or Kruskal–Wallis test, was applied to test for difference in performance among the models.”
§ Enhancing diagnostic capability with multi-agents conversational large language models > Methods > Statistical analysis
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Flessi-Kappa was used to assess the consistency of model’ outputs among multiple runs.”
§ Enhancing diagnostic capability with multi-agents conversational large language models > Methods > Statistical analysis
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Any disagreements were resolved through a group discussion.”
§ Enhancing diagnostic capability with multi-agents conversational large language models > Methods > Data acquirement
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper relies on this without support.
The curated clinical cases simulate real-world clinical consultations and are representative of real clinical practice.
“was tested on curated clinical cases simulating real-world clinical consultations (Fig. 3).”
The paper relies on this without support.
The evaluation metrics (accuracy of most likely diagnosis, possible diagnoses, helpfulness of further diagnostic tests) validly measure diagnostic capability.
“MAC significantly outperformed single-agent models GPT-3.5 and GPT-4 in terms of diagnostic accuracy and helpfulness of further recommended test in both primary and follow-up consultation.”
The paper relies on this without support.
The rare disease cases sampled are representative of rare diseases generally.
“This study included 302 kinds of rare disease from 33 different disease categories.”
The paper relies on this without support.
The multi-agent conversation framework’s consensus output reflects a correct or clinically useful diagnostic process analogous to a real multi-disciplinary team discussion.
“different agents can simulate the collaborative nature of a medical team, simulating a Multi-Disciplinary Team (MDT) discussion that is commonly adopted in clinical practice to solve complex medical tasks.”
The paper relies on this without support.
GPT-3.5-turbo and GPT-4 are commonly tested for medical purposes.
“GPT-3.5-turbo and GPT-4 are commonly tested for medical purposes and were selected as the base model for the Multi-Agent Conversation Framework (MAC).”
The paper relies on this without support.
LLMs are unlikely to surpass human capabilities in domains requiring extensive medical knowledge and sophisticated analytical skills in the short term.
“it is unlikely that LLMs will surpass human capabilities in domains requiring extensive medical knowledge and sophisticated analytical skills in the short term.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“and GPT-3.5 (16.23%, p < 0.001)”
From A Review of Multi-Agent AI Systems for Biological and Clinical Data Analysis (2026) · cites this paper as [35]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The helpfulness rate for recommended further diagnostic tests reached 78.26%, a marked improvement over the GPT-4 baseline of 58.17% (p < 0.001).”
From A Review of Multi-Agent AI Systems for Biological and Clinical Data Analysis (2026) · cites this paper as [35]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“four AI “doctor” agents, each with a specialized knowledge focus or reasoning style, discuss a patient case, while a fifth agent acts as a supervisor to coordinate the conversation”
From A Review of Multi-Agent AI Systems for Biological and Clinical Data Analysis (2026) · cites this paper as [35]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“and GPT-3.5 (29.36%, p < 0.001)”
From A Review of Multi-Agent AI Systems for Biological and Clinical Data Analysis (2026) · cites this paper as [35]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“significantly exceeding GPT-4 (37.86%, p = 0.010)”
From A Review of Multi-Agent AI Systems for Biological and Clinical Data Analysis (2026) · cites this paper as [35]
Experimental — accuracy not yet measured
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“compared to 34.55% for GPT-4, p = 0.001”
From A Review of Multi-Agent AI Systems for Biological and Clinical Data Analysis (2026) · cites this paper as [35]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“token consumption for MAS reached over 6000 tokens compared to just 118 for standard input/output prompting”
From A Review of Multi-Agent AI Systems for Biological and Clinical Data Analysis (2026) · cites this paper as [35]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“nearly doubling the performance of standalone GPT-4 (19.65%, p < 0.001)”
From A Review of Multi-Agent AI Systems for Biological and Clinical Data Analysis (2026) · cites this paper as [35]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“the MAC’s 34.11% primary accuracy beat CoT (27.81%)”
From A Review of Multi-Agent AI Systems for Biological and Clinical Data Analysis (2026) · cites this paper as [35]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The authors developed an MAS framework for differential diagnosis, inspired by the multidisciplinary team meetings common in clinical practice.”
From A Review of Multi-Agent AI Systems for Biological and Clinical Data Analysis (2026) · cites this paper as [35]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Qualitatively, the dialogues revealed that agents effectively introduced diverse perspectives, such as identifying rare syndromes or flagging lab abnormalities that contradicted specific hypotheses, leading to a more thorough analysis than a single model’s monologue.”
From A Review of Multi-Agent AI Systems for Biological and Clinical Data Analysis (2026) · cites this paper as [35]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“and 67.88% in follow-ups (p = 0.042)”
From A Review of Multi-Agent AI Systems for Biological and Clinical Data Analysis (2026) · cites this paper as [35]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“and Self-Consistency (32.45%) methods”
From A Review of Multi-Agent AI Systems for Biological and Clinical Data Analysis (2026) · cites this paper as [35]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Self-Refine (32.01%)”
From A Review of Multi-Agent AI Systems for Biological and Clinical Data Analysis (2026) · cites this paper as [35]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“While early evaluations such as the MAC framework reported a 50-fold increase in token consumption (6000 vs. 118) for a roughly 15% absolute accuracy gain, recent large-scale clinical evidence suggests this ‘accuracy premium’ is vital in high-stakes domains [35].”
From A Review of Multi-Agent AI Systems for Biological and Clinical Data Analysis (2026) · cites this paper as [35]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“While early evaluations such as the MAC framework reported a 50-fold increase in token consumption (6000 vs. 118) for a roughly 15% absolute accuracy gain, recent large-scale clinical evidence suggests this ‘accuracy premium’ is vital in high-stakes domains [35].”
From A Review of Multi-Agent AI Systems for Biological and Clinical Data Analysis (2026) · cites this paper as [35]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“However, current frameworks rely on a single model family (e.g., GPT series6), limiting epistemic diversity and opportunities for cross-validation.”
From Model confrontation and collaboration: A debate intelligence framework for enhancing medical reasoning in large language models (2026) · cites this paper as [6]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“With complete information provided in follow-up consultations, the MAS accuracy for the most-likely diagnosis rose to 53.86%”
From A Review of Multi-Agent AI Systems for Biological and Clinical Data Analysis (2026) · cites this paper as [35]
The cited source says what the claim attributes to it.
In the citing paper
“On a dataset of 302 curated rare-disease cases (spanning categories such as abdominal surgery, cardiac, and genetic disorders), the MAS—referred to as the Multi-agent Conversation (MAC) framework in their paper—attained a 34.11% ‘most-likely diagnosis’ accuracy during the primary consultation”
From A Review of Multi-Agent AI Systems for Biological and Clinical Data Analysis (2026) · cites this paper as [35]
The cited source says what the claim attributes to it.
In the citing paper
“the ‘possible diagnoses’ accuracy reached 48.12% in primary consultations”
From A Review of Multi-Agent AI Systems for Biological and Clinical Data Analysis (2026) · cites this paper as [35]
The cited source says what the claim attributes to it.
In the citing paper
“The multi-agent approach significantly outperformed single-model baselines in both initial diagnosis and follow-up consultations.”
From A Review of Multi-Agent AI Systems for Biological and Clinical Data Analysis (2026) · cites this paper as [35]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“This study included 302 kinds of rare disease from 33 different disease categories. One to nine kinds of rare disease were randomly selected for each category.”
§ Enhancing diagnostic capability with multi-agents conversational large language models > Results > Study sample
Reported 302 · recomputed 297
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Enhancing diagnostic capability with multi-agents conversational large language models > Results > Performance of MAC on diagnostic ability
“helpfulness of further diagnostic tests (78.26% vs 77.37%)”
§ Enhancing diagnostic capability with multi-agents conversational large language models > Results > Performance of MAC on diagnostic ability
“| GPT-3.5 | 4 | 24.28% | 36.64% | 77.59% |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Enhancing diagnostic capability with multi-agents conversational large language models > Results > Performance of MAC on diagnostic ability
“Using GPT-4 as the base model, the Most Likely Diagnosis accuracy in primary consultations was 31.31% for 2 agents”
§ Enhancing diagnostic capability with multi-agents conversational large language models > Results > Performance of MAC on diagnostic ability
“| GPT-4 | 2 | 31.13% | 45.03% | 73.51% |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Enhancing diagnostic capability with multi-agents conversational large language models > Results > Performance of MAC on diagnostic ability
“For MAC using GPT-3.5 as the base model, settings with more than 2 doctor agents were not feasible due to token output limitation.”
§ Enhancing diagnostic capability with multi-agents conversational large language models > Results > Performance of MAC on diagnostic ability
“| GPT-3.5 | 3 | 24.17% | 35.43% | 79.14% | | GPT-3.5 | 4 | 24.28% | 36.64% | 77.59% | | GPT-3.5 | 5 | 22.85% | 36.09% | 79.47% |”
Citations
Cites
The Measurement of Observer Agreement for Categorical Data
Landis, J. R., & Koch, G. G. (1977) · Biometrics · doi:10.2307/2529310 · Cited by 81,148
Chain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263
Large language models encode clinical knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2023) · Nature · doi:10.1038/s41586-023-06291-2 · Cited by 4,083
Large Language Models Encode Clinical Knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2212.13138 · Cited by 4,083
Large language models in medicine
Thirunavukarasu, A. J., Ting, D. S. J., Elangovan, K., Gutierrez, L., Tan, T. F., & Ting, D. S. W. (2023) · Nature Medicine · doi:10.1038/s41591-023-02448-8 · Cited by 3,944
Self-Refine: Iterative Refinement with Self-Feedback
Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye, S., Yang, Y., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.17651 · Cited by 2,314
Estimating cumulative point prevalence of rare diseases: analysis of the Orphanet database
Nguengang Wakap, S., Lambert, D. M., Olry, A., Rodwell, C., Gueydan, C., Lanneau, V., Murphy, D., Le Cam, Y., & Rath, A. (2020) · European Journal of Human Genetics · doi:10.1038/s41431-019-0508-0 · Cited by 1,752
Self-Consistency Improves Chain of Thought Reasoning in Language Models
Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., Chowdhery, A., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2203.11171 · Cited by 829
Evaluation and mitigation of the limitations of large language models in clinical decision-making
Hager, P., Jungmann, F., Holland, R., Bhagat, K., Hubrecht, I., Knauer, M., Vielhauer, J., Makowski, M., Braren, R., Kaissis, G., et al. (2024) · Nature Medicine · doi:10.1038/s41591-024-03097-1 · Cited by 798 · cited 1×
“However,itshould be noted that this study used older models such as Llama-2, which may perform less optimally than more advanced models like GPT-4 30 .”
Research works printed but not linked: 2
- Smit, A., Duckworth, P., Grinsztajn, N., Barrett, T. & Pretorius, A. Are we going MAD? Benchmarking Multi-Agent Debate between Language Models for Medical Q&A. In: Deep Generative Models for Health Workshop NeurIPS 2023) (NeurIPS, 2023).
- Pal, A., Umapathi, L. K. & Sankarasubbu, M. Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering. In: Conference on Health, Inference, and Learning) (PMLR, 2022).
Cited by
Sorted by citations
Items 101–121 of 121
Artificial intelligence agents and agentic artificial intelligence applied to precision medicine
Costa, F. F. (2026) · Drug Discovery Today · doi:10.1016/j.drudis.2026.104781
Bouqdir, K., Nfaoui, E. H., & Abouaomar, A. (2026) · International Conference on Circuit, Systems and Communication (ICCSC) · doi:10.1109/iccsc71566.2026.11650471
Simon, B. F., Dénes-Fazakas, L., Eigner, G., Kovács, L., & Szilágyi, L. (2026) · IEEE International Conference on Intelligent Engineering Systems (INES) · doi:10.1109/ines69513.2026.11661038
Blackwell: An Explainable Anamnesis-Based Multi-agent Framework for Remote Clinical Diagnosis
Torquato, L., Haque, R., Stynes, P., Doyle, G., & Cawley, O. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37249-9_52
Generative large language models in medicine: a scoping review of recent methodological advances
LI, F., Li, J., Cao, W., Li, H., Li, Y., Sun, Z., Chen, D., Duan, T., Li, P., Tizhoosh, H., et al. (2026) · npj Health Systems · doi:10.1038/s44401-026-00137-5
Self-citationMedRef-KG: Multi-agent KG-grounded reasoning framework for medical specialist referrals
Elshaer, Z., Mohamed, E. H., Selim, S., & Al‐Atabany, W. (2026) · Telematics and Informatics Reports · doi:10.1016/j.teler.2026.100363
Yildiz, M., Demir, A., & Kaya, E. (2026) · Journal of Health Informatics and Digital Systems · doi:10.68159/a800893285
Anderson, C. (2026) · medRxiv · doi:10.64898/2026.09.01.26361960
Potenzial und Herausforderungen von Large Language Models in der klinischen Entscheidungsfindung
Jungmann, F., & Hager, P. (2026) · Journal für Gastroenterologische und Hepatologische Erkrankungen · doi:10.1007/s41971-026-00265-7
MDT-guided language-model reasoning for spinal infection diagnosisPreprint
Chen, J., Zhang, R., Xu, C., Li, H., 胡维维, Cai, L., Ma, S., Meng, Y., Li, X., Chen, Y., et al. (2026) · Research Square · doi:10.21203/rs.3.rs-10692703/v1
Self-citationShalmani, M. A., Khani, M., Harris, M. S., Lu, Q., & Luo, J. (2026) · Journal of Medical Systems · doi:10.1007/s10916-026-02460-8
CReaTe-LM: A large language model for stepwise, heuristic clinical reasoning instruction
Zhao, Z., Wang, S., Chen, R., Tian, S., Wu, F., Hu, H., & Hao, P. (2026) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2026.103540
Liu, F., Liu, Z., Fei, X., He, J., Xing, J., Li, J., & Chan, P. (2026) · Journal of Medical Systems · doi:10.1007/s10916-026-02459-1
Self-citationZhang, R., Dong, J., Xin, J., Ballard, J., Beeche, C., Sharma, R., Zhao, X., Konz, N., Long, Q., Witschey, W. R., et al. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-38079-1_16
AI in Proton Therapy Treatment Planning: A Narrative Review
Ding, Y., Feng, H., Bues, M., Fatyga, M., Liu, T., Zhang, W., Whitaker, T. J., Lin, H., Lee, N. Y., Simone, C. B., et al. (2026) · International Journal of Particle Therapy · doi:10.1016/j.ijpt.2026.101965
Self-citationCai, T., Zhang, N., Li, Y., Li, X., Li, X., & Liu, B. (2026) · Research Square · doi:10.21203/rs.3.rs-10586813/v1
Feng, J., Jiao, Y., Xie, H., Wang, J., Li, Y., Xie, L., Peng, W., & Sun, X. (2026) · medRxiv · doi:10.64898/2026.09.24.26363897
Worth, J. E., Perez, A., Wu, D. J., Brodeur, P. G., Tat, E., Manrai, A. K., Wu, D., Walton, C., McCoy, L. G., Jain, P., et al. (2026) · BMJ Digital Health & AI · doi:10.1136/bmjdh-2026-000113 · Cited by 1
AI Assisted Development of Interactive Multimedia Modules for Resident Self-Directed Learning
Khan, S., Tippareddy, C., & Faraji, N. (2026) · Current Problems in Diagnostic Radiology · doi:10.1067/j.cpradiol.2026.10.003
Multi-Agent Debate Strategies: Survey, Taxonomy, and Challenges
Motger, Q., Oriol, M., Marco, J., & Franch, X. (2026) · arXiv · doi:10.48550/arxiv.2607.26212 · Cited by 5
Metadata
- Authors
- Chen, X., Yi, H., You, M., Liu, W., Wang, L., Li, H., Zhang, X., Guo, Y., Fan, L., Chen, G., Lao, Q., Fu, W., Li, K., & Li, J.
- Affiliations
- Sichuan University (CN)
- First Affiliated Hospital Zhejiang University (CN)
- Beijing University of Posts and Telecommunications (CN)
- Corresponding author
- Fu, Weili
- Li, Kang
- Li, Jian
- ORCID
- Venue
- npj Digital Medicine
- Published
- 2025-03-13
- Volume
- 8
- Issue
- 1
- Pages
- 159-159
- ISSN
- 2398-6352 (ISSN-L)
- PMC
- PMC11906805
- OpenAlex
- W4408411093
- Cited by
- 147
- References
- 40
- Citations by year
- 2026: 114 · 2025: 33
- FWCI
- 27.84
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- npj Digital Medicine (publishedVersion, cc-by-nc-nd)
- PubMed Central (publishedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Keywords
- multi-agent LLM systems, disease diagnosis, rare diseases, clinical reasoning, GPT-4, clinical decision support
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-07, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-07 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
