CoChat
Sign Up
Cited by 81Artificial Intelligence in Healthcare and EducationMachine Learning in HealthcareArtificial Intelligence in Healthcare

MDAgents: an adaptive collaboration of LLMs for medical decision-making

Open access
Kim, Yubin, Park, Chanwoo, Jeong, Hyewon, Chan, Yik, Xu, Xuhai, McDuff, Daniel, Lee, Hyeonhoon, Ghassemi, Marzyeh +2 more·2024·Advances in Neural Information Processing Systems 37
DOI 10.52202/079017-2522
Open this paper in CoChat — free accountOpen PDF

Abstract

Foundation models are becoming valuable tools in medicine. Yet despite their promise, the best way to leverage Large Language Models (LLMs) in complex medical tasks remains an open question. We introduce a novel multi-agent framework, named Medical Decision-making Agents (MDAgents) that helps address this gap by automatically assigning a collaboration structure to a team of LLMs. The assigned solo or group collaboration structure is tailored to the medical task at hand, emulating real-world medical decision-making processes adapted to tasks of varying complexities. We evaluate our framework and baseline methods using state-of-the-art LLMs across a suite of real-world medical knowledge and medical diagnosis benchmarks, including a comparison of LLMs' medical complexity classification against human physicians. MDAgents achieved the best performance in seven out of ten benchmarks on tasks requiring an understanding of medical knowledge and multi-modal reasoning, showing a significant improvement of up to 4.2% (p < 0.05) compared to previous methods' best performances. Ablation studies reveal that MDAgents effectively determines medical complexity to optimize for efficiency and accuracy across diverse medical tasks. Notably, the combination of moderator review and external medical knowledge in group collaboration resulted in an average accuracy improvement of 11.8%. Our code can be found at https://github.com/mitmedialab/MDAgents.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.52202/079017-2522

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • MDAgents demonstrated a significant improvement of up to 4.2% (p < 0.05) over the best-performing prior methods.

    “MDAgents demonstrated a significant improvement of up to 4.2% (p < 0.05) over the best-performing prior methods”

    From A Review of Multi-Agent AI Systems for Biological and Clinical Data Analysis (2026) · cites this paper as [42]

    Section 4.2 (Results; page not provided) reports that MDAgents significantly outperformed solo and group methods (p < 0.05) and performed best on seven of ten benchmarks. The Results passage does not establish the claimed maximum improvement of 4.2% over prior methods; that figure appears only in the abstract, which is excluded as evidence under the supplied rubric. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

  • Specific accuracy gains reached 5.2% on diagnostic reasoning datasets like DDXPlus.

    “with specific accuracy gains reaching 5.2% on diagnostic reasoning datasets like DDXPlus”

    From A Review of Multi-Agent AI Systems for Biological and Clinical Data Analysis (2026) · cites this paper as [42]

  • In evaluations across ten diverse medical benchmarks, MDAgents achieved superior performance in seven out of ten tasks.

    “In evaluations across ten diverse medical benchmarks, including medical knowledge retrieval (MedQA, PubMedQA) and multimodal reasoning (Path-VQA, PMC-VQA), the framework achieved superior performance in seven out of ten tasks.”

    From A Review of Multi-Agent AI Systems for Biological and Clinical Data Analysis (2026) · cites this paper as [42]

  • Static multi-agent settings often result in high computational costs (averaging 20.3 API calls).

    “Static multi-agent settings often result in high computational costs (averaging 20.3 API calls)”

    From A Review of Multi-Agent AI Systems for Biological and Clinical Data Analysis (2026) · cites this paper as [42]

  • MDAgents introduces an adaptive coordination framework that automatically assigns collaboration structures to a team of agents based on task complexity, mirroring tiered referral processes used in clinical practice.

    “MDAgents (Kim et al., 2024), introduces an adaptive coordination framework that automatically assigns collaboration structures to a team of agents based on task complexity, mirroring the tiered referral processes used in clinical practice [42].”

    From A Review of Multi-Agent AI Systems for Biological and Clinical Data Analysis (2026) · cites this paper as [42]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 60

  1. Monica Agrawal, Stefan Hegselmann, Hunter Lang, Yoon Kim, and David Sontag. Large language models are few-shot clinical information extractors. arXiv preprint arXiv:2205.12689, 2022. · arXiv
  2. Seongsu Bae, Daeun Kyung, Jaehee Ryu, Eunbyeol Cho, Gyubok Lee, Sunjun Kweon, Jungwoo Oh, Lei Ji, Eric Chang, Tackeun Kim, et al. Ehrxqa: A multi-modal question answering dataset for electronic health records with chest x-ray images. Advances in Neural Information Processing Systems, 36, 2024.
  3. Michael L Barnett, Dhruv Boddupalli, Shantanu Nundy, and David W Bates. Comparative accuracy of diagnosis by collective intelligence of multiple physicians vs individual physicians. JAMA network open, 2(3):e190096-e190096, 2019.
  4. Michael L Barnett, Nancy L Keating, Nicholas A Christakis, A James O'malley, and Bruce E Landon. Reasons for choice of referral physician among primary care and specialist physicians. Journal of general internal medicine, 27:506-512, 2012.
  5. Ofir Ben-Assuli, Nanda Kumar, Ofer Arazy, and Itamar Shabtai. The use of analytic hierarchy process for measuring the complexity of medical diagnosis. Health Informatics Journal, 26(1):218-232, 2020.
  6. Justin Bitter, Elizabeth van Veen-Berkx, Hein G Gooszen, and Pierre van Amelsvoort. Mul- tidisciplinary teamwork is an important issue to healthcare professionals. Team Performance Management: An International Journal, 19(5/6):263-278, 2013.
  7. Justin Chih-Yao Chen, Swarnadeep Saha, and Mohit Bansal. Reconcile: Round-table conference improves reasoning via consensus among diverse llms, 2024.
  8. Michael Christ, Florian Grossmann, Daniela Winter, Roland Bingisser, and Elke Platz. Modern triage in the emergency department. Deutsches Ärzteblatt International, 107(50):892, 2010.
  9. Jane Cioffi and Roslyn Markham. Clinical decision-making by midwives: managing case complexity. Journal of advanced nursing, 25(2):265-272, 1997.
  10. Jan Clusmann, Fiona R Kolbinger, Hannah Sophie Muti, Zunamys I Carrero, Jan-Niklas Eckardt, Narmin Ghaffari Laleh, Chiara Maria Lavinia Löffler, Sophie-Caroline Schwarzkopf, Michaela Unger, Gregory P Veldhuizen, et al. The future landscape of large language models in medicine. Communications medicine, 3(1):141, 2023.
  11. Nick Craddock and Laurence Mynors-Wallis. Psychiatric diagnosis: impersonal, imperfect and important. The British Journal of Psychiatry, 204(2):93-95, 2014.
  12. Lesley F Degner and Jeffrey A Sloan. Decision making during serious illness: what role do patients really want to play? Journal of clinical epidemiology, 45(9):941-950, 1992.
  13. Carolyn Ee, James Lake, Joseph Firth, Fiona Hargraves, M De Manincor, Tanya Meade, Wolfgang Marx, and Jerome Sarris. An integrative collaborative care model for people with mental illness and physical comorbidities. International Journal of Mental Health Systems, 14:1-16, 2020.
  14. Arthur S Elstein and Alan Schwarz. Clinical problem solving and diagnostic decision making: selective review of the cognitive literature. BMJ, 324(7339):729-732, 2002.
  15. Glyn Elwyn, Dominick Frosch, Richard Thomson, Natalie Joseph-Williams, Amy Lloyd, Paul Kinnersley, Emma Cording, Dave Tomson, Carole Dodd, Stephen Rollnick, et al. Shared decision making: a model for clinical practice. Journal of general internal medicine, 27:1361- 1367, 2012.
  16. Nancy E Epstein. Multidisciplinary in-hospital teams improve patient outcomes: A review. Surgical neurology international, 5(Suppl 7):S295, 2014.
  17. Isaac R Galatzer-Levy, Daniel McDuff, Vivek Natarajan, Alan Karthikesalingam, and Matteo Malgaroli. The capability of large language models to measure psychiatric functioning. arXiv preprint arXiv:2308.01834, 2023. · arXiv
  18. Amy L Garcia. Variability in acuity in acute care. JONA: The Journal of Nursing Administration, 47(10):476-483, 2017.
  19. Nicki Gilboy, Paula Tanabe, Debbie Travers, Alexander M Rosenau, et al. Emergency severity index (esi): A triage tool for emergency department care, version 4. implementation handbook 2012 edition. AHRQ publication, 12, 2011.
  20. Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. Domain-specific language model pretraining for biomedical natural language processing. ACM Transactions on Computing for Healthcare, 3(1):1-23, October 2021.
  21. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874, 2021. · arXiv
  22. Zhiyuan Hu, Chumin Liu, Xidong Feng, Yilun Zhao, See-Kiong Ng, Anh Tuan Luu, Junxian He, Pang Wei Koh, and Bryan Hooi. Uncertainty of thoughts: Uncertainty-aware planning enhances information seeking in large language models. arXiv preprint arXiv:2402.03271, 2024. · arXiv
  23. M. Jimenez-Lara. Reaping the benefits of integrated health care. stanford social innovation review. Stanford Social Innovation Review., 2016.
  24. Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. Pubmedqa: A dataset for biomedical research question answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2567-2577, 2019.
  25. Qiao Jin, Zhizheng Wang, Yifan Yang, Qingqing Zhu, Donald Wright, Thomas Huang, W John Wilbur, Zhe He, Andrew Taylor, Qingyu Chen, et al. Agentmd: Empowering language agents for risk prediction with large-scale clinical tool learning. arXiv preprint arXiv:2402.13225, 2024. · arXiv
  26. Emre Kıcıman, Robert Ness, Amit Sharma, and Chenhao Tan. Causal reasoning and large language models: Opening a new frontier for causality. arXiv preprint arXiv:2305.00050, 2023. · arXiv
  27. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199-22213, 2022.
  28. Aviral Kumar, Vincent Zhuang, Rishabh Agarwal, Yi Su, John D Co-Reyes, Avi Singh, Kate Baumli, Shariq Iqbal, Colton Bishop, Rebecca Roelofs, et al. Training language models to self-correct via reinforcement learning. arXiv preprint arXiv:2409.12917, 2024. · arXiv
  29. Tiffany H. Kung, Morgan Cheatham, Arielle Medenilla, Czarina Sillos, Lorie De Leon, Camille Elepaño, Maria Madriaga, Rimel Aggabao, Giezel Diaz-Candido, James Maningo, and Victor Tseng. Performance of chatgpt on usmle: Potential for ai-assisted medical education using large language models. PLOS Digital Health, 2(2):1-12, 02 2023.
  30. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453-466, 2019.
  31. Danielle L LaFrance, Mary Jane Weiss, Ellie Kazemi, Joanne Gerenser, and Jacqueline Dobres. Multidisciplinary teaming: Enhancing collaboration through increased understanding. Behavior analysis in practice, 12(3):709-726, 2019.
  32. Beibin Li, Konstantina Mellou, Bo Zhang, Jeevan Pathuri, and Ishai Menache. Large language models for supply chain optimization. arXiv preprint arXiv:2307.03875, 2023. · arXiv
  33. Junkai Li, Siyu Wang, Meng Zhang, Weitao Li, Yunghwei Lai, Xinhui Kang, Weizhi Ma, and Yang Liu. Agent hospital: A simulacrum of hospital with evolvable medical agents. arXiv preprint arXiv:2405.02957, 2024. · arXiv
  34. Shuyue Stella Li, Vidhisha Balachandran, Shangbin Feng, Jonathan Ilgen, Emma Pierson, Pang Wei Koh, and Yulia Tsvetkov. Mediq: Question-asking llms for adaptive and reliable medical reasoning. arXiv preprint arXiv:2406.00922, 2024. · arXiv
  35. Daniel McDuff, Mike Schaekermann, Tao Tu, Anil Palepu, Amy Wang, Jake Garrison, Karan Singhal, Yash Sharma, Shekoofeh Azizi, Kavita Kulkarni, et al. Towards accurate differential diagnosis with large language models. arXiv preprint arXiv:2312.00164, 2023. · arXiv
  36. Ateev Mehrotra, Christopher B Forrest, and Caroline Y Lin. Dropping the baton: specialty referrals in the united states. The Milbank Quarterly, 89(1):39-68, 2011.
  37. Michael Moor, Oishi Banerjee, Zahra Shakeri Hossein Abad, Harlan M. Krumholz, Jure Leskovec, Eric J. Topol, and Pranav Rajpurkar. Foundation models for generalist medical artificial intelligence. Nature, 616(7956):259-265, 2023.
  38. Tong Mu, Alec Helyar, Johannes Heidecke, Joshua Achiam, Andrea Vallone, Ian Kivlichan, Molly Lin, Alex Beutel, John Schulman, and Lilian Weng. Rule based rewards for language model safety. Advances in Neural Information Processing Systems, 2024.
  39. Mark A Musen, Blackford Middleton, and Robert A Greenes. Clinical decision-support systems. In Biomedical informatics: computer applications in health care and biomedicine, pages 795-840. Springer, 2021.
  40. Anand K Parekh, Richard A Goodman, Catherine Gordon, Howard K Koh, and HHS Intera- gency Workgroup on Multiple Chronic Conditions. Managing multiple chronic conditions: a strategic framework for improving health outcomes and quality of life. Public health reports, 126(4):460-471, 2011.
  41. Chanwoo Park, Xiangyu Liu, Asuman Ozdaglar, and Kaiqing Zhang. Do llm agents have regret? a case study in online learning and games. arXiv preprint arXiv:2403.16843, 2024. · arXiv
  42. Khaled Saab, Tao Tu, Wei-Hung Weng, Ryutaro Tanno, David Stutz, Ellery Wulczyn, Fan Zhang, Tim Strother, Chunjong Park, Elahe Vedadi, et al. Capabilities of gemini models in medicine. arXiv preprint arXiv:2404.18416, 2024. · arXiv
  43. Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. Alfworld: Aligning text and embodied environments for interactive learning. arXiv preprint arXiv:2010.03768, 2020. · arXiv
  44. Karan Singhal, Shekoofeh Azizi, Tao Tu, S. Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, Perry Payne, Martin Senevi- ratne, Paul Gamble, Chris Kelly, Abubakr Babiker, Nathanael Schärli, Aakanksha Chowdhery, Philip Mansfield, Dina Demner-Fushman, Blaise Agüera y Arcas, Dale Webster, Greg S. Corrado, Yossi Matias, Katherine Chou, Juraj Gottweis, Nenad Tomasev, Yun Liu, Alvin Rajko- mar, Joelle Barral, Christopher Semturs, Alan Karthikesalingam, and Vivek Natarajan. Large language models encode clinical knowledge. Nature, 620(7972):172-180, 2023.
  45. Harold C Sox, Michael C Higgins, Douglas K Owens, and Gillian Sanders Schmidler. Medical decision making. John Wiley & Sons, 2024.
  46. J Stairmand, Louise Signal, D Sarfati, C Jackson, L Batten, M Holdaway, and C Cunningham. Consideration of comorbidity in treatment decision making in multidisciplinary cancer team meetings: a systematic review. Annals of Oncology, 26(7):1325-1332, 2015.
  47. Arun James Thirunavukarasu, Darren Shu Jeng Ting, Kabilan Elangovan, Laura Gutierrez, Ting Fang Tan, and Daniel Shu Wei Ting. Large language models in medicine. Nature medicine, 29(8):1930-1940, 2023.
  48. Arun James Thirunavukarasu, Darren Shu Jeng Ting, Kabilan Elangovan, Laura Gutierrez, Ting Fang Tan, and Daniel Shu Wei Ting. Large language models in medicine. Nature Medicine, 29(8):1930-1940, 2023.
  49. Daniel Truhn, Jan-Niklas Eckardt, Dyke Ferber, and Jakob Nikolas Kather. Large language models and multimodal foundation models for precision oncology. NPJ Precision Oncology, 8(1):72, 2024.
  50. Sean R Tunis, Daniel B Stryer, and Carolyn M Clancy. Practical clinical trials: increasing the value of clinical research for decision making in clinical and health policy. Jama, 290(12):1624- 1632, 2003.
  51. Dave Van Veen, Cara Van Uden, Louis Blankemeier, Jean-Benoit Delbrouck, Asad Aali, Chris- tian Bluethgen, Anuj Pareek, Malgorzata Polacin, Eduardo Pontes Reis, Anna Seehofnerová, Nidhi Rohatgi, Poonam Hosamani, William Collins, Neera Ahuja, Curtis P. Langlotz, Jason Hom, Sergios Gatidis, John Pauly, and Akshay S. Chaudhari. Adapted large language models can outperform medical experts in clinical text summarization. Nature Medicine, 30(4):1134-1142, February 2024.
  52. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171, 2022. · arXiv
  53. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24837, 2022.
  54. Thomas Weida and Jane Weida. Outpatient e/m coding simplified. Family Practice Management, 29(1):26-31, 2022.
  55. Robin M Weinick, Rachel M Burns, and Ateev Mehrotra. Many emergency department visits could be managed at urgent care centers and retail clinics. Health affairs, 29(9):1630-1636, 2010.
  56. Richard C Wuerz, Leslie W Milne, David R Eitel, Debbie Travers, and Nicki Gilboy. Reliability and validity of a new five-level triage instrument. Academic emergency medicine, 7(3):236-242, 2000.
  57. Guangzhi Xiong, Qiao Jin, Zhiyong Lu, and Aidong Zhang. Benchmarking retrieval-augmented generation for medicine. arXiv preprint arXiv:2402.13178, 2024. · arXiv
  58. Weixiang Yan, Haitian Liu, Tengxiao Wu, Qian Chen, Wen Wang, Haoyuan Chai, Jiayi Wang, Weishan Zhao, Yixin Zhang, Renjun Zhang, et al. Clinicallab: Aligning agents for multi- departmental clinical diagnostics in the real world. arXiv preprint arXiv:2406.13890, 2024. · arXiv
  59. Haodi Zhang, Jiahong Li, Yichi Wang, and Yuanfeng Songi. Integrating automated knowledge extraction with large language models for explainable medical decision-making. In 2023 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), pages 1710-1717. IEEE, 2023.
  60. Junbin Zhou and Xiao Xu. The difficulty of medical decision-making: should patients be involved? Hepatobiliary Surgery and Nutrition, 12(3):407, 2023.

Cited by

Sorted by citations

Authors
Kim, Y., Park, C., Jeong, H., Chan, Y., Xu, X., McDuff, D., Lee, H., Ghassemi, M., Breazeal, C., & Park, H.
ORCID
Venue
Advances in Neural Information Processing Systems 37
Publisher
Neural Information Processing Systems Foundation, Inc. (NeurIPS)
Published
2024
Pages
79410-79452
ISSN
1049-5258 (ISSN-L)
DOI
10.52202/079017-2522
OpenAlex
W4415796712
Cited by
81
References
0
Citations by year
2026: 66 · 2025: 14 · 2024: 1
FWCI
21.78
Type
conference paper
Language
en
Open access
Licence
cc-by
Locations
  • neural information processing systems (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University)
Funders
  • Seoul National University
  • Seoul National University Hospital
  • Korea Foundation for Advanced Studies
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
  • Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
Keywords
large language models, multi-agent systems, clinical decision-making, adaptive collaboration, clinical decision support

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review