CoChat
Sign Up
Cited by 10Artificial Intelligence in Healthcare and EducationOrthopedic Infections and TreatmentsMachine Learning in Healthcare

Battle of the Bots: Solving Clinical Cases in Osteoarticular Infections With Large Language Models

Open access
Borgonovo, Fabio, Matsuo, Takahiro, Petri, Francesco, Amin Alavi, Seyed Mohammad, Mazudie Ndjonko, Laura Chelsea, Gori, Andrea, Berbari, Elie F.·2025·Mayo Clinic Proceedings: Digital Health
DOI 10.1016/j.mcpdig.2025.100230PMC PMC12205795
Open this paper in CoChat — free accountOpen PDF

Abstract

Objective: To evaluate the ability of 15 different large language models (LLMs) to solve clinical cases with osteoarticular infections following published guidelines. Materials and Methods: The study evaluated 15 LLMs across 5 categories of osteoarticular infections: periprosthetic joint infection, diabetic foot infection, native vertebral osteomyelitis, fracture-related infections, and septic arthritis. Models were selected systematically, including general-purpose and medical-specific systems, ensuring robust English support. In total, 126 text-based questions, developed by the authors from published guidelines and validated by experts, assessed diagnostic, management, and treatment strategies. Each model answered individually, with responses classified as correct or incorrect based on guidelines. All tests were conducted between April 17, 2025, and April 28, 2025. Results, presented as percentages of correct answers and aggregated scores, highlight performance trends. Mixed-effects logistic regression with a random question effect was used to quantify how each LLM compared in answering the study questions. Results: The performance of 15 LLMs was evaluated, with the percentage of correct answers reported. OpenEvidence and Microsoft Copilot achieved the highest score (119/126 [94.4%]), excelling in multiple categories. ChatGPT-4o and Gemini 2.5 Pro scored 117 of the 126 (92.8%). When used as references, OpenEvidence was not inferior to any comparator and was superior to 5 LLMs. Performance varied across categories, highlighting the strengths and limitations of individual models. Conclusion: OpenEvidence and Miccrosoft Copilot achieved the highest accuracy among evaluated LLMs, highlighting their potential for precisely addressing complex clinical cases. This study emphasizes the need for specialized, validated artificial intelligence tools in medical practice. Although promising, current models face limitations in real-world applications, requiring further refinement to support clinical decision making reliably.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1016/j.mcpdig.2025.100230

Cites

7 of 7 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 39

  1. [1]Sloan M. Premkumar A. Sheth N.P. Projected volume of primary total joint arthroplasty in the U.S., 2014 to 2030 J Bone Joint Surg Am 100 17 2018 1455 1460 10.2106/JBJS.17.01617 30180053 · DOI
  2. [2]Kurtz S.M. Lau E.C. Son M.S. Chang E.T. Zimmerli W. Parvizi J. Are we winning or losing the battle with periprosthetic joint infection: trends in periprosthetic joint infection and mortality risk for the medicare population J Arthroplasty 33 10 2018 3238 3245 10.1016/j.arth.2018.05.042 29914821 · DOI
  3. [3]Weinstein E.J. Stephens-Shields A.J. Newcomb C.W. Incidence, microbiological studies, and factors associated with prosthetic joint infection after total knee arthroplasty JAMA Netw Open 6 10 2023 e2340457 10.1001/jamanetworkopen.2023.40457 PMC10618849 37906194 · DOI
  4. [4]Premkumar A. Kolin D.A. Farley K.X. Projected economic burden of periprosthetic joint infection of the hip and knee in the United States J Arthroplasty 36 5 2021 1484 1489.e3 10.1016/j.arth.2020.12.005 33422392 · DOI
  5. [5]Patel R. Periprosthetic joint infection N Engl J Med 388 3 2023 251 262 10.1056/NEJMra2203477 36652356 · DOI
  6. [6]Armstrong D.G. Swerdlow M.A. Armstrong A.A. Five year mortality and direct costs of care for people with diabetic foot complications are comparable to cancer J Foot Ankle Res 13 1 2020 16 10.1186/s13047-020-00383-2 32209136 PMC7092527 · DOI · Paper page
  7. [7]Cortés-Penfield N. Tai D.B.G. Hewlett A. Orthopedic infectious diseases: a survey on the composition and perceived value of an emerging subspecialty clinical service J Bone Jt Infect 9 3 2024 161 165 10.5194/jbji-9-161-2024 38903858 PMC11187702 · DOI
  8. [8]Hewlett A.L. Cortés-Penfield N. Ohnoutka C.J. Tande A.J. What orthopaedic infectious diseases specialists want J Bone Joint Surg Am 106 22 2024 2175 2178 10.2106/JBJS.24.00166 38875347 · DOI
  9. [9]Denecke K. May R. LLMHealthGroup Rivera Romero O. Potential of large language models in health care: Delphi study J Med Internet Res 26 2024 e52399 10.2196/52399 PMC11130776 38739445 · DOI · Paper page
  10. [11]Sahoo S.S. Plasek J.M. Xu H. Large language models for biomedicine: foundations, opportunities, challenges, and best practices J Am Med Inform Assoc 31 9 2024 2114 2124 10.1093/jamia/ocae074 38657567 PMC11339493 · DOI
  11. [12]Di Ieva A. Stewart C. Suero Molina E. Large language models in neurosurgery Adv Exp Med Biol 1462 2024 177 198 10.1007/978-3-031-64892-2_11 39523266 · DOI
  12. [13]Li J. Deng Y. Sun Q. Benchmarking large language models in evidence-based medicine IEEE J Biomed Health Inform Published online October 21, 2024 10.1109/JBHI.2024.3483816 39437276 · DOI · Paper page
  13. [14]Lammert J. Dreyer T. Mathes S. Expert-guided large language models for clinical decision support in precision oncology JCO Precis Oncol 8 2024 e2400478 10.1200/PO-24-00478 39475661 · DOI
  14. [16]Quer G. Topol E.J. The potential for large language models to transform cardiovascular medicine Lancet Digit Health 6 10 2024 e767 e771 10.1016/S2589-7500(24)00151-1 39214760 · DOI
  15. [17]Benary M. Wang X.D. Schmidt M. Leveraging large language models for decision support in personalized oncology JAMA Netw Open 6 11 2023 e2343689 10.1001/jamanetworkopen.2023.43689 PMC10656647 37976064 · DOI · Paper page
  16. [18]Wong F. de la Fuente-Nunez C. Collins J.J. Leveraging artificial intelligence in the fight against infectious diseases Science 381 6654 2023 164 170 10.1126/science.adh1114 37440620 PMC10663167 · DOI
  17. [19]Brownstein J.S. Rader B. Astley C.M. Tian H. Advances in artificial intelligence for infectious-disease surveillance N Engl J Med 388 17 2023 1597 1607 10.1056/NEJMra2119215 37099342 · DOI
  18. [20]Biswas S. ChatGPT and the future of medical writing Radiology 307 2 2023 e223312 10.1148/radiol.223312 36728748 · DOI · Paper page
  19. [21]Draschl A. Hauer G. Fischerauer S.F. Are ChatGPT’s free-text responses on periprosthetic joint infections of the hip and knee reliable and useful? J Clin Med 12 20 2023 6655 10.3390/jcm12206655 37892793 PMC10607052 · DOI
  20. [22]Lum Z.C. Can artificial intelligence pass the American Board of Orthopaedic Surgery examination? Orthopaedic residents versus ChatGPT Clin Orthop 481 8 2023 1623 1630 10.1097/CORR.0000000000002704 37220190 PMC10344569 · DOI
  21. [23]Senneville É. Albalawi Z. van Asten S.A. IWGDF/IDSA guidelines on the diagnosis and treatment of diabetes-related foot infections (IWGDF/IDSA 2023) Diabetes Metab Res Rev 40 3 2024 e3687 10.1002/dmrr.3687 37779323 · DOI
  22. [25]Depypere M. Kuehl R. Metsemakers W.J. Recommendations for systemic antimicrobial therapy in fracture-related infection: a consensus from an international expert group J Orthop Trauma 34 1 2020 30 41 10.1097/BOT.0000000000001626 31567902 PMC6903362 · DOI
  23. [27]Miller J.M. Binnicker M.J. Campbell S. A guide to utilization of the microbiology laboratory for diagnosis of infectious diseases: 2018 update by the Infectious Diseases Society of America and the American Society for Microbiology Clin Infect Dis 67 6 2018 e1 e94 10.1093/cid/ciy381 29955859 PMC7108105 · DOI · Paper page
  24. [28]Ravn C. Neyt J. Benito N. Guideline for management of septic arthritis in native joints (SANJO) J Bone Jt Infect 8 1 2023 29 37 10.5194/jbji-8-29-2023 36756304 PMC9901514 · DOI
  25. [29]Expert Panel on Musculoskeletal Imaging Pierce J.L. Perry M.T. ACR Appropriateness Criteria® suspected osteomyelitis, septic arthritis, or soft tissue infection (excluding spine and diabetic foot): 2022 update J Am Coll Radiol 19 11S 2022 S473 S487 10.1016/j.jacr.2022.09.013 36436971 · DOI
  26. [30]Berbari E.F. Kanj S.S. Kowalski T.J. 2015 Infectious Diseases Society of America (IDSA) Clinical Practice Guidelines for the Diagnosis and Treatment of Native Vertebral Osteomyelitis in Adults Clin Infect Dis 61 6 2015 e26 e46 10.1093/cid/civ482 26229122 · DOI
  27. [31]Expert Panel on Neurological Imaging Ortiz A.O. Levitt A. ACR Appropriateness Criteria® Suspected Spine Infection J Am Coll Radiol 18 11S 2021 S488 S501 10.1016/j.jacr.2021.09.001 34794603 · DOI
  28. [32]Agbareia R. Omar M. Soffer S. Glicksberg B.S. Nadkarni G.N. Klang E. Visual-textual integration in LLMs for medical diagnosis: a preliminary quantitative analysis Comput Struct Biotechnol J 27 2025 184 189 10.1016/j.csbj.2024.12.019 39850658 PMC11754970 · DOI
  29. [33]Agbareia R. Omar M. Zloto O. The role of prompt engineering for multimodal LLM glaucoma diagnosis Preprint Posted online November 1, 2024 medRxiv 24316434 10.1101/2024.10.30.24316434 · DOI
  30. [36]Miao J. Thongprayoon C. Suppadungsuk S. Garcia Valencia O.A. Cheungpasitporn W. Integrating retrieval-augmented generation with large language models in nephrology: advancing practical applications Medicina (Kaunas) 60 3 2024 445 10.3390/medicina60030445 38541171 PMC10972059 · DOI
  31. [37]Kim Y. Jeong H. Chen S. Medical hallucinations in foundation models and their impact on healthcare Preprint Posted online March 3, 2025 medRxiv 10.48550/arXiv.2503.05777 · DOI · Paper page
  32. [39]Harrer S. Attention is not all you need: the complicated case of ethically using large language models in healthcare and medicine EBioMedicine 90 2023 104512 10.1016/j.ebiom.2023.104512 PMC10025985 36924620 · DOI · Paper page
  33. [41]Salloch S. Eriksen A. What are humans doing in the loop? Co-reasoning and practical judgment when using machine learning-driven decision aids Am J Bioeth 24 9 2024 67 78 10.1080/15265161.2024.2353800 38767971 · DOI
  34. [43]Gilson A. Safranek C.W. Huang T. How does ChatGPT perform on the United States medical licensing exams? The implications of large language models for medical education and knowledge assessment JMIR Med Educ 9 2023 e45312 10.2196/45312 PMC9947764 36753318 · DOI · Paper page
  35. [44]Al-Shakarchi N.J. Haq I.U. ChatGPT performance in the UK Medical Licensing Assessment: how to train the next generation? Mayo Clin Proc Digit Health 1 3 2023 309 310 10.1016/j.mcpdig.2023.06.004 40206625 PMC11975728 · DOI
  36. [45]Johnson D. Goodman R. Patrinely J. Assessing the Accuracy and reliability of ai-generated medical responses: an evaluation of the Chat-GPT Model Preprint. Posted online February 28 2023 Res Sq rs.3.rs-2566942 10.21203/rs.3.rs-2566942/v1 · DOI · Paper page
  37. [48]Wilhelm T.I. Roos J. Kaczmarczyk R. Large language models for therapy recommendations across 3 clinical specialties: comparative study J Med Internet Res 25 2023 e49324 10.2196/49324 PMC10644179 37902826 · DOI
  38. [49]Rodman A. Zwaan L. Olson A. Manrai A.K. When it comes to benchmarks, humans are the only way NEJM AI 2 4 2025 AIe2500143 10.1056/AIe2500143 · DOI
  39. [50]Tariq R. Voth E. Khanna S. Integrating clinical guidelines with ChatGPT-4 enhances its’ skills Mayo Clin Proc Digit Health 2 2 2024 177 180 10.1016/j.mcpdig.2024.02.004 40207175 PMC11976014 · DOI

Cited by

Sorted by citations

Authors
Borgonovo, F., Matsuo, T., Petri, F., Amin Alavi, S. M., Mazudie Ndjonko, L. C., Gori, A., & Berbari, E. F.
Affiliations
  • Mayo Clinic (US)
  • University of Milan (IT)
Corresponding author
Berbari, Elie F.
ORCID
Venue
Mayo Clinic Proceedings: Digital Health
Publisher
Elsevier BV
Published
2025-09
Volume
3
Issue
3
Pages
100230
ISSN
2949-7612 (ISSN-L)
DOI
10.1016/j.mcpdig.2025.100230
PMC
PMC12205795
OpenAlex
W4410629445
Cited by
10
References
7
Citations by year
2026: 8 · 2025: 2
FWCI
2.41
Type
journal article
Language
en
Open access
Licence
Locations
  • Mayo Clinic Proceedings Digital Health (publishedVersion, other-oa)
  • Europe PMC (PubMed Central) (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • Archivio Istituzionale della Ricerca (Universita Degli Studi Di Milano) (submittedVersion, cc-by)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
  • National Center for Advancing Translational Sciences
  • National Institutes of Health
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Health Sciences › Medicine › Surgery › Orthopedic Infections and Treatments
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
Keywords
large language models, bone and joint infections, periprosthetic joint infection, diabetic foot infection, vertebral osteomyelitis, fracture-related infection, septic arthritis, clinical decision support, guideline adherence, LLM performance comparison, OpenEvidence, Microsoft Copilot, GPT-4o, Gemini

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review