CoChat
Sign Up
Cited by 37Artificial Intelligence in Healthcare and EducationClinical practice guidelines implementationArtificial Intelligence in Healthcare

ChatGPT versus NASS clinical guidelines for degenerative spondylolisthesis: a comparative analysis.

Ahmed, Wasil, Saturno, Michael, Rajjoub, Rami, Duey, Akiro H., Zaidat, Bashar, Hoang, Timothy, Restrepo Mejia, Mateo, Gallate, Zachary S. +5 more·2024·European Spine Journal
DOI 10.1007/s00586-024-08198-6
Open this paper in CoChat — free account

Abstract

Background context Clinical guidelines, developed in concordance with the literature, are often used to guide surgeons' clinical decision making. Recent advancements of large language models and artificial intelligence (AI) in the medical field come with exciting potential. OpenAI's generative AI model, known as ChatGPT, can quickly synthesize information and generate responses grounded in medical literature, which may prove to be a useful tool in clinical decision-making for spine care. The current literature has yet to investigate the ability of ChatGPT to assist clinical decision making with regard to degenerative spondylolisthesis.Purpose The study aimed to compare ChatGPT's concordance with the recommendations set forth by The North American Spine Society (NASS) Clinical Guideline for the Diagnosis and Treatment of Degenerative Spondylolisthesis and assess ChatGPT's accuracy within the context of the most recent literature.Methods ChatGPT-3.5 and 4.0 was prompted with questions from the NASS Clinical Guideline for the Diagnosis and Treatment of Degenerative Spondylolisthesis and graded its recommendations as "concordant" or "nonconcordant" relative to those put forth by NASS. A response was considered "concordant" when ChatGPT generated a recommendation that accurately reproduced all major points made in the NASS recommendation. Any responses with a grading of "nonconcordant" were further stratified into two subcategories: "Insufficient" or "Over-conclusive," to provide further insight into grading rationale. Responses between GPT-3.5 and 4.0 were compared using Chi-squared tests.Results ChatGPT-3.5 answered 13 of NASS's 28 total clinical questions in concordance with NASS's guidelines (46.4%). Categorical breakdown is as follows: Definitions and Natural History (1/1, 100%), Diagnosis and Imaging (1/4, 25%), Outcome Measures for Medical Intervention and Surgical Treatment (0/1, 0%), Medical and Interventional Treatment (4/6, 66.7%), Surgical Treatment (7/14, 50%), and Value of Spine Care (0/2, 0%). When NASS indicated there was sufficient evidence to offer a clear recommendation, ChatGPT-3.5 generated a concordant response 66.7% of the time (6/9). However, ChatGPT-3.5's concordance dropped to 36.8% when asked clinical questions that NASS did not provide a clear recommendation on (7/19). A further breakdown of ChatGPT-3.5's nonconcordance with the guidelines revealed that a vast majority of its inaccurate recommendations were due to them being "over-conclusive" (12/15, 80%), rather than "insufficient" (3/15, 20%). ChatGPT-4.0 answered 19 (67.9%) of the 28 total questions in concordance with NASS guidelines (P = 0.177). When NASS indicated there was sufficient evidence to offer a clear recommendation, ChatGPT-4.0 generated a concordant response 66.7% of the time (6/9). ChatGPT-4.0's concordance held up at 68.4% when asked clinical questions that NASS did not provide a clear recommendation on (13/19, P = 0.104).Conclusions This study sheds light on the duality of LLM applications within clinical settings: one of accuracy and utility in some contexts versus inaccuracy and risk in others. ChatGPT was concordant for most clinical questions NASS offered recommendations for. However, for questions NASS did not offer best practices, ChatGPT generated answers that were either too general or inconsistent with the literature, and even fabricated data/citations. Thus, clinicians should exercise extreme caution when attempting to consult ChatGPT for clinical recommendations, taking care to ensure its reliability within the context of recent literature.

Integrity signals

Registry

Crossref · 10.1007/s00586-024-08198-6

Cites

34 of 34 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Several chatbots have experienced inconsistent accuracy by clinical practice guideline standards.

    “Currently, few studies use clinical practice guidelines to define successful performance, and several chatbots have experienced inconsistent accuracy by these standards.168,169”

    From Large Language Models for Chatbot Health Advice Studies: A Systematic Review (2025) · cites this paper as [168]

Cites

Flagged first, then by citations

Cited by

Sorted by citations

Related

Authors
Ahmed, W., Saturno, M., Rajjoub, R., Duey, A. H., Zaidat, B., Hoang, T., Restrepo Mejia, M., Gallate, Z. S., Shrestha, N., Tang, J., Zapolsky, I., Kim, J. S., & Cho, S. K.
Affiliations
  • Icahn School of Medicine at Mount Sinai (US)
  • Rosalind Franklin University of Medicine and Science (US)
Corresponding author
Cho, Samuel K.
ORCID
Venue
European Spine Journal
Publisher
Springer Science and Business Media LLC
Published
2024-03-15
Volume
33
Issue
11
Pages
4182-4203
ISSN
  • 0940-6719 (ISSN-L)
  • 1432-0932
DOI
10.1007/s00586-024-08198-6
OpenAlex
W4392859069
Cited by
37
References
34
Citations by year
2026: 11 · 2025: 21 · 2024: 5
FWCI
3.47
Type
journal article
Language
en
Open access
closed
Licence
Locations
  • European Spine Journal (publishedVersion)
  • PubMed (publishedVersion)
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Clinical practice guidelines implementation
  • Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
Keywords
ChatGPT, large language models, degenerative spondylolisthesis, guideline adherence, clinical decision-making, artificial intelligence in healthcare, spinal surgery, intervertebral disc degeneration, outcome measures, natural history
MeSH terms
Clinical Decision-Making, Artificial Intelligence, Humans, Spondylolisthesis, Practice Guidelines as Topic

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review