CoChat

A cross-sectional study on ChatGPT’s alignment with clinical practice guidelines in musculoskeletal rehabilitation

Open access
Safran, Ertuğrul, Yildirim, Sefa · 2025 · BMC Musculoskeletal Disorders

The DOI matches its Crossref record; 35 of the 35 references it cites were found in a registry or an index.

Abstract

BACKGROUND: AI models like ChatGPT have the potential to support musculoskeletal rehabilitation by providing clinical insights. However, their alignment with evidence-based guidelines needs evaluation before integration into physiotherapy practice. OBJECTIVE: To evaluate the performance of ChatGPT (GPT-4 model) in generating responses to musculoskeletal rehabilitation queries by comparing its recommendations with evidence-based clinical practice guidelines (CPGs). DESIGN: This study was designed as a cross-sectional observational study. METHODS: Twenty questions covering disease information, assessment, and rehabilitation were developed by two experienced physiotherapists specializing in musculoskeletal disorders. The questions were distributed across three anatomical regions: upper extremity (7 questions), lower extremity (9 questions), and spine (4 questions). ChatGPT's responses were obtained and evaluated independently by two raters using a 5-point Likert scale assessing relevance, accuracy, clarity, completeness, and consistency. Weighted kappa values were calculated to assess inter-rater agreement and consistency within each category. RESULTS: ChatGPT's responses received the highest average score for clarity (4.85), followed by accuracy (4.62), relevance (4.50), and completeness (4.20). Consistency received the lowest score (3.85). The highest agreement (weighted kappa = 0.90) was observed in the disease information category, whereas rehabilitation displayed relatively lower agreement (weighted kappa = 0.56). Variability in consistency and moderate weighted kappa values in relevance and clarity highlighted areas requiring improvement. CONCLUSIONS: This study demonstrates ChatGPT's potential in providing guideline-aligned information in musculoskeletal rehabilitation. However, due to observed limitations in consistency, completeness, and the ability to replicate nuanced clinical reasoning, its use should remain supplementary rather than as a primary decision-making tool. While it performed better in disease information, as evidenced by higher inter-rater agreement and scores, its performance in the rehabilitation category was comparatively lower, highlighting challenges in addressing complex, nuanced therapeutic interventions. This variability in consistency and domain-specific reasoning underscores the need for further refinement to ensure reliability in complex clinical scenarios. CLINICAL TRIAL NUMBER: Not applicable.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1186/s12891-025-08650-8

Cites

35 of 35 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

1 retracted

Show all 35 references

Research works printed but not linked: 1

  1. Jain K, Prajapati V. NLP/deep learning techniques in healthcare for decision making. Prim Health Care: Open Access. 2021;11(3):373–80.

Cited by

See all 25

Metadata

Authors
Safran, E., & Yildirim, S.
Affiliations
Bezmiâlem Vakıf Üniversitesi (TR)
Corresponding author
Safran, Ertuğrul
Venue
BMC Musculoskeletal Disorders
Published
2025-04-24
Volume
26
Issue
1
Pages
411-411
ISSN
1471-2474 (ISSN-L)
OpenAlex
W4409737737
Cited by
21
References
35
Citations by year
2026: 15 · 2025: 6
FWCI
5.02
Type
journal article
Language
en
Open access
Licence
Locations
  • BMC Musculoskeletal Disorders (publishedVersion, cc-by-nc-nd)
  • Europe PMC (PubMed Central) (publishedVersion, cc-by-nc-nd)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Clinical practice guidelines implementation
  • Health Sciences › Medicine › Rehabilitation › Musculoskeletal Disorders and Rehabilitation
Keywords
ChatGPT, large language models, musculoskeletal rehabilitation, physiotherapy, guideline alignment, upper limb, lower extremity, spine, interobserver agreement, clinical reasoning
MeSH terms
Generative Artificial Intelligence, Cross-Sectional Studies, Humans, Musculoskeletal Diseases, Practice Guidelines as Topic, Physical Therapy Modalities

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: A cross-sectional study on ChatGPT’s alignment with clinical practice guidelines in musculoskeletal rehabilitation. https://app.cochat.ai/papers/10.1186/s12891-025-08650-8
The paper
Safran, E., & Yildirim, S. (2025). A cross-sectional study on ChatGPT’s alignment with clinical practice guidelines in musculoskeletal rehabilitation. BMC Musculoskeletal Disorders. https://doi.org/10.1186/s12891-025-08650-8

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.