Student or educator? Sign up with your school email and get 6 months of Pro free.

The DOI matches its Crossref record; 37 of the 40 references it cites were found in a registry or an index; 19 of the 43 sources its claims cite were checked for support.

Fabricated or accurate? Ethical concerns and citation hallucination in aI-generated scientific writing on musculoskeletal topics

Open access
Safran, Ertuğrul, Çalı, Adem·2025·Anatolian Current Medical Journal

Abstract

Aims: Large language models (LLMs) such as ChatGPT are increasingly used in academic and clinical writing. While these tools can generate coherent and domain-specific text, concerns persist regarding the accuracy of their automatically generated references. In musculoskeletal rehabilitation—a field heavily reliant on current evidence—the reliability of citations is especially critical. Yet, systematic evaluations of citation accuracy in AI-generated scientific content are lacking. To evaluate the reference accuracy of scientific texts generated by ChatGPT (GPT-4) in response to musculoskeletal rehabilitation prompts, and to determine whether reference accuracy improves following structured post-generation verification. Methods: ChatGPT was prompted to generate four scientific paragraphs on musculoskeletal rehabilitation topics (manual therapy, ACL reconstruction, low back pain, and rotator cuff repair), each including 10 references with DOIs. A total of 40 references were analyzed using a 3-point scoring system (0=fabricated, 1=partially correct, 2=fully accurate), which was used to assess citation quality. After initial evaluation, ChatGPT was asked to verify and revise its references. Scores before and after this step were compared descriptively and with Wilcoxon signed-rank tests to assess statistical significance, and effect sizes (r) were calculated to estimate the magnitude of improvement. Results: Only 7.5% of references were fully accurate in the initial generation, while 42.5% were completely fabricated. The remaining 50% were partially correct. After verification, the proportion of fully accurate references rose to 77.5%. Wilcoxon signed-rank testing confirmed a statistically significant improvement in accuracy across all prompts (W=561.0, p

Overview

CoChat Summary

TL;DR

This study evaluated citation accuracy in ChatGPT (GPT-4)-generated musculoskeletal rehabilitation texts and found that most initial references were inaccurate or fabricated. A structured post-generation verification prompt substantially improved reference accuracy, but the authors conclude that LLM-assisted writing still requires strict human-led citation verification.

Key findings

  • In the initial generation, only 3/40 references (7.5%) were fully accurate, 20/40 (50%) were partially correct, and 17/40 (42.5%) were completely fabricated or had major errors.
  • After ChatGPT was prompted to verify and revise its references, 31/40 (77.5%) were fully accurate; median accuracy increased from 1 (IQR 0–1) to 2 (IQR 2–2), with Wilcoxon signed-rank test W=561.0, p<0.001, r=0.60. The figure reports fabricated references declined from 42.5% to 2.5%.
  • Prompt-level improvements were statistically significant: manual therapy W=32.5, p=0.017; ACL reconstruction W=45.0, p=0.003; low back pain W=48.0, p=0.019; rotator cuff repair W=34.5, p=0.008.
  • Common errors included invalid DOIs, fabricated article titles, and mismatched metadata, often combining real author names with invented bibliographic details. Baseline accuracy was lowest for rotator cuff repair, where only 1/10 references (10%) was fully accurate initially.

Methods

  • Structured observational analysis using four ChatGPT (GPT-4) prompts on manual therapy, ACL reconstruction, low back pain, and rotator cuff repair; each prompt requested a 300–400 word scientific paragraph with 10 references including DOIs, yielding 40 references. Prompts were administered sequentially in fixed order in separate, context-cleared sessions via the OpenAI web interface in June 2025.
  • References were scored 0=fabricated or completely mismatched, 1=partially correct, 2=fully accurate. One researcher designed prompts and generated content; another independently evaluated references and DOIs, with manual verification via Crossref, PubMed, Google Scholar, and publisher websites when DOIs did not resolve.
  • Descriptive statistics compared score frequencies before and after a verification prompt; Wilcoxon signed-rank tests assessed changes in accuracy, with effect sizes (r) calculated. Statistical significance was set at p<0.05, using IBM SPSS Statistics 25.

Limitations

  • The paper states that the sample was limited to 40 references across four prompts, which may limit generalizability; the prompts do not cover the full scope of musculoskeletal rehabilitation.
  • The paper states that reference accuracy was assessed manually and remains subject to interpretation bias despite predefined criteria, and that the verification step relied on ChatGPT’s own response to follow-up prompts rather than external validation.
  • The paper states that only GPT-4 was tested, so findings may differ across other models or platforms. The text also makes evident that prompts were not randomized and that the evaluator’s blinding to condition or verification status is not reported.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.38053/acmj.1746227

Cites

37 of 40 references verified0 citation errors · 3 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

12 Valid · 0 Not valid · 10 Not checked · 79 Own results and interpretations

56 more claims · 1 with their cited sources checked

Create a free account to see all 101 claims and every source verdict.

See all claims — free account
  • Probably not a substantial claim. 23
  • It may state two separate claims as one. 27

Create a free account to see these claims.

Citations

Cites

Show all 32 references

Cited by

See all 4

Metadata

Authors
Safran, E., & Çalı, A.
Affiliations
  • Bezmiâlem Vakıf Üniversitesi (TR)
  • Istanbul Topkapi University (TR)
Venue
Anatolian Current Medical Journal
Published
2025-09-15
Volume
7
Issue
5
Pages
695-702
ISSN
2718-0115 (ISSN-L)
OpenAlex
W4414428938
Cited by
4
References
32
Citations by year
2026: 4
FWCI
0.97
Type
journal article
Language
en
Open access
Licence
cc-by-nc-nd
Locations
  • Anatolian Current Medical Journal (publishedVersion, cc-by-nc-nd)
  • DergiPark (Istanbul University) (submittedVersion)
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Social Sciences › Library and Information Sciences › Academic Writing and Publishing
  • Social Sciences › Education › Academic integrity and plagiarism
Keywords
citation hallucination, ChatGPT, large language models, citation accuracy, musculoskeletal rehabilitation, manual therapy, anterior cruciate ligament reconstruction, low back pain, rotator cuff repair, research ethics

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol and qwen3.8-omni-flash).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Fabricated or accurate? Ethical concerns and citation hallucination in aI-generated scientific writing on musculoskeletal topics. https://app.cochat.ai/papers/10.38053/acmj.1746227
The paper
Safran, E., & Çalı, A. (2025). Fabricated or accurate? Ethical concerns and citation hallucination in aI-generated scientific writing on musculoskeletal topics. Anatolian Current Medical Journal. https://doi.org/10.38053/acmj.1746227

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.