Student or educator? Sign up with your school email and get 6 months of Pro free.

The DOI matches its Crossref record; 40 of the 40 references it cites were found in a registry or an index.

GPT for RCTs?: Using AI to determine adherence to reporting guidelines

Open access

Preprint — published as 10.1136/bmjopen-2024-088735

Abstract

Background: Adherence to established reporting guidelines can improve clinical trial reporting standards, but attempts to improve adherence have produced mixed results. This exploratory study aimed to determine how accurate a Large Language Model generative AI system (AI-LLM) was for determining reporting guideline compliance in a sample of sports medicine clinical trial reports. Design and Methods: This study was an exploratory retrospective data analysis. The OpenAI GPT-4 and Meta LLama2 AI-LLMa were evaluated for their ability to determine reporting guideline adherence in a sample of 113 published sports medicine and exercise science clinical trial reports. For each paper, the GPT-4-Turbo and Llama 2 70B models were prompted to answer a series of nine reporting guideline questions about the text of the article. The GPT-4-Vision model was prompted to answer two additional reporting guideline questions about the participant flow diagram in a subset of articles. The dataset was randomly split (80/20) into a TRAIN and TEST dataset. Hyperparameter and fine-tuning were performed using the TRAIN dataset. The Llama2 model was fine-tuned using the data from the GPT-4-Turbo analysis of the TRAIN dataset. Primary outcome measure: Model performance (F1-score, classification accuracy) was assessed using the TEST dataset. Results: Across all questions about the article text, the GPT-4-Turbo AI-LLM demonstrated acceptable performance (F1-score = 0.89, accuracy[95% CI] = 90%[85-94%]). Accuracy for all reporting guidelines was > 80%. The Llama2 model accuracy was initially poor (F1-score = 0.63, accuracy[95%CI] = 64%[57-71%]), and improved with fine-tuning (F1-score = 0.84, accuracy[95%CI] = 83%[77-88%]). The GPT-4-Vision model accurately identified all participant flow diagrams (accuracy[95% CI] = 100%[89-100%]) but was less accurate at identifying when details were missing from the flow diagram (accuracy[95% CI] = 57%[39-73%]). Conclusions: Both the GPT-4 and fine-tuned Llama2 AI-LLMs showed promise as tools for assessing reporting guideline compliance. Next steps should include developing an efficent, open-source AI-LLM and exploring methods to improve model accuracy.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1101/2023.12.14.23299971

Cites

40 of 40 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1 Confusion matrix from the analysis of the TEST dataset. Actual (y-axis) represents the answers given by the GPT-4 Turbo model, and predicted (x-axis) represents the labels for each question extracted from Schulz et al.19
Figure 1 Confusion matrix from the analysis of the TEST dataset. Actual (y-axis) represents the answers given by the GPT-4 Turbo model, and predicted (x-axis) represents the labels for each question extracted from Schulz et al.19CC BY-NC · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 27 references

Cited by

See all 24

Metadata

Authors
Wrightson, J. G., Blazey, P., Moher, D., Khan, K. M., & Ardern, C. L.
Affiliations
  • University of British Columbia (CA)
  • University of Ottawa (CA)
Corresponding author
Wrightson, J.G.
Venue
BMJ Open
Publisher
openRxiv
Published
2023-12-15
Volume
15
Issue
3
Pages
e088735-e088735
ISSN
2044-6055 (ISSN-L)
OpenAlex
W4408685817
Cited by
24
References
27
Citations by year
2026: 10 · 2025: 12 · 2024: 2
FWCI
13.97
Type
journal article
Language
en
Open access
Licence
cc-by-nc
Locations
  • BMJ Open (publishedVersion, cc-by-nc)
  • medRxiv (acceptedVersion, cc-by-nc-nd)
  • PubMed Central (publishedVersion, cc-by-nc)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
Canadian Institutes of Health Research
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
Keywords
large language models, generative AI, GPT-4, Llama 2, trial reporting quality, F1 score
MeSH terms
Artificial Intelligence, Humans, Research Design, Retrospective Studies, Randomized Controlled Trials as Topic, Guideline Adherence

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: GPT for RCTs?: Using AI to determine adherence to reporting guidelines. https://app.cochat.ai/papers/10.1101/2023.12.14.23299971
The paper
Wrightson, J. G., Blazey, P., Moher, D., Khan, K. M., & Ardern, C. L. (2023). GPT for RCTs?: Using AI to determine adherence to reporting guidelines. BMJ Open. https://doi.org/10.1101/2023.12.14.23299971

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.