The DOI matches its Crossref record; 40 of the 40 references it cites were found in a registry or an index.
GPT for RCTs?: Using AI to determine adherence to reporting guidelines
Open accessPreprint — published as 10.1136/bmjopen-2024-088735
Abstract
Background: Adherence to established reporting guidelines can improve clinical trial reporting standards, but attempts to improve adherence have produced mixed results. This exploratory study aimed to determine how accurate a Large Language Model generative AI system (AI-LLM) was for determining reporting guideline compliance in a sample of sports medicine clinical trial reports. Design and Methods: This study was an exploratory retrospective data analysis. The OpenAI GPT-4 and Meta LLama2 AI-LLMa were evaluated for their ability to determine reporting guideline adherence in a sample of 113 published sports medicine and exercise science clinical trial reports. For each paper, the GPT-4-Turbo and Llama 2 70B models were prompted to answer a series of nine reporting guideline questions about the text of the article. The GPT-4-Vision model was prompted to answer two additional reporting guideline questions about the participant flow diagram in a subset of articles. The dataset was randomly split (80/20) into a TRAIN and TEST dataset. Hyperparameter and fine-tuning were performed using the TRAIN dataset. The Llama2 model was fine-tuned using the data from the GPT-4-Turbo analysis of the TRAIN dataset. Primary outcome measure: Model performance (F1-score, classification accuracy) was assessed using the TEST dataset. Results: Across all questions about the article text, the GPT-4-Turbo AI-LLM demonstrated acceptable performance (F1-score = 0.89, accuracy[95% CI] = 90%[85-94%]). Accuracy for all reporting guidelines was > 80%. The Llama2 model accuracy was initially poor (F1-score = 0.63, accuracy[95%CI] = 64%[57-71%]), and improved with fine-tuning (F1-score = 0.84, accuracy[95%CI] = 83%[77-88%]). The GPT-4-Vision model accurately identified all participant flow diagrams (accuracy[95% CI] = 100%[89-100%]) but was less accurate at identifying when details were missing from the flow diagram (accuracy[95% CI] = 57%[39-73%]). Conclusions: Both the GPT-4 and fine-tuned Llama2 AI-LLMs showed promise as tools for assessing reporting guideline compliance. Next steps should include developing an efficent, open-source AI-LLM and exploring methods to improve model accuracy.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“a recent study used GPT-3.5 to check RCT reports on sports medicine and exercise science for adherence with 9 CONSORT checklist items, reporting accuracy in the range of 70–100%54.”
From Text classification models for assessing the completeness of randomized controlled trial publications based on CONSORT reporting guidelines (2024) · cites this paper as [54]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
CONSORT 2010 Statement: updated guidelines for reporting parallel group randomised trials
the CONSORT Group, Schulz, K. F., Altman, D. G., & Moher, D. (2010) · BMC Medicine · doi:10.1186/1741-7015-8-18 · Cited by 13,371
Self-citationMoher, D., Hopewell, S., Schulz, K. F., Montori, V., Gotzsche, P. C., Devereaux, P. J., Elbourne, D., Egger, M., & Altman, D. G. (2010) · BMJ · doi:10.1136/bmj.c869 · Cited by 8,883 · cited 1×
“The relevance of these questions for reporting standards is detailed elsewhere [16,25] Data labelling Each text-question pair was labelled with a single-word answer to the question: "YES” or “NO”.”
Self-citationTHE USE OF CONFIDENCE OR FIDUCIAL LIMITS ILLUSTRATED IN THE CASE OF THE BINOMIAL
CLOPPER, C. J., & PEARSON, E. S. (1934) · Biometrika · doi:10.1093/biomet/26.4.404 · Cited by 5,003 · cited 1×
“The model classification accuracy (the ratio of true positives to the total number of cases) and associated 95% confidence interval (95%CI, [27]) were also calculated.”
MethodHow to increase value and reduce waste when research priorities are set
Chalmers, I., Bracken, M. B., Djulbegovic, B., Garattini, S., Grant, J., Gülmezoglu, A. M., Howells, D. W., Ioannidis, J. P. A., & Oliver, S. (2014) · The Lancet · doi:10.1016/s0140-6736(13)62229-1 · Cited by 1,436
The scandal of poor medical research.
Altman, D. G. (1994) · BMJ · doi:10.1136/bmj.308.6924.283 · Cited by 897 · cited 1×
“Poor reporting of clinical trials negatively impacts care and is unethical [3,4,28,29].”
BackgroundTurner, L., Shamseer, L., Altman, D. G., Weeks, L., Peters, J., Kober, T., Dias, S., Schulz, K. F., Plint, A. C., & Moher, D. (2012) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000030.pub2 · Cited by 740
Self-citationMinimum information about clinical artificial intelligence modeling: the MI-CLAIM checklist.
Norgeot, B., Quer, G., Beaulieu-Jones, B. K., Torkamani, A., Dias, R., Gianfrancesco, M., Arnaout, R., Kohane, I. S., Saria, S., Topol, E., et al. (2020) · Nature Medicine · doi:10.1038/s41591-020-1041-y · Cited by 665
Turner, L., Shamseer, L., Altman, D. G., Schulz, K. F., & Moher, D. (2012) · Systematic Reviews · doi:10.1186/2046-4053-1-60 · Cited by 628 · cited 1×
“Medical journals often attempt to improve reporting standards by instructing authors to complete and submit reporting guideline checklists with their trial reports [11].”
BackgroundSelf-citationSimera, I., Moher, D., Hirst, A., Hoey, J., Schulz, K. F., & Altman, D. G. (2010) · BMC Medicine · doi:10.1186/1741-7015-8-24 · Cited by 590
Self-citationUnderreporting Research Is Scientific Misconduct
Chalmers, L. (1990) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.1990.03440100121018 · Cited by 551 · cited 2×
“Improving trial reporting, therefore, is an ethical imperative [4,5].”
Background
Cited by
Tran, V.-T., Gartlehner, G., Yaacoub, S., Boutron, I., Schwingshackl, L., Stadelmaier, J., Sommer, I., Alebouyeh, F., Afach, S., Meerpohl, J., et al. (2024) · Annals of Internal Medicine · doi:10.7326/m23-3389 · Cited by 66
Jiang, L., Lan, M., Menke, J. D., Vorland, C. J., & Kilicoglu, H. (2024) · Scientific Reports · doi:10.1038/s41598-024-72130-7 · Cited by 18
“5 to check RCT reports on sports medicine and exercise science for adherence with 9 CONSORT checklist items, reporting accuracy in the range of 70–100% 54 .”
Background(2024) · doi:10.1101/2024.03.31.24305138 · Cited by 18
“5 to check RCT reports on sports medicine and exercise science for adherence with 9 CONSORT checklist items, reporting accuracy in the range of 70-100% 54 .”
BackgroundCybersecurity Compliance for Operational Technology Using Large Language Models
Bolton, R., Sheikhfathollahi, M., Parkinson, S., et al. (2025) · IEEE Open Journal of the Computer Society · doi:10.1109/ojcs.2026.3684079 · Cited by 1
“An example, in the medical field, involves an investigation of how compliance can be assessed against reporting guidelines in clinical trials [26].”
Metadata
- Authors
- Wrightson, J. G., Blazey, P., Moher, D., Khan, K. M., & Ardern, C. L.
- Affiliations
- University of British Columbia (CA)
- University of Ottawa (CA)
- Corresponding author
- Wrightson, J.G.
- Venue
- BMJ Open
- Publisher
- openRxiv
- Published
- 2023-12-15
- Volume
- 15
- Issue
- 3
- Pages
- e088735-e088735
- ISSN
- 2044-6055 (ISSN-L)
- PMC
- PMC11927406
- OpenAlex
- W4408685817
- Cited by
- 24
- References
- 27
- Citations by year
- 2026: 10 · 2025: 12 · 2024: 2
- FWCI
- 13.97
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by-nc
- Locations
- BMJ Open (publishedVersion, cc-by-nc)
- medRxiv (acceptedVersion, cc-by-nc-nd)
- PubMed Central (publishedVersion, cc-by-nc)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Canadian Institutes of Health Research
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Keywords
- large language models, generative AI, GPT-4, Llama 2, trial reporting quality, F1 score
- MeSH terms
- Artificial Intelligence, Humans, Research Design, Retrospective Studies, Randomized Controlled Trials as Topic, Guideline Adherence
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: GPT for RCTs?: Using AI to determine adherence to reporting guidelines. https://app.cochat.ai/papers/10.1101/2023.12.14.23299971
- The paper
- Wrightson, J. G., Blazey, P., Moher, D., Khan, K. M., & Ardern, C. L. (2023). GPT for RCTs?: Using AI to determine adherence to reporting guidelines. BMJ Open. https://doi.org/10.1101/2023.12.14.23299971
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
