GPT for RCTs?: Using AI to determine adherence to reporting guidelines
Open accessPreprint — published as 10.1136/bmjopen-2024-088735 · Paper page
The DOI matches its Crossref record; 40 of the 40 references it cites were found in a registry or an index.
Abstract
Background: Adherence to established reporting guidelines can improve clinical trial reporting standards, but attempts to improve adherence have produced mixed results. This exploratory study aimed to determine how accurate a Large Language Model generative AI system (AI-LLM) was for determining reporting guideline compliance in a sample of sports medicine clinical trial reports. Design and Methods: This study was an exploratory retrospective data analysis. The OpenAI GPT-4 and Meta LLama2 AI-LLMa were evaluated for their ability to determine reporting guideline adherence in a sample of 113 published sports medicine and exercise science clinical trial reports. For each paper, the GPT-4-Turbo and Llama 2 70B models were prompted to answer a series of nine reporting guideline questions about the text of the article. The GPT-4-Vision model was prompted to answer two additional reporting guideline questions about the participant flow diagram in a subset of articles. The dataset was randomly split (80/20) into a TRAIN and TEST dataset. Hyperparameter and fine-tuning were performed using the TRAIN dataset. The Llama2 model was fine-tuned using the data from the GPT-4-Turbo analysis of the TRAIN dataset. Primary outcome measure: Model performance (F1-score, classification accuracy) was assessed using the TEST dataset. Results: Across all questions about the article text, the GPT-4-Turbo AI-LLM demonstrated acceptable performance (F1-score = 0.89, accuracy[95% CI] = 90%[85-94%]). Accuracy for all reporting guidelines was > 80%. The Llama2 model accuracy was initially poor (F1-score = 0.63, accuracy[95%CI] = 64%[57-71%]), and improved with fine-tuning (F1-score = 0.84, accuracy[95%CI] = 83%[77-88%]). The GPT-4-Vision model accurately identified all participant flow diagrams (accuracy[95% CI] = 100%[89-100%]) but was less accurate at identifying when details were missing from the flow diagram (accuracy[95% CI] = 57%[39-73%]). Conclusions: Both the GPT-4 and fine-tuned Llama2 AI-LLMs showed promise as tools for assessing reporting guideline compliance. Next steps should include developing an efficent, open-source AI-LLM and exploring methods to improve model accuracy.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“a recent study used GPT-3.5 to check RCT reports on sports medicine and exercise science for adherence with 9 CONSORT checklist items, reporting accuracy in the range of 70–100%54.”
From Text classification models for assessing the completeness of randomized controlled trial publications based on CONSORT reporting guidelines (2024) · cites this paper as [54]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
CONSORT 2010 Statement: updated guidelines for reporting parallel group randomised trials
the CONSORT Group, Schulz, K. F., Altman, D. G., & Moher, D. (2010) · BMC Medicine · doi:10.1186/1741-7015-8-18 · Cited by 13,371
Self-citationMoher, D., Hopewell, S., Schulz, K. F., Montori, V., Gotzsche, P. C., Devereaux, P. J., Elbourne, D., Egger, M., & Altman, D. G. (2010) · BMJ · doi:10.1136/bmj.c869 · Cited by 8,883 · cited 1×
“The relevance of these questions for reporting standards is detailed elsewhere [16,25] Data labelling Each text-question pair was labelled with a single-word answer to the question: "YES” or “NO”.”
Self-citationTHE USE OF CONFIDENCE OR FIDUCIAL LIMITS ILLUSTRATED IN THE CASE OF THE BINOMIAL
CLOPPER, C. J., & PEARSON, E. S. (1934) · Biometrika · doi:10.1093/biomet/26.4.404 · Cited by 5,003 · cited 1×
“The model classification accuracy (the ratio of true positives to the total number of cases) and associated 95% confidence interval (95%CI, [27]) were also calculated.”
MethodHow to increase value and reduce waste when research priorities are set
Chalmers, I., Bracken, M. B., Djulbegovic, B., Garattini, S., Grant, J., Gülmezoglu, A. M., Howells, D. W., Ioannidis, J. P. A., & Oliver, S. (2014) · The Lancet · doi:10.1016/s0140-6736(13)62229-1 · Cited by 1,436
The scandal of poor medical research.
Altman, D. G. (1994) · BMJ · doi:10.1136/bmj.308.6924.283 · Cited by 897 · cited 1×
“Poor reporting of clinical trials negatively impacts care and is unethical [3,4,28,29].”
BackgroundTurner, L., Shamseer, L., Altman, D. G., Weeks, L., Peters, J., Kober, T., Dias, S., Schulz, K. F., Plint, A. C., & Moher, D. (2012) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000030.pub2 · Cited by 740
Self-citationMinimum information about clinical artificial intelligence modeling: the MI-CLAIM checklist.
Norgeot, B., Quer, G., Beaulieu-Jones, B. K., Torkamani, A., Dias, R., Gianfrancesco, M., Arnaout, R., Kohane, I. S., Saria, S., Topol, E., et al. (2020) · Nature Medicine · doi:10.1038/s41591-020-1041-y · Cited by 665
Turner, L., Shamseer, L., Altman, D. G., Schulz, K. F., & Moher, D. (2012) · Systematic Reviews · doi:10.1186/2046-4053-1-60 · Cited by 628 · cited 1×
“Medical journals often attempt to improve reporting standards by instructing authors to complete and submit reporting guideline checklists with their trial reports [11].”
BackgroundSelf-citationSimera, I., Moher, D., Hirst, A., Hoey, J., Schulz, K. F., & Altman, D. G. (2010) · BMC Medicine · doi:10.1186/1741-7015-8-24 · Cited by 590
Self-citationUnderreporting Research Is Scientific Misconduct
Chalmers, L. (1990) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.1990.03440100121018 · Cited by 551 · cited 2×
“Improving trial reporting, therefore, is an ethical imperative [4,5].”
BackgroundIncreasing value and reducing waste in biomedical research: who's listening?
Moher, D., Glasziou, P., Chalmers, I., Nasser, M., Bossuyt, P. M. M., Korevaar, D. A., Graham, I. D., Ravaud, P., & Boutron, I. (2016) · The Lancet · doi:10.1016/s0140-6736(15)00307-4 · Cited by 462
Self-citationHopewell, S., Ravaud, P., Baron, G., & Boutron, I. (2012) · BMJ · doi:10.1136/bmj.e4178 · Cited by 235 · cited 1×
“Using reporting guidelines, such as the CONsolidated Standards of Reporting Trials (CONSORT), can improve trial reporting standards [6–8], but adherence is often poor [9].”
MethodA systematic scoping review of adherence to reporting guidelines in health care literature
Thabane, L., Samaan, Mbuagbaw, L., Kosa, S., Debono, B., Dillenburg, Zhang, Fruci, V., Dennis, & Bawor (2013) · Journal of Multidisciplinary Healthcare · doi:10.2147/jmdh.s43952 · Cited by 192
Dechartres, A., Trinquart, L., Atal, I., Moher, D., Dickersin, K., Boutron, I., Perrodeau, E., Altman, D. G., & Ravaud, P. (2017) · BMJ · doi:10.1136/bmj.j2490 · Cited by 171
Self-citationScoping review on interventions to improve adherence to reporting guidelines in health research
Blanco, D., Altman, D., Moher, D., Boutron, I., Kirkham, J. J., & Cobo, E. (2019) · BMJ Open · doi:10.1136/bmjopen-2018-026589 · Cited by 154 · cited 4×
“%) of author-submitted CONSORT checklists reported by Blanco et al. (2019) [12].”
BackgroundMethodInfluentialSelf-citationArtificial intelligence hallucinations
Salvagno, M., Taccone, F. S., & Gerli, A. G. (2023) · Critical Care · doi:10.1186/s13054-023-04473-y · Cited by 136 · cited 1×
“The less-than-perfect accuracy of the model and the tendency of the current generation of AI-LLMs to “hallucinate” content [31] means that human confirmation of compliance is required.”
BackgroundEvidence based medicine manifesto for better healthcare
Heneghan, C., Mahtani, K. R., Goldacre, B., Godlee, F., Macdonald, H., & Jarvies, D. (2017) · BMJ · doi:10.1136/bmj.j2973 · Cited by 120
Mesquida, C., Murphy, J., Lakens, D., & Warne, J. P. (2022) · Royal Society Open Science · doi:10.1098/rsos.220946 · Cited by 116 · cited 1×
“Reporting in sports medicine and exercise science papers is often inadequate, and there are concerns about the reproducibility and veracity of many findings [16–20].”
BackgroundConsortium for Transparency in Exercise Science (COTES) Collaborators, Caldwell, A. R., Vigotsky, A. D., Tenan, M. S., Radel, R., Mellor, D. T., Kreutzer, A., Lahart, I. M., Mills, J. P., & Boisgontier, M. P. (2020) · Sports Medicine · doi:10.1007/s40279-019-01227-1 · Cited by 114
Duff, J. M., Leather, H., Walden, E. O., LaPlant, K. D., & George, T. J. (2010) · JNCI: Journal of the National Cancer Institute · doi:10.1093/jnci/djq117 · Cited by 103 · cited 2×
“Poor reporting of clinical trials is common [1], threatens the reliability and credibility of medical research [2] and affects patient care [3].”
BackgroundPandis, N., Shamseer, L., Kokich, V. G., Fleming, P. S., & Moher, D. (2014) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2014.04.001 · Cited by 92
Self-citationEvaluating the Robustness of Neural Language Models to Input Perturbations
Moradi, M., & Samwald, M. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.117 · Cited by 69 · cited 1×
“Model robustness/sensitivity was assessed using text perturbations [26] (see Supplementary material for rationale and methods).”
MethodMethods to assess research misconduct in health-related research: A scoping review
Bordewijk, E. M., Li, W., van Eekelen, R., Wang, R., Showell, M., Mol, B. W., & van Wely, M. (2021) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2021.05.012 · Cited by 68 · cited 1×
“There are many automated tools that journal publishers and editors use to screen manuscripts for errors and misconduct [30].”
BackgroundMiRoR network, Blanco, D., Biggane, A. M., & Cobo, E. (2018) · Trials · doi:10.1186/s13063-018-2475-0 · Cited by 68 · cited 2×
“%) of author-submitted CONSORT checklists reported by Blanco et al. (2019) [12].”
BackgroundMethodBlanco, D., Schroter, S., Aldcroft, A., Moher, D., Boutron, I., Kirkham, J. J., & Cobo, E. (2020) · BMJ Open · doi:10.1136/bmjopen-2020-036799 · Cited by 62 · cited 2×
“For example, some of the ‘core’ CONSORT questions [24] require analysis of the CONSORT flow figures.”
BackgroundSelf-citationCONSORT endorsement improves the quality of reports of randomized clinical trials in dentistry
Sarkis-Onofre, R., Poletto-Neto, V., Cenci, M. S., Moher, D., & Pereira-Cenci, T. (2020) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2020.01.020 · Cited by 59
Self-citationHeneghan, C., Mahtani, K. R., Goldacre, B. M., Godlee, F., Macdonald, H., & Jarvies, D. (2017) · BMJ Evidence-Based Medicine · doi:10.1136/ebmed-2017-j2973rep · Cited by 55 · cited 1×
“Poor reporting of clinical trials negatively impacts care and is unethical [3,4,28,29].”
BackgroundImpact of Pretraining Term Frequencies on Few-Shot Reasoning
Razeghi, Y., Logan, R. L., Gardner, M., & Singh, S. (2022) · arXiv · doi:10.48550/arxiv.2202.07206 · Cited by 53
Twomey, R., Harlley, S., Medina, C. R., Murphy, J., Atkins, W. C., McCrum, C. L., Schneider, C., Warne, J. P., Yingling, V. R., & Caldwell, A. R. (2021) · Communications in Kinesiology · doi:10.51224/cik.v1i3.43 · Cited by 43
Speich, B., Mann, E., Schönenberger, C. M., Mellor, K., Griessbach, A. N., Dhiman, P., Gandhi, P., Lohner, S., Agarwal, A., Odutayo, A., et al. (2023) · JAMA Network Open · doi:10.1001/jamanetworkopen.2023.17651 · Cited by 41 · cited 3×
“Other recent attempts to improve reporting standards involve training peer reviewers, authors, or editors, but results have been mixed [13,14].”
BackgroundSelf-citationMcErlean, M., Samways, J., Godolphin, P. J., & Chen, Y. (2022) · Irish Journal of Medical Science (1971 -) · doi:10.1007/s11845-022-02955-6 · Cited by 38
Ioannidis, J. P. A., Berkwits, M., Flanagin, A., & Bloom, T. (2023) · JAMA · doi:10.1001/jama.2023.17607 · Cited by 25 · cited 1×
“Following recent calls to evaluate the role of Artificial Intelligence (AI) in facilitating editorial and peer review decisions [10], we assessed how well an AI model could determine reporting guideline adherence in clinical trial reports.”
MethodInfluentialWang, F., Schilsky, R. L., Page, D., Califf, R. M., Cheung, K., Wang, X., & Pang, H. (2020) · JAMA Network Open · doi:10.1001/jamanetworkopen.2020.14661 · Cited by 24
Schulz, R., Langen, G., Prill, R., Cassel, M., & Weissgerber, T. L. (2022) · BMJ Open · doi:10.1136/bmjopen-2021-059347 · Cited by 23 · cited 4×
“Improving reporting practices in sports medicine should be a priority for researchers and publishers [16,21].”
BackgroundInfluentialReviewerGPT? An Exploratory Study on Using Large Language Models for Paper Reviewing
Liu, R., & Shah, N. B. (2023) · arXiv · doi:10.48550/arxiv.2306.00622 · Cited by 23 · cited 4×
“Instead, we followed the example of Liu and Shah [15] and split each paper into three sections: the Introduction, Method and Results.”
BackgroundMethodResultInfluentialWang, P., Wolfram, D., & Gilbert, E. (2024) · PLoS ONE · doi:10.1371/journal.pone.0299806 · Cited by 15
Open and transparent sports science research: the role of journals to move the field forward
Hansford, H. J., Cashin, A. G., Wewege, M. A., Ferraro, M. C., McAuley, J. H., Jones, M. D., & Collaborators, T. T. S. S. (2022) · Knee Surgery Sports Traumatology Arthroscopy · doi:10.1007/s00167-022-06893-9 · Cited by 14 · cited 1×
“Improving reporting practices in sports medicine should be a priority for researchers and publishers [16,21].”
BackgroundBullock, G., Ward, P., Impellizzeri, F., Kluzek, S., Hughes, T., Hillman, C., Waterman, B., Danelson, K., Henry, K., Barr, E. T., et al. (2023) · medRxiv · doi:10.1101/2023.03.30.23287959 · Cited by 14
Murphy, J., Mesquida, C., & Warne, J. P. (2023) · Communications in Kinesiology · doi:10.51224/cik.2023.53 · Cited by 12
OpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.08774
Cited by
Tran, V.-T., Gartlehner, G., Yaacoub, S., Boutron, I., Schwingshackl, L., Stadelmaier, J., Sommer, I., Alebouyeh, F., Afach, S., Meerpohl, J., et al. (2024) · Annals of Internal Medicine · doi:10.7326/m23-3389 · Cited by 66
(2024) · doi:10.1101/2024.03.31.24305138 · Cited by 18
“5 to check RCT reports on sports medicine and exercise science for adherence with 9 CONSORT checklist items, reporting accuracy in the range of 70-100% 54 .”
BackgroundJiang, L., Lan, M., Menke, J. D., Vorland, C. J., & Kilicoglu, H. (2024) · Scientific Reports · doi:10.1038/s41598-024-72130-7 · Cited by 18
“5 to check RCT reports on sports medicine and exercise science for adherence with 9 CONSORT checklist items, reporting accuracy in the range of 70–100% 54 .”
BackgroundCybersecurity Compliance for Operational Technology Using Large Language Models
Bolton, R., Sheikhfathollahi, M., Parkinson, S., et al. (2025) · IEEE Open Journal of the Computer Society · doi:10.1109/ojcs.2026.3684079 · Cited by 1
“An example, in the medical field, involves an investigation of how compliance can be assessed against reporting guidelines in clinical trials [26].”
Metadata
- Authors
- Wrightson, J. G., Blazey, P., Moher, D., Khan, K. M., & Ardern, C. L.
- Affiliations
- University of British Columbia (CA)
- University of Ottawa (CA)
- Corresponding author
- Wrightson, J.G.
- Venue
- BMJ Open
- Publisher
- openRxiv
- Published
- 2023-12-15
- Volume
- 15
- Issue
- 3
- Pages
- e088735-e088735
- ISSN
- 2044-6055 (ISSN-L)
- PMC
- PMC11927406
- OpenAlex
- W4408685817
- Cited by
- 24
- References
- 27
- Citations by year
- 2026: 10 · 2025: 12 · 2024: 2
- FWCI
- 13.97
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by-nc
- Locations
- BMJ Open (publishedVersion, cc-by-nc)
- medRxiv (acceptedVersion, cc-by-nc-nd)
- PubMed Central (publishedVersion, cc-by-nc)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Canadian Institutes of Health Research
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Keywords
- large language models, generative AI, GPT-4, Llama 2, trial reporting quality, F1 score
- MeSH terms
- Artificial Intelligence, Humans, Research Design, Retrospective Studies, Randomized Controlled Trials as Topic, Guideline Adherence
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
