The DOI matches its Crossref record; 89 of the 89 references it cites were found in a registry or an index.
Concordance of randomised controlled trials for artificial intelligence interventions with the CONSORT-AI reporting guidelines
Open accessAbstract
The Consolidated Standards of Reporting Trials extension for Artificial Intelligence interventions (CONSORT-AI) was published in September 2020. Since its publication, several randomised controlled trials (RCTs) of AI interventions have been published but their completeness and transparency of reporting is unknown. This systematic review assesses the completeness of reporting of AI RCTs following publication of CONSORT-AI and provides a comprehensive summary of RCTs published in recent years. 65 RCTs were identified, mostly conducted in China (37%) and USA (18%). Median concordance with CONSORT-AI reporting was 90% (IQR 77–94%), although only 10 RCTs explicitly reported its use. Several items were consistently under-reported, including algorithm version, accessibility of the AI intervention or code, and references to a study protocol. Only 3 of 52 included journals explicitly endorsed or mandated CONSORT-AI. Despite a generally high concordance amongst recent AI RCTs, some AI-specific considerations remain systematically poorly reported. Further encouragement of CONSORT-AI adoption by journals and funders may enable more complete adoption of the full CONSORT-AI guidelines.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures



Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Second, there are also concerns about the quality and transparency of published clinical trials, which often lack the methodological rigor and detailed reporting necessary to inspire clinical confidence and enable replication9–11.”
From A quantitative analysis of global AI medical studies: gaps in randomized controlled trials (2026) · cites this paper as [9]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“However, these guidelines do not cover RAG applications in clinical settings, emphasizing the need for a more specific guideline.”
From Improving large language model applications in biomedicine with retrieval-augmented generation: a systematic review, meta-analysis, and clinical development guidelines (2025) · cites this paper as [13]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
3 retracted
Zhu, S., Niu, Y., Wang, J., Xu, D., & Li, Y. (2022) · Computational and Mathematical Methods in Medicine · doi:10.1155/2022/3021320 · Cited by 10
RetractedWang, T., Chen, Y., Du, H., Liu, Y., Zhang, L., & Meng, M. (2021) · Computational and Mathematical Methods in Medicine · doi:10.1155/2021/3584034 · Cited by 7
RetractedZhang, F., Wu, S., Qu, M., & Zhou, L. (2022) · Contrast Media & Molecular Imaging · doi:10.1155/2022/1013241 · Cited by 5
RetractedThe PRISMA 2020 statement: An updated guideline for reporting systematic reviews
Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., et al. (2021) · BMJ · doi:10.1136/bmj.n71 · Cited by 108,283
Self-citationCONSORT 2010 Statement: updated guidelines for reporting parallel group randomised trials
the CONSORT Group, Schulz, K. F., Altman, D. G., & Moher, D. (2010) · BMC Medicine · doi:10.1186/1741-7015-8-18 · Cited by 13,371
Self-citationCONSORT 2010 Statement: updated guidelines for reporting parallel group randomised trials
Schulz, K. F., Altman, D. G., Moher, D., & for the CONSORT Group (2010) · BMJ · doi:10.1136/bmj.c332 · Cited by 9,329
Self-citationLarge Language Models Encode Clinical Knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2212.13138 · Cited by 4,083
Self-citationLarge language models encode clinical knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2023) · Nature · doi:10.1038/s41586-023-06291-2 · Cited by 4,083
Self-citationInternational evaluation of an AI system for breast cancer screening
McKinney, S. M., Sieniek, M., Godbole, V., Godwin, J., Antropova, N., Ashrafian, H., Back, T., Chesus, M., Corrado, G. S., Darzi, A., et al. (2020) · Nature · doi:10.1038/s41586-019-1799-6 · Cited by 3,287
Self-citationCONSORT 2010 statement: Updated guidelines for reporting parallel group randomised trials
Schulz, K. F., Altman, D. G., & Moher, D. (2010) · Journal of Pharmacology and Pharmacotherapeutics · doi:10.4103/0976-500x.72352 · Cited by 2,165
Self-citation
Research works printed but not linked: 3
- Available from: https://link.springer.com/10.1007/s00345-022-04086-0 (2022). · DOI
- Wornow, M. et al. The Shaky Foundations of Clinical Foundation Models: A Survey of Large Language Models and Foundation Models for EMRs [Internet]. arXiv; [cited 2023 Jul 5]. Available from: http://arxiv.org/abs/2303.12961 (2023). · arXiv
- Lefebvre, C. et al. Chapter 4: Searching for and selecting studies. in Cochrane Handbook for Systematic Reviews of Interventions [Internet]. Version 6.3 (updated February 2022). Cochrane; 2022. Available from: www.training.cochrane.org/handbook.
Cited by
Tam, T. Y. C., Sivarajkumar, S., Kapoor, S., Stolyar, A. V., Polanska, K., McCarthy, K. R., Osterhoudt, H., Wu, X., Visweswaran, S., Fu, S., et al. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01258-7 · Cited by 424
AI, Health, and Health Care Today and Tomorrow: The JAMA Summit Report on Artificial Intelligence
Angus, D. C., Khera, R., Lieu, T., Liu, V., Ahmad, F. S., Anderson, B., Bhavani, S. V., Bindman, A., Brennan, T., Celi, L. A., et al. (2025) · JAMA · doi:10.1001/jama.2025.18490 · Cited by 195
Self-citationLiu, S., McCoy, A. B., & Wright, A. (2025) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaf008 · Cited by 200
Multimodal artificial intelligence in ophthalmology: Applications, challenges, and future directions
Jin, K., Yu, T., & Grzybowski, A. E. (2025) · Survey of Ophthalmology · doi:10.1016/j.survophthal.2025.07.003 · Cited by 46
El-Sayed, A., Lovat, L. B., & Ahmad, O. F. (2025) · Gastroenterology · doi:10.1053/j.gastro.2025.01.254 · Cited by 46
Metadata
- Authors
- Martindale, A. P. L., Llewellyn, C. D., de Visser, R. O., Ng, B., Ngai, V., Kale, A. U., di Ruffano, L. F., Golub, R. M., Collins, G. S., Moher, D., McCradden, M. D., Oakden-Rayner, L., Rivera, S. C., Calvert, M., Kelly, C. J., Lee, C. S., Yau, C., Chan, A.-W., Keane, P. A., Beam, A. L., Denniston, A. K., & Liu, X.
- Affiliations
- Brighton and Sussex Medical School (GB)
- University of Oxford (GB)
- University College London (GB)
- University Hospitals Birmingham NHS Foundation Trust (GB)
- Northwestern University (US)
- Ottawa Hospital Research Institute (CA)
- University of Toronto (CA)
- University of Birmingham (GB)
- National Institute for Health and Care Research (GB)
- Google (United Kingdom) (GB)
- University of Washington (US)
- Moorfields Eye Hospital NHS Foundation Trust (GB)
- Harvard University (US)
- Corresponding author
- Liu, Xiaoxuan
- ORCID
- Martindale, Alexander P. L.
- Llewellyn, Carrie D.
- Ng, Benjamin
- Ngai, Victoria
- Kale, Aditya U.
- di Ruffano, Lavinia Ferrante
- Collins, Gary S.
- Moher, David
- McCradden, Melissa D.
- Rivera, Samantha Cruz
- Calvert, Melanie
- Kelly, Christopher J.
- Lee, Cecilia S.
- Yau, Christopher
- Chan, An-Wen
- Keane, Pearse A.
- Beam, Andrew L.
- Denniston, Alastair K.
- Liu, Xiaoxuan
- Venue
- Nature Communications
- Published
- 2024-02-22
- Volume
- 15
- Issue
- 1
- Pages
- 1619-1619
- ISSN
- 2041-1723 (ISSN-L)
- PMC
- PMC10883966
- OpenAlex
- W4392058364
- Semantic Scholar
- 284bb9ee87010245a3237804f7f3adda2b6a327a
- Cited by
- 58
- References
- 87
- Citations by year
- 2026: 28 · 2025: 26 · 2024: 4
- FWCI
- 16.13
- Type
- journal article
- Language
- en
- Update notices
- correction (2024-07-29)
- Open access
- gold
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- Nature Communications (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (publishedVersion)
- University of Birmingham Research Portal (University of Birmingham) (publishedVersion, cc-by)
- UCL Discovery (University College London) (submittedVersion, cc-by)
- Adelaide Research & Scholarship (AR&S) (University of Adelaide) (submittedVersion, cc-by)
- Figshare (submittedVersion, cc-by)
- Funders
- Cancer Research UK (27294)
- National Institute for Health and Care Research (NIHR205109)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Keywords
- CONSORT-AI, reporting guidelines, artificial intelligence interventions, randomized controlled trials, reporting completeness, guideline adherence, research transparency
- MeSH terms
- Artificial Intelligence, China, Humans, Research Design, Randomized Controlled Trials as Topic, Guidelines as Topic, Research Report
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Concordance of randomised controlled trials for artificial intelligence interventions with the CONSORT-AI reporting guidelines. https://app.cochat.ai/papers/10.1038/s41467-024-45355-3
- The paper
- Martindale, A. P. L., Llewellyn, C. D., de Visser, R. O., Ng, B., Ngai, V., Kale, A. U., di Ruffano, L. F., Golub, R. M., Collins, G. S., Moher, D., McCradden, M. D., Oakden-Rayner, L., Rivera, S. C., Calvert, M., Kelly, C. J., Lee, C. S., Yau, C., Chan, A.-W., Keane, P. A., . . . Liu, X. (2024). Concordance of randomised controlled trials for artificial intelligence interventions with the CONSORT-AI reporting guidelines. Nature Communications. https://doi.org/10.1038/s41467-024-45355-3
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.