Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 62 of the 62 references it cites were found in a registry or an index; 18 of the 33 sources its claims cite were checked for support.

Reporting guidelines for clinical trial reports for interventions involving artificial intelligence: the CONSORT-AI extension

Open access
Liu, Xiaoxuan, Cruz Rivera, Samantha, Moher, David, Calvert, Melanie J., Denniston, Alastair K., The SPIRIT-AI and CONSORT-AI Working Group, SPIRIT-AI and CONSORT-AI Steering Group, Chan, An-Wen, Darzi, Ara, Holmes, Christopher, Yau, Christopher, Ashrafian, Hutan, Deeks, Jonathan J., Ferrante di Ruffano, Lavinia, Faes, Livia, Keane, Pearse A., Vollmer, Sebastian J., SPIRIT-AI and CONSORT-AI Consensus Group, Lee, Aaron Y., Jonas, Adrian, Esteva, Andre, Beam, Andrew L., Chan, An-Wen, Panico, Maria Beatrice, Lee, Cecilia S., Haug, Charlotte, Kelly, Christopher J., Yau, Christopher, Mulrow, Cynthia, Espinoza, Cyrus, Fletcher, John, Paltoo, Dina, Manna, Elaine, Price, Gary, Collins, Gary S., Harvey, Hugh, Matcham, James, Monteiro, Joao, ElZarrad, M. Khair, Ferrante di Ruffano, Lavinia, Oakden-Rayner, Luke, McCradden, Melissa, Keane, Pearse A., Savage, Richard, Golub, Robert, Sarkar, Rupa, Rowley, Samuel · 2020 · Nature Medicine

Abstract

The CONSORT 2010 statement provides minimum guidelines for reporting randomized trials. Its widespread use has been instrumental in ensuring transparency in the evaluation of new interventions. More recently, there has been a growing recognition that interventions involving artificial intelligence (AI) need to undergo rigorous, prospective evaluation to demonstrate impact on health outcomes. The CONSORT-AI (Consolidated Standards of Reporting Trials–Artificial Intelligence) extension is a new reporting guideline for clinical trials evaluating interventions with an AI component. It was developed in parallel with its companion statement for clinical trial protocols: SPIRIT-AI (Standard Protocol Items: Recommendations for Interventional Trials–Artificial Intelligence). Both guidelines were developed through a staged consensus process involving literature review and expert consultation to generate 29 candidate items, which were assessed by an international multi-stakeholder group in a two-stage Delphi survey (103 stakeholders), agreed upon in a two-day consensus meeting (31 stakeholders) and refined through a checklist pilot (34 participants). The CONSORT-AI extension includes 14 new items that were considered sufficiently important for AI interventions that they should be routinely reported in addition to the core CONSORT 2010 items. CONSORT-AI recommends that investigators provide clear descriptions of the AI intervention, including instructions and skills required for use, the setting in which the AI intervention is integrated, the handling of inputs and outputs of the AI intervention, the human–AI interaction and provision of an analysis of error cases. CONSORT-AI will help promote transparency and completeness in reporting clinical trials for AI interventions. It will assist editors and peer reviewers, as well as the general readership, to understand, interpret and critically appraise the quality of clinical trial design and risk of bias in the reported outcomes.

Overview

CoChat Summary

TL;DR

CONSORT-AI is a reporting-guideline extension that adds 14 AI-specific items to the CONSORT 2010 checklist for randomized trials of interventions involving artificial intelligence. It was developed through literature review, a two-stage Delphi survey, a consensus meeting, and a checklist pilot to improve transparency and completeness in AI trial reports.

Key findings

  • The CONSORT-AI extension includes 14 new checklist items (11 extensions and 3 elaborations) considered sufficiently important for AI interventions to be routinely reported alongside core CONSORT 2010 items.
  • The development process generated 29 candidate items; the two-stage Delphi survey received 103 responses in round one and 91 responses in round two (88% of round-one participants), and the consensus meeting involved 31 stakeholders.
  • The final checklist recommends clear reporting of the AI intervention, including instructions and skills required for use, integration setting, handling of inputs and outputs, human–AI interaction, and analysis of error cases.
  • All 14 included items passed the pre-specified 80% inclusion threshold; CONSORT-AI 5(iii) initially received 77% support, then passed with 97% after rewording and a re-vote.

Methods

  • CONSORT-AI was developed in parallel with SPIRIT-AI according to the EQUATOR Network’s methodological framework, overseen by a 15-member international Steering Group.
  • A literature search on 13 May 2019 using “artificial intelligence,” “machine learning,” and “deep learning” identified 316 registered trials on ClinicalTrials.gov, of which 62 were completed and 7 had published results; after consultation with 19 experts, 29 candidate items were generated.
  • Candidate items were assessed in a two-stage online Delphi survey (169 experts invited; 103 and 91 responses), discussed at a two-day consensus meeting in January 2020 (31 stakeholders; 41 items discussed; 80% inclusion threshold), and refined through a checklist pilot with 34 participants.

Limitations

  • The authors state that relatively few published interventional AI trials existed, so discussions and decisions were not always supported by completed trial examples.
  • The authors state that the literature search used terms such as “artificial intelligence,” “machine learning,” and “deep learning,” but not “clinical decision support systems” or “expert systems,” so such systems were not actively discussed.
  • The authors state that initial candidate items were generated by a relatively small expert group, though wider Delphi participants proposed additional items; the text also makes evident that CONSORT-AI is a minimum reporting guidance and will need updating as AI applications evolve.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1038/s41591-020-1034-x

Cites

62 of 62 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Fig. 1 CONSORT 2010 flow diagram — adapted for AI clinical trials. CONSORT-AI 4a (i): State the inclusion and exclusion criteria at the level of participants. CONSORT-AI 4a (ii): State the inclusion and exclusion criteria at the level of the input data. CONSORT 13b (core CONSORT item): For each group, losses and exclusions after randomization, together with reasons.
Fig. 1 CONSORT 2010 flow diagram — adapted for AI clinical trials. CONSORT-AI 4a (i): State the inclusion and exclusion criteria at the level of participants. CONSORT-AI 4a (ii): State the inclusion and exclusion criteria at the level of the input data. CONSORT 13b (core CONSORT item): For each group, losses and exclusions after randomization, together with reasons.CC BY · PubMed Central

Verification

Claims the paper makes

9 Valid · 7 Not valid · 6 Not checked · 36 Own results and interpretations

3 more claims

Create a free account to see all 58 claims and every source verdict.

See all claims — free account
  • Probably not a substantial claim. 45
  • It may state two separate claims as one. 16

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

6 more claims about this paper

Create a free account to see every claim other papers make about this one.

See all claims — free account

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

1 of 1

Citations

Cites

Show all 60 references

Cited by

See all 1,421

Metadata

Authors
Liu, X., Cruz Rivera, S., Moher, D., Calvert, M. J., Denniston, A. K., The SPIRIT-AI and CONSORT-AI Working Group, SPIRIT-AI and CONSORT-AI Steering Group, Chan, A.-W., Darzi, A., Holmes, C., Yau, C., Ashrafian, H., Deeks, J. J., Ferrante di Ruffano, L., Faes, L., Keane, P. A., Vollmer, S. J., SPIRIT-AI and CONSORT-AI Consensus Group, Lee, A. Y., Jonas, A., Esteva, A., Beam, A. L., Chan, A.-W., Panico, M. B., Lee, C. S., Haug, C., Kelly, C. J., Yau, C., Mulrow, C., Espinoza, C., Fletcher, J., Paltoo, D., Manna, E., Price, G., Collins, G. S., Harvey, H., Matcham, J., Monteiro, J., ElZarrad, M. K., Ferrante di Ruffano, L., Oakden-Rayner, L., McCradden, M., Keane, P. A., Savage, R., Golub, R., Sarkar, R., & Rowley, S.
Affiliations
  • University Hospitals Birmingham NHS Foundation Trust (GB)
  • University of Ottawa (CA)
  • University of Toronto (CA)
  • Imperial College London (GB)
  • The Alan Turing Institute (GB)
  • University of Oxford (GB)
  • Luzerner Kantonsspital (CH)
  • National Institute for Health and Care Excellence (GB)
  • Salesforce (United States) (US)
  • Harvard University (US)
  • Google (United Kingdom) (GB)
  • University of Washington (US)
  • National Institutes of Health (US)
  • BMJ Group (United Kingdom) (GB)
  • AstraZeneca (United Kingdom) (GB)
  • United States Food and Drug Administration (US)
  • American Medical Association (US)
  • Medical Research Council (GB)
Corresponding author
Denniston, Alastair K.
Venue
Nature Medicine
Published
2020-09
Volume
26
Issue
9
Pages
1364-1374
ISSN
  • 1078-8956 (ISSN-L)
  • 1546-170X
OpenAlex
W3136552952
Cited by
1,421
References
60
Citations by year
2026: 596 · 2025: 243 · 2024: 162 · 2023: 129 · 2022: 145 · 2021: 129 · 2020: 16
FWCI
21.75
Type
journal article
Language
en
Open access
Licence
Locations
  • Nature Medicine (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • University of Birmingham Research Portal (University of Birmingham) (acceptedVersion, cc-by)
  • University of Birmingham Research Portal (University of Birmingham) (submittedVersion)
Funders
  • Wellcome Trust
  • Alan Turing Institute
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
Keywords
CONSORT-AI, reporting guidelines, artificial intelligence interventions, clinical trial reporting, human-AI interaction, risk of bias
MeSH terms
Artificial Intelligence, Humans, Research Design, Randomized Controlled Trials as Topic, Peer Review, Research, Research Report

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-15, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-15 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Reporting guidelines for clinical trial reports for interventions involving artificial intelligence: the CONSORT-AI extension. https://app.cochat.ai/papers/10.1038/s41591-020-1034-x
The paper
Liu, X., Cruz Rivera, S., Moher, D., Calvert, M. J., Denniston, A. K., The SPIRIT-AI and CONSORT-AI Working Group, SPIRIT-AI and CONSORT-AI Steering Group, Chan, A.-W., Darzi, A., Holmes, C., Yau, C., Ashrafian, H., Deeks, J. J., Ferrante di Ruffano, L., Faes, L., Keane, P. A., Vollmer, S. J., SPIRIT-AI and CONSORT-AI Consensus Group, Lee, A. Y., . . . Rowley, S. (2020). Reporting guidelines for clinical trial reports for interventions involving artificial intelligence: the CONSORT-AI extension. Nature Medicine. https://doi.org/10.1038/s41591-020-1034-x

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.