Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 55 of the 56 references it cites were found in a registry or an index.

Reporting guideline for the early-stage clinical evaluation of decision support systems driven by artificial intelligence: DECIDE-AI

Open access
Vasey, Baptiste, Nagendran, Myura, Campbell, Bruce, Clifton, David A., Collins, Gary S., Denaxas, Spiros, Denniston, Alastair K., Faes, Livia, Geerts, Bart, Ibrahim, Mudathir, Liu, Xiaoxuan, Mateen, Bilal A., Mathur, Piyush, McCradden, Melissa D., Morgan, Lauren, Ordish, Johan, Rogers, Campbell, Saria, Suchi, Ting, Daniel S. W., Watkinson, Peter, Weber, Wim, Wheatstone, Peter, McCulloch, Peter, the DECIDE-AI expert group, Lee, Aaron Y., Fraser, Alan G., Connell, Ali, Vira, Alykhan, Esteva, Andre, Althouse, Andrew D., Beam, Andrew L., de Hond, Anne, Boulesteix, Anne-Laure, Bradlow, Anthony, Ercole, Ari, Paez, Arsenio, Tsanas, Athanasios, Kirby, Barry, Glocker, Ben, Velardo, Carmelo, Park, Chang Min, Hehakaya, Charisma, Baber, Chris, Paton, Chris, Johner, Christian, Kelly, Christopher J., Vincent, Christopher J., Yau, Christopher, McGenity, Clare, Gatsonis, Constantine, Faivre-Finn, Corinne, Simon, Crispin, Sent, Danielle, Bzdok, Danilo, Treanor, Darren, Wong, David C., Steiner, David F., Higgins, David, Benson, Dawn, O’Regan, Declan P., Gunasekaran, Dinesh V., Danks, Dominic, Neri, Emanuele, Kyrimi, Evangelia, Schwendicke, Falk, Magrabi, Farah, Ives, Frances, Rademakers, Frank E., Fowler, George E., Frau, Giuseppe, Hogg, H. D. Jeffry, Marcus, Hani J., Chan, Heang-Ping, Xiang, Henry, McIntyre, Hugh F., Harvey, Hugh, Kim, Hyungjin, Habli, Ibrahim, Fackler, James C., Shaw, James, Higham, Janet, Wohlgemut, Jared M., Chong, Jaron, Bibault, Jean-Emmanuel, Cohen, Jérémie F., Kers, Jesper, Morley, Jessica, Krois, Joachim, Monteiro, Joao, Horovitz, Joel, Fletcher, John, Taylor, Jonathan, Yoon, Jung Hyun, Singh, Karandeep, Moons, Karel G. M., Karpathakis, Kassandra, Catchpole, Ken, Hood, Kerenza, Balaskas, Konstantinos, Kamnitsas, Konstantinos, Militello, Laura, Wynants, Laure, Oakden-Rayner, Lauren, Lovat, Laurence B., Smits, Luc J. M., Hinske, Ludwig C., ElZarrad, M. Khair, van Smeden, Maarten, Giavina-Bianchi, Mara, Daley, Mark, Sendak, Mark P., Sujan, Mark, Rovers, Maroeska, DeCamp, Matthew, Woodward, Matthew, Komorowski, Matthieu, Marsden, Max, Mackintosh, Maxine, Abramoff, Michael D., de la Hoz, Miguel Ángel Armengol, Hambidge, Neale, Daly, Neil, Peek, Niels, Redfern, Oliver, Ahmad, Omer F., Bossuyt, Patrick M., Keane, Pearse A., Ferreira, Pedro N. P., Schnell-Inderst, Petra, Mascagni, Pietro, Dasgupta, Prokar, Guan, Pujun, Barnett, Rachel, Kader, Rawen, Chopra, Reena, Mann, Ritse M., Sarkar, Rupa, Mäenpää, Saana M., Finlayson, Samuel G., Vollam, Sarah, Vollmer, Sebastian J., Park, Seong Ho, Laher, Shakir, Joshi, Shalmali, van der Meijden, Siri L., Shelmerdine, Susan C., Tan, Tien-En, Stocker, Tom J. W., Giannini, Valentina, Madai, Vince I., Newcombe, Virginia, Ng, Wei Yan, Rogers, Wendy A., Ogallo, William, Park, Yoonyoung, Perkins, Zane B. · 2022 · Nature Medicine

Abstract

A growing number of artificial intelligence (AI)-based clinical decision support systems are showing promising performance in preclinical, in silico evaluation, but few have yet demonstrated real benefit to patient care. Early-stage clinical evaluation is important to assess an AI system's actual clinical performance at small scale, ensure its safety, evaluate the human factors surrounding its use and pave the way to further large-scale trials. However, the reporting of these early studies remains inadequate. The present statement provides a multi-stakeholder, consensus-based reporting guideline for the Developmental and Exploratory Clinical Investigations of DEcision support systems driven by Artificial Intelligence (DECIDE-AI). We conducted a two-round, modified Delphi process to collect and analyze expert opinion on the reporting of early clinical evaluation of AI systems. Experts were recruited from 20 pre-defined stakeholder categories. The final composition and wording of the guideline was determined at a virtual consensus meeting. The checklist and the Explanation & Elaboration (E&E) sections were refined based on feedback from a qualitative evaluation process. In total, 123 experts participated in the first round of Delphi, 138 in the second round, 16 in the consensus meeting and 16 in the qualitative evaluation. The DECIDE-AI reporting guideline comprises 17 AI-specific reporting items (made of 28 subitems) and ten generic reporting items, with an E&E paragraph provided for each. Through consultation and consensus with a range of stakeholders, we developed a guideline comprising key items that should be reported in early-stage clinical studies of AI-based decision support systems in healthcare. By providing an actionable checklist of minimal reporting items, the DECIDE-AI guideline will facilitate the appraisal of these studies and replicability of their findings.

Overview

Integrity signals

Registry

Crossref · 10.1038/s41591-022-01772-9

Cites

55 of 55 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

The text we found for this paper looks like another document, so its claims and self-check weren’t run. The registry and citation checks still apply.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The text we found for this paper looks like another document, so the self-check wasn’t run.

Citations

Cites

Show all 56 references

Research works printed but not linked: 1

  1. Sujan, M., Baber, C., Salmon, P., Pool, R. & Chozos, N. Human factors and ergonomics in healthcare AI. https://www.researchgate.net/publication/354728442_Human_Factors_and_Ergonomics_in_Healthcare_AI (2021).

Cited by

See all 1,039

Metadata

Authors
Vasey, B., Nagendran, M., Campbell, B., Clifton, D. A., Collins, G. S., Denaxas, S., Denniston, A. K., Faes, L., Geerts, B., Ibrahim, M., Liu, X., Mateen, B. A., Mathur, P., McCradden, M. D., Morgan, L., Ordish, J., Rogers, C., Saria, S., Ting, D. S. W., Watkinson, P., Weber, W., Wheatstone, P., McCulloch, P., the DECIDE-AI expert group, Lee, A. Y., Fraser, A. G., Connell, A., Vira, A., Esteva, A., Althouse, A. D., Beam, A. L., de Hond, A., Boulesteix, A.-L., Bradlow, A., Ercole, A., Paez, A., Tsanas, A., Kirby, B., Glocker, B., Velardo, C., Park, C. M., Hehakaya, C., Baber, C., Paton, C., Johner, C., Kelly, C. J., Vincent, C. J., Yau, C., McGenity, C., Gatsonis, C., Faivre-Finn, C., Simon, C., Sent, D., Bzdok, D., Treanor, D., Wong, D. C., Steiner, D. F., Higgins, D., Benson, D., O’Regan, D. P., Gunasekaran, D. V., Danks, D., Neri, E., Kyrimi, E., Schwendicke, F., Magrabi, F., Ives, F., Rademakers, F. E., Fowler, G. E., Frau, G., Hogg, H. D. J., Marcus, H. J., Chan, H.-P., Xiang, H., McIntyre, H. F., Harvey, H., Kim, H., Habli, I., Fackler, J. C., Shaw, J., Higham, J., Wohlgemut, J. M., Chong, J., Bibault, J.-E., Cohen, J. F., Kers, J., Morley, J., Krois, J., Monteiro, J., Horovitz, J., Fletcher, J., Taylor, J., Yoon, J. H., Singh, K., Moons, K. G. M., Karpathakis, K., Catchpole, K., Hood, K., Balaskas, K., Kamnitsas, K., Militello, L., Wynants, L., Oakden-Rayner, L., Lovat, L. B., Smits, L. J. M., Hinske, L. C., ElZarrad, M. K., van Smeden, M., Giavina-Bianchi, M., Daley, M., Sendak, M. P., Sujan, M., Rovers, M., DeCamp, M., Woodward, M., Komorowski, M., Marsden, M., Mackintosh, M., Abramoff, M. D., de la Hoz, M. Á. A., Hambidge, N., Daly, N., Peek, N., Redfern, O., Ahmad, O. F., Bossuyt, P. M., Keane, P. A., Ferreira, P. N. P., Schnell-Inderst, P., Mascagni, P., Dasgupta, P., Guan, P., Barnett, R., Kader, R., Chopra, R., Mann, R. M., Sarkar, R., Mäenpää, S. M., Finlayson, S. G., Vollam, S., Vollmer, S. J., Park, S. H., Laher, S., Joshi, S., van der Meijden, S. L., Shelmerdine, S. C., Tan, T.-E., Stocker, T. J. W., Giannini, V., Madai, V. I., Newcombe, V., Ng, W. Y., Rogers, W. A., Ogallo, W., Park, Y., & Perkins, Z. B.
Affiliations
  • University of Oxford (GB)
  • Imperial College London (GB)
  • University of Exeter (GB)
  • UCL Biomedical Research Centre (GB)
  • University Hospitals Birmingham NHS Foundation Trust (GB)
  • Moorfields Eye Hospital NHS Foundation Trust (GB)
  • Maimonides Medical Center (US)
  • The Alan Turing Institute (GB)
  • Cleveland Clinic (US)
  • University of Toronto (CA)
  • Medicines and Healthcare products Regulatory Agency (GB)
  • HeartFlow (United States) (US)
  • Johns Hopkins University (US)
  • National University of Singapore (SG)
  • BMJ Group (United Kingdom) (GB)
  • University of Washington (US)
  • Google (United Kingdom) (GB)
  • University of Pittsburgh (US)
  • Harvard University (US)
  • Royal Berkshire Hospital (GB)
  • University of Cambridge (GB)
  • University of Edinburgh (GB)
  • Seoul National University (KR)
  • University Medical Center Utrecht (NL)
  • University of Leeds (GB)
  • Brown University (US)
  • Amsterdam University Medical Centers (NL)
  • Leeds Teaching Hospitals NHS Trust (GB)
  • Google (United States) (US)
  • Berlin Institute of Health at Charité - Universitätsmedizin Berlin (DE)
  • MRC London Institute of Medical Sciences (GB)
  • Singapore National Eye Center (SG)
  • University of Pisa (IT)
  • Queen Mary University of London (GB)
  • Charité – Universitätsmedizin Berlin (DE)
  • Macquarie University (AU)
  • KU Leuven (BE)
  • University of Bristol (GB)
  • Deep Blue (Italy) (IT)
  • National Hospital for Neurology and Neurosurgery (GB)
  • University of Michigan (US)
  • Nationwide Children's Hospital (US)
  • East Sussex Healthcare NHS Trust (GB)
  • Western University (CA)
  • Inserm (FR)
  • Sheffield Teaching Hospitals NHS Foundation Trust (GB)
  • Yonsei University (KR)
  • Utrecht University (NL)
  • Medical University of South Carolina (US)
  • Moorfields Eye Hospital (GB)
  • Applied Decision Science (United States) (US)
  • Maastricht University (NL)
  • United States Food and Drug Administration (US)
  • Duke University (US)
  • Radboud University Nijmegen (NL)
  • University of Colorado Boulder (US)
  • University of Iowa (US)
  • University of Manchester (GB)
  • Wellcome / EPSRC Centre for Interventional and Surgical Sciences (GB)
  • University of Lisbon (PT)
  • King's College London (GB)
  • The University of Texas MD Anderson Cancer Center (US)
  • University College London (GB)
  • Radboud University Medical Center (NL)
  • Helsinki University Hospital (FI)
  • German Research Centre for Artificial Intelligence (DE)
  • Asan Medical Center (KR)
  • University of Turin (IT)
  • IBM (United States) (US)
Corresponding author
Vasey, Baptiste
ORCID
Venue
Nature Medicine
Published
2022-05
Volume
28
Issue
5
Pages
924-933
ISSN
  • 1078-8956 (ISSN-L)
  • 1546-170X
OpenAlex
W4280572991
Cited by
1,039
References
56
Citations by year
2026: 623 · 2025: 173 · 2024: 129 · 2023: 95 · 2022: 19
FWCI
31.36
Type
journal article
Language
en
Licence
Locations
  • Nature Medicine (publishedVersion)
  • PubMed (publishedVersion)
  • Research Publications (Maastricht University) (publishedVersion, other-oa)
  • Pure Amsterdam UMC (publishedVersion, other-oa)
  • Research Explorer (The University of Manchester) (publishedVersion, other-oa)
  • University of Birmingham Research Portal (University of Birmingham) (publishedVersion)
  • Explore Bristol Research (publishedVersion)
  • Institutional Research Information System University of Turin (University of Turin) (submittedVersion, other-oa)
  • Data Archiving and Networked Services (DANS) (submittedVersion, other-oa)
  • Utrecht University Repository (Utrecht University) (submittedVersion, other-oa)
  • Lirias (submittedVersion)
  • Queen Mary Research Online (Queen Mary University of London) (submittedVersion)
  • Royal Devon and Exeter Research Repository (Royal Devon and Exeter) (submittedVersion, public-domain)
  • Radboud Repository (Radboud University) (submittedVersion)
  • Seoul National University Open Repository (Seoul National University) (submittedVersion)
  • Spiral (Imperial College London) (submittedVersion)
Funders
  • National Science Foundation
  • Alfred P. Sloan Foundation
  • American Heart Association
  • Pioneer Fund
  • AstraZeneca
  • GlaxoSmithKline
  • Ohio State University
  • Nationwide Children's Hospital
  • Johns Hopkins University
  • University of South Carolina
  • Queen Mary University of London
  • Wellcome Trust
  • Alan Turing Institute (TU/ASG/R-SPEH-105)
  • UK Research and Innovation (P/S023283/1)
  • Duke-NUS Medical School (Duke‐NUS/RSF/2021/0018, NUS/RSF/2021/0018)
  • Cancer Research UK (27294, C49297/A27294, C49297)
  • National Institute for Health and Care Research (C49297/A27294, ACF-2009-14-007)
  • Research Councils UK
  • Academy of Medical Sciences (AMS-CSF4-Newcombe)
  • Imperial College London (P/S023283/1)
  • University College London
  • University of Oxford
  • Newcastle University
  • Cardiff University
  • Macquarie University
  • Agency for Science, Technology and Research (A20H4g2141, H20C6a0032)
  • Institut National de la Santé et de la Recherche Médicale
  • Universiteit van Amsterdam
  • Universiteit Utrecht
  • Seoul National University
  • Yonsei University
  • University of Toronto
  • National Health Insurance Service
  • KU Leuven
  • Seoul National University Hospital
  • Russian Science Foundation
  • Università di Pisa
  • Moorfields Eye Hospital NHS Foundation Trust
  • Berlin Institute of Health
  • Sheffield Teaching Hospitals NHS Foundation Trust
  • Université de Paris
  • National Institutes of Health
  • Medical School, University of Michigan
  • School of Electronic Engineering and Computer Science, Queen Mary University of London
  • Department of Anesthesiology and Critical Care Medicine, Johns Hopkins Medicine
  • Medical Research Council (MR/K006584/1, C49297/A27294)
  • Engineering and Physical Sciences Research Council (N510129)
  • National Medical Research Council (NMRC/HSRG/0087/2018, MOH‐000655‐00, Duke-NUS/RSF/2021/0018)
  • Yonsei University College of Medicine
  • NIHR Oxford Biomedical Research Centre (C49297/ A27294)
  • National Heart, Lung, and Blood Institute (K01HL141771)
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Clinical practice guidelines implementation
Keywords
artificial intelligence, clinical decision support systems, reporting guidelines, AI safety, human factors, stakeholder consensus, clinical performance assessment, replicability
MeSH terms
Artificial Intelligence, Humans, Research Design, Consensus, Checklist, Research Report

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Reporting guideline for the early-stage clinical evaluation of decision support systems driven by artificial intelligence: DECIDE-AI. https://app.cochat.ai/papers/10.1038/s41591-022-01772-9
The paper
Vasey, B., Nagendran, M., Campbell, B., Clifton, D. A., Collins, G. S., Denaxas, S., Denniston, A. K., Faes, L., Geerts, B., Ibrahim, M., Liu, X., Mateen, B. A., Mathur, P., McCradden, M. D., Morgan, L., Ordish, J., Rogers, C., Saria, S., Ting, D. S. W., . . . Perkins, Z. B. (2022). Reporting guideline for the early-stage clinical evaluation of decision support systems driven by artificial intelligence: DECIDE-AI. Nature Medicine. https://doi.org/10.1038/s41591-022-01772-9

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.