The DOI matches its Crossref record; 55 of the 56 references it cites were found in a registry or an index.
Reporting guideline for the early-stage clinical evaluation of decision support systems driven by artificial intelligence: DECIDE-AI
Open accessAbstract
A growing number of artificial intelligence (AI)-based clinical decision support systems are showing promising performance in preclinical, in silico evaluation, but few have yet demonstrated real benefit to patient care. Early-stage clinical evaluation is important to assess an AI system's actual clinical performance at small scale, ensure its safety, evaluate the human factors surrounding its use and pave the way to further large-scale trials. However, the reporting of these early studies remains inadequate. The present statement provides a multi-stakeholder, consensus-based reporting guideline for the Developmental and Exploratory Clinical Investigations of DEcision support systems driven by Artificial Intelligence (DECIDE-AI). We conducted a two-round, modified Delphi process to collect and analyze expert opinion on the reporting of early clinical evaluation of AI systems. Experts were recruited from 20 pre-defined stakeholder categories. The final composition and wording of the guideline was determined at a virtual consensus meeting. The checklist and the Explanation & Elaboration (E&E) sections were refined based on feedback from a qualitative evaluation process. In total, 123 experts participated in the first round of Delphi, 138 in the second round, 16 in the consensus meeting and 16 in the qualitative evaluation. The DECIDE-AI reporting guideline comprises 17 AI-specific reporting items (made of 28 subitems) and ten generic reporting items, with an E&E paragraph provided for each. Through consultation and consensus with a range of stakeholders, we developed a guideline comprising key items that should be reported in early-stage clinical studies of AI-based decision support systems in healthcare. By providing an actionable checklist of minimal reporting items, the DECIDE-AI guideline will facilitate the appraisal of these studies and replicability of their findings.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The text we found for this paper looks like another document, so the self-check wasn’t run.
Citations
Cites
The PRISMA 2020 statement: An updated guideline for reporting systematic reviews
Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., et al. (2021) · BMJ · doi:10.1136/bmj.n71 · Cited by 108,283
Self-citationHarris, P. A., Taylor, R., Thielke, R., Payne, J., Gonzalez, N., & Conde, J. G. (2009) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2008.08.010 · Cited by 51,384
The REDCap consortium: building an international community of software platform partners
Harris, P. A., Taylor, R., Minor, B. L., Elliott, V., Fernandez, M., O'Neal, L., McLeod, L., Delacqua, G., Delacqua, F., Kirby, J., et al. (2019) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2019.103208 · Cited by 24,013
von Elm, E., Altman, D. G., Egger, M., Pocock, S. J., Gøtzsche, P. C., Vandenbroucke, J. P., & Initiative, F. T. S. (2007) · Annals of Internal Medicine · doi:10.7326/0003-4819-147-8-200710160-00010 · Cited by 17,796
S., N. L. (2017) · doi:10.1177/1609406917733847 · Cited by 14,051
International Organization for Standardization
(2011) · SpringerReference · doi:10.1007/springerreference_16847 · Cited by 12,534
Elm, E. V., Altman, D. G., Egger, M., Pocock, S. J., Gøtzsche, P. C., & Vandenbroucke, J. P. (2007) · BMJ · doi:10.1136/bmj.39335.541782.ad · Cited by 10,766
An Experimental Application of the DELPHI Method to the Use of Experts
Dalkey, N., & Helmer, O. (1963) · Management Science · doi:10.1287/mnsc.9.3.458 · Cited by 6,061
Skivington, K., Matthews, L., Simpson, S. A., Craig, P., Baird, J., Blazeby, J. M., Boyd, K. A., Craig, N., French, D. P., McIntosh, E., et al. (2021) · BMJ · doi:10.1136/bmj.n2061 · Cited by 5,808
Lipton, Z. C. (2018) · Queue · doi:10.1145/3236386.3241340 · Cited by 5,427
Research works printed but not linked: 1
- Sujan, M., Baber, C., Salmon, P., Pool, R. & Chozos, N. Human factors and ergonomics in healthcare AI. https://www.researchgate.net/publication/354728442_Human_Factors_and_Ergonomics_in_Healthcare_AI (2021).
Cited by
Kung, T. H., Cheatham, M., Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., Maningo, J., et al. (2023) · PLOS Digital Health · doi:10.1371/journal.pdig.0000198 · Cited by 4,115
Kung, T. H., Cheatham, M., ChatGPT, Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., et al. (2022) · doi:10.1101/2022.12.19.22283643 · Cited by 4,115
Large language models in medicine
Thirunavukarasu, A. J., Ting, D. S. J., Elangovan, K., Gutierrez, L., Tan, T. F., & Ting, D. S. W. (2023) · Nature Medicine · doi:10.1038/s41591-023-02448-8 · Cited by 3,944
Self-citationCollins, G. S., Moons, K. G. M., Dhiman, P., Riley, R. D., Beam, A. L., Van Calster, B., Ghassemi, M., Liu, X., Reitsma, J. B., van Smeden, M., et al. (2024) · BMJ · doi:10.1136/bmj-2023-078378 · Cited by 4,718
Self-citationThe TRIPOD-LLM reporting guideline for studies using large language models
Gallifant, J., Afshar, M., Ameen, S., Aphinyanaphongs, Y., Chen, S., Cacciamani, G., Demner-Fushman, D., Dligach, D., Daneshjou, R., Fernandes, C., et al. (2025) · Nature Medicine · doi:10.1038/s41591-024-03425-5 · Cited by 615
Self-citation
Metadata
- Authors
- Vasey, B., Nagendran, M., Campbell, B., Clifton, D. A., Collins, G. S., Denaxas, S., Denniston, A. K., Faes, L., Geerts, B., Ibrahim, M., Liu, X., Mateen, B. A., Mathur, P., McCradden, M. D., Morgan, L., Ordish, J., Rogers, C., Saria, S., Ting, D. S. W., Watkinson, P., Weber, W., Wheatstone, P., McCulloch, P., the DECIDE-AI expert group, Lee, A. Y., Fraser, A. G., Connell, A., Vira, A., Esteva, A., Althouse, A. D., Beam, A. L., de Hond, A., Boulesteix, A.-L., Bradlow, A., Ercole, A., Paez, A., Tsanas, A., Kirby, B., Glocker, B., Velardo, C., Park, C. M., Hehakaya, C., Baber, C., Paton, C., Johner, C., Kelly, C. J., Vincent, C. J., Yau, C., McGenity, C., Gatsonis, C., Faivre-Finn, C., Simon, C., Sent, D., Bzdok, D., Treanor, D., Wong, D. C., Steiner, D. F., Higgins, D., Benson, D., O’Regan, D. P., Gunasekaran, D. V., Danks, D., Neri, E., Kyrimi, E., Schwendicke, F., Magrabi, F., Ives, F., Rademakers, F. E., Fowler, G. E., Frau, G., Hogg, H. D. J., Marcus, H. J., Chan, H.-P., Xiang, H., McIntyre, H. F., Harvey, H., Kim, H., Habli, I., Fackler, J. C., Shaw, J., Higham, J., Wohlgemut, J. M., Chong, J., Bibault, J.-E., Cohen, J. F., Kers, J., Morley, J., Krois, J., Monteiro, J., Horovitz, J., Fletcher, J., Taylor, J., Yoon, J. H., Singh, K., Moons, K. G. M., Karpathakis, K., Catchpole, K., Hood, K., Balaskas, K., Kamnitsas, K., Militello, L., Wynants, L., Oakden-Rayner, L., Lovat, L. B., Smits, L. J. M., Hinske, L. C., ElZarrad, M. K., van Smeden, M., Giavina-Bianchi, M., Daley, M., Sendak, M. P., Sujan, M., Rovers, M., DeCamp, M., Woodward, M., Komorowski, M., Marsden, M., Mackintosh, M., Abramoff, M. D., de la Hoz, M. Á. A., Hambidge, N., Daly, N., Peek, N., Redfern, O., Ahmad, O. F., Bossuyt, P. M., Keane, P. A., Ferreira, P. N. P., Schnell-Inderst, P., Mascagni, P., Dasgupta, P., Guan, P., Barnett, R., Kader, R., Chopra, R., Mann, R. M., Sarkar, R., Mäenpää, S. M., Finlayson, S. G., Vollam, S., Vollmer, S. J., Park, S. H., Laher, S., Joshi, S., van der Meijden, S. L., Shelmerdine, S. C., Tan, T.-E., Stocker, T. J. W., Giannini, V., Madai, V. I., Newcombe, V., Ng, W. Y., Rogers, W. A., Ogallo, W., Park, Y., & Perkins, Z. B.
- Affiliations
- University of Oxford (GB)
- Imperial College London (GB)
- University of Exeter (GB)
- UCL Biomedical Research Centre (GB)
- University Hospitals Birmingham NHS Foundation Trust (GB)
- Moorfields Eye Hospital NHS Foundation Trust (GB)
- Maimonides Medical Center (US)
- The Alan Turing Institute (GB)
- Cleveland Clinic (US)
- University of Toronto (CA)
- Medicines and Healthcare products Regulatory Agency (GB)
- HeartFlow (United States) (US)
- Johns Hopkins University (US)
- National University of Singapore (SG)
- BMJ Group (United Kingdom) (GB)
- University of Washington (US)
- Google (United Kingdom) (GB)
- University of Pittsburgh (US)
- Harvard University (US)
- Royal Berkshire Hospital (GB)
- University of Cambridge (GB)
- University of Edinburgh (GB)
- Seoul National University (KR)
- University Medical Center Utrecht (NL)
- University of Leeds (GB)
- Brown University (US)
- Amsterdam University Medical Centers (NL)
- Leeds Teaching Hospitals NHS Trust (GB)
- Google (United States) (US)
- Berlin Institute of Health at Charité - Universitätsmedizin Berlin (DE)
- MRC London Institute of Medical Sciences (GB)
- Singapore National Eye Center (SG)
- University of Pisa (IT)
- Queen Mary University of London (GB)
- Charité – Universitätsmedizin Berlin (DE)
- Macquarie University (AU)
- KU Leuven (BE)
- University of Bristol (GB)
- Deep Blue (Italy) (IT)
- National Hospital for Neurology and Neurosurgery (GB)
- University of Michigan (US)
- Nationwide Children's Hospital (US)
- East Sussex Healthcare NHS Trust (GB)
- Western University (CA)
- Inserm (FR)
- Sheffield Teaching Hospitals NHS Foundation Trust (GB)
- Yonsei University (KR)
- Utrecht University (NL)
- Medical University of South Carolina (US)
- Moorfields Eye Hospital (GB)
- Applied Decision Science (United States) (US)
- Maastricht University (NL)
- United States Food and Drug Administration (US)
- Duke University (US)
- Radboud University Nijmegen (NL)
- University of Colorado Boulder (US)
- University of Iowa (US)
- University of Manchester (GB)
- Wellcome / EPSRC Centre for Interventional and Surgical Sciences (GB)
- University of Lisbon (PT)
- King's College London (GB)
- The University of Texas MD Anderson Cancer Center (US)
- University College London (GB)
- Radboud University Medical Center (NL)
- Helsinki University Hospital (FI)
- German Research Centre for Artificial Intelligence (DE)
- Asan Medical Center (KR)
- University of Turin (IT)
- IBM (United States) (US)
- Corresponding author
- Vasey, Baptiste
- ORCID
- Vasey, Baptiste
- Nagendran, Myura
- Campbell, Bruce
- Clifton, David A.
- Collins, Gary S.
- Denaxas, Spiros
- Denniston, Alastair K.
- Faes, Livia
- Geerts, Bart
- Ibrahim, Mudathir
- Liu, Xiaoxuan
- Mateen, Bilal A.
- Mathur, Piyush
- McCradden, Melissa D.
- Morgan, Lauren
- Ordish, Johan
- Rogers, Campbell
- Saria, Suchi
- Ting, Daniel S. W.
- Watkinson, Peter
- Weber, Wim
- McCulloch, Peter
- Lee, Aaron Y.
- Fraser, Alan G.
- Esteva, Andre
- Althouse, Andrew D.
- Beam, Andrew L.
- Boulesteix, Anne-Laure
- Bradlow, Anthony
- Ercole, Ari
- Tsanas, Athanasios
- Glocker, Ben
- Velardo, Carmelo
- Park, Chang Min
- Hehakaya, Charisma
- Baber, Chris
- Paton, Chris
- Kelly, Christopher J.
- Yau, Christopher
- McGenity, Clare
- Sent, Danielle
- Bzdok, Danilo
- Treanor, Darren
- Wong, David C.
- Steiner, David F.
- Higgins, David
- O’Regan, Declan P.
- Danks, Dominic
- Neri, Emanuele
- Kyrimi, Evangelia
- Schwendicke, Falk
- Magrabi, Farah
- Rademakers, Frank E.
- Fowler, George E.
- Hogg, H. D. Jeffry
- Marcus, Hani J.
- Chan, Heang-Ping
- Xiang, Henry
- McIntyre, Hugh F.
- Harvey, Hugh
- Kim, Hyungjin
- Habli, Ibrahim
- Fackler, James C.
- Shaw, James
- Wohlgemut, Jared M.
- Chong, Jaron
- Bibault, Jean-Emmanuel
- Cohen, Jérémie F.
- Kers, Jesper
- Morley, Jessica
- Krois, Joachim
- Fletcher, John
- Yoon, Jung Hyun
- Singh, Karandeep
- Moons, Karel G. M.
- Karpathakis, Kassandra
- Catchpole, Ken
- Hood, Kerenza
- Balaskas, Konstantinos
- Kamnitsas, Konstantinos
- Militello, Laura
- Wynants, Laure
- Lovat, Laurence B.
- Smits, Luc J. M.
- Hinske, Ludwig C.
- ElZarrad, M. Khair
- Daley, Mark
- Sendak, Mark P.
- Sujan, Mark
- Rovers, Maroeska
- DeCamp, Matthew
- Woodward, Matthew
- Komorowski, Matthieu
- Marsden, Max
- Mackintosh, Maxine
- Peek, Niels
- Redfern, Oliver
- Ahmad, Omer F.
- Bossuyt, Patrick M.
- Keane, Pearse A.
- Ferreira, Pedro N. P.
- Mascagni, Pietro
- Guan, Pujun
- Kader, Rawen
- Chopra, Reena
- Mann, Ritse M.
- Sarkar, Rupa
- Mäenpää, Saana M.
- Finlayson, Samuel G.
- Vollam, Sarah
- Vollmer, Sebastian J.
- Park, Seong Ho
- Joshi, Shalmali
- Shelmerdine, Susan C.
- Tan, Tien-En
- Giannini, Valentina
- Madai, Vince I.
- Newcombe, Virginia
- Rogers, Wendy A.
- Ogallo, William
- Park, Yoonyoung
- Perkins, Zane B.
- Venue
- Nature Medicine
- Published
- 2022-05
- Volume
- 28
- Issue
- 5
- Pages
- 924-933
- ISSN
- 1078-8956 (ISSN-L)
- 1546-170X
- OpenAlex
- W4280572991
- Cited by
- 1,039
- References
- 56
- Citations by year
- 2026: 623 · 2025: 173 · 2024: 129 · 2023: 95 · 2022: 19
- FWCI
- 31.36
- Type
- journal article
- Language
- en
- Update notices
- correction (2022-08-12)
- Open access
- Licence
- other-oa
- www.springer.com/tdm (tdm)
- www.springer.com/tdm (vor)
- Locations
- Nature Medicine (publishedVersion)
- PubMed (publishedVersion)
- Research Publications (Maastricht University) (publishedVersion, other-oa)
- Pure Amsterdam UMC (publishedVersion, other-oa)
- Research Explorer (The University of Manchester) (publishedVersion, other-oa)
- University of Birmingham Research Portal (University of Birmingham) (publishedVersion)
- Explore Bristol Research (publishedVersion)
- Institutional Research Information System University of Turin (University of Turin) (submittedVersion, other-oa)
- Data Archiving and Networked Services (DANS) (submittedVersion, other-oa)
- Utrecht University Repository (Utrecht University) (submittedVersion, other-oa)
- Lirias (submittedVersion)
- Queen Mary Research Online (Queen Mary University of London) (submittedVersion)
- Royal Devon and Exeter Research Repository (Royal Devon and Exeter) (submittedVersion, public-domain)
- Radboud Repository (Radboud University) (submittedVersion)
- Seoul National University Open Repository (Seoul National University) (submittedVersion)
- Spiral (Imperial College London) (submittedVersion)
- Funders
- National Science Foundation
- Alfred P. Sloan Foundation
- American Heart Association
- Pioneer Fund
- AstraZeneca
- GlaxoSmithKline
- Ohio State University
- Nationwide Children's Hospital
- Johns Hopkins University
- University of South Carolina
- Queen Mary University of London
- Wellcome Trust
- Alan Turing Institute (TU/ASG/R-SPEH-105)
- UK Research and Innovation (P/S023283/1)
- Duke-NUS Medical School (Duke‐NUS/RSF/2021/0018, NUS/RSF/2021/0018)
- Cancer Research UK (27294, C49297/A27294, C49297)
- National Institute for Health and Care Research (C49297/A27294, ACF-2009-14-007)
- Research Councils UK
- Academy of Medical Sciences (AMS-CSF4-Newcombe)
- Imperial College London (P/S023283/1)
- University College London
- University of Oxford
- Newcastle University
- Cardiff University
- Macquarie University
- Agency for Science, Technology and Research (A20H4g2141, H20C6a0032)
- Institut National de la Santé et de la Recherche Médicale
- Universiteit van Amsterdam
- Universiteit Utrecht
- Seoul National University
- Yonsei University
- University of Toronto
- National Health Insurance Service
- KU Leuven
- Seoul National University Hospital
- Russian Science Foundation
- Università di Pisa
- Moorfields Eye Hospital NHS Foundation Trust
- Berlin Institute of Health
- Sheffield Teaching Hospitals NHS Foundation Trust
- Université de Paris
- National Institutes of Health
- Medical School, University of Michigan
- School of Electronic Engineering and Computer Science, Queen Mary University of London
- Department of Anesthesiology and Critical Care Medicine, Johns Hopkins Medicine
- Medical Research Council (MR/K006584/1, C49297/A27294)
- Engineering and Physical Sciences Research Council (N510129)
- National Medical Research Council (NMRC/HSRG/0087/2018, MOH‐000655‐00, Duke-NUS/RSF/2021/0018)
- Yonsei University College of Medicine
- NIHR Oxford Biomedical Research Centre (C49297/ A27294)
- National Heart, Lung, and Blood Institute (K01HL141771)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Clinical practice guidelines implementation
- Keywords
- artificial intelligence, clinical decision support systems, reporting guidelines, AI safety, human factors, stakeholder consensus, clinical performance assessment, replicability
- MeSH terms
- Artificial Intelligence, Humans, Research Design, Consensus, Checklist, Research Report
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Reporting guideline for the early-stage clinical evaluation of decision support systems driven by artificial intelligence: DECIDE-AI. https://app.cochat.ai/papers/10.1038/s41591-022-01772-9
- The paper
- Vasey, B., Nagendran, M., Campbell, B., Clifton, D. A., Collins, G. S., Denaxas, S., Denniston, A. K., Faes, L., Geerts, B., Ibrahim, M., Liu, X., Mateen, B. A., Mathur, P., McCradden, M. D., Morgan, L., Ordish, J., Rogers, C., Saria, S., Ting, D. S. W., . . . Perkins, Z. B. (2022). Reporting guideline for the early-stage clinical evaluation of decision support systems driven by artificial intelligence: DECIDE-AI. Nature Medicine. https://doi.org/10.1038/s41591-022-01772-9
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.