Guidelines for clinical trial protocols for interventions involving artificial intelligence: the SPIRIT-AI Extension
Open accessAbstract
The SPIRIT 2013 (The Standard Protocol Items: Recommendations for Interventional Trials) statement aims to improve the completeness of clinical trial protocol reporting, by providing evidence-based recommendations for the minimum set of items to be addressed. This guidance has been instrumental in promoting transparent evaluation of new interventions. More recently, there is a growing recognition that interventions involving artificial intelligence need to undergo rigorous, prospective evaluation to demonstrate their impact on health outcomes. The SPIRIT-AI extension is a new reporting guideline for clinical trials protocols evaluating interventions with an AI component. It was developed in parallel with its companion statement for trial reports: CONSORT-AI. Both guidelines were developed using a staged consensus process, involving a literature review and expert consultation to generate 26 candidate items, which were consulted on by an international multi-stakeholder group in a 2-stage Delphi survey (103 stakeholders), agreed on in a consensus meeting (31 stakeholders) and refined through a checklist pilot (34 participants). The SPIRIT-AI extension includes 15 new items, which were considered sufficiently important for clinical trial protocols of AI interventions. These new items should be routinely reported in addition to the core SPIRIT 2013 items. SPIRIT-AI recommends that investigators provide clear descriptions of the AI intervention, including instructions and skills required for use, the setting in which the AI intervention will be integrated, considerations around the handling of input and output data, the human-AI interaction and analysis of error cases. SPIRIT-AI will help promote transparency and completeness for clinical trial protocols for AI interventions. Its use will assist editors and peer-reviewers, as well as the general readership, to understand, interpret and critically appraise the design and risk of bias for a planned clinical trial.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Such regional bias could inadvertently exacerbate existing health disparities if models were not robustly validated in varied clinical contexts.
“Such regional bias could inadvertently exacerbate existing health disparities if models were not robustly validated in varied clinical contexts36–38.”
From A quantitative analysis of global AI medical studies: gaps in randomized controlled trials (2026) · cites this paper as [36]
In Results, SPIRIT-AI 9 (page unavailable), the paper says AI algorithms may not generalise beyond their development environment. SPIRIT-AI 6a (ii) calls for validation evidence relevant to the intended setting and population, and the Discussion warns of harmful consequences from systematic errors. The paper does not specifically say that regional bias could exacerbate existing health disparities. (LLM, conf=0.83)
Experimental — accuracy not yet measuredReporting, evaluation, and appraisal frameworks including SPIRIT-AI, CONSORT-AI, DECIDE-AI, TRIPOD+AI, and PROBAST+AI have substantially strengthened the conceptual and reporting architecture for trustworthy clinical AI.
“Reporting, evaluation, and appraisal frameworks including SPIRIT-AI, CONSORT-AI, DECIDE-AI, TRIPOD+AI, and PROBAST+AI have substantially strengthened the conceptual and reporting architecture for trustworthy clinical AI [10,11,12,13,14].”
From Beyond Model Development in Healthcare AI: Post-Development Robustness, Post-Deployment Monitoring, and Lifecycle Governance—A Scoping Review of Reviews (2026) · cites this paper as [10]
The Abstract identifies SPIRIT-AI as a reporting guideline for AI trial protocols, and the Discussion says it is expected, alongside CONSORT-AI, to improve trial quality. The source does not establish that these frameworks have already substantially strengthened clinical AI reporting, and it does not address the other frameworks named in the sentence. (LLM, conf=0.84)
Experimental — accuracy not yet measuredThe SPIRIT 2025 update identified relevant SPIRIT or CONSORT extensions and related guidelines.
“identified relevant SPIRIT or CONSORT extensions and related guidelines (eg,21 22 23 24)”
From SPIRIT 2025 explanation and elaboration: updated guideline for protocols of randomised trials (2025) · cites this paper as [21]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Dermatologist-level classification of skin cancer with deep neural networks
Esteva, A., Kuprel, B., Novoa, R. A., Ko, J., Swetter, S. M., Blau, H. M., & Thrun, S. (2017) · Nature · doi:10.1038/nature21056 · Cited by 14,362 · see citing works
SPIRIT 2013 Statement: Defining Standard Protocol Items for Clinical Trials
Chan, A.-W., Tetzlaff, J. M., Altman, D. G., Laupacis, A., Gøtzsche, P. C., Krleža-Jerić, K., Hróbjartsson, A., Mann, H., Dickersin, K., Berlin, J. A., et al. (2013) · Annals of Internal Medicine · doi:10.7326/0003-4819-158-3-201302050-00583 · Cited by 7,363 · see citing works
Self-citationSPIRIT 2013 explanation and elaboration: guidance for protocols of clinical trials
Chan, A.-W., Tetzlaff, J. M., Gotzsche, P. C., Altman, D. G., Mann, H., Berlin, J. A., Dickersin, K., Hrobjartsson, A., Schulz, K. F., Parulekar, W. R., et al. (2013) · BMJ · doi:10.1136/bmj.e7586 · Cited by 6,251 · see citing works
Self-citationInternational evaluation of an AI system for breast cancer screening
McKinney, S. M., Sieniek, M., Godbole, V., Godwin, J., Antropova, N., Ashrafian, H., Back, T., Chesus, M., Corrado, G. S., Darzi, A., et al. (2020) · Nature · doi:10.1038/s41586-019-1799-6 · Cited by 3,287 · see citing works
Key challenges for delivering clinical impact with artificial intelligence
Kelly, C. J., Karthikesalingam, A., Suleyman, M., Corrado, G., & King, D. (2019) · BMC Medicine · doi:10.1186/s12916-019-1426-2 · Cited by 3,136 · see citing works
An overview of clinical decision support systems: Benefits, risks, and strategies for success
Sutton, R. T., Pincock, D., Baumgart, D. C., Sadowski, D. C., Fedorak, R. N., & Kroeker, K. I. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-0221-y · Cited by 3,019 · see citing works
Clinically applicable deep learning for diagnosis and referral in retinal disease
De Fauw, J., Ledsam, J. R., Romera-Paredes, B., Nikolov, S., Tomasev, N., Blackwell, S., Askham, H., Glorot, X., O’Donoghue, B., Visentin, D., et al. (2018) · Nature Medicine · doi:10.1038/s41591-018-0107-6 · Cited by 2,639 · see citing works
The practical implementation of artificial intelligence technologies in medicine
He, J., Baxter, S. L., Xu, J., Xu, J., Zhou, X., & Zhang, K. (2019) · Nature Medicine · doi:10.1038/s41591-018-0307-0 · Cited by 2,246 · see citing works
Liu, X., Faes, L., Kale, A. U., Wagner, S. K., Fu, D. J., Bruynseels, A., Mahendiran, T., Moraes, G., Shamdas, M., Kern, C., et al. (2019) · The Lancet Digital Health · doi:10.1016/s2589-7500(19)30123-2 · Cited by 1,883 · see citing works
Self-citationZech, J. R., Badgeley, M. A., Liu, M., Costa, A. B., Titano, J. J., & Oermann, E. K. (2018) · PLOS Medicine · doi:10.1371/journal.pmed.1002683 · Cited by 1,719 · see citing works
Rajpurkar, P., Irvin, J., Ball, R. L., Zhu, K., Yang, B., Mehta, H., Duan, T., Ding, D., Bagul, A., Langlotz, C. P., et al. (2018) · PLOS Medicine · doi:10.1371/journal.pmed.1002686 · Cited by 1,449 · see citing works
Guidance for Developers of Health Research Reporting Guidelines
Moher, D., Schulz, K. F., Simera, I., & Altman, D. G. (2010) · PLoS Medicine · doi:10.1371/journal.pmed.1000217 · Cited by 1,437 · see citing works
Do no harm: a roadmap for responsible machine learning for health care
Wiens, J., Saria, S., Sendak, M., Ghassemi, M., Liu, V. X., Doshi-Velez, F., Jung, K., Heller, K., Kale, D., Saeed, M., et al. (2019) · Nature Medicine · doi:10.1038/s41591-019-0548-6 · Cited by 1,266 · see citing works
Nagendran, M., Chen, Y., Lovejoy, C. A., Gordon, A. C., Komorowski, M., Harvey, H., Topol, E. J., Ioannidis, J. P. A., Collins, G. S., & Maruthappu, M. (2020) · BMJ · doi:10.1136/bmj.m689 · Cited by 1,152 · see citing works
Abràmoff, M. D., Lou, Y., Erginay, A., Clarida, W., Amelon, R., Folk, J. C., & Niemeijer, M. (2016) · Investigative Opthalmology & Visual Science · doi:10.1167/iovs.16-19964 · Cited by 1,101 · see citing works
Adversarial attacks on medical machine learning
Finlayson, S. G., Bowers, J. D., Ito, J., Zittrain, J. L., Beam, A. L., & Kohane, I. S. (2019) · Science · doi:10.1126/science.aaw4399 · Cited by 1,088 · see citing works
Wang, P., Berzin, T. M., Glissen Brown, J. R., Bharadwaj, S., Becq, A., Xiao, X., Liu, P., Li, L., Song, Y., Zhang, D., et al. (2019) · Gut · doi:10.1136/gutjnl-2018-317500 · Cited by 865 · see citing works
Self-citationCalvert, M., Kyte, D., Mercieca-Bebber, R., Slade, A., Chan, A.-W., King, M. T., and the SPIRIT-PRO Group, Hunn, A., Bottomley, A., Regnault, A., et al. (2018) · JAMA · doi:10.1001/jama.2017.21903 · Cited by 814 · see citing works
Self-citationFleuren, L. M., Klausch, T. L. T., Zwager, C. L., Schoonmade, L. J., Guo, T., Roggeveen, L. F., Swart, E. L., Girbes, A. R. J., Thoral, P., Ercole, A., et al. (2020) · Intensive Care Medicine · doi:10.1007/s00134-019-05872-y · Cited by 755 · see citing works
Reporting of artificial intelligence prediction models
Collins, G. S., & Moons, K. G. M. (2019) · The Lancet · doi:10.1016/s0140-6736(19)30037-6 · Cited by 701 · see citing works
Wijnberge, M., Geerts, B. F., Hol, L., Lemmers, N., Mulder, M. P., Berge, P., Schenk, J., Terwindt, L. E., Hollmann, M. W., Vlaar, A. P., et al. (2020) · JAMA · doi:10.1001/jama.2020.0592 · Cited by 586 · see citing works
Wang, P., Liu, X., Berzin, T. M., Glissen Brown, J. R., Liu, P., Zhou, C., Lei, L., Li, L., Guo, Z., Lei, S., et al. (2020) · The Lancet Gastroenterology & Hepatology · doi:10.1016/s2468-1253(19)30411-x · Cited by 415 · see citing works
Self-citationHidden stratification causes clinically meaningful failures in machine learning for medical imaging
Oakden-Rayner, L., Dunnmon, J., Carneiro, G., & Re, C. (2020) · Proceedings of the ACM Conference on Health, Inference, and Learning · doi:10.1145/3368555.3384468 · Cited by 411 · see citing works
Gong, D., Wu, L., Zhang, J., Mu, G., Shen, L., Liu, J., Wang, Z., Zhou, W., An, P., Huang, X., et al. (2020) · The Lancet Gastroenterology & Hepatology · doi:10.1016/s2468-1253(19)30413-3 · Cited by 393 · see citing works
Artificial intelligence in health care: accountability and safety
Habli, I., Lawton, T., & Porter, Z. (2020) · Bulletin of the World Health Organization · doi:10.2471/blt.19.237487 · Cited by 357 · see citing works
Why deep-learning AIs are so easy to fool
Heaven, D. (2019) · Nature · doi:10.1038/d41586-019-03013-5 · Cited by 342 · see citing works
Impact of a deep learning assistant on the histopathologic classification of liver cancer
Kiani, A., Uyumazturk, B., Rajpurkar, P., Wang, A., Gao, R., Jones, E., Yu, Y., Langlotz, C. P., Ball, R. L., Montine, T. J., et al. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-0232-8 · Cited by 325 · see citing works
Wu, L., Zhang, J., Zhou, W., An, P., Shen, L., Liu, J., Jiang, X., Huang, X., Mu, G., Wan, X., et al. (2019) · Gut · doi:10.1136/gutjnl-2018-317366 · Cited by 319 · see citing works
The Effect of Image Resolution on Deep Learning in Radiography
Sabottke, C. F., & Spieler, B. M. (2020) · Radiology: Artificial Intelligence · doi:10.1148/ryai.2019190015 · Cited by 312 · see citing works
Labovitz, D. L., Shafner, L., Reyes Gil, M., Virmani, D., & Hanina, A. (2017) · Stroke · doi:10.1161/strokeaha.116.016281 · Cited by 311 · see citing works
Predicting conversion to wet age-related macular degeneration using deep learning
Yim, J., Chopra, R., Spitz, T., Winkens, J., Obika, A., Kelly, C., Askham, H., Lukic, M., Huemer, J., Fasler, K., et al. (2020) · Nature Medicine · doi:10.1038/s41591-020-0867-7 · Cited by 297 · see citing works
Su, J.-R., Li, Z., Shao, X.-J., Ji, C.-R., Ji, R., Zhou, R.-C., Li, G.-C., Liu, G.-Q., He, Y.-S., Zuo, X.-L., et al. (2020) · Gastrointestinal Endoscopy · doi:10.1016/j.gie.2019.08.026 · Cited by 286 · see citing works
Sounderajah, V., Ashrafian, H., Aggarwal, R., De Fauw, J., Denniston, A. K., Greaves, F., Karthikesalingam, A., King, D., Liu, X., Markar, S. R., et al. (2020) · Nature Medicine · doi:10.1038/s41591-020-0941-1 · Cited by 270 · see citing works
Self-citationClinical applications of continual learning machine learning
Lee, C. S., & Lee, A. Y. (2020) · The Lancet Digital Health · doi:10.1016/s2589-7500(20)30102-3 · Cited by 254 · see citing works
STARE-HI—Statement on reporting of evaluation studies in Health Informatics
Talmon, J. L., Ammenwerth, E., Brender, J., DEKEIZER, N., Nykänen, P., & Rigby, M. (2008) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2008.09.002 · Cited by 237 · see citing works
Choi, K. J., Jang, J. K., Lee, S. S., Sung, Y. S., Shim, W. H., Kim, H. S., Yun, J., Choi, J., Lee, Y., Kang, B.-K., et al. (2018) · Radiology · doi:10.1148/radiol.2018180763 · Cited by 219 · see citing works
Mendes-Soares, H., Raveh-Sadka, T., Azulay, S., Edens, K., Ben-Shlomo, Y., Cohen, Y., Ofek, T., Bachrach, D., Stevens, J., Colibaseanu, D., et al. (2019) · JAMA Network Open · doi:10.1001/jamanetworkopen.2018.8102 · Cited by 212 · see citing works
Voss, C., Schwartz, J., Daniels, J., Kline, A., Haber, N., Washington, P., Tariq, Q., Robinson, T. N., Desai, M., Phillips, J. M., et al. (2019) · JAMA Pediatrics · doi:10.1001/jamapediatrics.2019.0285 · Cited by 206 · see citing works
Lin, H., Li, R., Liu, Z., Chen, J., Yang, Y., Chen, H., Lin, Z., Lai, W., Long, E., Wu, X., et al. (2019) · EClinicalMedicine · doi:10.1016/j.eclinm.2019.03.001 · Cited by 199 · see citing works
Self-citationAn artificial intelligence decision support system for the management of type 1 diabetes
Tyler, N. S., Mosquera-Lopez, C. M., Wilson, L. M., Dodier, R., Branigan, D., Gabo, V., Guillot, F. H., Hilts, W. W., Youssef, J. E., Castle, J., et al. (2020) · Nature Metabolism · doi:10.1038/s42255-020-0212-y · Cited by 147 · see citing works
Dai, L., Cheng, C.-W., Tian, R., Zhong, L. L., Li, Y.-P., Lyu, A.-P., Chan, A.-W., Shang, H.-C., & Bian, Z.-X. (2019) · Chinese Journal of Integrative Medicine · doi:10.1007/s11655-018-2999-x · Cited by 145 · see citing works
Self-citationReporting guidelines for clinical trials evaluating artificial intelligence interventions are needed
The CONSORT-AI and SPIRIT-AI Steering Group (2019) · Nature Medicine · doi:10.1038/s41591-019-0603-3 · Cited by 143 · see citing works
Caballero-Ruiz, E., García-Sáez, G., Rigla, M., Villaplana, M., Pons, B., & Hernando, M. E. (2017) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2017.02.014 · Cited by 136 · see citing works
Kim, H., Goo, J. M., Lee, K. H., Kim, Y. T., & Park, C. M. (2020) · Radiology · doi:10.1148/radiol.2020192764 · Cited by 131 · see citing works
Use of guidelines to improve the quality and transparency of reporting oral health research
Sarkis-Onofre, R., Cenci, M. S., Demarco, F. F., Lynch, C. D., Fleming, P. S., Pereira-Cenci, T., & Moher, D. (2015) · Journal of Dentistry · doi:10.1016/j.jdent.2015.01.006 · Cited by 73 · see citing works
Nicolae, A., Morton, G., Chung, H., Loblaw, A., Jain, S., Mitchell, D., Lu, L., Helou, J., Al-Hanaqta, M., Heath, E., et al. (2017) · International Journal of Radiation Oncology*Biology*Physics · doi:10.1016/j.ijrobp.2016.11.036 · Cited by 69 · see citing works
Extension of the CONSORT and SPIRIT statements
Liu, X., Faes, L., Calvert, M. J., & Denniston, A. K. (2019) · The Lancet · doi:10.1016/s0140-6736(19)31819-7 · Cited by 65 · see citing works
Self-citationProceedings of the ACM Conference on Health, Inference, and Learning
(2020) · doi:10.1145/3368555 · Cited by 53 · see citing works
Pooch, E. H. P., Ballester, P. L., & Barros, R. C. (2019) · arXiv · doi:10.48550/arxiv.1909.01940 · Cited by 32 · see citing works
Gregory, J., Welliver, S., & Chong, J. (2020) · Journal of Magnetic Resonance Imaging · doi:10.1002/jmri.27035 · Cited by 27 · see citing works
Kim, T. W. B., Gay, N., Khemka, A., & Garino, J. (2016) · JMIR Rehabilitation and Assistive Technologies · doi:10.2196/rehab.5148 · Cited by 24 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Collins, G. S., Moons, K. G. M., Dhiman, P., Riley, R. D., Beam, A. L., Van Calster, B., Ghassemi, M., Liu, X., Reitsma, J. B., van Smeden, M., et al. (2024) · BMJ · doi:10.1136/bmj-2023-078378 · Cited by 4,718 · see citing works
Self-citationRevolutionizing healthcare: the role of artificial intelligence in clinical practice
Alowais, S. A., Alghamdi, S. S., Alsuhebany, N., Alqahtani, T., Alshaya, A. I., Almohareb, S. N., Aldairem, A., Alrashed, M., Bin Saleh, K., Badreldin, H. A., et al. (2023) · BMC Medical Education · doi:10.1186/s12909-023-04698-z · Cited by 3,146 · see citing works
Deep learning-enabled medical computer vision
Esteva, A., Chou, K., Yeung, S., Naik, N., Madani, A., Mottaghi, A., Liu, Y., Topol, E., Dean, J., & Socher, R. (2021) · npj Digital Medicine · doi:10.1038/s41746-020-00376-2 · Cited by 1,406 · see citing works
Ethical and Bias Considerations in Artificial Intelligence/Machine Learning
Hanna, M. G., Pantanowitz, L., Jackson, B., Palmer, O., Visweswaran, S., Pantanowitz, J., Deebajah, M., & Rashidi, H. H. (2025) · Modern Pathology · doi:10.1016/j.modpat.2024.100686 · Cited by 535 · see citing works
The STARD-AI reporting guideline for diagnostic accuracy studies using artificial intelligence
Sounderajah, V., Guni, A., Liu, X., Collins, G. S., Karthikesalingam, A., Markar, S. R., Golub, R. M., Denniston, A. K., Shetty, S., Moher, D., et al. (2025) · Nature Medicine · doi:10.1038/s41591-025-03953-8 · Cited by 352 · see citing works
Self-citationThe Fallacy of AI Functionality
Raji, I. D., Kumar, I. E., Horowitz, A., & Selbst, A. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533158 · Cited by 251 · see citing works
Liu, X., Glocker, B., McCradden, M. M., Ghassemi, M., Denniston, A. K., & Oakden-Rayner, L. (2022) · The Lancet Digital Health · doi:10.1016/s2589-7500(22)00003-6 · Cited by 240 · see citing works
Self-citationESMO guidance for reporting oncology real-world evidence (GROW)
Castelo-Branco, L., Pellat, A., Martins-Branco, D., Valachis, A., Derksen, J. W. G., Suijkerbuijk, K. P. M., Dafni, U., Dellaporta, T., Vogel, A., Prelaj, A., et al. (2023) · Annals of Oncology · doi:10.1016/j.annonc.2023.10.001 · Cited by 247 · see citing works
Methods for clinical evaluation of artificial intelligence algorithms for medical diagnosis
Park, S. H., Han, K., Jang, H. Y., Park, J. E., Lee, J.-G., Kim, D. W., & Choi, J. (2023) · Radiology · doi:10.1148/radiol.220182 · Cited by 163 · see citing works
Manyara, A. M., Purvis, A., Ciani, O., Collins, G. S., & Taylor, R. S. (2024) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2024.111485 · Cited by 163 · see citing works
Vasey, B., Ursprung, S., Beddoe, B., Taylor, E. H., Marlow, N., Bilbro, N., Watkinson, P., & McCulloch, P. (2021) · JAMA Network Open · doi:10.1001/jamanetworkopen.2021.1276 · Cited by 154 · see citing works
SPIRIT 2025 explanation and elaboration: updated guideline for protocols of randomised trials
Hróbjartsson, A., Boutron, I., Hopewell, S., Moher, D., Schulz, K. F., Collins, G. S., Tunn, R., Aggarwal, R., Berkwits, M., Berlin, J. A., et al. (2025) · BMJ · doi:10.1136/bmj-2024-081660 · Cited by 171 · see citing works
Self-citationCai, Y., Cai, Y.-Q., Tang, L.-Y., Wang, Y.-H., Gong, M., Jing, T.-C., Li, H.-J., Li-Ling, J., Hu, W., Yin, Z., et al. (2024) · BMC Medicine · doi:10.1186/s12916-024-03273-7 · Cited by 122 · see citing works
Empirical data drift detection experiments on real-world medical imaging data
Kore, A., Abbasi Bavil, E., Subasri, V., Abdalla, M., Fine, B., Dolatabadi, E., & Abdalla, M. (2024) · Nature Communications · doi:10.1038/s41467-024-46142-w · Cited by 124 · see citing works
Continual learning in medical devices: FDA's action plan and beyond
Vokinger, K. N., Feuerriegel, S., & Kesselheim, A. S. (2021) · The Lancet Digital Health · doi:10.1016/s2589-7500(21)00076-5 · Cited by 95 · see citing works
Enhancing trust in AI through industry self-governance.
Roski, J., Maier, E. J., Vigilante, K., Kane, E. A., & Matheny, M. E. (2021) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocab065 · Cited by 83 · see citing works
Chen, Q., Chan, K. L., Guo, S., Chen, M., Lo, C. K.-M., & Ip, P. (2023) · Trauma, Violence, & Abuse · doi:10.1177/15248380221082090 · Cited by 60 · see citing works
You, J. G., Hernandez-Boussard, T., Pfeffer, M. A., Landman, A., & Mishuris, R. G. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01506-4 · Cited by 59 · see citing works
Machine Learning for Health: Algorithm Auditing & Quality Control
Oala, L., Murchison, A. G., Balachandran, P., Choudhary, S., Fehr, J., Leite, A. W., Goldschmidt, P. G., Johner, C., Schörverth, E. D. M., Nakasi, R., et al. (2021) · Journal of Medical Systems · doi:10.1007/s10916-021-01783-y · Cited by 48 · see citing works
Self-citationKoçak, B., Keleş, A., & Köse, F. (2024) · Diagnostic and Interventional Radiology · doi:10.4274/dir.2024.232604 · Cited by 12 · see citing works
Loaiza-Bonilla, A., Yost, C., Kurnaz, S., Tuysuz, E., Thaker, N. G., Giritlioglu, D., & Noel Meza, J. P. (2026) · ESMO Real World Data and Digital Oncology · doi:10.1016/j.esmorw.2026.100706 · Cited by 8 · see citing works
El Arab, R. A., Mustafa, M. H., Almagharbeh, W. T., Saleem, N. H., Al Abdulmohsen, S., Boathab, R., & Bu Washl, M. (2026) · Healthcare · doi:10.3390/healthcare14111459 · Cited by 4 · see citing works
“Reporting, evaluation, and appraisal frameworks including SPIRIT-AI, CONSORT-AI, DECIDE-AI, TRIPOD+AI, and PROBAST+AI have substantially strengthened the conceptual and reporting architecture for trustworthy clinical AI [10–14].”
InfluentialValidation is not enough: Longitudinal evidence of post-deployment fragility in clinical AI systems
Kopanitsa, G. (2026) · PLOS Digital Health · doi:10.1371/journal.pdig.0001534 · Cited by 3 · see citing works
“…primarily focus on evidence generation at the point of evaluation and provide limited guidance on how performance degradation should be detected and managed during routine clinical operation, particularly when outcome labels are delayed, incomplete, or influenced by the model’s own use [25–27].”
A quantitative analysis of global AI medical studies: gaps in randomized controlled trials
Wang, Q., Yang, Y., Qiu, L., Xu, Y., Sun, R., Xue, P., & Qiao, Y. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-02698-z · Cited by 4 · see citing works
“Such regional bias could inadvertently exacerbate existing health disparities if models were not robustly validated in varied clinical contexts 36 – 38 .”
Shen, Z., Shi, S., Mao, L., Li, C., & Xie, J. (2026) · Journal of Medical Internet Research · doi:10.2196/105599
- Authors
- Rivera, S. C., Liu, X., Chan, A.-W., Denniston, A. K., & Calvert, M. J.
- Affiliations
- University of Birmingham (GB)
- University Hospitals Birmingham NHS Foundation Trust (GB)
- University of Toronto (CA)
- Corresponding author
- Denniston, Alastair K
- Venue
- BMJ
- Publisher
- BMJ
- Published
- 2020-09-09
- Volume
- 370
- Pages
- m3210
- ISSN
- 1756-1833 (ISSN-L)
- PMC
- PMC7490785
- OpenAlex
- W3137245688
- Cited by
- 463
- References
- 51
- Citations by year
- 2026: 179 · 2025: 84 · 2024: 61 · 2023: 49 · 2022: 42 · 2021: 41 · 2020: 6 · 2018: 1
- FWCI
- 21.04
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- BMJ (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- University of Birmingham Research Portal (University of Birmingham) (publishedVersion, cc-by)
- HighWire Press Open Archive (submittedVersion)
- University of Birmingham Research Portal (University of Birmingham) (submittedVersion, cc-by)
- Spiral (Imperial College London) (submittedVersion, other-oa)
- Funders
- Microsoft Research
- Brown University
- Vanderbilt University
- University of Pennsylvania
- Harvard University
- University of Washington
- Harvard T.H. Chan School of Public Health
- McGill University
- Ottawa Hospital Research Institute
- London School of Hygiene and Tropical Medicine
- Université de Sherbrooke
- Wellcome Trust
- Green Templeton College, University of Oxford
- National Institute for Health and Care Excellence
- King's College Hospital NHS Foundation Trust
- Alan Turing Institute (TU/ASG/R-SPEH-113)
- University Hospitals Birmingham NHS Foundation Trust
- UK Research and Innovation
- DeepMind
- Amsterdam University Medical Centers
- Cancer Research UK (27294, 16895)
- National Institute for Health and Care Research (CS-2014-14-023, NIHR201334)
- University of Exeter
- Imperial College London
- King's College London
- University College London
- University of Oxford
- University of Leeds
- University of Birmingham
- Institut National de la Santé et de la Recherche Médicale
- Universiteit Leiden
- Universiteit van Amsterdam
- Women's College Hospital
- University of Toronto
- Indian Institute of Technology Madras
- Hospital for Sick Children
- Moorfields Eye Hospital NHS Foundation Trust
- Berlin Institute of Health
- Nature
- Université de Paris
- Sickkids Research Institute
- Goddard Space Flight Center
- School of Medicine, Vanderbilt University
- City, University of London
- Medical Research Council
- Engineering and Physical Sciences Research Council
- Research England
- National Heart, Lung, and Blood Institute (K01HL141771)
- National Eye Institute (K23EY029246)
- Health Data Research UK
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Clinical practice guidelines implementation
- Keywords
- artificial intelligence interventions, reporting guidelines, human-AI interaction, risk of bias, CONSORT-AI, Delphi consensus
- MeSH terms
- Artificial Intelligence, Clinical Protocols, Clinical Trials as Topic, Humans, Research Design, Consensus, Checklist
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
