Regulatory Frameworks for Development and Evaluation of Artificial Intelligence–Based Diagnostic Imaging Algorithms: Summary and Recommendations
Open accessAbstract
Although artificial intelligence (AI)-based algorithms for diagnosis hold promise for improving care, their safety and effectiveness must be ensured to facilitate wide adoption. Several recently proposed regulatory frameworks provide a solid foundation but do not address a number of issues that may prevent algorithms from being fully trusted. In this article, we review the major regulatory frameworks for software as a medical device applications, identify major gaps, and propose additional strategies to improve the development and evaluation of diagnostic AI algorithms. We identify the following major shortcomings of the current regulatory frameworks: (1) conflation of the diagnostic task with the diagnostic algorithm, (2) superficial treatment of the diagnostic task definition, (3) no mechanism to directly compare similar algorithms, (4) insufficient characterization of safety and performance elements, (5) lack of resources to assess performance at each installed site, and (6) inherent conflicts of interest. We recommend the following additional measures: (1) separate the diagnostic task from the algorithm, (2) define performance elements beyond accuracy, (3) divide the evaluation process into discrete steps, (4) encourage assessment by a third-party evaluator, (5) incorporate these elements into the manufacturers' development process. Specifically, we recommend four phases of development and evaluation, analogous to those that have been applied to pharmaceuticals and proposed for software applications, to help ensure world-class performance of all algorithms at all installed sites. In the coming years, we anticipate the emergence of a substantial body of research dedicated to ensuring the accuracy, reliability, and safety of the algorithms.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Given the high stakes nature of medical AI, human oversight is essential and increasingly required by policy makers and regulators.
“Given the high stakes nature of medical AI, human oversight is essential and increasingly required by policy makers and regulators.17 26”
From FUTURE-AI: international consensus guideline for trustworthy and deployable artificial intelligence in healthcare (2025) · cites this paper as [17]
The Introduction (page unavailable) emphasizes reducing patient risk, and Recommendation 2 (page unavailable) says the operator should be able to accept or reject an algorithm’s conclusion and monitor its performance. These passages support a role for human oversight, but the paper does not say that policymakers or regulators increasingly require it. (LLM, conf=0.82)
Experimental — accuracy not yet measuredExternal validation is ideally performed by independent researchers from other institutions or settings.
“External validation is ideally performed by independent researchers from other institutions or settings3,8,10,16,27,68,72.”
From Guidelines and quality criteria for artificial intelligence-based prediction models in healthcare: a scoping review (2022) · cites this paper as [10]
To ensure generalizability to the intended healthcare setting, developers are advised to extensively validate the model for representative data from that setting.
“To ensure the generalizability of the AIPM to the intended healthcare setting, developers are advised to extensively validate the model for representative data from that setting6–8,10,13–15,24,27,56,64,66,68,72,77,79,87,88 (see phase 1, Representativeness).”
From Guidelines and quality criteria for artificial intelligence-based prediction models in healthcare: a scoping review (2022) · cites this paper as [10]
The literature recommends assessing generalizability on external data from a different time period, place, or healthcare setting.
“The literature recommends to assess generalizability on external data from a different time period, place, or healthcare setting3,7,8,10,16,17,27,56,68,72,79.”
From Guidelines and quality criteria for artificial intelligence-based prediction models in healthcare: a scoping review (2022) · cites this paper as [10]
The AIPM should be evaluated according to how it interfaces with the end user and how well the AIPM and user perform together in a typical environment.
“The AIPM should be evaluated according to how it interfaces with the end user, and how well the AIPM and the user perform together in a typical environment10,100,102,103.”
From Guidelines and quality criteria for artificial intelligence-based prediction models in healthcare: a scoping review (2022) · cites this paper as [10]
Designing a good user interface and interaction requires consideration of cognitive load, showing only relevant information in the right context, and allowing adjustment by end users.
“Designing a good user interface and interaction requires careful consideration of the cognitive load of the end users10,18,68,99,100, by showing only relevant information in the right context, and by allowing adjustment of its behavior by end users99.”
From Guidelines and quality criteria for artificial intelligence-based prediction models in healthcare: a scoping review (2022) · cites this paper as [10]
Such evaluations can be done via reader and user studies.
“It was proposed that such evaluations can, for example, be done via reader and user studies10,102,103.”
From Guidelines and quality criteria for artificial intelligence-based prediction models in healthcare: a scoping review (2022) · cites this paper as [10]
It has been recommended that decisions are deterministic and that updates or adaptations to the AIPM happen cautiously.
“From a user interaction design perspective, it has been recommended that decisions are deterministic (consistently giving the same output for a certain input)10, and that updates of or adaptations to the AIPM should happen cautiously99.”
From Guidelines and quality criteria for artificial intelligence-based prediction models in healthcare: a scoping review (2022) · cites this paper as [10]
These mechanisms should detect situations beyond the capabilities of the AIPM and share confidence in predictions with the user.
“These mechanisms should detect situations beyond the capabilities of the AIPM10,99, and share the confidence in the predictions with the user10,22,26,99.”
From Guidelines and quality criteria for artificial intelligence-based prediction models in healthcare: a scoping review (2022) · cites this paper as [10]
As monitoring and auditing of AIPMs in practice are widely recommended, the developed software should facilitate this.
“Finally, as monitoring and auditing of AIPMs in practice are widely recommended (covered in more detail in phase 6), the developed software should facilitate this10,22,26,32,58,62,104.”
From Guidelines and quality criteria for artificial intelligence-based prediction models in healthcare: a scoping review (2022) · cites this paper as [10]
New software vulnerabilities and changes made in response to an attack should be documented and reported after thorough testing.
“When new software vulnerabilities come to light, they should be documented and reported32,61, and so should any changes made to the AIPM in response to an attack after thorough testing10,23,35,61.”
From Guidelines and quality criteria for artificial intelligence-based prediction models in healthcare: a scoping review (2022) · cites this paper as [10]
Deliberate stress tests such as load testing, penetration testing, integration testing, and unit testing are important for software verification of the AIPM.
“Deliberate stress tests like load testing, penetration testing, integration testing and unit testing are important for the verification of the AIPM from a software perspective10,26,35,46,66,79.”
From Guidelines and quality criteria for artificial intelligence-based prediction models in healthcare: a scoping review (2022) · cites this paper as [10]
Adequate logging and traceability of predictions and decisions is required, and the AIPM interface should share performance data with end users to enable ongoing monitoring of individual and aggregated cases.
“This means adequate logging and traceability of predictions and decisions is required and the AIPM interface should provide sharing of performance data with end users to enable ongoing monitoring of both individual and aggregated cases, quickly highlighting any significant deviations in performance10,26,61,66.”
From Guidelines and quality criteria for artificial intelligence-based prediction models in healthcare: a scoping review (2022) · cites this paper as [10]
A feasibility study or implementation pilot is recommended preceding an impact study to ensure correct and safe use in healthcare practice.
“A feasibility study or implementation pilot is recommended preceding an impact study to ensure correct and safe use in healthcare practice10,18,72.”
From Guidelines and quality criteria for artificial intelligence-based prediction models in healthcare: a scoping review (2022) · cites this paper as [10]
Pilot studies are recommended to examine potential pitfalls during implementation, considering software and hardware issues.
“Comparable to the feasibility study of phase 5, pilot studies are recommended to examine the potential pitfalls during implementation, considering both software and hardware issues10,18,72.”
From Guidelines and quality criteria for artificial intelligence-based prediction models in healthcare: a scoping review (2022) · cites this paper as [10]
Monitoring dataset shift is repeatedly mentioned in the literature.
“Second, the monitoring of dataset shift is also repeatedly mentioned in the literature5,10,22,72,79,91.”
From Guidelines and quality criteria for artificial intelligence-based prediction models in healthcare: a scoping review (2022) · cites this paper as [10]
Developers are recommended to regularly update AIPMs to improve predictive performance and mitigate dataset shift.
“Developers are recommended to regularly update their AIPMs over time to improve the AIPM’s predictive performance as new improvements become available and to mitigate dataset shift10,19,23.”
From Guidelines and quality criteria for artificial intelligence-based prediction models in healthcare: a scoping review (2022) · cites this paper as [10]
Monitoring an AI algorithm's performance helps identify potential biases or unintended consequences that may arise during its use.
“Additionally, monitoring the algorithm’s performance helps identify any potential biases or unintended consequences that may arise during its use [43].”
From Twelve tips for addressing ethical concerns in the implementation of artificial intelligence in medical education (2024) · cites this paper as [43]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Ai, T., Yang, Z., Hou, H., Zhan, C., Chen, C., Lv, W., Tao, Q., Sun, Z., & Xia, L. (2020) · Radiology · doi:10.1148/radiol.2020200642 · Cited by 5,697 · see citing works
Sensitivity of Chest CT for COVID-19: Comparison to RT-PCR
Fang, Y., Zhang, H., Xie, J., Lin, M., Ying, L., Pang, P., & Ji, W. (2020) · Radiology · doi:10.1148/radiol.2020200432 · Cited by 3,340 · see citing works
Key challenges for delivering clinical impact with artificial intelligence
Kelly, C. J., Karthikesalingam, A., Suleyman, M., Corrado, G., & King, D. (2019) · BMC Medicine · doi:10.1186/s12916-019-1426-2 · Cited by 3,136 · see citing works
Zech, J. R., Badgeley, M. A., Liu, M., Costa, A. B., Titano, J. J., & Oermann, E. K. (2018) · PLOS Medicine · doi:10.1371/journal.pmed.1002683 · Cited by 1,719 · see citing works
Enhancing The Reliability of Out-of-distribution Image Detection in Neural NetworksPreprint
Liang, S., Li, Y., & Srikant, R. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1706.02690 · Cited by 1,117 · see citing works
Adversarial attacks on medical machine learning
Finlayson, S. G., Bowers, J. D., Ito, J., Zittrain, J. L., Beam, A. L., & Kohane, I. S. (2019) · Science · doi:10.1126/science.aaw4399 · Cited by 1,088 · see citing works
The current radiological approach to renal cysts.
Bosniak, M. A. (1986) · Radiology · doi:10.1148/radiology.158.1.3510019 · Cited by 1,051 · see citing works
Temporal Changes of CT Findings in 90 Patients with COVID-19 Pneumonia: A Longitudinal Study
Wang, Y., Dong, C., Hu, Y., Li, C., Ren, Q., Zhang, X., Shi, H., & Zhou, M. (2020) · Radiology · doi:10.1148/radiol.2020200843 · Cited by 959 · see citing works
Prokop, M., van Everdingen, W. M., Vellinga, T. V. R., van Ufford, H. M. E. Q., Stöger, J. L., Beenen, L. F. M., Geurts, B. H. J., Gietema, H. A., Krdzalic, J., Schaefer‐Prokop, C. M., et al. (2020) · Radiology · doi:10.1148/radiol.2020201473 · Cited by 922 · see citing works
Pesapane, F., Codari, M., & Sardanelli, F. (2018) · European Radiology Experimental · doi:10.1186/s41747-018-0061-6 · Cited by 839 · see citing works
On instabilities of deep learning in image reconstruction and the potential costs of AI
Antun, V., Renna, F., Poon, C., Adcock, B., & Hansen, A. C. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1907377117 · Cited by 792 · see citing works
Simpson, S. A., Kay, F. U., Abbara, S., Bhalla, S., Chung, J. H., Chung, M., Henry, T. S., Kanne, J. P., Kligerman, S. J., Ko, J. P., et al. (2020) · Journal of Thoracic Imaging · doi:10.1097/rti.0000000000000524 · Cited by 774 · see citing works
Chest CT Features of COVID-19 in Rome, Italy
Caruso, D., Zerunian, M., Polici, M., Pucciarelli, F., Polidori, T., Rucci, C., Guido, G., Bracci, B., Dominicis, C. D., & Laghi, A. (2020) · Radiology · doi:10.1148/radiol.2020201237 · Cited by 609 · see citing works
Training Confidence-calibrated Classifiers for Detecting Out-of-Distribution SamplesPreprint
Lee, K., Lee, H., Lee, K., & Shin, J. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.09325 · Cited by 509 · see citing works
Kazerooni, E. A., Martínez, F. J., Flint, A., Jamadar, D. A., Gross, B. H., Spizarny, D. L., Cascade, P. N., Whyte, R. I., Lynch, J. P., & Toews, G. (1997) · American Journal of Roentgenology · doi:10.2214/ajr.169.4.9308447 · Cited by 468 · see citing works
Key Concepts of Clinical Trials: A Narrative Review
Umscheid, C. A., Margolis, D. J., & Grossman, C. E. (2011) · Postgraduate Medicine · doi:10.3810/pgm.2011.09.2475 · Cited by 414 · see citing works
Chang, Y., Yu, C., Chang, S., Galvin, J. R., Liu, H., Hsiao, C.-H., Kuo, P.-H., Chen, K., Franks, T. J., Huang, K.-M., et al. (2005) · Radiology · doi:10.1148/radiol.2363040958 · Cited by 370 · see citing works
Winkler, J. K., Fink, C., Toberer, F., Enk, A., Deinlein, T., Hofmann-Wellenhof, R., Thomas, L., Lallas, A., Blum, A., Stolz, W., et al. (2019) · JAMA Dermatology · doi:10.1001/jamadermatol.2019.1735 · Cited by 365 · see citing works
Learning Confidence for Out-of-Distribution Detection in Neural NetworksPreprint
DeVries, T., & Taylor, G. W. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.04865 · Cited by 354 · see citing works
Severe Acute Respiratory Syndrome: Temporal Lung Changes at Thin-Section CT in 30 Patients
OOI, G. C., Khong, P. L., Müller, N. L., Yiu, W. C., Zhou, L. J., Ho, J., LAM, B. K., Nicolaou, S., & TSANG, K. W. T. (2004) · Radiology · doi:10.1148/radiol.2303030853 · Cited by 329 · see citing works
From development to deployment: dataset shift, causality, and shift-stable models in health AI
Subbaswamy, A., & Saria, S. (2019) · Biostatistics · doi:10.1093/biostatistics/kxz041 · Cited by 276 · see citing works
Inconsistent Performance of Deep Learning Models on Mammogram Classification
Wang, X., Liang, G., Zhang, Y., Blanton, H., Bessinger, Z., & Jacobs, N. (2020) · Journal of the American College of Radiology · doi:10.1016/j.jacr.2020.01.006 · Cited by 176 · see citing works
Designing Medical Informatics Research and Library--Resource Projects to Increase What Is Learned
Stead, W. W., Haynes, R. B., Fuller, S. S., Friedman, C. P., Travis, L. E., Beck, J. R., Fenichel, C. H., Chandrasekaran, B., Buchanan, B. G., Abola, E. E., et al. (1994) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.1994.95236134 · Cited by 127 · see citing works
Huang, G.-Q., Gong, T., Wang, G., Wang, J., Guo, X., Erpeng, C., Li, S., Li, X., Yu, Y., & Lin, L. (2020) · American Journal of Roentgenology · doi:10.2214/ajr.20.23078 · Cited by 108 · see citing works
Is digital medicine different?
Lancet, T. (2018) · The Lancet · doi:10.1016/s0140-6736(18)31562-9 · Cited by 73 · see citing works
Beckett, M. K., Quiter, E. S., Ryan, G. W., Berrebi, C., Taylor, S. L., Cho, M., Pincus, H. A., & Kahn, K. L. (2011) · Implementation Science · doi:10.1186/1748-5908-6-35 · Cited by 60 · see citing works
Evaluation of clinical decision‐aids, with an application to a system for dyspepsia
Spiegelhalter, D. J. (1983) · Statistics in Medicine · doi:10.1002/sim.4780020215 · Cited by 48 · see citing works
Radiology in 2018: Are You Working with AI or Being Replaced by AI?
Bluemke, D. A. (2018) · Radiology · doi:10.1148/radiol.2018184007 · Cited by 45 · see citing works
Imaging Quality Control in the Era of Artificial Intelligence
Larson, D. B., & Boland, G. W. (2019) · Journal of the American College of Radiology · doi:10.1016/j.jacr.2019.05.048 · Cited by 44 · see citing works
Self-citationLearning Predictive Models That TransportPreprint
Subbaswamy, A., Schulam, P. F., & Saria, S. (2018) · arXiv.org · doi:10.48550/arxiv.1812.04597 · Cited by 20 · see citing works
Hecker, S. J., Preston, C., & Foote, M. (2003) · Biotechnology annual review · doi:10.1016/s1387-2656(03)09006-9 · Cited by 4 · see citing works
Flagged first, then by citations
Research works printed but not linked: 1
- [37]Ren J. Liu P.J. Fertig E. Snoek J. Poplin R. DePristo M.A. Dillon J.V. Lakshminarayanan B. Likelihood ratios for out-of-distribution detection. arXiv:1906.02845 Available at: https://arxiv.org/abs/1906.02845 Submitted June 7, 2019. Updated December 5, 2019. Accessed May 10, 2020 · arXiv
Cited by
Sorted by citations
The Role of AI in Hospitals and Clinics: Transforming Healthcare in the 21st Century
Varnosfaderani, S. M., & Forouzanfar, M. (2024) · Bioengineering · doi:10.3390/bioengineering11040337 · Cited by 1,069 · see citing works
Lekadir, K., Frangi, A. F., Porras, A. R., Glocker, B., Cintas, C., Langlotz, C. P., Weicken, E., Asselbergs, F. W., Prior, F., Collins, G. S., et al. (2025) · BMJ · doi:10.1136/bmj-2024-081554 · Cited by 957 · see citing works
“17 26 Human-in-the-loop mechanisms should be designed and implemented to perform specific quality checks (eg, to flag biases, errors, or implausible explanations), and to overrule the AI predictions when necessary.”
Self-citationde Hond, A. A. H., Leeuwenberg, A. M., Hooft, L., Kant, I. M. J., Nijman, S. W. J., van Os, H. J. A., Aardoom, J. J., Debray, T. P. A., Schuit, E., van Smeden, M., et al. (2022) · npj Digital Medicine · doi:10.1038/s41746-021-00549-7 · Cited by 499 · see citing works
Deep learning for diabetic retinopathy detection and classification based on fundus images: A review
Tsiknakis, N., Theodoropoulos, D., Manikis, G. C., Ktistakis, E., Boutsora, O., Bertó, A., Scarpa, F., Scarpa, A., Fotiadis, D. I., & Kostas Marias, A. (2021) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2021.104599 · Cited by 347 · see citing works
Qin, Z. Z., Ahmed, S., Sarker, M. S., Paul, K., Adel, A. S. S., Naheyan, T., Barrett, R., Banu, S., & Creswell, J. (2021) · The Lancet Digital Health · doi:10.1016/s2589-7500(21)00116-3 · Cited by 290 · see citing works
How does artificial intelligence in radiology improve efficiency and health outcomes?
van Leeuwen, K. G., de Rooij, M., Schalekamp, S., van Ginneken, B., & Rutten, M. J. C. M. (2022) · Pediatric Radiology · doi:10.1007/s00247-021-05114-8 · Cited by 236 · see citing works
Technology readiness levels for machine learning systems
Lavin, A., Lee, C. M., Visnjic, A., Ganju, S., Newman, D. J., Ganguly, S., Lange, D., Baydin, A. G., Sharma, A., Gibson, A., et al. (2022) · Nature Communications · doi:10.1038/s41467-022-33128-9 · Cited by 247 · see citing works
Yelne, S., Chaudhary, M., Dod, K., Sayyad, A., & Sharma, R. (2023) · Cureus · doi:10.7759/cureus.49252 · Cited by 190 · see citing works
The digital scribe in clinical practice: a scoping review and research agenda
van Buchem, M. M., Boosman, H., Bauer, M. P., Kant, I. M. J., Cammel, S. A., & Steyerberg, E. W. (2021) · npj Digital Medicine · doi:10.1038/s41746-021-00432-5 · Cited by 152 · see citing works
Franco D’Souza, R., Mathew, M., Mishra, V., & Surapaneni, K. M. (2024) · Medical Education Online · doi:10.1080/10872981.2024.2330250 · Cited by 147 · see citing works
Morley, J., Murphy, L., Mishra, A., Joshi, I., & Karpathakis, K. (2021) · JMIR Formative Research · doi:10.2196/31623 · Cited by 119 · see citing works
Parvin, N., Joo, S., Jung, J. H., & Mandal, T. K. (2025) · Nanomaterials · doi:10.3390/nano15120895 · Cited by 130 · see citing works
Jha, A. K., Bradshaw, T., Buvat, I., Hatt, M., Prabhat, K. C., Liu, C., Obuchowski, N., Saboury, B., Slomka, P. J., Sunderland, J. J., et al. (2022) · Journal of Nuclear Medicine · doi:10.2967/jnumed.121.263239 · Cited by 113 · see citing works
Optimization of Radiology Workflow with Artificial Intelligence
Ranschaert, E. R., Topff, L., & Pianykh, O. S. (2021) · Radiologic Clinics of North America · doi:10.1016/j.rcl.2021.06.006 · Cited by 103 · see citing works
Fraser, A. G., Biasin, E., Bijnens, B. H., Bruining, N., Caiani, E. G., Cobbaert, K., Davies, R. H., Gilbert, S., Hovestadt, L., Kamenjašević, E., et al. (2023) · Expert Review of Medical Devices · doi:10.1080/17434440.2023.2184685 · Cited by 103 · see citing works
Jacob, C., Brasier, N., Laurenzi, E., Heuss, S., Mougiakakou, S. G., Çöltekin, A., & Peter, M. K. (2024) · Journal of Medical Internet Research · doi:10.2196/67485 · Cited by 113 · see citing works
Voter, A. F., Meram, E., Garrett, J. W., & Yu, J. J. (2021) · Journal of the American College of Radiology · doi:10.1016/j.jacr.2021.03.005 · Cited by 89 · see citing works
Big data and predictive analytics in healthcare in Bangladesh: regulatory challenges
Hassan, S., Dhali, M., Zaman, F., & Tanveer, M. (2021) · Heliyon · doi:10.1016/j.heliyon.2021.e07179 · Cited by 82 · see citing works
Allen, B., Dreyer, K. J., Stibolt, R., Agarwal, S., Coombs, L. P., Treml, C., Elkholy, M., Brink, L., & Wald, C. (2021) · Journal of the American College of Radiology · doi:10.1016/j.jacr.2021.08.022 · Cited by 80 · see citing works
Petersen, E., Potdevin, Y., Mohammadi, E., Zidowitz, S., Breyer, S., Nowotka, D., Henn, S., Pechmann, L., Leucker, M., Rostalski, P., et al. (2022) · IEEE Access · doi:10.1109/access.2022.3178382 · Cited by 83 · see citing works
Olawade, D. B., Plabon, S. B., Ojo, A., Ogunbona, M. A., Makanjuola, B. D., & Olasilola, R. (2026) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2026.106362 · Cited by 86 · see citing works
Areas of research to support the system of radiological protection
Laurier, D., Rühm, W., Paquet, F., Applegate, K. E., Cool, D. A., Clement, C. H., & the International Commission on Radiological Protection (ICRP) (2021) · Radiation and Environmental Biophysics · doi:10.1007/s00411-021-00947-1 · Cited by 77 · see citing works
Lambin, P., Woodruff, H. C., Mali, S. A., Zhong, X., Kuang, S., Lavrova, E., Khan, H., Lekadir, K., Zwanenburg, A., Deasy, J. O., et al. (2025) · Nature Reviews Clinical Oncology · doi:10.1038/s41571-025-01067-1 · Cited by 69 · see citing works
Artificial Intelligence in Clinical Oncology: From Data to Digital Pathology and Treatment
Kumar, K. S., Miskovic, V., Blasiak, A., Sundar, R., Pedrocchi, A., Pearson, A. T., Prelaj, A., & Ho, D. (2023) · American Society of Clinical Oncology Educational Book · doi:10.1200/edbk_390084 · Cited by 64 · see citing works
Data Governance in AI - Enabled Healthcare Systems: A Case of the Project Nightingale
Arigbabu, A. T., Olaniyi, O. O., Adigwe, C. S., Adebiyi, O. O., & Ajayi, S. A. (2024) · Asian Journal of Research in Computer Science · doi:10.9734/ajrcos/2024/v17i5441 · Cited by 64 · see citing works
Stogiannos, N., Malik, R., Kumar, A., Barnes, A., Pogose, M., Harvey, H., McEntee, M. F., & Malamateniou, C. (2023) · British Journal of Radiology · doi:10.1259/bjr.20221157 · Cited by 62 · see citing works
Self-citationArtificial intelligence in smart drug delivery systems: a step toward personalized medicine
Panchpuri, M., Painuli, R., & Kumar, C. (2025) · RSC Pharmaceutics · doi:10.1039/d5pm00089k · Cited by 65 · see citing works
Lekadir, K., Osuala, R., Gallin, C., Lazrak, N., Kushibar, K., Tsakou, G., Aussó, S., Alberich, L. C., Marias, K., Tsiknakis, M., et al. (2021) · arXiv · doi:10.48550/arxiv.2109.09658 · Cited by 58 · see citing works
Expectations for Artificial Intelligence (AI) in Psychiatry
Monteith, S., Glenn, T., Geddes, J. R., Whybrow, P. C., Achtyes, E. D., & Bauer, M. (2022) · Current Psychiatry Reports · doi:10.1007/s11920-022-01378-5 · Cited by 56 · see citing works
Hassan, N., Slight, R. D., Morgan, G., Bates, D. W., Gallier, S., Sapey, E., & Slight, S. P. (2023) · BMJ Health & Care Informatics · doi:10.1136/bmjhci-2023-100784 · Cited by 56 · see citing works
Implementing Machine Learning in Interventional Cardiology: The Benefits Are Worth the Trouble
Ali, W. B., Pesaranghader, A., Avram, R., Overtchouk, P., Perrin, N., Laffite, S., Cartier, R., Ibrahim, R., Modine, T., & Hussin, J. (2021) · Frontiers in Cardiovascular Medicine · doi:10.3389/fcvm.2021.711401 · Cited by 56 · see citing works
MAIC–10 brief quality checklist for publications using artificial intelligence and medical images
Cerdá-Alberich, L., Solana, J., Mallol, P., Ribas, G., García-Junco, M., Alberich‐Bayarri, Á., & Martí‐Bonmatí, L. (2023) · Insights into Imaging · doi:10.1186/s13244-022-01355-9 · Cited by 55 · see citing works
Artificial intelligence in stroke rehabilitation: From acute care to long-term recovery
Kopalli, S. R., Shukla, M., Jayaprakash, B., Kundlas, M., Srivastava, A., Jagtap, J., Gulati, M., Chigurupati, S., Ibrahim, E., Khandige, P. S., et al. (2025) · Neuroscience · doi:10.1016/j.neuroscience.2025.03.017 · Cited by 54 · see citing works
Adversarial attacks in radiology – A systematic review
Sorin, V., Soffer, S., Glicksberg, B. S., Barash, Y., Konen, E., & Klang, E. (2023) · European Journal of Radiology · doi:10.1016/j.ejrad.2023.111085 · Cited by 47 · see citing works
Machine Learning Algorithms to Detect Subclinical Keratoconus: Systematic Review
Maile, H. P., Li, J.-P. O., Gore, D. M., Leucci, M. T., Mulholland, P. J., Hau, S., Szabó, A., Moghul, I., Balaskas, K., Fujinami, K., et al. (2021) · JMIR Medical Informatics · doi:10.2196/27363 · Cited by 46 · see citing works
Uwimana, A., Gnecco, G., & Riccaboni, M. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.109391 · Cited by 46 · see citing works
Real-World and Regulatory Perspectives of Artificial Intelligence in Cardiovascular Imaging
Wellnhofer, E. (2022) · Frontiers in Cardiovascular Medicine · doi:10.3389/fcvm.2022.890809 · Cited by 48 · see citing works
Morley, J., Morton, C. E., Karpathakis, K., Taddeo, M., & Floridi, L. (2021) · SSRN Electronic Journal · doi:10.2139/ssrn.3826358 · Cited by 46 · see citing works
van Leeuwen, K. G., Schalekamp, S., Rutten, M. J. C. M., Huisman, M., Schaefer‐Prokop, C. M., de Rooij, M., van Ginneken, B., Maresch, B., Geurts, B. H. J., van Dijke, C. F., et al. (2024) · Radiology · doi:10.1148/radiol.230981 · Cited by 38 · see citing works
Artificial intelligence in cancer research: learning at different levels of data granularity
Cirillo, D., Núñez‐Carpintero, I., & Valencia, A. (2021) · Molecular Oncology · doi:10.1002/1878-0261.12920 · Cited by 41 · see citing works
Martí‐Bonmatí, L., Koh, D., Riklund, K., Bobowicz, M., Roussakis, Y., Vilanova, J. C., Fütterer, J. J., Rimola, J., Mallol, P., Ribas, G., et al. (2022) · Insights into Imaging · doi:10.1186/s13244-022-01220-9 · Cited by 33 · see citing works
Macrae, C. (2024) · Risk Analysis · doi:10.1111/risa.14273 · Cited by 33 · see citing works
Farah, L., Borget, I., Martelli, N., & Vallée, A. (2024) · Journal of Medical Internet Research · doi:10.2196/51514 · Cited by 35 · see citing works
Artificial intelligence (AI) and ethical concerns: a review and research agenda
Ghosh, M. (2025) · Cogent Business & Management · doi:10.1080/23311975.2025.2551809 · Cited by 41 · see citing works
Kukhareva, P., Weir, C. R., Fiol, G. D., Aarons, G. A., Taft, T., Schlechter, C. R., Reese, T. J., Curran, R. L., Nanjo, C. J., Borbolla, D. A., et al. (2022) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2022.104014 · Cited by 37 · see citing works
Reliable and Safe Use of Machine Translation in Medical Settings
Mehandru, N., Robertson, S., & Salehi, N. (2022) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3531146.3533244 · Cited by 37 · see citing works
External Validation of e‐ASPECTS Software for Interpreting Brain CT in Stroke
Mair, G., White, P., Bath, P. M., Muir, K. W., Salman, R. A., Martin, C., Dye, D., Chappell, F. M., Vacek, A., von Kummer, R., et al. (2022) · Annals of Neurology · doi:10.1002/ana.26495 · Cited by 31 · see citing works
Changes in software as a medical device based on artificial intelligence technologies
Zinchenko, V. V., Четвериков, С. Ф., Akhmad, E. S., Arzamasov, K. M., Vladzymyrskyy, A. V., Андрейченко, А. Е., & Morozov, S. P. (2022) · International Journal of Computer Assisted Radiology and Surgery · doi:10.1007/s11548-022-02669-1 · Cited by 30 · see citing works
Vasiliev, Y. A., Vlazimirsky, A. V., Omelyanskaya, O. V., Arzamasov, K. M., Четвериков, С. Ф., Андреевич, Р. Д., & Zelenova, M. A. (2023) · Digital Diagnostics · doi:10.17816/dd321971 · Cited by 28 · see citing works
Alsadoun, L., Ali, H., Mushtaq, M. M., Mushtaq, M., Burhanuddin, M., Anwar, R., Liaqat, M., Bokhari, S. F. H., Hasan, A. H., & Ahmed, F. (2024) · Cureus · doi:10.7759/cureus.67844 · Cited by 31 · see citing works
Lessons learned in transitioning to AI in the medical imaging of COVID-19
Naqa, I. M. E., Hu, Q., Chen, W., Li, H., Fuhrman, J. D., Gorre, N., & Giger, M. L. (2021) · Journal of medical imaging · doi:10.1117/1.jmi.8.s1.010902 · Cited by 29 · see citing works
Zhelev, Z., Peters, J., Rogers, M., Allen, M. J., Kijauskaite, G., Seedat, F., Wilkinson, E. C., & Hyde, C. (2023) · Journal of Medical Screening · doi:10.1177/09691413221144382 · Cited by 30 · see citing works
Chauhan, A. S., Singh, R., Priyadarshi, N., Twala, B., Suthar, S., & Swami, S. (2024) · Discover Artificial Intelligence · doi:10.1007/s44163-024-00161-0 · Cited by 28 · see citing works
RetractedAkintuyi, O. B. (2024) · Open Access Research Journal of Engineering and Technology · doi:10.53022/oarjet.2024.6.2.0017 · Cited by 24 · see citing works
Hayat, H., Nukala, A., Nyamira, A., Fan, J., & Wang, P. (2021) · Biomedical Materials · doi:10.1088/1748-605x/ac15b2 · Cited by 26 · see citing works
van der Werff, S. D., van Rooden, S. M., Henriksson, A., Behnke, M., Aghdassi, S. J. S., van Mourik, M. S. M., & Nauclér, P. (2025) · Journal of Internal Medicine · doi:10.1111/joim.20100 · Cited by 25 · see citing works
Tozzi, A. E., Fabozzi, F., Eckley, M., Croci, I., Dell’Anna, V. A., Colantonio, E., & Mastronuzzi, A. (2022) · Frontiers in Oncology · doi:10.3389/fonc.2022.905770 · Cited by 22 · see citing works
AI-Based CXR First Reading: Current Limitations to Ensure Practical Value
Vasilev, Y. A., Vladzymyrskyy, A. V., Omelyanskaya, O. V., Blokhin, I. A., Kirpichev, Y., & Arzamasov, K. M. (2023) · Diagnostics · doi:10.3390/diagnostics13081430 · Cited by 22 · see citing works
Zippel, C., & Bohnet-Joschko, S. (2021) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph18105072 · Cited by 21 · see citing works
NAVIGATOR: an Italian regional imaging biobank to promote precision medicine for oncologic patients
Borgheresi, R., Barucci, A., Colantonio, S., Aghakhanyan, G., Assante, M., Bertelli, E., Carlini, E., Carpi, R., Caudai, C., Cavallero, D., et al. (2022) · European Radiology Experimental · doi:10.1186/s41747-022-00306-9 · Cited by 20 · see citing works
Młynarska, E., Bojdo, K., Frankenstein, H., Kustosik, N., Mstowska, W., Przybylak, A., Rysz, J., & Franczyk, B. (2025) · Journal of Clinical Medicine · doi:10.3390/jcm14030887 · Cited by 20 · see citing works
Federated Learning in Healthcare: From Research to Real-World Deployment
Bakas, S., Li, X., Shah, P., & Roth, H. R. (2026) · Annual Review of Biomedical Engineering · doi:10.1146/annurev-bioeng-080125-041414 · Cited by 19 · see citing works
The ABC recommendations for validation of supervised machine learning results in biomedical sciences
Chicco, D., & Jurman, G. (2022) · Frontiers in Big Data · doi:10.3389/fdata.2022.979465 · Cited by 21 · see citing works
Regulatory challenges in ai-based diagnostics: Legal implications of ai use in medical diagnostics
Naili, Y. T., Mangkunegara, I. S., Purwono, P., & Baballe, M. A. (2025) · BIO Web of Conferences · doi:10.1051/bioconf/202515201034 · Cited by 17 · see citing works
The Road Map for ACR Practice Accreditation for Radiology Artificial Intelligence
Larson, D. B., Bhargavan, M., Tilkin, M., Coombs, L. P., & Wald, C. (2025) · Journal of the American College of Radiology · doi:10.1016/j.jacr.2025.02.008 · Cited by 15 · see citing works
Self-citationResponsible adoption of multimodal artificial intelligence in health care: promises and challenges
Azarfar, G., Naimimohasses, S., Rambhatla, S., Komorowski, M., Ferro, D., Lewis, P. R., Gates, D., Shara, N. M., Gascon, G. M., Chang, A. C., et al. (2025) · The Lancet Digital Health · doi:10.1016/j.landig.2025.100917 · Cited by 14 · see citing works
Larson, D. B., Doo, F. X., Allen, B., Mongan, J., Flanders, A. E., & Wald, C. (2024) · Journal of the American College of Radiology · doi:10.1016/j.jacr.2024.01.024 · Cited by 14 · see citing works
Self-citationA nested model for AI design and validation
Dubey, A., Yang, Z., & Hattab, G. (2024) · iScience · doi:10.1016/j.isci.2024.110603 · Cited by 15 · see citing works
Human Dignity and Artificial Intelligence in Healthcare: A Basis for a Catholic Ethics on AI
Gozum, I. E. A., & Flake, C. C. (2024) · Journal of Religion and Health · doi:10.1007/s10943-024-02206-1 · Cited by 15 · see citing works
Hao, X., Wang, Y., Li, K., Zhu, T., & Herasevich, V. (2025) · Anesthesiology and Perioperative Science · doi:10.1007/s44254-025-00136-z · Cited by 14 · see citing works
Challenges of Machine Learning and AI (What Is Next?), Responsible and Ethical AI
Gkontra, P., Quaglio, G., Garmendia, A. T., & Lekadir, K. (2023) · Springer eBooks · doi:10.1007/978-3-031-36678-9_17 · Cited by 17 · see citing works
Sindhu, A., Jadhav, U. S., Ghewade, B., Bhanushali, J. D., & Yadav, P. (2024) · Cureus · doi:10.7759/cureus.57657 · Cited by 17 · see citing works
Exploring bias risks in artificial intelligence and targeted medicines manufacturing
Nwebonyi, N., & McKay, F. (2024) · BMC Medical Ethics · doi:10.1186/s12910-024-01112-1 · Cited by 15 · see citing works
Bhimani, M., Miller, A., Agnew, J. D., Ausin, M. S., Raglow-Defranco, M., Mangat, H. S., Voisard, M., Taylor, M., Bierman-Lytle, S., Parikh, V., et al. (2025) · medRxiv · doi:10.1101/2025.03.17.25324157 · Cited by 13 · see citing works
Ammann, C., Hadler, T., Gröschel, J., Kolbitsch, C., & Schulz‐Menger, J. (2023) · Frontiers in Cardiovascular Medicine · doi:10.3389/fcvm.2023.1118499 · Cited by 12 · see citing works
Mehrtabar, S., Marey, A., Desai, A., Saad, A. M., Desai, V. S., Goñi, J., Pal, B., & Umair, M. (2025) · Journal of Imaging Informatics in Medicine · doi:10.1007/s10278-025-01656-7 · Cited by 14 · see citing works
A review of big data technology and its application in cancer care
Xiao, T., Kong, S., Zhang, Z., Hua, D., & Liu, F. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.108577 · Cited by 12 · see citing works
Artificial intelligence methods available for cancer research
Murmu, A., & Győrffy, B. (2024) · Frontiers of Medicine · doi:10.1007/s11684-024-1085-3 · Cited by 12 · see citing works
Regulating AI Adaptation: An Analysis of AI Medical Device UpdatesPreprint
Wu, K., Wu, E., Rodolfa, K. T., Ho, D. E., & Zou, J. (2024) · medRxiv · doi:10.1101/2024.06.26.24309506 · Cited by 11 · see citing works
Federated multimodal AI for precision-equitable diabetes care
Bai, B., Liu, X., & Li, H. Q. (2026) · Frontiers in Digital Health · doi:10.3389/fdgth.2025.1678047 · Cited by 10 · see citing works
Risk Assessment and Classification of Medical Device Software for the Internet of Medical Things
Brass, I., & Mkwashi, A. (2022) · International Conference on the Internet of Things (IoT) · doi:10.1145/3567445.3571104 · Cited by 11 · see citing works
Antaki, F., Hammana, I., Tessier, M.-C., Boucher, A., Jetté, M. L. D., Beauchemin, C., Hammamji, K., Ong, A. Y., Rhéaume, M., Gauthier, D., et al. (2024) · JMIR Diabetes · doi:10.2196/59867 · Cited by 11 · see citing works
NESTANets: stable, accurate and efficient neural networks for analysis-sparse inverse problems
Neyra-Nesterenko, M., & Adcock, B. (2022) · Sampling Theory Signal Processing and Data Analysis · doi:10.1007/s43670-022-00043-5 · Cited by 10 · see citing works
Ethical and societal aspects of radiological protection for offspring and next generations
Zölzer, F., Schneider, T., Ainsbury, E. A., Goto, A., Liutsko, L., O’Reilly, G., & Lochard, J. (2023) · International Journal of Radiation Biology · doi:10.1080/09553002.2023.2281523 · Cited by 9 · see citing works
Advancing UK Regulatory Science Strategy in the Context of Global Regulation: a Stakeholder Survey
Rivera, S. C., Torlińska, B., Marston, E., Denniston, A. K., Oliver, K. R., Hoare, S., & Calvert, M. (2021) · Therapeutic Innovation & Regulatory Science · doi:10.1007/s43441-021-00263-2 · Cited by 9 · see citing works
Artificial intelligence optimizes the standardized diagnosis and treatment of chronic sinusitis
Liu, Y.-Y., Jiang, S.-P., & Wang, Y. (2025) · Frontiers in Physiology · doi:10.3389/fphys.2025.1522090 · Cited by 10 · see citing works
Sorin, V., Korfiatis, P. D., Bratt, A. K., Leiner, T., Wald, C., Butler, C. S., Cook, C. J., Kline, T. L., & Collins, J. D. (2025) · Journal of the American College of Radiology · doi:10.1016/j.jacr.2025.06.036 · Cited by 9 · see citing works
Generative spaces: collaboration, learning, and innovation in a regulatory sandbox
Macrae, C., & Ansell, C. (2026) · Journal of Public Administration Research and Theory · doi:10.1093/jopart/muag024 · Cited by 10 · see citing works
Skepu, A., Phakathi, B., Makgoka, M., Mbita, Z., Damane, B. P., Demetriou, D., & Dlamini, Z. (2023) · Springer eBooks · doi:10.1007/978-3-031-21506-3_9 · Cited by 8 · see citing works
Fogleman, B. M., Goldman, M., Holland, A. B., Dyess, G. A., & Patel, A. (2024) · Cureus · doi:10.7759/cureus.58032 · Cited by 12 · see citing works
You, J., Hu, J., & Jiang, B. (2025) · Sustainable Cities and Society · doi:10.1016/j.scs.2025.106746 · Cited by 8 · see citing works
Yilmaz, R., Alsayegh, A., Bakhaidar, M., Fazlollahi, A. M., Hamdan, N. A., Tee, T., Shalmiev, A., Laroche, D., Santaguida, C., Donoho, D. A., et al. (2025) · npj Artificial Intelligence · doi:10.1038/s44387-025-00032-8 · Cited by 10 · see citing works
Diagnostic and Prognostic Electrocardiogram-Based Models for Rapid Clinical Applications
Islam, M. S., Kalmady, S. V., Hindle, A., Sandhu, R. K., Sun, W., Sepehrvand, N., Greiner, R., & Kaul, P. (2024) · Canadian Journal of Cardiology · doi:10.1016/j.cjca.2024.07.003 · Cited by 9 · see citing works
Impact of random outliers in auto-segmented targets on radiotherapy treatment plans for glioblastoma
Poel, R., Rüfenacht, E., Ermiş, E., Müller, M., Fix, M. K., Aebersold, D. M., Manser, P., & Reyes, M. (2022) · Radiation Oncology · doi:10.1186/s13014-022-02137-9 · Cited by 7 · see citing works
Regulatory and Implementation Considerations for Artificial Intelligence
Chen, S., & Lobo, B. C. (2024) · Otolaryngologic Clinics of North America · doi:10.1016/j.otc.2024.04.007 · Cited by 6 · see citing works
Doria, A. (2023) · Expert Review of Hematology · doi:10.1080/17474086.2023.2192474 · Cited by 9 · see citing works
Wang, Y., Fu, W., Zhang, Y., Wang, D., Gu, Y., Wang, W., Xu, H. M., Ge, X., Ye, C., Fang, J., et al. (2024) · Scientific Reports · doi:10.1038/s41598-024-64893-w · Cited by 6 · see citing works
Openness and Transparency in the Evaluation of Bias in Artificial Intelligence
Larson, D. B. (2022) · Radiology · doi:10.1148/radiol.222263 · Cited by 6 · see citing works
Self-citationBuilding Purpose-Driven Radiology Departments: Distributive and Servant Leadership in Action
Kamran, R., Patlas, M. N., Cecere, M., & Doria, A. S. (2026) · Canadian Association of Radiologists Journal · doi:10.1177/08465371261437071 · Cited by 5 · see citing works
Controversies in Artificial Intelligence in Neurosurgery
Yilmaz, R., Browd, S. R., & Donoho, D. A. (2024) · Neurosurgery Clinics of North America · doi:10.1016/j.nec.2024.08.008 · Cited by 5 · see citing works
- Authors
- Larson, D. B., Harvey, H., Rubin, D. L., Irani, N., Tse, J. R., & Langlotz, C. P.
- Affiliations
- Stanford Medicine (US)
- University College London (GB)
- University of Kansas Medical Center (US)
- University of California, Los Angeles (US)
- Corresponding author
- Larson, David B.
- Venue
- Journal of the American College of Radiology
- Publisher
- Elsevier BV
- Published
- 2021-03
- Volume
- 18
- Issue
- 3
- Pages
- 413-424
- ISSN
- 1546-1440 (ISSN-L)
- PMC
- PMC7574690
- OpenAlex
- W3093602247
- Cited by
- 198
- References
- 28
- Citations by year
- 2026: 38 · 2025: 51 · 2024: 36 · 2023: 24 · 2022: 29 · 2021: 20
- FWCI
- 1.39
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Journal of the American College of Radiology (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- Funders
- International Business Machines Corporation
- GE Healthcare
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Health Professions › Health Information Management › Healthcare Regulation
- Physical Sciences › Engineering › Safety, Risk, Reliability and Quality › Safety Systems Engineering in Autonomy
- Keywords
- artificial intelligence, diagnostic imaging algorithm, regulatory framework, software as a medical device, algorithm evaluation, performance characterization, third-party evaluation, conflicts of interest, medical device regulation
- MeSH terms
- Algorithms, Artificial Intelligence, Diagnostic Imaging, Software, Reproducibility of Results
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
