Evaluating artificial intelligence in medicine: phases of clinical research
Open accessAbstract
Increased scrutiny of artificial intelligence (AI) applications in healthcare highlights the need for real-world evaluations for effectiveness and unintended consequences. The complexity of healthcare, compounded by the user- and context-dependent nature of AI applications, calls for a multifaceted approach beyond traditional in silico evaluation of AI. We propose an interdisciplinary, phased research framework for evaluation of AI implementations in healthcare. We draw analogies to and highlight differences from the clinical trial phases for drugs and medical devices, and we present study design and methodological guidance for each stage.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Successful pilot deployments act as a springboard for broader adoption.
“Successful pilot deployments act as a springboard for broader adoption [58].”
From Use of artificial intelligence in critical care: opportunities and obstacles (2024) · cites this paper as [58]
Table 1 (Introduction, ‘AI versus AI implementation in healthcare’; page unavailable) lists a clinical pilot study as a phase 2 evaluation method. The ‘Phase 2: Efficacy and side effects’ section says validated improvements can justify larger phase 3 trials, but it does not say successful pilot deployments lead to broader adoption. (LLM, conf=0.81)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Austin, P. C. (2011) · Multivariate Behavioral Research · doi:10.1080/00273171.2011.568786 · Cited by 12,242 · see citing works
High-performance medicine: the convergence of human and artificial intelligence
Topol, E. J. (2019) · Nature Medicine · doi:10.1038/s41591-018-0300-7 · Cited by 10,450 · see citing works
I.—COMPUTING MACHINERY AND INTELLIGENCE
TURING, A. M. (1950) · Mind · doi:10.1093/mind/lix.236.433 · Cited by 10,290 · see citing works
Gulshan, V., Peng, L., Coram, M., Stumpe, M. C., Wu, D., Narayanaswamy, A., Venugopalan, S., Widner, K., Madams, T., Cuadros, J., et al. (2016) · JAMA · doi:10.1001/jama.2016.17216 · Cited by 7,697 · see citing works
Computing Machinery and Intelligence (1950)
Turing, A. M. (2004) · doi:10.1093/oso/9780198250791.003.0017 · Cited by 6,120 · see citing works
Marginal Structural Models and Causal Inference in Epidemiology
Robins, J. M., Hernán, M. Á., & Brumback, B. (2000) · Epidemiology · doi:10.1097/00001648-200009000-00011 · Cited by 5,581 · see citing works
Matching Methods for Causal Inference: A Review and a Look Forward
Stuart, E. A. (2010) · Statistical Science · doi:10.1214/09-sts313 · Cited by 5,478 · see citing works
Estimating causal effects from epidemiological data
Hernán, M. Á., & Robins, J. M. (2006) · Journal of Epidemiology & Community Health · doi:10.1136/jech.2004.029496 · Cited by 1,029 · see citing works
The ethics of artificial intelligence
Bostrom, N., & Yudkowsky, E. (2014) · The Cambridge Handbook of Artificial Intelligence · doi:10.1017/cbo9781139046855.020 · Cited by 918 · see citing works
Cognitive and usability engineering methods for the evaluation of clinical information systems.
Kushniruk, A. W., & Patel, V. L. (2004) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2004.01.003 · Cited by 760 · see citing works
The coming of age of artificial intelligence in medicine
Patel, V. L., Shortliffe, E. H., Mario Stefanelli, A., Peter Szolovits, B., Berthold, M. R., Bellazzi, R., & Abu–Hanna, A. (2008) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2008.07.017 · Cited by 629 · see citing works
Adapting to Artificial Intelligence: Radiologists and Pathologists as Information Specialists
Jha, S., & Topol, E. J. (2016) · JAMA · doi:10.1001/jama.2016.17438 · Cited by 622 · see citing works
Bahrammirzaee, A. (2010) · Neural Computing and Applications · doi:10.1007/s00521-010-0362-z · Cited by 565 · see citing works
How artificial intelligence is changing drug discovery
Fleming, N. (2018) · Nature · doi:10.1038/d41586-018-05267-x · Cited by 546 · see citing works
Introduction to Artificial Intelligence
Ertel, W. (2017) · Undergraduate Topics in Computer Science · doi:10.1007/978-3-319-58487-4 · Cited by 524 · see citing works
Google DeepMind and healthcare in an age of algorithms
Powles, J., & Hodson, H. (2017) · Health and Technology · doi:10.1007/s12553-017-0179-1 · Cited by 511 · see citing works
What This Computer Needs Is a Physician: Humanism and Artificial Intelligence
Verghese, A., Shah, N. H., & Harrington, R. A. (2018) · JAMA · doi:10.1001/jama.2017.19198 · Cited by 468 · see citing works
The use of artificial neural networks in decision support in cancer: A systematic review
Lisboa, P. J., & Taktak, A. F. G. (2006) · Neural Networks · doi:10.1016/j.neunet.2005.10.007 · Cited by 462 · see citing works
Evaluation of health information systems—problems and challenges
Ammenwerth, E., Gräber, S., Herrmann, G., Bürkle, T., & König, J. (2003) · International Journal of Medical Informatics · doi:10.1016/s1386-5056(03)00131-x · Cited by 414 · see citing works
Kaplan, B. (2001) · International Journal of Medical Informatics · doi:10.1016/s1386-5056(01)00184-8 · Cited by 380 · see citing works
Takahashi, H., Tampo, H., Arai, Y., Inoue, Y., & Kawashima, H. (2017) · PLoS ONE · doi:10.1371/journal.pone.0179790 · Cited by 259 · see citing works
Safe and Sound: Artificial Intelligence in Hazardous Applications
Fox, J. P., & Das, S. K. (2000) · Medical Entomology and Zoology · Cited by 234 · see citing works
Evaluation Methods in Biomedical Informatics
Friedman, C. P., & Wyatt, J. C. (2006) · Computers in health care · doi:10.1007/0-387-30677-3 · Cited by 187 · see citing works
Hendler, J. (2008) · IEEE Intelligent Systems · doi:10.1109/mis.2008.20 · Cited by 170 · see citing works
From idea to market: the drug approval process.
Lipsky, M. S., & Sharp, L. K. (2001) · PubMed · Cited by 170 · see citing works
An FDA Viewpoint on Unique Considerations for Medical-Device Clinical Trials
Faris, O., & Shuren, J. E. (2017) · New England Journal of Medicine · doi:10.1056/nejmra1512592 · Cited by 152 · see citing works
The Cambridge handbook of artificial intelligence
(2015) · Choice Reviews Online · doi:10.5860/choice.187061 · Cited by 146 · see citing works
Designing Medical Informatics Research and Library--Resource Projects to Increase What Is Learned
Stead, W. W., Haynes, R. B., Fuller, S. S., Friedman, C. P., Travis, L. E., Beck, J. R., Fenichel, C. H., Chandrasekaran, B., Buchanan, B. G., Abola, E. E., et al. (1994) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.1994.95236134 · Cited by 127 · see citing works
Artificial intelligence in health care: within touching distance
Lancet, T. (2017) · The Lancet · doi:10.1016/s0140-6736(17)31540-4 · Cited by 117 · see citing works
AI TURNS FIFTY: REVISITING ITS ORIGINS
Cordeschi, R. (2007) · Applied Artificial Intelligence · doi:10.1080/08839510701252304 · Cited by 108 · see citing works
Kaufman, D. R., Roberts, W. D., Merrill, J. A., Lai, T.-Y., & Henry, S. B. (2006) · Nursing Research · doi:10.1097/00006199-200603001-00007 · Cited by 81 · see citing works
Is digital medicine different?
Lancet, T. (2018) · The Lancet · doi:10.1016/s0140-6736(18)31562-9 · Cited by 73 · see citing works
Activity Recognition for Diabetic Patients Using a Smartphone
Cvetković, B., Janko, V., Romero, A. E., Kafalı, Ö., Stathis, K., & Luštrek, M. (2016) · Journal of medical systems · doi:10.1007/s10916-016-0598-y · Cited by 33 · see citing works
Introduction to Artificial Intelligence
Wu, A., Caddell, J., & MacNeil, C. (2019) · Cited by 32 · see citing works
Flagged first, then by citations
Research works printed but not linked: 4
- [21]Core MG, Lane HC, van Lent M, Gomboc D, Solomon S, Rosenberg M. Building explainable artificial intelligence systems. In: Proceedings of the 18th Conference on Innovative Applications of Artificial Intelligence, vol. 2; 2006; 1766– 73.
- [32]Liao QV, Gruen D, Miller S. Questioning the Ai: toward design practices for explainable AI user experiences.In: proceedings of the 2020 CHI Conference on Human Factors in Computing Systems; 2020.
- [33]Saleiro P, Kuester B, Hinkson L, et al Aequitas: a bias and fairness audit toolkit. arXiv: 1811.05577. 2018. · arXiv
- [34]Bellamy RKE, Dey K, Hind M, et al AI fairness 360: an extensible toolkit for detecting, understanding, and mitigating unwanted algorithmic bias. arXiv: 1810.01943. 2018. · arXiv
Cited by
Sorted by citations
Vasey, B., Nagendran, M., Campbell, B., Clifton, D. A., Collins, G. S., Denaxas, S., Denniston, A. K., Faes, L., Geerts, B., Ibrahim, M., et al. (2022) · Nature Medicine · doi:10.1038/s41591-022-01772-9 · Cited by 1,039 · see citing works
Vasey, B., Nagendran, M., Campbell, B., Clifton, D. A., Collins, G. S., Denaxas, S., Denniston, A. K., Faes, L., Geerts, B., Ibrahim, M., et al. (2022) · BMJ · doi:10.1136/bmj-2022-070904 · Cited by 665 · see citing works
Artificial intelligence for clinical oncology
Kann, B. H., Hosny, A., & Aerts, H. J. W. L. (2021) · Cancer Cell · doi:10.1016/j.ccell.2021.04.002 · Cited by 387 · see citing works
Competencies for the Use of Artificial Intelligence–Based Tools by Health Care Professionals
Russell, R. G., Lovett Novak, L., Patel, M., Garvey, K. V., Craig, K. J. T., Jackson, G. P., Moore, D., & Miller, B. M. (2023) · Academic Medicine · doi:10.1097/acm.0000000000004963 · Cited by 320 · see citing works
Self-citationArtificial intelligence in oncology: path to implementation
Chua, I. S., Gaziel‐Yablowitz, M., Korach, Z. T., Kehl, K. L., Levitan, N. A., Arriaga, Y. E., Jackson, G. P., Bates, D. W., & Hassett, M. (2021) · Cancer Medicine · doi:10.1002/cam4.3935 · Cited by 179 · see citing works
Self-citationUse of artificial intelligence in critical care: opportunities and obstacles
Pinsky, M. R., Bedoya, A., Bihorac, A., Celi, L., Churpek, M., Economou-Zavlanos, N. J., Elbers, P., Saria, S., Liu, V., Lyons, P. G., et al. (2024) · Critical Care · doi:10.1186/s13054-024-04860-z · Cited by 160 · see citing works
“Successful pilot deployments act as a springboard for broader adoption [58].”
BackgroundCrossnohere, N. L., Elsaid, M. I., Paskett, J., Bose‐Brill, S., & Bridges, J. F. P. (2022) · Journal of Medical Internet Research · doi:10.2196/36823 · Cited by 131 · see citing works
Tsopra, R., Fernández, X. M., Luchinat, C., Alberghina, L., Lehrach, H. R., Vanoni, M., Dreher, F., Sezerman, O. U., Marc Cuggia, A., de Tayrac, M., et al. (2021) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-021-01634-3 · Cited by 117 · see citing works
A Research Ethics Framework for the Clinical Translation of Healthcare Machine Learning
McCradden, M. D., Anderson, J. A., Stephenson, E. A., Drysdale, E., Erdman, L., Goldenberg, A., & Shaul, R. Z. (2022) · The American Journal of Bioethics · doi:10.1080/15265161.2021.2013977 · Cited by 117 · see citing works
Implementation frameworks for end-to-end clinical AI: derivation of the SALIENT framework
van der Vegt, A. H., Scott, I. A., Dermawan, K., Schnetler, R. J., Kalke, V. R., & Lane, P. J. (2023) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocad088 · Cited by 109 · see citing works
Stidham, R. W., & Takenaka, K. (2022) · Gastroenterology · doi:10.1053/j.gastro.2021.12.238 · Cited by 108 · see citing works
Embi, P. J. (2021) · JAMA Network Open · doi:10.1001/jamanetworkopen.2021.4622 · Cited by 99 · see citing works
Syrowatka, A., Song, W., Amato, M. G., Foer, D., Edrees, H. H., Co, Z., Kuznetsova, M., Dulgarian, S., Seger, D. L., Simona, A., et al. (2021) · The Lancet Digital Health · doi:10.1016/s2589-7500(21)00229-6 · Cited by 95 · see citing works
Self-citationFraser, A. G., Biasin, E., Bijnens, B. H., Bruining, N., Caiani, E. G., Cobbaert, K., Davies, R. H., Gilbert, S., Hovestadt, L., Kamenjašević, E., et al. (2023) · Expert Review of Medical Devices · doi:10.1080/17434440.2023.2184685 · Cited by 103 · see citing works
Drummond, D., & Gonsard, A. (2024) · Journal of Medical Internet Research · doi:10.2196/58504 · Cited by 95 · see citing works
Artificial Intelligence–enabled Decision Support in Surgery
Loftus, T. J., Altieri, M. S., Balch, J. A., Abbott, K. L., Choi, J., Marwaha, J. S., Hashimoto, D. A., Brat, G. A., Raftopoulos, Y., Evans, H. L., et al. (2023) · Annals of Surgery · doi:10.1097/sla.0000000000005853 · Cited by 92 · see citing works
Self-citationHah, H., & Goldin, D. S. (2021) · Journal of Medical Internet Research · doi:10.2196/33540 · Cited by 81 · see citing works
Translation of AI into oncology clinical practice
Naqa, I. E., Karolak, A. M., Luo, Y., Folio, L. R., Tarhini, A. A., Rollison, D. E. M., & Parodi, K. (2023) · Oncogene · doi:10.1038/s41388-023-02826-z · Cited by 83 · see citing works
Jain, A., Brooks, J. R., Alford, C. C., Chang, C. S., Mueller, N. M., Umscheid, C. A., & Bierman, A. S. (2023) · JAMA Health Forum · doi:10.1001/jamahealthforum.2023.1197 · Cited by 79 · see citing works
Informed consent for artificial intelligence in emergency medicine: A practical guide
Iserson, K. V. (2023) · The American Journal of Emergency Medicine · doi:10.1016/j.ajem.2023.11.022 · Cited by 73 · see citing works
Lekadir, K., Osuala, R., Gallin, C., Lazrak, N., Kushibar, K., Tsakou, G., Aussó, S., Alberich, L. C., Marias, K., Tsiknakis, M., et al. (2021) · arXiv · doi:10.48550/arxiv.2109.09658 · Cited by 58 · see citing works
Vasey, B., Novak, A., Ather, S., Ibrahim, M., & McCulloch, P. G. (2023) · Clinical Radiology · doi:10.1016/j.crad.2022.09.131 · Cited by 58 · see citing works
The use of AI in mental health services to support decision-making: scoping review
Auf, H., Svedberg, P., Nygren, J., Nair, M., & Lundgren, L. E. (2025) · Journal of Medical Internet Research · doi:10.2196/63548 · Cited by 56 · see citing works
Lu, J., Callahan, A., Patel, B., Morse, K., Dash, D., Pfeffer, M. A., & Shah, N. H. (2022) · JAMA Network Open · doi:10.1001/jamanetworkopen.2022.27779 · Cited by 52 · see citing works
A normative framework for artificial intelligence as a sociotechnical system in healthcare
McCradden, M. D., Joshi, S. D., Anderson, J. A., & London, A. J. (2023) · Patterns · doi:10.1016/j.patter.2023.100864 · Cited by 49 · see citing works
Scoping review of the current landscape of AI-based applications in clinical trials
Cascini, F., Beccia, F., Causio, F. A., Melnyk, A., Zaino, A., & Ricciardi, W. (2022) · Frontiers in Public Health · doi:10.3389/fpubh.2022.949377 · Cited by 45 · see citing works
Clinical use of artificial intelligence requires AI-capable organizations
Novak, L. L., Russell, R. G., Garvey, K., Patel, M., Thomas Craig, K. J., Snowdon, J., & Miller, B. (2023) · JAMIA Open · doi:10.1093/jamiaopen/ooad028 · Cited by 43 · see citing works
Performance of a Chest Radiograph AI Diagnostic Tool for COVID-19: A Prospective Observational Study
Sun, J., Peng, L., Li, T., Adila, D., Zaiman, Z., Melton, G. B., Ingraham, N. E., Murray, E., Boley, D. L., Switzer, S., et al. (2022) · Radiology Artificial Intelligence · doi:10.1148/ryai.210217 · Cited by 39 · see citing works
Landau, A. Y., Ferrarello, S., Blanchard, A. M., Cato, K., Atkins, N., Salazar, S. N., Patton, D. U., & Topaz, M. (2021) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocab286 · Cited by 37 · see citing works
Cresswell, K. M., Rigby, M., Magrabi, F., Scott, P., Brender, J., Craven, C. K., Wong, Z. S.-Y., Kukhareva, P., Ammenwerth, E., Georgiou, A., et al. (2023) · Health Policy · doi:10.1016/j.healthpol.2023.104889 · Cited by 44 · see citing works
Khan, S. D., Hoodbhoy, Z., Raja, M. H. R., Kim, J. Y., Hogg, H. D. J., Manji, A. A. A., Gulamali, F., Hasan, A., Shaikh, A., Tajuddin, S., et al. (2024) · PLOS Digital Health · doi:10.1371/journal.pdig.0000514 · Cited by 36 · see citing works
Kukhareva, P., Weir, C. R., Fiol, G. D., Aarons, G. A., Taft, T., Schlechter, C. R., Reese, T. J., Curran, R. L., Nanjo, C. J., Borbolla, D. A., et al. (2022) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2022.104014 · Cited by 37 · see citing works
Randomized Controlled Trials in Medical AI
Genin, K., & Grote, T. (2021) · Philosophy of Medicine · doi:10.5195/philmed.2021.27 · Cited by 40 · see citing works
Artificial Intelligence in Pediatrics: Learning to Walk Together
Demirbaş, K. C., Yıldız, M., Saygılı, S., Canpolat, N., & Kasapçopur, Ö. (2024) · Turkish Archives of Pediatrics · doi:10.5152/turkarchpediatr.2024.24002 · Cited by 30 · see citing works
Suwanvecho, S., Suwanrusme, H., Jirakulaporn, T., Issarachai, S. S., Taechakraichana, N., Lungchukiet, P., Decha, W., Boonpakdee, W., Thanakarn, N., Wongrattananon, P., et al. (2021) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaa334 · Cited by 30 · see citing works
Self-citationShould the Use of Adaptive Machine Learning Systems in Medicine be Classified as Research?
Sparrow, R., Hatherley, J. J., Oakley, J. G., & Bain, C. (2024) · The American Journal of Bioethics · doi:10.1080/15265161.2024.2337429 · Cited by 29 · see citing works
Chaiteerakij, R., Ariyaskul, D., Kulkraisri, K., Apiparakoon, T., Sukcharoen, S., Chaichuen, O., Pensuwan, P., Tiyarattanachai, T., Rerknimitr, R., & Marukatat, S. (2024) · Scientific Reports · doi:10.1038/s41598-024-71657-z · Cited by 29 · see citing works
Zinchenko, V. V., Arzamasov, K. M., Четвериков, С. Ф., Maltsev, A. V., Петрович, Н. В., Akhmad, E. S., Е., Ш. Д., Андрейченко, А. Е., Vladzymyrskyy, A. V., & Morozov, S. P. (2022) · Современные технологии в медицине · doi:10.17691/stm2022.14.5.02 · Cited by 28 · see citing works
AI Quality Standards in Health Care: Rapid Umbrella Review
Kuziemsky, C. E., Chrimes, D., Minshall, S. R., Mannerow, M., & Lau, F. (2024) · Journal of Medical Internet Research · doi:10.2196/54705 · Cited by 28 · see citing works
Masters, K., & Salcedo, D. (2024) · Medical Teacher · doi:10.1080/0142159x.2023.2298756 · Cited by 26 · see citing works
Romagnoli, A., Ferrara, F., Langella, R., & Zovi, A. (2024) · Pharmaceutical Research · doi:10.1007/s11095-024-03685-3 · Cited by 28 · see citing works
Cho, I., Cho, J., Hong, J. H., Choe, W. S., & Shin, H. (2023) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocad145 · Cited by 26 · see citing works
Nashwan, A. J., & Hani, S. B. (2023) · Contemporary Clinical Trials Communications · doi:10.1016/j.conctc.2023.101223 · Cited by 30 · see citing works
Pham, N., Hill, V. B., Rauschecker, A. M., Lui, Y. W., Niogi, S. N., Filippi, C. G., Chang, P., Zaharchuk, G., & Wintermark, M. (2023) · American Journal of Neuroradiology · doi:10.3174/ajnr.a7850 · Cited by 24 · see citing works
Evaluation and Regulation of Artificial Intelligence Medical Devices for Clinical Decision Support
Weissman, G. E. (2025) · Annual Review of Biomedical Data Science · doi:10.1146/annurev-biodatasci-103123-095824 · Cited by 23 · see citing works
Esmaeilzadeh, P. (2025) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-025-03182-6 · Cited by 26 · see citing works
Machine learning in cardiac surgery: a narrative review
Miles, T. J., & Ghanta, R. K. (2024) · Journal of Thoracic Disease · doi:10.21037/jtd-23-1659 · Cited by 21 · see citing works
Lovejoy, C. A., Arora, A., Buch, V., & Dayan, I. (2021) · Future Healthcare Journal · doi:10.7861/fhj.2021-0128 · Cited by 22 · see citing works
Boussi Rahmouni, H., Hassine, N. B. E. H., Chouchen, M., Ceylan, H. İ., Muntean, R. I., Bragazzi, N. L., & Dergaa, I. (2025) · Healthcare · doi:10.3390/healthcare13202553 · Cited by 25 · see citing works
Artificial intelligence (AI) for neurologists: do digital neurones dream of electric sheep?
Yeung, J. A., Wang, Y. Y., Kraljević, Ž., & Teo, J. (2023) · Practical Neurology · doi:10.1136/pn-2023-003757 · Cited by 23 · see citing works
Dwyer, B., Flathers, M., Sano, A., Dempsey, A. G., Cipriani, A., Gazi, A. H., Hill, B., Gorban, C., Rodríguez, C. I., Stromeyer, C., et al. (2025) · NPP—Digital Psychiatry and Neuroscience · doi:10.1038/s44277-025-00049-6 · Cited by 22 · see citing works
Digital public health leadership in the global fight for health security
Al-Knawy, B., Kozlakidis, Z., Tarkoma, S., Bates, D. W., Honkela, A., Crooks, G., Rhee, K. B. L., & McKillop, M. M. (2023) · BMJ Global Health · doi:10.1136/bmjgh-2022-011454 · Cited by 19 · see citing works
Artificial Intelligence in Medical Education
Savage, T. (2021) · Academic Medicine · doi:10.1097/acm.0000000000004183 · Cited by 19 · see citing works
Bias in Artificial Intelligence
Park, Y., & Hu, J. (2023) · Clinical Journal of the American Society of Nephrology · doi:10.2215/cjn.0000000000000078 · Cited by 18 · see citing works
Self-citationKim, J. P., Ryan, K., Kasun, M., Hogg, J., Dunn, L. B., & Roberts, L. W. (2023) · JMIR AI · doi:10.2196/47449 · Cited by 18 · see citing works
Tsai, C.-C., Kim, J. Y., Chen, Q., Rowell, B. E., Yang, X. J., Kontar, R. A., Whitaker, M., & Lester, C. A. (2024) · Journal of Medical Internet Research · doi:10.2196/59946 · Cited by 20 · see citing works
A scoping review of silent trials for medical artificial intelligence
Tikhomirov, L., Semmler, C., Prizant, N., Bhasin, S., Kenyon, G., van der Vegt, A., Erdman, L., Kurian, N. C., Thompson, H., Palmer, L. J., et al. (2026) · Nature Health · doi:10.1038/s44360-025-00048-z · Cited by 17 · see citing works
Zheng, L., Ohde, J. W., Overgaard, S. M., Brereton, T. A., Jose, K. A., Wi, C., Peterson, K. J., & Juhn, Y. J. (2023) · JMIR Formative Research · doi:10.2196/45391 · Cited by 17 · see citing works
Approaches to Sampling for Quality Control of Artificial Intelligence in Biomedical Research
Четвериков, С. Ф., Arzamasov, K. M., Андрейченко, А. Е., Петрович, Н. В., Bobrovskaya, T., & Vladzimirsky, A. V. (2023) · Современные технологии в медицине · doi:10.17691/stm2023.15.2.02 · Cited by 16 · see citing works
Chari, S., Seneviratne, O. W., Ghalwash, M. F., Shirai, S. S., Gruen, D. M., Meyer, P., Chakraborty, P., & McGuinness, D. L. (2023) · Semantic Web · doi:10.3233/sw-233282 · Cited by 16 · see citing works
Self-citationNooraie, R. Y., Lyons, P. G., Baumann, A. A., & Saboury, B. (2021) · PET Clinics · doi:10.1016/j.cpet.2021.07.002 · Cited by 16 · see citing works
Shanmugam, U., Rajendran, M. K., Natarajan, J., & Karri, V. V. S. R. (2026) · Journal of Clinical Medicine · doi:10.3390/jcm15051937 · Cited by 14 · see citing works
Zulkifli, B., Fakri, F., Odigie, J., Nnabuife, L. C., Isitua, C. C., & Chiari, W. (2023) · Narra X · doi:10.52225/narrax.v1i1.80 · Cited by 14 · see citing works
Evaluation criteria for artificial intelligence
Montrosse‐Moorhead, B. (2023) · New Directions for Evaluation · doi:10.1002/ev.20566 · Cited by 14 · see citing works
The impact of artificial intelligence on Ukrainian medicine: benefits and challenges for the future
Sofilkanych, N., Vesova, O., Kaminskyy, V., & Kryvosheieva, A. (2023) · Futurity Medicine. · doi:10.57125/fem.2023.12.30.04 · Cited by 20 · see citing works
Randomized Controlled Trials in Medical AI
Genin, K., & Grote, T. (2021) · Philosophy of Medicine · doi:10.5195/pom.2021.27 · Cited by 14 · see citing works
Torous, J. B., Ledley, K. T., Gorban, C., Strudwick, G., Schwarz, J., Choudhary, S., Emerson, M. R., Patriquin, M. A., Dempsey, A. G., Bantjes, J. R., et al. (2025) · JMIR Mental Health · doi:10.2196/84501 · Cited by 15 · see citing works
Challenges of Machine Learning and AI (What Is Next?), Responsible and Ethical AI
Gkontra, P., Quaglio, G., Garmendia, A. T., & Lekadir, K. (2023) · Springer eBooks · doi:10.1007/978-3-031-36678-9_17 · Cited by 17 · see citing works
Hua, D., Petrina, N., Sacks, A. J., Young, N., Cho, J., Smith, R. C., & Poon, S. K.-S. (2024) · JAMIA Open · doi:10.1093/jamiaopen/ooae151 · Cited by 12 · see citing works
Risk Assessment and Classification of Medical Device Software for the Internet of Medical Things
Brass, I., & Mkwashi, A. (2022) · International Conference on the Internet of Things (IoT) · doi:10.1145/3567445.3571104 · Cited by 11 · see citing works
Nolin-Lapalme, A., Corbin, D., Tastet, O., Avram, R., & Hussin, J. (2024) · Canadian Journal of Cardiology · doi:10.1016/j.cjca.2024.04.026 · Cited by 11 · see citing works
Savage, S. A., Seth, I., Angus, Z. G., & Rozen, W. M. (2024) · Journal of Plastic Reconstructive & Aesthetic Surgery · doi:10.1016/j.bjps.2024.11.023 · Cited by 13 · see citing works
Kim, R. Y. (2024) · Cancer Biomarkers · doi:10.3233/cbm-230360 · Cited by 9 · see citing works
Joo, B. (2025) · Neurointervention · doi:10.5469/neuroint.2025.00283 · Cited by 9 · see citing works
Bobak, C. A., Svoboda, M., Giffin, K. A., Wall, D. P., & Moore, J. H. (2020) · Pacific Symposium on Biocomputing (PSB) · doi:10.1142/9789811232701_0035 · Cited by 9 · see citing works
Overgaard, S. M., Peterson, K. J., Wi, C. I., Kshatriya, B. S. A., Ohde, J. W., Brereton, T. A., Zheng, L., Rost, L. M., Zink, J., Nikakhtar, A., et al. (2022) · PubMed Central · Cited by 9 · see citing works
Pharmacy Futures: Summit on Artificial Intelligence in Pharmacy Practice
Luchen, G. G., Fera, T., Anderson, S. V., & Chen, D. C. (2024) · American Journal of Health-System Pharmacy · doi:10.1093/ajhp/zxae279 · Cited by 8 · see citing works
The Role of Artificial Intelligence in Gynecologic Oncology Decision-Making: A Feasibility Study
Psilopatis, I., Sipulina, N., Stuebs, F. A., Heindl, F., Poeschke, P., Bader, S., Krueckel, A., Fasching, P. A., Beckmann, M. W., & Emons, J. (2025) · Gynecologic and Obstetric Investigation · doi:10.1159/000544751 · Cited by 8 · see citing works
Tripathi, S., Augustin, A. I., Dako, F., & Kim, E. (2022) · AI and Ethics · doi:10.1007/s43681-022-00227-8 · Cited by 8 · see citing works
Strategies to Improve Real-World Applicability of Laparoscopic Anatomy Segmentation Models
Kolbinger, F. R., He, J., Ma, J., & Zhu, F. M. (2024) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw63382.2024.00233 · Cited by 8 · see citing works
Understanding the evidence for artificial intelligence in healthcare
Jackson, G. P., & Shortliffe, E. H. (2025) · BMJ Quality & Safety · doi:10.1136/bmjqs-2025-018559 · Cited by 9 · see citing works
Self-citationKočo, L., Siebers, C. C. N., Schlooz, M. S., Meeuwis, C., Oldenburg, H. S. A., Prokop, M., & Mann, R. M. (2024) · Cancers · doi:10.3390/cancers16020401 · Cited by 8 · see citing works
Ueda, D., Walston, S. L., Takita, H., Mitsuyama, Y., & Miki, Y. (2024) · Japanese Journal of Radiology · doi:10.1007/s11604-024-01716-y · Cited by 7 · see citing works
Artificial intelligence in healthcare: A focus on the best practices
Haddiya, I., & Ramdani, S. (2024) · ITM Web of Conferences · doi:10.1051/itmconf/20246902010 · Cited by 7 · see citing works
Ruberg, S. J., Menon, S., & Demanuele, C. (2023) · Harvard Data Science Review · doi:10.1162/99608f92.c1292c54 · Cited by 7 · see citing works
Patient digital twins: an introduction based on a scoping reviewPreprint
Drummond, D., & Gonsard, A. (2024) · medRxiv · doi:10.1101/2024.02.20.24303096 · Cited by 8 · see citing works
Balaha, H. M., El-Gendy, E. M., & Saafan, M. M. (2024) · Soft Computing · doi:10.1007/s00500-024-09873-y · Cited by 6 · see citing works
Validation of an Artificial Intelligence-Powered Virtual Assistant for Emergency Triage in Neurology
Alessandro, L., Crema, S., Castiglione, J. I., Dossi, D., Eberbach, F., Köhler, A., Laffue, A., Marone, A., Nagel, V., Rueda, J. M. P., et al. (2025) · The Neurologist · doi:10.1097/nrl.0000000000000594 · Cited by 6 · see citing works
Translational health technology and system schemes: enhancing the dynamics of health informatics
Rissanen, M. (2020) · Health Information Science and Systems · doi:10.1007/s13755-020-00133-5 · Cited by 6 · see citing works
Cell projection plots: A novel visualization of bone marrow aspirate cytology
Dehkharghanian, T., Mu, Y., Ross, C. A., Sur, M., Tizhoosh, H. R., & Campbell, C. J. V. (2023) · Journal of Pathology Informatics · doi:10.1016/j.jpi.2023.100334 · Cited by 6 · see citing works
Artificial intelligence and the future of pharmacy
Nelson, S. D. (2023) · American Journal of Health-System Pharmacy · doi:10.1093/ajhp/zxad316 · Cited by 6 · see citing works
Lakkimsetti, M., Devella, S. G., Patel, K. B., Dhandibhotla, S., Kaur, J., Mathew, M., Kataria, J., Nallani, M., Farwa, U. E., Patel, T. D., et al. (2024) · Cureus · doi:10.7759/cureus.58400 · Cited by 8 · see citing works
An international consensus panel on the potential value of Digital Surgery
Erskine, J., Abrishami, P., Bernhard, J., Charter, R., Culbertson, R., Hiatt, J. C., Igarashi, A., Jackson, G. P., Lien, M., Maddern, G. J., et al. (2024) · BMJ Open · doi:10.1136/bmjopen-2023-082875 · Cited by 6 · see citing works
Self-citationJiang, Z., Chen, H., Wu, Y., Qin, Y., Pei, C., Zeng, D., Sheng, B., & Wong, T. Y. (2026) · Intelligent Medicine · doi:10.1016/j.imed.2026.01.001 · Cited by 5 · see citing works
Wilhelm, C., Steckelberg, A., & Rebitschek, F. G. (2024) · Systematic Reviews · doi:10.1186/s13643-024-02646-6 · Cited by 5 · see citing works
Carter, R. E., Weston, A. D., Wieczorek, M. A., Pacheco-Spann, L. M., Fahad, S., Caruso, M. A., Aristizabal, M., Bruce, A. J., Hall, M. R., & Bruce, C. J. (2024) · Dermatitis · doi:10.1089/derm.2024.0302 · Cited by 5 · see citing works
Framework for the Evaluation of Clinical AI Systems
Shortliffe, E. H., Sepúlveda, M.-J., & Patel, V. L. (2022) · Cognitive informatics in biomedicine and healthcare · doi:10.1007/978-3-031-09108-7_17 · Cited by 5 · see citing works
Artificial intelligence and the future of the pharmacist’s role in drug information
Patel, R., Shirk, R., Advani, A., Kar, I., Munir, F. M., Nelson, S. D., Saad, M., & Aungst, T. D. (2025) · American Journal of Health-System Pharmacy · doi:10.1093/ajhp/zxaf175 · Cited by 4 · see citing works
Essential Concepts in Artificial Intelligence: A Guide for Pediatric Providers
Bolivar, L. E. M., & Satzer, M. B. (2025) · Children · doi:10.3390/children12101386 · Cited by 4 · see citing works
Pillai, M., Shumway, J., Adapa, K., Dooley, J., McGurk, R., Mazur, L. M., Das, S. K., & Chera, B. S. (2023) · Advances in Radiation Oncology · doi:10.1016/j.adro.2023.101234 · Cited by 4 · see citing works
- Authors
- Park, Y., Jackson, G. P., Foreman, M. A., Gruen, D., Hu, J., & Das, A. K.
- Affiliations
- IBM Research - Cambridge (United States) (US)
- IBM (United States) (US)
- IBM Research - Thomas J. Watson Research Center (US)
- Corresponding author
- Park, Yoonyoung
- Venue
- JAMIA Open
- Publisher
- Oxford University Press (OUP)
- Published
- 2020-09-08
- Volume
- 3
- Issue
- 3
- Pages
- 326-331
- ISSN
- 2574-2531 (ISSN-L)
- PMC
- PMC7660958
- OpenAlex
- W3083901094
- Cited by
- 150
- References
- 34
- Citations by year
- 2026: 19 · 2025: 26 · 2024: 33 · 2023: 35 · 2022: 20 · 2021: 15 · 2020: 2
- FWCI
- 5.30
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- JAMIA Open (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Keywords
- artificial intelligence in healthcare, AI evaluation, clinical trial phases, research design, real-world effectiveness, unintended consequences, AI implementation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review