Monitoring performance of clinical artificial intelligence in health care: a scoping review
Open accessAbstract
Objective: The objective of this review was to provide an overview of the diverse methods described, tested, or implemented for monitoring performance of clinical artificial intelligence (AI) systems, while also summarizing the arguments given for or against these methods. Introduction: The integration of AI in clinical decision-making is steadily growing. Performances of AI systems evolve over time, necessitating ongoing performance monitoring. However, the evidence on specific monitoring methods is sparse and heterogeneous. Thus, an overview of the evidence on this topic is warranted to guide further research on clinical AI monitoring. Inclusion criteria: We included publications detailing metrics or statistical processes employed in systematic, continuous, or repeated initiatives aimed at evaluating or predicting the clinical performance of AI models with direct implications for patient management in health care. No limitations on language or publication date were enforced. Methods: We performed systematic database searches in MEDLINE (Ovid), Embase (Ovid), Scopus, and ProQuest Dissertations and Theses Global, supplemented by backward and forward citation searches and gray literature searches. Two or more independent reviewers conducted title and abstract screening, full-text evaluation, and data extraction using a tool developed by the authors. During extraction, the methods identified were divided into subcategories. The results are presented narratively and summarized in tables and graphs. Results: Thirty-nine sources of evidence were included in the review, with the most abundant source types being opinion papers/narrative reviews (33%) and simulation studies (33%). One guideline on the topic was identified, offering limited guidance on specific metrics and statistical methods. The number of sources included increased year by year, with almost 4 times as many sources included in 2023 compared with 2019. The most commonly reported performance metrics were traditional metrics from the medical literature, including area under the receiver operating characteristics curve (AUROC), sensitivity, specificity, and predictive values, although few arguments were given supporting these choices. Some studies reported on metrics and statistical processing specifically designed to monitor clinical AI. Conclusion: This review provides a summary of the methods described for monitoring AI in health care. It reveals a relative scarcity of evidence and guidance for specific practical implementation of performance monitoring of clinical AI. This underscores the imperative for further research, discussion, and guidance regarding the specifics of implementing monitoring for clinical AI. The steady increase in the number of relevant sources published per year suggests that this area of research is gaining increased focus, and the amount of evidence and guidance available will likely increase significantly over the coming years. Review registration: Open Science Framework https://osf.io/afkrn
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Much of this literature addresses governance architectures, robustness concepts, implementation barriers, or monitoring frameworks, whereas only a subset is centred on longitudinal or operational evidence generated during routine deployment use.
“Much of this literature addresses governance architectures, robustness concepts, implementation barriers, or monitoring frameworks, whereas only a subset is centred on longitudinal or operational evidence generated during routine deployment use [18,19,20].”
From Beyond Model Development in Healthcare AI: Post-Development Robustness, Post-Deployment Monitoring, and Lifecycle Governance—A Scoping Review of Reviews (2026) · cites this paper as [18]
Split into 2 atoms — 0 confirmed, 2 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.
Experimental — accuracy not yet measuredExisting reviews have examined specific dimensions of the problem, including post-deployment monitoring, temporal dataset shift, implementation frameworks, healthcare MLOps, governance, human-AI interaction, and deployment-proximal evaluation.
“Existing reviews have examined specific dimensions of the problem, including post-deployment monitoring, temporal dataset shift, implementation frameworks, healthcare MLOps, governance, human–AI interaction, and deployment-proximal evaluation”
From Beyond Model Development in Healthcare AI: Post-Development Robustness, Post-Deployment Monitoring, and Lifecycle Governance—A Scoping Review of Reviews (2026) · cites this paper as [18]
The Discussion describes this paper as a scoping review of methods for monitoring clinical AI performance, supporting the post-deployment monitoring example. The References also identify a separate review of temporal dataset shift. The cited paper does not establish that reviews have examined all the other listed dimensions, including healthcare MLOps, governance, and human–AI interaction; no page numbers are available in our copy. (LLM, conf=0.85)
Experimental — accuracy not yet measuredNo previous review has integrated review-level and review-oriented evidence across post-development robustness, deployment-proximal evaluation, post-deployment monitoring, adaptive updating, and lifecycle governance in clinical AI practice.
“to our knowledge, no previous review has integrated review-level and review-oriented evidence across post-development robustness, deployment-proximal evaluation, post-deployment monitoring, adaptive updating, and lifecycle governance in clinical AI practice [18,19,20].”
From Beyond Model Development in Healthcare AI: Post-Development Robustness, Post-Deployment Monitoring, and Lifecycle Governance—A Scoping Review of Reviews (2026) · cites this paper as [18]
The Introduction says preliminary searches found no completed or ongoing reviews on clinical AI performance monitoring in three named databases. That supports the monitoring portion, but the paper does not establish that no previous review integrated all five areas named in the claim. Its Strengths and limitations section also notes that the search may have missed work on validation or implementation. (LLM, conf=0.84)
Experimental — accuracy not yet measuredThe validation-deployment gap arises within a single institution over time, not only through cross-site transportability failures.
“This extends critiques of retrospective validation by showing the validation–deployment gap arises within a single institution over time, not only through cross-site transportability failures [15,33].”
From Validation is not enough: Longitudinal evidence of post-deployment fragility in clinical AI systems (2026) · cites this paper as [33]
The Introduction (page unavailable) says clinical AI performance may change during deployment and that acceptable performance at validation does not guarantee sustained adequacy. This supports a gap developing over time, but our copy does not establish the more specific claim that it arises within a single institution rather than through cross-site differences. (LLM, conf=0.82)
Experimental — accuracy not yet measuredDegradation may be linked to how data are produced within evolving workflows and information systems, not only to who the patients are.
“This supports a workflow-aware interpretation of post-deployment fragility: degradation may be linked to how data are produced within evolving workflows and information systems, not only to who the patients are [8,18,33,35].”
From Validation is not enough: Longitudinal evidence of post-deployment fragility in clinical AI systems (2026) · cites this paper as [33]
The Introduction says clinical AI performance may change during deployment when its operating environment changes. The Results’ input-monitoring section also discusses detecting shifts in model inputs. Neither passage specifically links degradation to evolving clinical workflows or information systems, so the source supports the broader point but not that proposed mechanism. (LLM, conf=0.79)
Experimental — accuracy not yet measured
Much of the review literature addressed governance architectures, robustness concepts, implementation barriers, monitoring frameworks, MLOps, or deployment-proximal evaluation, rather than mature evidence generated during sustained routine deployment.
“Much of the review literature instead addressed governance architectures, robustness concepts, implementation barriers, monitoring frameworks, MLOps, or deployment-proximal evaluation, rather than mature evidence generated during sustained routine deployment [18,19,20,29,35,37,47,49].”
From Beyond Model Development in Healthcare AI: Post-Development Robustness, Post-Deployment Monitoring, and Lifecycle Governance—A Scoping Review of Reviews (2026) · cites this paper as [18]
These reviews differ substantially in design, scope, evidential intent, and proximity to live clinical use.
“but these reviews differ substantially in design, scope, evidential intent, and proximity to live clinical use [18,19,20].”
From Beyond Model Development in Healthcare AI: Post-Development Robustness, Post-Deployment Monitoring, and Lifecycle Governance—A Scoping Review of Reviews (2026) · cites this paper as [18]
Calibration failure is structural and progressive rather than sporadic.
“This extends prior reports of temporal miscalibration [32–34] by showing calibration failure is structural and progressive rather than sporadic.”
From Validation is not enough: Longitudinal evidence of post-deployment fragility in clinical AI systems (2026) · cites this paper as [33]
Technical constraints including stability of platform and need for real-time feedback require robust system monitoring.
“Third, technical constraints – including the stability of the platform and the need for real-time feedback – require robust system monitoring.”
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The PRISMA 2020 statement: An updated guideline for reporting systematic reviews
Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., et al. (2021) · BMJ · doi:10.1136/bmj.n71 · Cited by 108,283 · see citing works
PRISMA Extension for Scoping Reviews (PRISMA-ScR): Checklist and Explanation
Tricco, A. C., Lillie, E., Zarin, W., O'Brien, K. K., Colquhoun, H., Levac, D., Moher, D., Peters, M. D. J., Horsley, T., Weeks, L., et al. (2018) · Annals of Internal Medicine · doi:10.7326/m18-0850 · Cited by 43,565 · see citing works
From local explanations to global understanding with explainable AI for trees
Lundberg, S. M., Erion, G., Chen, H., DeGrave, A., Prutkin, J. M., Nair, B., Katz, R., Himmelfarb, J., Bansal, N., & Lee, S.-I. (2020) · Nature Machine Intelligence · doi:10.1038/s42256-019-0138-9 · Cited by 10,551 · see citing works
Guidance for conducting systematic scoping reviews
Peters, M. D. J., Godfrey, C. M., Khalil, H., McInerney, P., Parker, D., & Soares, C. B. (2015) · International Journal of Evidence-Based Healthcare · doi:10.1097/xeb.0000000000000050 · Cited by 7,472 · see citing works
International evaluation of an AI system for breast cancer screening
McKinney, S. M., Sieniek, M., Godbole, V., Godwin, J., Antropova, N., Ashrafian, H., Back, T., Chesus, M., Corrado, G. S., Darzi, A., et al. (2020) · Nature · doi:10.1038/s41586-019-1799-6 · Cited by 3,287 · see citing works
Fleuren, L. M., Klausch, T. L. T., Zwager, C. L., Schoonmade, L. J., Guo, T., Roggeveen, L. F., Swart, E. L., Girbes, A. R. J., Thoral, P., Ercole, A., et al. (2020) · Intensive Care Medicine · doi:10.1007/s00134-019-05872-y · Cited by 755 · see citing works
Feng, J., Phillips, R. V., Malenica, I., Bishara, A., Hubbard, A. E., Celi, L. A., & Pirracchio, R. (2022) · npj Digital Medicine · doi:10.1038/s41746-022-00611-y · Cited by 423 · see citing works
Magrabi, F., Ammenwerth, E., McNair, J. B., De Keizer, N. F., Hyppönen, H., Nykänen, P., Rigby, M., Scott, P. J., Vehko, T., Wong, Z. S.-Y., et al. (2019) · Yearbook of Medical Informatics · doi:10.1055/s-0039-1677903 · Cited by 347 · see citing works
Measuring Domain Shift for Deep Learning in Histopathology
Stacke, K., Eilertsen, G., Unger, J., & Lundström, C. F. (2020) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2020.3032060 · Cited by 334 · see citing works
Calibration drift in regression and machine learning models for acute kidney injury
Davis, S. E., Lasko, T. A., Chen, G., Siew, E. D., & Matheny, M. E. (2017) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocx030 · Cited by 312 · see citing works
Data drift in medical machine learning: implications and potential remedies
Sahiner, B., Chen, W., Samala, R. K., & Petrick, N. (2023) · The British Journal of Radiology · doi:10.1259/bjr.20220878 · Cited by 279 · see citing works
Explainable AI for clinical and remote health applications: A survey on tabular and time series data
Di Martino, F., & Delmastro, F. (2023) · Artificial Intelligence Review · doi:10.1007/s10462-022-10304-3 · Cited by 216 · see citing works
Use of gpt-4 to diagnose complex clinical cases
Eriksen, A. V., Möller, S., & Ryg, J. (2024) · NEJM AI · doi:10.1056/aip2300031 · Cited by 200 · see citing works
Temporal quality degradation in AI models
Jarquin, D. V., Sharp, A., Zhang, R., Nguyen, T., Hoang, A. T., & Pianykh, O. S. (2022) · Scientific Reports · doi:10.1038/s41598-022-15245-z · Cited by 187 · see citing works
Detection of calibration drift in clinical prediction models to inform model updating
Davis, S. E., Greevy, R. A., Lasko, T. A., Walsh, C. G., & Matheny, M. E. (2020) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2020.103611 · Cited by 160 · see citing works
A framework for the oversight and local deployment of safe and high-quality prediction models
Bedoya, A. D., Economou-Zavlanos, N. J., Goldstein, B. A., Young, A., Jelovsek, J. E., O’Brien, C., Parrish, A. B., Elengold, S., Lytle, K., Balu, S., et al. (2022) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocac078 · Cited by 119 · see citing works
Rahmani, K., Thapa, R., Tsou, P., Chetty, S. C., Barnes, G., Lam, C., & Tso, C. F. (2022) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2022.104930 · Cited by 115 · see citing works
Mahadevaiah, G., RV, P., Bermejo, I., Jaffray, D., Dekker, A., & Wee, L. (2020) · Medical Physics · doi:10.1002/mp.13562 · Cited by 114 · see citing works
Duckworth, C., Chmiel, F. P., Burns, D. K., Zlatev, Z. D., White, N. M., Daniels, T. W. V., Kiuber, M., & Boniface, M. J. (2021) · Scientific Reports · doi:10.1038/s41598-021-02481-y · Cited by 109 · see citing works
Guo, L. L., Pfohl, S. R., Fries, J., Posada, J., Fleming, S. L., Aftandilian, C., Shah, N., & Sung, L. (2021) · Applied Clinical Informatics · doi:10.1055/s-0041-1735184 · Cited by 96 · see citing works
Prognostic models will be victims of their own success, unless…
Lenert, M. C., Matheny, M. E., & Walsh, C. G. (2019) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocz145 · Cited by 90 · see citing works
Quality Assurance for AI-Based Applications in Radiation Therapy
Claessens, M., Oria, C. S., Brouwer, C. L., Ziemer, B. P., Scholey, J., Lin, H., Witztum, A., Morin, O., Naqa, I. E., van Elmpt, W., et al. (2022) · Seminars in Radiation Oncology · doi:10.1016/j.semradonc.2022.06.011 · Cited by 86 · see citing works
Integrating Al Algorithms into the Clinical Workflow
Juluru, K., Shih, H.-H., Murthy, K. N. K., Elnajjar, P., El-Rowmeim, A., Roth, C. J., Genereaux, B. W., Fox, J. J., Siegel, E. L., & Rubin, D. L. (2021) · Radiology Artificial Intelligence · doi:10.1148/ryai.2021210013 · Cited by 85 · see citing works
Allen, B., Dreyer, K. J., Stibolt, R., Agarwal, S., Coombs, L. P., Treml, C., Elkholy, M., Brink, L., & Wald, C. (2021) · Journal of the American College of Radiology · doi:10.1016/j.jacr.2021.08.022 · Cited by 80 · see citing works
Corbin, C. K., Maclay, R., Acharya, A., Mony, S., Punnathanam, S., Thapa, R., Kotecha, N., Shah, N. H., & Chen, J. H. (2023) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocad114 · Cited by 59 · see citing works
Cha, Y., Kim, J.-T., Park, C. H., Kim, J.-W., Lee, S.-Y., & Yoo, J. (2022) · Journal of Orthopaedic Surgery and Research · doi:10.1186/s13018-022-03408-7 · Cited by 55 · see citing works
Simulated clinical deployment of fully automatic deep learning for clinical prostate MRI assessment
Schelb, P., Wang, X., Radtke, J. P., Wiesenfarth, M., Kickingereder, P., Stenzinger, A., Hohenfellner, M., Schlemmer, H., Maier-Hein, K. H., & Bonekamp, D. (2020) · European Radiology · doi:10.1007/s00330-020-07086-z · Cited by 50 · see citing works
Open questions and research gaps for monitoring and updating AI-enabled tools in clinical settings
Davis, S. E., Walsh, C. G., & Matheny, M. E. (2022) · Frontiers in Digital Health · doi:10.3389/fdgth.2022.958284 · Cited by 33 · see citing works
Nakatsugawa, M., Cheng, Z., Kiess, A. P., Choflet, A., Bowers, M., Utsunomiya, K., Sugiyama, S., Wong, J. W., Quon, H., & McNutt, T. (2018) · International Journal of Radiation Oncology*Biology*Physics · doi:10.1016/j.ijrobp.2018.09.038 · Cited by 31 · see citing works
Bartels, R., Dudink, J., Haitjema, S., Oberski, D. L., & Veen, A. V. '. (2022) · Frontiers in Digital Health · doi:10.3389/fdgth.2022.942588 · Cited by 31 · see citing works
Yang, R., Yang, X., Wang, L., Li, D., Guo, Y., Li, Y., Guan, Y.-M., Wu, X., Xu, S., Zhang, S., et al. (2021) · Radiotherapy and Oncology · doi:10.1016/j.radonc.2021.06.024 · Cited by 30 · see citing works
The Role of the ACR Data Science Institute in Advancing Health Equity in Radiology
Allen, B., & Dreyer, K. J. (2019) · Journal of the American College of Radiology · doi:10.1016/j.jacr.2018.12.038 · Cited by 28 · see citing works
Zinchenko, V. V., Arzamasov, K. M., Четвериков, С. Ф., Maltsev, A. V., Петрович, Н. В., Akhmad, E. S., Е., Ш. Д., Андрейченко, А. Е., Vladzymyrskyy, A. V., & Morozov, S. P. (2022) · Современные технологии в медицине · doi:10.17691/stm2022.14.5.02 · Cited by 28 · see citing works
Vasiliev, Y. A., Vlazimirsky, A. V., Omelyanskaya, O. V., Arzamasov, K. M., Четвериков, С. Ф., Андреевич, Р. Д., & Zelenova, M. A. (2023) · Digital Diagnostics · doi:10.17816/dd321971 · Cited by 28 · see citing works
Mapping the Landscape of Care Providers’ Quality Assurance Approaches for AI in Diagnostic Imaging
Lundström, C. F., & Lindvall, M. (2022) · Journal of Imaging Informatics in Medicine · doi:10.1007/s10278-022-00731-7 · Cited by 26 · see citing works
Davis, S. E., Greevy, R. A., Lasko, T. A., Walsh, C. G., & Matheny, M. E. (2019) · PubMed Central · Cited by 24 · see citing works
Eslami, S., de Keizer, N. F., de Jonge, E., de Rooij, S. E., Abu–Hanna, A., & Minne, L. (2012) · Methods of Information in Medicine · doi:10.3414/me11-02-0044 · Cited by 21 · see citing works
Detecting changes in the performance of a clinical machine learning tool over time
Schinkel, M., Boerman, A. W., Paranjape, K., Wiersinga, W. J., & Nanayakkara, P. W. B. (2023) · EBioMedicine · doi:10.1016/j.ebiom.2023.104823 · Cited by 19 · see citing works
Rahmani, K., Thapa, R., Tsou, P., et al. (2022) · medRxiv · doi:10.1101/2022.06.06.22276062 · Cited by 18 · see citing works
Approaches to Sampling for Quality Control of Artificial Intelligence in Biomedical Research
Четвериков, С. Ф., Arzamasov, K. M., Андрейченко, А. Е., Петрович, Н. В., Bobrovskaya, T., & Vladzimirsky, A. V. (2023) · Современные технологии в медицине · doi:10.17691/stm2023.15.2.02 · Cited by 16 · see citing works
Detecting Drift in Healthcare AI Models Based on Data Availability
Rotalinti, Y., Tucker, A., Lonergan, M., Myles, P. R., & Branson, R. D. (2023) · Communications in computer and information science · doi:10.1007/978-3-031-23633-4_17 · Cited by 15 · see citing works
Levy, T. J., Coppa, K., Cang, J., Barnaby, D. P., Paradis, M. D., Cohen, S. L., Makhnevich, A., van Klaveren, D., Kent, D. M., Davidson, K. W., et al. (2022) · Nature Communications · doi:10.1038/s41467-022-34646-2 · Cited by 14 · cited 3× · see citing works
“38,47,57 Calibration Six sources described monitoring of calibration measures.”
An Explainable and Actionable Mistrust Scoring Framework for Model Monitoring
Bhaskhar, N., Rubin, D. L., & Lee‐Messer, C. (2023) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2023.3272876 · Cited by 13 · see citing works
Kerf, G. D., Claessens, M., Raouassi, F., Mercier, C., Stas, D., Ost, P., Dirix, P. R., & Verellen, D. (2023) · Physics and Imaging in Radiation Oncology · doi:10.1016/j.phro.2023.100494 · Cited by 13 · see citing works
CheXstray: A Real-Time Multi-Modal Monitoring Workflow for Medical Imaging AI
Merkow, J., Soin, A., Long, J., Cohen, J. P., Saligrama, S., Bridge, C., Yang, X., Kaiser, S., Borg, S., Tarapov, I., et al. (2023) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-43898-1_32 · Cited by 13 · see citing works
Ethics framework for predictive clinical AI model updating
Pruski, M. (2023) · Ethics and Information Technology · doi:10.1007/s10676-023-09721-x · Cited by 10 · see citing works
George, R. D., Ellis, B., West, A., Graff, A., Weaver, S., Abramowski, M., Brown, K., Kerr, L., Lu, S.-C., Swisher, C. L., et al. (2023) · Communications Medicine · doi:10.1038/s43856-023-00317-6 · Cited by 8 · see citing works
Integration and evaluation of chest X-ray artificial intelligence in clinical practice
Wong, K. P., Homer, S. Y., Wei, S. H., Yaghmai, N., Estrada, O., Young, T. J., Buhr, R. G., Barjaktarević, I., Shrestha, L., Daly, M., et al. (2023) · Journal of medical imaging · doi:10.1117/1.jmi.10.5.051805 · Cited by 7 · see citing works
Monitoring performance of clinical artificial intelligence: a scoping review protocol
Andersen, E. S., Birk‐Korch, J. B., Röttger, R., Brasen, C. L., Brandslund, I., & Madsen, J. S. (2024) · JBI Evidence Synthesis · doi:10.11124/jbies-23-00390 · Cited by 5 · see citing works
Self-citationMonitoring Methodology for an AI Tool for Breast Cancer Screening Deployed in Clinical Centers
Aguilar, C., Pacilè, S., Weber, N., & Fillard, P. (2023) · Life · doi:10.3390/life13020440 · Cited by 4 · see citing works
Kuriki, P. E. D. A., & Kitamura, F. (2023) · Seminars in Roentgenology · doi:10.1053/j.ro.2023.01.006 · Cited by 4 · see citing works
Corbin, C. K., Maclay, R., Acharya, A., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2303.06269 · Cited by 2 · see citing works
Quality control of artificial intelligence for medical imaging during clinical use
Wang, H., Xiao, Y., Meng, X., Ren, H., & Liu, S. (2019) · Zhonghua fangshexian yixue zazhi · doi:10.3760/cma.j.issn.1005-1201.2019.09.002 · Cited by 1 · see citing works
Flagged first, then by citations
Research works printed but not linked: 2
- [24]Andreychenko AE Logunova TA Gombolevskiy VA Nikolaev AE Vladzymyrskyy AV Sinitsyn VE. A methodology for selection and quality control of the radiological computer vision deployment at the megalopolis scale [internet]. medRxiv; 2022 [cited 2023 Dec 5]. Available from: https://www.medrxiv.org/content/10.1101/2022.02.12.22270663v1.full. · DOI
- [44]Elguindi S Li A Zhu M Cervino L Veeraraghavan H Jiang J. Automated contour edit tracking to improve AI auto-segmentation. Med Phys 2022; 49(6): e133.
Cited by
Sorted by citations
Arab, R. A. E., Moosa, O. A. A., & Sagbakken, M. (2025) · Frontiers in Public Health · doi:10.3389/fpubh.2025.1617138 · Cited by 58 · see citing works
Odone, A., Barbati, C., Amadasi, S., Schultz, T., & Resnik, D. B. (2025) · The Lancet Infectious Diseases · doi:10.1016/s1473-3099(25)00412-8 · Cited by 39 · see citing works
Assudani, P. J., Bhurgy, A. S., Kollem, S., Bhurgy, B. S., Ahmad, M. O., Kulkarni, M. B., & Bhaiyya, M. L. (2025) · Microchemical Journal · doi:10.1016/j.microc.2025.115802 · Cited by 35 · see citing works
Adversarial artificial intelligence in radiology: Attacks, defenses, and future considerations
Dietrich, N. T., Gong, B., & Patlas, M. N. (2025) · Diagnostic and Interventional Imaging · doi:10.1016/j.diii.2025.05.006 · Cited by 28 · see citing works
Meinlschmidt, G., Koc, S., Boerner, E., Tegethoff, M., Simacek, T., Schirmer, L., & Schneider, M. (2025) · BMC Medical Education · doi:10.1186/s12909-025-07307-3 · Cited by 15 · see citing works
Chow, J., Lee, R., & Wu, H. (2025) · Journal of Imaging Informatics in Medicine · doi:10.1007/s10278-025-01493-8 · Cited by 18 · see citing works
Canino, G., Costanzo, A. D., Salerno, N., Leo, I., Cannataro, M., Guzzi, P. H., Veltri, P., Sorrentino, S., Rosa, S. D., & Torella, D. (2025) · Bioengineering · doi:10.3390/bioengineering12101102 · Cited by 14 · see citing works
Mehrtabar, S., Marey, A., Desai, A., Saad, A. M., Desai, V. S., Goñi, J., Pal, B., & Umair, M. (2025) · Journal of Imaging Informatics in Medicine · doi:10.1007/s10278-025-01656-7 · Cited by 14 · see citing works
Regulating, implementing and evaluating AI in Singapore healthcare: AI governance roundtable’s view
Goh, W. W. B., Tan, C. H., Tan, C. M., Prahl, A., Lwin, M. O., & Sung, J. J. �. (2025) · Annals of the Academy of Medicine Singapore · doi:10.47102/annals-acadmedsg.202556 · Cited by 10 · see citing works
Artificial intelligence and physician burnout: A productivity paradox
Goodson, D. A., Garcia, B., Hogarth, M. A., & Tu, S.-P. (2025) · Learning Health Systems · doi:10.1002/lrh2.70013 · Cited by 10 · see citing works
Suri, C., Ratre, Y. K., Pande, B., Bhaskar, L. V., & Verma, H. K. (2026) · World Journal of Gastroenterology · doi:10.3748/wjg.v32.i1.111428 · Cited by 10 · see citing works
Shojaei, S., Yakar, D., Vellinga, N., Bozgo, V., Kwee, T. C., Huisman, H. J., & Bonnici, J. P. M. (2026) · Abdominal Radiology · doi:10.1007/s00261-026-05434-z · Cited by 7 · see citing works
Societal factors influencing the implementation of AI-driven technologies in (smart) hospitals
de Graaf, Y., Ahmed, A., Sanges, C., Herbst, L., & Vrijhoef, H. J. (2025) · PLoS ONE · doi:10.1371/journal.pone.0325718 · Cited by 6 · see citing works
Khan, A. S., Galea, S., & Mendez, I. M. (2025) · Digital Health · doi:10.1177/20552076251334422 · Cited by 6 · see citing works
Shah, C., Ghodasara, S., Chen, D., & Chen, P. (2025) · Journal of the American College of Radiology · doi:10.1016/j.jacr.2025.09.032 · Cited by 5 · see citing works
State of the AI: Post-Deployment Monitoring of Radiology-Focused Internally Developed AI
Cook, C. J., Klug, J. R., Kandler, B. W., Suárez, A. B., Dachowicz, A. P., Blezek, D. J., Missert, A. D., Conte, G. M., Benfield, J. D., Mensing-Diggs, A., et al. (2026) · Mayo Clinic Proceedings Digital Health · doi:10.1016/j.mcpdig.2026.100342 · Cited by 5 · see citing works
Perdomo-Lampignano, J. A., Kale, A. U., Kumar, S., Shah, D., Reddy, B., & Lowe, D. J. (2026) · Journal of Medical Systems · doi:10.1007/s10916-026-02426-w · Cited by 5 · see citing works
El Arab, R. A., Mustafa, M. H., Almagharbeh, W. T., Saleem, N. H., Al Abdulmohsen, S., Boathab, R., & Bu Washl, M. (2026) · Healthcare · doi:10.3390/healthcare14111459 · Cited by 4 · see citing works
“Much of the review literature instead addressed governance architectures, robustness concepts, implementation barriers, monitoring frameworks, MLOps, or deployment-proximal evaluation, rather than mature evidence generated during sustained routine deployment [18–20”
InfluentialChang, Q., Chen, F., Chen, Y., Cheng, L., Dong, D., Dong, J., Feng, X., Ge, J., He, J., He, Y., et al. (2025) · Intelligent Medicine · doi:10.1016/j.imed.2025.09.001 · Cited by 4 · see citing works
Wenderott, K., Krups, J., & Weigl, M. (2026) · BMJ Open Quality · doi:10.1136/bmjoq-2025-003639 · Cited by 4 · see citing works
Validation is not enough: Longitudinal evidence of post-deployment fragility in clinical AI systems
Kopanitsa, G. (2026) · PLOS Digital Health · doi:10.1371/journal.pdig.0001534 · Cited by 3 · see citing works
“This supports a workflow-aware interpretation of post-deployment fragility: degradation may be linked to how data are produced within evolving workflows and information systems, not only to who the patients are [8,18,33,35].”
Ameri, A., Salmanizadeh, F., Samzadeh, H. R., & Ghaemi, M. M. (2025) · Health Science Reports · doi:10.1002/hsr2.71679 · Cited by 3 · see citing works
Raghupathi, W., & Raghupathi, V. (2026) · Health Information Science and Systems · doi:10.1007/s13755-026-00433-2 · Cited by 4 · see citing works
Li, J. D., Xiang, W., Shang, D., Li, S., & Li, Q. (2026) · Frontiers in Cardiovascular Medicine · doi:10.3389/fcvm.2026.1831342 · Cited by 3 · see citing works
Postill, G., Dent, A., Dombroski, J. D., Verma, A. A., Myers, J., & Apramian, T. (2025) · PLOS Digital Health · doi:10.1371/journal.pdig.0000917 · Cited by 2 · see citing works
Strategies for Embedding Prediction Models in Clinical Decision‑Making Workflows
Amusa, T., Okunola, D., Izinyon, O. C., Alabi, A., & Akinpeloye, O. (2026) · Cureus · doi:10.7759/cureus.101185 · Cited by 4 · see citing works
Ethical Considerations in Personal Health Large Language Models
Liu, J., & Liu, S. (2026) · Journal of Medical Internet Research · doi:10.2196/92240 · Cited by 2 · see citing works
Arab, R. A. E., Mustafa, M. H., Almagharbeh, W. T., Ayoub, M. Y., Alsanawi, F., Almathen, F., Almosabeh, R., & Talaq, M. A. (2026) · Healthcare · doi:10.3390/healthcare14142052 · Cited by 2 · see citing works
AI-Driven Cloud Automation in Healthcare: Enhancing Patient Data Processing and Compliance
Pallaprolu, S. (2025) · European Journal of Computer Science and Information Technology · doi:10.37745/ejcsit.2013/vol13n426474 · Cited by 1 · see citing works
Out of the Laboratory and Into the Clinic
Liu, W., Qu, B., Stone, A., Powell, M. G., Dufresne, S., Braun, S., Galdyn, I. A., Golinko, M. S., Yin, Z., & Pontell, M. E. (2026) · Annals of Plastic Surgery · doi:10.1097/sap.0000000000004633 · Cited by 1 · see citing works
Milicevic, F., Mahmoud, M. A., Ghandour, M. A., Khasawneh, M. Y., Ghasemi, A. R., Kabir, K., & Mert, Ü. (2026) · Die Orthopädie · doi:10.1007/s00132-025-04765-x · Cited by 1 · see citing works
Bunney, G., Miller, K., Ryu, K., Graber‐Naidich, A., Wessels, A. J., Pasao, M. A., Rizvi, M., Brown, I. M., & Yiadom, M. Y. A. B. (2026) · BMJ Digital Health & AI · doi:10.1136/bmjdhai-2025-000203 · Cited by 1 · see citing works
Curlej, P., & Soldera, J. (2026) · World Journal of Gastrointestinal Endoscopy · doi:10.4253/wjge.v18.i3.116381 · Cited by 1 · see citing works
Zhou, Q., Sun, L., Ou, Y., Liu, H., Gao, J., Dai, R., Li, Y., Cheng, G., Tao, Y., Pan, Y., et al. (2026) · Briefings in Bioinformatics · doi:10.1093/bib/bbag107 · Cited by 1 · see citing works
Baranwal, H. V., Amrita, Ranjan, R., & Paridhi (2026) · International Journal of Community Medicine and Public Health · doi:10.18203/2394-6040.ijcmph20261452 · Cited by 1 · see citing works
Ahmed, M. M., Okesanya, O. J., Othman, Z. K., Adebayo, U. O., Oso, T. A., & Lucero‐Prisno, D. E. (2026) · Informatics and Health · doi:10.1016/j.infoh.2026.03.002 · Cited by 1 · see citing works
ChatGPT as a digital pharmacist: A systematic review of medication counselling accuracy
Azmakan, H., Nabipour, A., Tehrani, N. G., Najari, N., hafshjani, P. F., Marvast, A. F., Mani, S., Sichani, N. A., Pakdaman, S. F., Shieh, M., et al. (2026) · Intelligence-Based Medicine · doi:10.1016/j.ibmed.2026.100417 · Cited by 1 · see citing works
Validating medical digital twins for clinical decision support: beyond predictive accuracy
Vallée, A. (2026) · JAMIA Open · doi:10.1093/jamiaopen/ooag160 · Cited by 1 · see citing works
Echo Chambers: Bias and Representation in Cardiac Imaging Datasets for Artificial Intelligence
Gorijavolu, R., Jaiswal, N., Pratap, J. S., Madapati, K., López, M. D. P. A., Leite, M., Muwanguzi, D. P., Izath, N., Samsel, K., Shoyombo, I., et al. (2026) · Current Cardiology Reports · doi:10.1007/s11886-026-02406-z · Cited by 1 · see citing works
Bergonzo, S., Mouchabac, S., Auxéméry, Y., Kulakovskyi, A., Altunlu, E., Leboyer, M., Bonnot, O., Bourla, A., Koutsouleris, N., & Frileux, S. (2026) · L Encéphale · doi:10.1016/j.encep.2026.08.007 · Cited by 1 · see citing works
Wang, S.-S., Gao, H., Lin, P., Qian, T.-C., Du, Y., & Xu, L. (2025) · World Journal of Gastrointestinal Oncology · doi:10.4251/wjgo.v17.i10.109792 · Cited by 1 · see citing works
Bringing artificial intelligence safely to the clinics: hope is not a strategy
Andersen, E. S. (2024) · JBI Evidence Synthesis · doi:10.11124/jbies-24-00501 · Cited by 1 · see citing works
Self-citationCotte, F., Schmude, M., Bode, P., Suliman, O., Lourenço, F. D., Pereira, M. P., Kini, N., Hartenstein, V., Muscoloni, A., Stroux, L., et al. (2025) · medRxiv · doi:10.1101/2025.09.20.25336227 · Cited by 1 · see citing works
Supporting quadruple aim in primary care using artificial intelligence
Tan, N. C., Lim, R. H. L., & Ng, D. C. C. (2025) · Annals of the Academy of Medicine Singapore · doi:10.47102/annals-acadmedsg.202515
Gupta, K. (2025) · Springer eBooks · doi:10.1007/978-3-032-02853-2_35
Cotte, F., Schmude, M., Bode, P., Suliman, O., Lourenço, F. D., Pereira, M. P., Kini, N., Hartenstein, V., Muscoloni, A., Stroux, L., et al. (2025) · Research Square · doi:10.21203/rs.3.rs-8022874/v1
Rafael-Palou, X., Munuera, J., Jiménez-Pastor, A., Osuala, R. O., Lekadir, K., & Diaz, O. (2026) · Lecture notes in electrical engineering · doi:10.1007/978-981-95-7425-4_40
A.I.’s Impact on Nursing Practice and Care
Walker, R., Khastar, S. M., Kuehnle, F., & Shen, M. (2026) · Springer eBooks · doi:10.1007/978-3-032-12402-9_6
Eskandar, K. (2026) · Journal of Addictive Diseases · doi:10.1080/10550887.2025.2607456
Mapping Economic Considerations in Clinical Artificial Intelligence Research
Özcan, H. (2026) · Information technology in economics and business. · doi:10.69882/adba.iteb.2026011
Zhao, X., Weng, Y., Huang, R., & Hu, Y. (2026) · Journal of Translational Medicine · doi:10.1186/s12967-026-07874-z
Threshold and Calibration Effects in Medical Machine Learning
Özcan, H. (2025) · International Conference on Algorithms, Computing and Artificial Intelligence (ACAI) · doi:10.1109/acai68217.2025.11406486
AI for monitoring clinical outcomes and adverse events
Hamid, S., Ali, R., Khanna, K., & Raza, K. (2026) · Elsevier eBooks · doi:10.1016/b978-0-443-44430-2.00002-5
Teymourifar, A. (2026) · Frontiers in Health Services · doi:10.3389/frhs.2026.1780458
Clinical Artificial Intelligence as a Novice Nurse
Ruffin, A. (2026) · Nurse Leader · doi:10.1016/j.mnl.2026.102778
Coluccia, S., Crispo, A., Ottaiano, A., Santorsola, M., Innamorato, M. A., Cerrone, V., Feo, R. D., Cascella, V., Esposito, D., Bruno, M. P., et al. (2026) · Pain Research and Management · doi:10.1155/prm/9211861
Ravindhran, B., Hatfield-Chetter, G., Morris-Jarvis, J. A., Stedman, N., Carradice, D., Smith, G., Thakker, D., Bicknell, C. D., Chetter, I. C., & Pymer, S. (2026) · Annals of Vascular Surgery · doi:10.1016/j.avsg.2026.05.071
Reply to “Beyond Procurement: Equity Considerations in Radiology Artificial Intelligence Adoption”
Do, B. H., & Beaulieu, C. F. (2026) · American Journal of Roentgenology · doi:10.2214/ajr.26.35244
Ventura, V. D. G. J., de Andrade, C. M. V., Almeida, J. M., Pessoa, B. P., Polanczyk, C. A., Fonseca, G., Boersma, E. H., Vianna, H. R., Farah, K. D. P., da Rocha, L. C. D., et al. (2026) · JMIR Formative Research · doi:10.2196/86379
Risk-adjusted conformity assessment of AI systems in healthcare: A scoping review
Reisinger, S., Kirmes, R., & Dockweiler, C. (2026) · Health Policy and Technology · doi:10.1016/j.hlpt.2026.101278
From appraisal to deployment: A governance framework for large language models in emergency medicine
Aykut, A. (2026) · Artificial Intelligence in Emergency Medicine · doi:10.1016/j.aiemed.2026.100029
Can you trust the algorithm? A clinical exploration of AI models in perioperative care
Cairns, K. P. (2026) · BMJ Innovations · doi:10.1136/bmjinnov-2025-001465
Lam, Q. T., Fan, F.-Y., Wang, Y., Feng, S., Vo, T. T. T., Minh, L. H. N., Vu, G. T., Le, N. Q. K., & Lee, I. (2026) · Diagnostics · doi:10.3390/diagnostics16162569
Ferre, R. M., Liu, R., Lam, H. S., Blaivas, M., Oh, L., Sangal, R. B., Baloescu, C., & Adhikari, S. (2026) · Journal of the American College of Emergency Physicians Open · doi:10.1016/j.acepjo.2026.100491
Wu, T., Cai, W., & Li, Z. (2026) · Frontiers in Endocrinology · doi:10.3389/fendo.2026.1887643
Post-Deployment Update Triggers for Clinical AI: An Error-Taxonomy Framework for Safe Model Revision
Herrera, L., Rojas, D., & Castro, A. (2024) · Journal of Health Informatics and Digital Systems · doi:10.68159/b329996013
Chi, H., Liu, X., Yan, Y., Wang, L., Luo, P., Su, L., Jiang, L., Zhou, X., Huang, G., Jiang, Y., et al. (2026) · Genes & Diseases · doi:10.1016/j.gendis.2026.102480 · Cited by 1 · see citing works
Mokoena, N., Novak, P., & Mendes, S. (2024) · Annals of Pharmacy Practice and Pharmacotherapy · doi:10.51847/nju0nk3jgz
Yu, S., Gu, X., Gao, Y., Zhao, F., & Zhang, J. (2026) · Integrated Diagnostics · doi:10.1016/j.intdia.2026.100006
SQC-Guard: A Statistical Quality Control Based Monitoring Framework for NLP Pipelines
Kawatra, J. S., Rajamani, S., Sabu, S., Sheldon, E. J., James, N., Booth, J., Eze, A., Key, D. A., Patel, S., Zenonos, A., et al. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37682-4_15
Jones, B. Z., Olsen, J. R., Miller, S. M., Herrera, E., Okelberry, D. R., Sitto, M. M., Brandenburg, T., Hoopes, P. C., & Moshirfar, M. (2026) · Cureus · doi:10.7759/cureus.117869
Yuji, K., & Yuji, W. (2026) · Expert Review of Medical Devices · doi:10.1080/17434440.2026.2753843
Artificial intelligence: Role in advancing public health
Meo, S. A. (2026) · Elsevier eBooks · doi:10.1016/b978-0-443-45973-3.00011-7
- Authors
- Andersen, E. S., Birk-Korch, J. B., Hansen, R. S., Fly, L. H., Röttger, R., Arcani, D. M. C., Brasen, C. L., Brandslund, I., & Madsen, J. S.
- Affiliations
- University of Southern Denmark (DK)
- Department of Biochemistry and Immunology, Lillebaelt Hospital – University Hospital of Southern Denmark, Vejle, Denmark
- Odense University Hospital (DK)
- First Affiliated Hospital Zhejiang University (CN)
- Corresponding author
- Andersen, Eline Sandvig
- ORCID
- Venue
- JBI Evidence Synthesis
- Published
- 2024-10-17
- Volume
- 22
- Issue
- 12
- Pages
- 2423-2446
- ISSN
- 2689-8381 (ISSN-L)
- PMC
- PMC11630661
- OpenAlex
- W4403498699
- Cited by
- 73
- References
- 52
- Citations by year
- 2026: 47 · 2025: 23 · 2024: 3
- FWCI
- 1.67
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- JBI Evidence Synthesis (publishedVersion, cc-by-nc-nd)
- PubMed Central (publishedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- University of Southern Denmark Research Portal (University of Southern Denmark) (publishedVersion, cc-by-nc-nd)
- Funders
- Syddansk Universitet
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Keywords
- clinical AI, performance monitoring, clinical decision support, continuous monitoring, receiver operating characteristic, sensitivity and specificity, predictive value, simulation study
- MeSH terms
- Clinical Decision-Making, Artificial Intelligence, Delivery of Health Care, Humans
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
