Not All Clinical AI Monitoring Systems Are Created Equal: Review and Recommendations
The DOI matches its Crossref record; 63 of the 63 references it cites were found in a registry or an index.
Abstract
AbstractWhile practices for the initial evaluation of clinical artificial intelligence (AI) algorithms are well established, there is little consensus on how to design effective monitoring systems for the post-deployment setting. In real-world case studies, design choices are often driven by practical constraints rather than their impact on the performance of the monitoring system. This narrative review critically examines the key decisions that shape the performance of AI monitoring systems, including the selection of monitoring criteria, the choice of data sources, and the statistical procedures employed. Our findings reveal significant variation in the designed systems, often with little transparency regarding how their design choices affect monitoring performance. To provide a more structured approach to designing monitoring systems that are both effective and practical, we introduce a road map for navigating the many options available. We bootstrap efforts in clinical AI monitoring by highlighting tools from three related fields that face similarly complex challenges: quality management, clinical trials, and real-world evidence generation.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Many organizations use model dashboards to track performance metrics, although consensus on what those metrics should be is hard to find [17].”
From Managing a 'responsibility vacuum' in AI monitoring and governance in healthcare: a qualitative study (2025) · cites this paper as [17]
Citations
Cites
International Organization for Standardization
(2011) · SpringerReference · doi:10.1007/springerreference_16847 · Cited by 12,534
Elm, E. V., Altman, D. G., Egger, M., Pocock, S. J., Gøtzsche, P. C., & Vandenbroucke, J. P. (2007) · BMJ · doi:10.1136/bmj.39335.541782.ad · Cited by 10,766
Estimating causal effects of treatments in randomized and nonrandomized studies
Rubin, D. B. (1974) · Journal of Educational Psychology · doi:10.1037/h0037350 · Cited by 9,436
SPIRIT 2013 Statement: Defining Standard Protocol Items for Clinical Trials
Chan, A.-W., Tetzlaff, J. M., Altman, D. G., Laupacis, A., Gøtzsche, P. C., Krleža-Jerić, K., Hróbjartsson, A., Mann, H., Dickersin, K., Berlin, J. A., et al. (2013) · Annals of Internal Medicine · doi:10.7326/0003-4819-158-3-201302050-00583 · Cited by 7,363
Pencina, M. J., D' Agostino, R. B., D' Agostino, R. B., & Vasan, R. S. (2008) · Statistics in Medicine · doi:10.1002/sim.2929 · Cited by 6,493
PAGE, E. S. (1954) · Biometrika · doi:10.1093/biomet/41.1-2.100 · Cited by 5,739
Causal inference in statistics: An overview
Pearl, J. (2009) · Statistics Surveys · doi:10.1214/09-ss057 · Cited by 2,336
Raji, I. D., Smart, A., White, R. N., Mitchell, M., Gebru, T., Hutchinson, B., Smith-Loud, J., Theron, D., & Barnes, P. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372873 · Cited by 2,245
Page, E. S. (1954) · Biometrika · doi:10.2307/2333009 · Cited by 1,813
Sanderson, S., Tatt, I. D., & Higgins, J. P. (2007) · International Journal of Epidemiology · doi:10.1093/ije/dym018 · Cited by 1,531
Research works printed but not linked: 10
- Van Belle G, Fisher LD, Heagerty PJ, Lumley T. Biostatistics: a methodology for the health sciences. Hoboken, NJ: John Wiley & Sons, Inc., 2004.
- Perdomo J Zrnic T Mendler-Dünner C Hardt M. Performative prediction. In: Daumè H III Singh A eds. Proceedings of the 37th International Conference on Machine Learning Virtual Meeting. PMLR 2020;119:7599-7609. (https://proceedings.mlr.press/v119/perdomo20a/perdomo20a.pdf)
- Liley J, Emerson S, Mateen B, Vallejos C, Aslett L, Vollmer S. Model updating after interventions paradoxically introduces bias. In: Banerjee A, Fukumizu K, eds. Proceedings of the 24th International Conference on Artificial Intelligence and Statistics, PMLR 2021;130:3916-3924 (https://proceedings.mlr.press/v130/liley21a.html).
- Pearl J. Causality. Cambridge: Cambridge University Press, 2009.
- Pyzdek T, Keller P. The six sigma handbook, sixth edition: a complete guide for green belts, black belts, and managers at all levels. New York, NY: McGraw-Hill Education, 2023.
- Montgomery DC. Statistical quality control. 7th ed. Nashville, TN: John Wiley & Sons, 2013.
- Feng J, Subbaswamy A, Gossmann A, et al. Designing monitoring strategies for deployed machine learning algorithms: navigating performativity through a causal lens. In: Locatello F, Didelez V, eds. Proceedings of the 3rd Conference on Causal Learning and Reasoning, PMLR 2024;236:587-608 (https://proceedings.mlr.press/v236/feng24a.html).
- Feng J, Gossmann A, Pennello GA, Petrick N, Sahiner B, Pirracchio R. Monitoring machine learning-based risk prediction algorithms in the presence of performativity. Int Conf Artif Intell Stat 2024;238:919-927 (https://proceedings.mlr.press/v238/feng24b.html).
- Boeken PA Zoeter O Mooij JM. Evaluating and correcting performative effects of decision support systems via causal domain shift. In: Locatello F Didelez V eds. Proceedings of the Third Conference on Causal Learning and Reasoning PMLR 2024:236:551-569 (https://proceedings.mlr.press/v236/boeken24a.html)
- Rotnitzky A. Inverse probability weighted methods. In: Fitzmaurice G, Davidian M, Verbeke G, Molendberghs G, eds. Longitudinal data analysis. Boca Raton, FL: CRC Press, 2009;453-476.
Cited by
Toward an Evaluation Science for Generative AI SystemsPreprint
Weidinger, L., Raji, I. D., Wallach, H., Mitchell, M., Wang, A., Salaudeen, O., Bommasani, R., Ganguli, D., Koyejo, S., & Isaac, W. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2503.05336 · Cited by 114
Bodien, Y. G., Busl, K. M., Chang, C. W. J., Claassen, J., Gaspard, N., Gosseries, O., Helbok, R., Massimini, M., Naccache, L., Newcombe, V., et al. (2025) · Intensive Care Medicine · doi:10.1007/s00134-025-08224-1 · Cited by 37
Toward governance of artificial intelligence in pediatric healthcare
Richter, F., Holmes, E., Richter, F., Guttmann, K., Duong, S. Q., Gangadharan, S., Schadt, E., Salmasian, H., Gelb, B. D., & Glicksberg, B. S. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-02000-7 · Cited by 37
WATCH: Adaptive Monitoring for AI Deployments via Weighted-Conformal MartingalesPreprint
Prinster, D., Han, X., Liu, A., & Saria, S. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2505.04608 · Cited by 25
Owens, K., Griffen, Z., & Damaraju, L. (2025) · BMC Health Services Research · doi:10.1186/s12913-025-13388-z · Cited by 19
Medical laboratory data-based models: opportunities, obstacles, and solutions
Meng, J., Wu, M., Shi, F., Xie, Y., Wang, H., & Guo, Y. (2025) · Journal of Translational Medicine · doi:10.1186/s12967-025-06802-x · Cited by 15
A global log for medical AIPreprint
Noori, A., Boussina, A. E., Bich, H. H., Anibal, J., Maslinski, J., Burger, M., Faltys, M., Rodman, A. M., Karthikesalingam, A., Blasimme, A., et al. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2510.04033 · Cited by 16
Self-citationSorin, V., Korfiatis, P. D., Bratt, A. K., Leiner, T., Wald, C., Butler, C. S., Cook, C. J., Kline, T. L., & Collins, J. D. (2025) · Journal of the American College of Radiology · doi:10.1016/j.jacr.2025.06.036 · Cited by 13
Shafran, R., Bond, L., Carlbring, P., Cohen, Z. D., Creed, T. A., Davey, E., Egan, S. J., Freeman, D. H., Hollon, S. D., Jacobson, N. C., et al. (2025) · Behaviour Research and Therapy · doi:10.1016/j.brat.2025.104945 · Cited by 13
Scaling medical device regulatory science using large language models
Li, H., He, X., Subbaswamy, A., Vossler, P., Gossmann, A., Singh, K., & Feng, J. (2026) · npj Digit. Medicine · doi:10.1038/s41746-026-02353-7 · Cited by 8
Self-citationHuman-AI co-design for clinical prediction models
Feng, J., Kothari, A., Vossler, P., Bishara, A. M., Zier, L. S., Addo, N., Kornblith, A. E., Tan, Y. S., & Singh, C. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-02838-5 · Cited by 12
Self-citationWhy and How to Monitor Deployed AI Systems in Health Care
Keyes, T., Callahan, A., Pandya, A. S., Ambers, N., Banda, J. M., Fuentes, M., Lugtu, C., Masariya, P., Nallan, S., O’Brien, C., et al. (2026) · NEJM Catalyst · doi:10.1056/cat.25.0372 · Cited by 10
Park, S. H., Dean, G., Ortiz, E. M., & Choi, J. (2025) · Korean Journal of Radiology · doi:10.3348/kjr.2025.0257 · Cited by 10
State of the AI: Post-Deployment Monitoring of Radiology-Focused Internally Developed AI
Cook, C. J., Klug, J. R., Kandler, B. W., Suárez, A. B., Dachowicz, A. P., Blezek, D. J., Missert, A. D., Conte, G. M., Benfield, J. D., Mensing-Diggs, A., et al. (2026) · Mayo Clinic Proceedings Digital Health · doi:10.1016/j.mcpdig.2026.100342 · Cited by 9
Clinical readiness of foundation and agentic AI in computational pathology
Da, Q., Chen, Y., Ju, M., Ji, Z., Zhou, A., Wang, W., Abikenari, M. A., Chikontwe, P., Larghero, G., Chen, B., et al. (2026) · Med · doi:10.48550/arxiv.2603.05884 · Cited by 7
Perdomo-Lampignano, J. A., Kale, A. U., Kumar, S., Shah, D., Reddy, B., & Lowe, D. J. (2026) · Journal of Medical Systems · doi:10.1007/s10916-026-02426-w · Cited by 7
Singh, H., Xia, F., Gossmann, A., Chuang, A., Hong, J. C., & Feng, J. (2025) · PubMed · doi:10.48550/arxiv.2506.00756 · Cited by 6
Self-citationDas, O., Tang, L., Oh, E. S., Suárez, J. I., Theodore, N., & Azad, T. D. (2025) · GeroScience · doi:10.1007/s11357-025-01997-9 · Cited by 5
Weng, N., Duan, W., Nguyen, H., Wang, Y., & McNeese, N. J. (2026) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3799437 · Cited by 4
Monitoring Deployed AI Systems in Health Care
Keyes, T., Callahan, A., Pandya, A. S., Ambers, N., Banda, J. M., Fuentes, M., Lugtu, C., Masariya, P., Nallan, S., O'Brien, C., et al. (2025) · arXiv · Cited by 3
Generative Artificial Intelligence and Large Language Models in Clinical Oncology
Yu, Y., Zhao, Z., Wang, Z., Lin, R., Tan, Y., Chen, Y., Li, T., Baptista‐Hon, D. T., Zhang, X., Wu, C., et al. (2026) · MedComm · doi:10.1002/mco2.70833 · Cited by 4
Arab, R. A. E., Mustafa, M. H., Almagharbeh, W. T., Ayoub, M. Y., Alsanawi, F., Almathen, F., Almosabeh, R., & Talaq, M. A. (2026) · Healthcare · doi:10.3390/healthcare14142052 · Cited by 3
Devarajan, Y., Acharya, S. K., Jakhar, S. K., Nagappan, B., Saxena, S. A., Naik, Y., Jayabal, R., & Mehar, K. (2026) · IET Cyber-Physical Systems Theory & Applications · doi:10.1049/cps2.70062 · Cited by 3
Boerman, A. W., van Ingen, M. M., Schotman, H. H. M., Visser, A., Janes, V. A., Nanayakkara, P. W. B., Schut, M. C., & de Jonge, R. (2025) · The Journal of Applied Laboratory Medicine · doi:10.1093/jalm/jfaf131 · Cited by 2
Characterizing industry payments for FDA-approved AI medical devices
Bergman, A., Patel, T. A., & Shah, K. P. (2025) · Health Affairs Scholar · doi:10.1093/haschl/qxaf211 · Cited by 2
Lipskiy, N., & Flowerday, S. (2026) · Future Internet · doi:10.3390/fi18050232 · Cited by 2
Scott, I. A., White, C., Shaw, T., Slade, C., Wan, T. T., Barmanray, R., Coorey, C. P., Johnson, S. L. J., Bell, L. M., Herd, M., et al. (2026) · Internal Medicine Journal · doi:10.1111/imj.70594 · Cited by 2
Cai, L., Meng, B., Huang, J., Ding, G., Ju, M., Wang, W.-W., Shi-jie, D., Lai, L., Wang, J., Yang, C.-X., et al. (2026) · LabMed discovery. · doi:10.1016/j.lmd.2026.100130 · Cited by 1
Kang, X., Ding, W., Matsumoto, K., Wang, L., Yu, H.-T., & Shi, X. (2026) · Information Fusion · doi:10.1016/j.inffus.2026.104702 · Cited by 1
Avoiding common failures in AI for health and medicine
Salaudeen, O., Zhang, H., Kim, E., Singh, K., & Ghassemi, M. (2026) · Cell · doi:10.1016/j.cell.2026.07.025 · Cited by 1
Self-citationMultimorbidity measurement strategies for predicting hospital visits
Neves, B. D., Moreira, J. M., Gonçalves, S., Cerejo, J., Mota, I., Silva, N. A., Leite, F., & Silva, M. J. (2025) · BMC Health Services Research · doi:10.1186/s12913-025-13817-z
Critical appraisal of the “ASHP Statement on the Use of Artificial Intelligence in Pharmacy”
Kharde, P. K., Kharde, Y., & Kanade, T. M. (2025) · American Journal of Health-System Pharmacy · doi:10.1093/ajhp/zxaf362
Deng, J., Patel, S. K., Fung, M., Heybati, K., Layard, B., Wang, B., Rubin, B., Simard, T., Hibbert, B., Anderson, T. J., et al. (2026) · BMJ Open · doi:10.1136/bmjopen-2025-108799
Jung, S., & Son, K. (2026) · JMIR Medical Informatics · doi:10.2196/94846
Umasugi, M. T., Fitriasari, E., & Tuharea, A. (2026) · Legal Medicine · doi:10.1016/j.legalmed.2026.102913
Fetal brain MRI analysis: Towards clinical translation and impact
Luis, A., Uus, A. U., Story, L., & Rutherford, M. (2026) · Developmental Medicine & Child Neurology · doi:10.1111/dmcn.70460
Big Problems, Big Data, Big Answers
Rider, N. L. (2026) · Immunology and Allergy Clinics of North America · doi:10.1016/j.iac.2026.08.010
Federated artificial intelligence monitoring service (FAMOS): an in silico feasibility study.
Luis, A., Scarsbrook, A. F., Grzeda, M. T., Perdomo-Lampignano, J. A., Clark, M., Wheller, B., Baldwin, J., Cairns, J., Dhesi, S. S., Hall, M., et al. (2026) · PubMed · doi:10.1093/bjrai/ubag016
A Framework to Monitor Editing of Artificial Intelligence-Generated Medical DocumentationPreprint
García-Agúndez, A., Zhou, S., Pourian, J., Blebea, C., Daneshpajouhnejad, P., Dente, E., Shi, K. X., Pollet, S., Xia, F., Shi, X., et al. (2026) · medRxiv · doi:10.64898/2026.09.30.26364427
Self-citationLearning to Detect Shifts in Unstructured Data Streams via Monitoring-by-BettingPreprint
Zhou, S., Ouyang, J., Xia, F., Subbaswamy, A., Prinster, D., Saria, S., García-Agúndez, A., Pollet, S., Yazdany, J., Vossler, P., et al. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.17399
Self-citationClinical AI in 2025–2026 — Six Themes from the SAIL Year in Review
Kim, E., Elia, M. V., Li, M. M., Healey, E., Erdman, L., Chandak, P., Argaw, P., Li, B., Black, K. C., Marwaha, J. S., et al. (2026) · NEJM AI · doi:10.1056/aip2600817
Self-citationYuji, K., & Yuji, W. (2026) · Expert Review of Medical Devices · doi:10.1080/17434440.2026.2753843
Generating Real‐World Evidence From AI Deployed in Health Systems
Poursoltan, L., Boussina, A. E., Cao, J., Kim, E., Bedmutha, P. S., Majmudar, H., Ramsis, M., Khalessi, A. A., & Singh, K. (2026) · Learning Health Systems · doi:10.1002/lrh2.70131
Self-citationMultimorbidity Measurement Strategies for Predicting Hospital Visits
Neves, B., Moreira, J. M., Gonçalves, S., Cerejo, J., Mota, I., Silva, N. A., Leite, F., & Silva, M. J. (2025) · doi:10.21203/rs.3.rs-6515465/v1
Metadata
- Authors
- Feng, J., Xia, F., Singh, K., & Pirracchio, R.
- Affiliations
- University of California, San Francisco (US)
- University of California San Diego (US)
- Venue
- NEJM AI
- Publisher
- Massachusetts Medical Society
- Published
- 2025-01-23
- Volume
- 2
- Issue
- 2
- Pages
- AIra2400657-AIra2400657
- ISSN
- 2836-9386 (ISSN-L)
- OpenAlex
- W4406740986
- Cited by
- 28
- References
- 61
- Citations by year
- 2026: 16 · 2025: 12
- FWCI
- 6.78
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- NEJM AI (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Health Sciences › Medicine › Surgery › Healthcare Technology and Patient Monitoring
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Keywords
- artificial intelligence in healthcare, clinical decision support, AI evaluation, clinical implementation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.