Not All Clinical AI Monitoring Systems Are Created Equal: Review and Recommendations
The DOI matches its Crossref record; 63 of the 63 references it cites were found in a registry or an index.
Abstract
AbstractWhile practices for the initial evaluation of clinical artificial intelligence (AI) algorithms are well established, there is little consensus on how to design effective monitoring systems for the post-deployment setting. In real-world case studies, design choices are often driven by practical constraints rather than their impact on the performance of the monitoring system. This narrative review critically examines the key decisions that shape the performance of AI monitoring systems, including the selection of monitoring criteria, the choice of data sources, and the statistical procedures employed. Our findings reveal significant variation in the designed systems, often with little transparency regarding how their design choices affect monitoring performance. To provide a more structured approach to designing monitoring systems that are both effective and practical, we introduce a road map for navigating the many options available. We bootstrap efforts in clinical AI monitoring by highlighting tools from three related fields that face similarly complex challenges: quality management, clinical trials, and real-world evidence generation.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Many organizations use model dashboards to track performance metrics, although consensus on what those metrics should be is hard to find [17].”
From Managing a 'responsibility vacuum' in AI monitoring and governance in healthcare: a qualitative study (2025) · cites this paper as [17]
Citations
Cites
International Organization for Standardization
(2011) · SpringerReference · doi:10.1007/springerreference_16847 · Cited by 12,534
Elm, E. V., Altman, D. G., Egger, M., Pocock, S. J., Gøtzsche, P. C., & Vandenbroucke, J. P. (2007) · BMJ · doi:10.1136/bmj.39335.541782.ad · Cited by 10,766
Estimating causal effects of treatments in randomized and nonrandomized studies
Rubin, D. B. (1974) · Journal of Educational Psychology · doi:10.1037/h0037350 · Cited by 9,436
SPIRIT 2013 Statement: Defining Standard Protocol Items for Clinical Trials
Chan, A.-W., Tetzlaff, J. M., Altman, D. G., Laupacis, A., Gøtzsche, P. C., Krleža-Jerić, K., Hróbjartsson, A., Mann, H., Dickersin, K., Berlin, J. A., et al. (2013) · Annals of Internal Medicine · doi:10.7326/0003-4819-158-3-201302050-00583 · Cited by 7,363
Pencina, M. J., D' Agostino, R. B., D' Agostino, R. B., & Vasan, R. S. (2008) · Statistics in Medicine · doi:10.1002/sim.2929 · Cited by 6,493
PAGE, E. S. (1954) · Biometrika · doi:10.1093/biomet/41.1-2.100 · Cited by 5,739
Causal inference in statistics: An overview
Pearl, J. (2009) · Statistics Surveys · doi:10.1214/09-ss057 · Cited by 2,336
Raji, I. D., Smart, A., White, R. N., Mitchell, M., Gebru, T., Hutchinson, B., Smith-Loud, J., Theron, D., & Barnes, P. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372873 · Cited by 2,245
Page, E. S. (1954) · Biometrika · doi:10.2307/2333009 · Cited by 1,813
Sanderson, S., Tatt, I. D., & Higgins, J. P. (2007) · International Journal of Epidemiology · doi:10.1093/ije/dym018 · Cited by 1,531
Research works printed but not linked: 10
- Van Belle G, Fisher LD, Heagerty PJ, Lumley T. Biostatistics: a methodology for the health sciences. Hoboken, NJ: John Wiley & Sons, Inc., 2004.
- Perdomo J Zrnic T Mendler-Dünner C Hardt M. Performative prediction. In: Daumè H III Singh A eds. Proceedings of the 37th International Conference on Machine Learning Virtual Meeting. PMLR 2020;119:7599-7609. (https://proceedings.mlr.press/v119/perdomo20a/perdomo20a.pdf)
- Liley J, Emerson S, Mateen B, Vallejos C, Aslett L, Vollmer S. Model updating after interventions paradoxically introduces bias. In: Banerjee A, Fukumizu K, eds. Proceedings of the 24th International Conference on Artificial Intelligence and Statistics, PMLR 2021;130:3916-3924 (https://proceedings.mlr.press/v130/liley21a.html).
- Pearl J. Causality. Cambridge: Cambridge University Press, 2009.
- Pyzdek T, Keller P. The six sigma handbook, sixth edition: a complete guide for green belts, black belts, and managers at all levels. New York, NY: McGraw-Hill Education, 2023.
- Montgomery DC. Statistical quality control. 7th ed. Nashville, TN: John Wiley & Sons, 2013.
- Feng J, Subbaswamy A, Gossmann A, et al. Designing monitoring strategies for deployed machine learning algorithms: navigating performativity through a causal lens. In: Locatello F, Didelez V, eds. Proceedings of the 3rd Conference on Causal Learning and Reasoning, PMLR 2024;236:587-608 (https://proceedings.mlr.press/v236/feng24a.html).
- Feng J, Gossmann A, Pennello GA, Petrick N, Sahiner B, Pirracchio R. Monitoring machine learning-based risk prediction algorithms in the presence of performativity. Int Conf Artif Intell Stat 2024;238:919-927 (https://proceedings.mlr.press/v238/feng24b.html).
- Boeken PA Zoeter O Mooij JM. Evaluating and correcting performative effects of decision support systems via causal domain shift. In: Locatello F Didelez V eds. Proceedings of the Third Conference on Causal Learning and Reasoning PMLR 2024:236:551-569 (https://proceedings.mlr.press/v236/boeken24a.html)
- Rotnitzky A. Inverse probability weighted methods. In: Fitzmaurice G, Davidian M, Verbeke G, Molendberghs G, eds. Longitudinal data analysis. Boca Raton, FL: CRC Press, 2009;453-476.
Cited by
Toward an Evaluation Science for Generative AI SystemsPreprint
Weidinger, L., Raji, I. D., Wallach, H., Mitchell, M., Wang, A., Salaudeen, O., Bommasani, R., Ganguli, D., Koyejo, S., & Isaac, W. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2503.05336 · Cited by 107
Bodien, Y. G., Busl, K. M., Chang, C. W. J., Claassen, J., Gaspard, N., Gosseries, O., Helbok, R., Massimini, M., Naccache, L., Newcombe, V., et al. (2025) · Intensive Care Medicine · doi:10.1007/s00134-025-08224-1 · Cited by 36
Toward governance of artificial intelligence in pediatric healthcare
Richter, F., Holmes, E., Richter, F., Guttmann, K., Duong, S. Q., Gangadharan, S., Schadt, E., Salmasian, H., Gelb, B. D., & Glicksberg, B. S. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-02000-7 · Cited by 33
WATCH: Adaptive Monitoring for AI Deployments via Weighted-Conformal MartingalesPreprint
Prinster, D., Han, X., Liu, A., & Saria, S. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2505.04608 · Cited by 25
Owens, K., Griffen, Z., & Damaraju, L. (2025) · BMC Health Services Research · doi:10.1186/s12913-025-13388-z · Cited by 19
Metadata
- Authors
- Feng, J., Xia, F., Singh, K., & Pirracchio, R.
- Affiliations
- University of California, San Francisco (US)
- University of California San Diego (US)
- Venue
- NEJM AI
- Publisher
- Massachusetts Medical Society
- Published
- 2025-01-23
- Volume
- 2
- Issue
- 2
- Pages
- AIra2400657-AIra2400657
- ISSN
- 2836-9386 (ISSN-L)
- OpenAlex
- W4406740986
- Cited by
- 28
- References
- 61
- Citations by year
- 2026: 16 · 2025: 12
- FWCI
- 6.78
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- NEJM AI (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Health Sciences › Medicine › Surgery › Healthcare Technology and Patient Monitoring
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Keywords
- artificial intelligence in healthcare, clinical decision support, AI evaluation, clinical implementation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Not All Clinical AI Monitoring Systems Are Created Equal: Review and Recommendations. https://app.cochat.ai/papers/10.1056/aira2400657
- The paper
- Feng, J., Xia, F., Singh, K., & Pirracchio, R. (2025). Not All Clinical AI Monitoring Systems Are Created Equal: Review and Recommendations. NEJM AI. https://doi.org/10.1056/aira2400657
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.