Not All Clinical AI Monitoring Systems Are Created Equal: Review and Recommendations
The DOI matches its Crossref record; 63 of the 63 references it cites were found in a registry or an index.
Abstract
AbstractWhile practices for the initial evaluation of clinical artificial intelligence (AI) algorithms are well established, there is little consensus on how to design effective monitoring systems for the post-deployment setting. In real-world case studies, design choices are often driven by practical constraints rather than their impact on the performance of the monitoring system. This narrative review critically examines the key decisions that shape the performance of AI monitoring systems, including the selection of monitoring criteria, the choice of data sources, and the statistical procedures employed. Our findings reveal significant variation in the designed systems, often with little transparency regarding how their design choices affect monitoring performance. To provide a more structured approach to designing monitoring systems that are both effective and practical, we introduce a road map for navigating the many options available. We bootstrap efforts in clinical AI monitoring by highlighting tools from three related fields that face similarly complex challenges: quality management, clinical trials, and real-world evidence generation.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Many organizations use model dashboards to track performance metrics, although consensus on what those metrics should be is hard to find [17].”
From Managing a 'responsibility vacuum' in AI monitoring and governance in healthcare: a qualitative study (2025) · cites this paper as [17]
Citations
Cites
International Organization for Standardization
(2011) · SpringerReference · doi:10.1007/springerreference_16847 · Cited by 12,534
Elm, E. V., Altman, D. G., Egger, M., Pocock, S. J., Gøtzsche, P. C., & Vandenbroucke, J. P. (2007) · BMJ · doi:10.1136/bmj.39335.541782.ad · Cited by 10,766
Estimating causal effects of treatments in randomized and nonrandomized studies
Rubin, D. B. (1974) · Journal of Educational Psychology · doi:10.1037/h0037350 · Cited by 9,436
SPIRIT 2013 Statement: Defining Standard Protocol Items for Clinical Trials
Chan, A.-W., Tetzlaff, J. M., Altman, D. G., Laupacis, A., Gøtzsche, P. C., Krleža-Jerić, K., Hróbjartsson, A., Mann, H., Dickersin, K., Berlin, J. A., et al. (2013) · Annals of Internal Medicine · doi:10.7326/0003-4819-158-3-201302050-00583 · Cited by 7,363
Pencina, M. J., D' Agostino, R. B., D' Agostino, R. B., & Vasan, R. S. (2008) · Statistics in Medicine · doi:10.1002/sim.2929 · Cited by 6,493
PAGE, E. S. (1954) · Biometrika · doi:10.1093/biomet/41.1-2.100 · Cited by 5,739
Causal inference in statistics: An overview
Pearl, J. (2009) · Statistics Surveys · doi:10.1214/09-ss057 · Cited by 2,336
Raji, I. D., Smart, A., White, R. N., Mitchell, M., Gebru, T., Hutchinson, B., Smith-Loud, J., Theron, D., & Barnes, P. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372873 · Cited by 2,245
Page, E. S. (1954) · Biometrika · doi:10.2307/2333009 · Cited by 1,813
Sanderson, S., Tatt, I. D., & Higgins, J. P. (2007) · International Journal of Epidemiology · doi:10.1093/ije/dym018 · Cited by 1,531
The Clinician and Dataset Shift in Artificial Intelligence
Finlayson, S. G., Subbaswamy, A., Singh, K., Bowers, J., Kupke, A., Zittrain, J., Kohane, I. S., & Saria, S. (2021) · New England Journal of Medicine · doi:10.1056/nejmc2104626 · Cited by 855
Self-citationThe design of simulation studies in medical statistics
Burton, A., Altman, D. G., Royston, P., & Holder, R. L. (2006) · Statistics in Medicine · doi:10.1002/sim.2673 · Cited by 752
Assessment of diagnostic tests when disease verification is subject to selection bias
Begg, C. B., & Greenes, R. A. (1983) · Biometrics · doi:10.2307/2530820 · Cited by 708
Monitor Alarm Fatigue : An Integrative Review
Cvach, M. (2012) · Biomedical Instrumentation & Technology · doi:10.2345/0899-8205-46.4.268 · Cited by 632
Multiple Testing Procedures with Applications to Genomics
Dudoit, S., & van der Laan, M. J. (2008) · Springer Series in Statistics · doi:10.1007/978-0-387-49317-6 · Cited by 441
Tartakovsky, A., Nikiforov, I., & Basseville, M. (2014) · Chapman & Hall/CRC eBooks · doi:10.1201/b17279 · Cited by 437
Feng, J., Phillips, R. V., Malenica, I., Bishara, A., Hubbard, A. E., Celi, L. A., & Pirracchio, R. (2022) · npj Digital Medicine · doi:10.1038/s41746-022-00611-y · Cited by 423
Self-citationLeading indicators of construction safety performance
Hinze, J., Thurman, S., & Wehle, A. (2013) · Safety Science · doi:10.1016/j.ssci.2012.05.016 · Cited by 411
Strategies for Handling Missing Data in Electronic Health Record Derived Data
Wells, B. J., Nowacki, A. S., Chagin, K., & Kattan, M. W. (2013) · eGEMs (Generating Evidence & Methods to improve patient outcomes) · doi:10.13063/2327-9214.1035 · Cited by 407
Tutorial on directed acyclic graphs
Digitale, J. C., Martin, J. N., & Glymour, M. M. (2022) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2021.08.001 · Cited by 403
Causality matters in medical imaging
Castro, D. C., Walker, I., & Glocker, B. (2020) · Nature Communications · doi:10.1038/s41467-020-17478-w · Cited by 364
How algorithmic confounding in recommendation systems increases homogeneity and decreases utility
Chaney, A. J. B., Stewart, B. M., & Engelhardt, B. E. (2018) · Proceedings of the 12th ACM Conference on Recommender Systems · doi:10.1145/3240323.3240370 · Cited by 331
Calibration drift in regression and machine learning models for acute kidney injury
Davis, S. E., Lasko, T. A., Chen, G., Siew, E. D., & Matheny, M. E. (2017) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocx030 · Cited by 312
Causality matters in medical imaging
Castro, D. C., Walker, I. D., & Glocker, B. (2020) · RePEc: Research Papers in Economics · Cited by 288
Data drift in medical machine learning: implications and potential remedies
Sahiner, B., Chen, W., Samala, R. K., & Petrick, N. (2023) · The British Journal of Radiology · doi:10.1259/bjr.20220878 · Cited by 279
Data Monitoring Committees in Clinical Trials
Ellenberg, S. S., Fleming, T. R., & DeMets, D. L. (2002) · Wiley eBooks · doi:10.1002/0470854162 · Cited by 268
Liu, X., Glocker, B., McCradden, M. M., Ghassemi, M., Denniston, A. K., & Oakden-Rayner, L. (2022) · The Lancet Digital Health · doi:10.1016/s2589-7500(22)00003-6 · Cited by 266
An Overview of Phase I Analysis for Process Improvement and Monitoring
Jones-Farmer, L. A., Woodall, W. H., Steiner, S. H., & Champ, C. W. (2014) · Journal of Quality Technology · doi:10.1080/00224065.2014.11917969 · Cited by 232
Statistical Challenges Facing Early Outbreak Detection in Biosurveillance
Shmueli, G., & Burkom, H. (2010) · Technometrics · doi:10.1198/tech.2010.06134 · Cited by 191
Raji, I. D., Smart, A., White, R. N., Mitchell, M., Gebru, T., Hutchinson, B., Smith-Loud, J., Theron, D., & Barnes, P. (2020) · arXiv · doi:10.48550/arxiv.2001.00973 · Cited by 179
Choosing primary endpoints for clinical trials of health care interventions
McLeod, C., Norman, R., Litton, E., Saville, B. R., Webb, S. A. R., & Snelling, T. (2019) · Contemporary Clinical Trials Communications · doi:10.1016/j.conctc.2019.100486 · Cited by 170
Temporal quality degradation in AI models
Vela, D., Sharp, A., Zhang, R., Nguyen, T., Hoang, A., & Pianykh, O. S. (2022) · Scientific Reports · doi:10.1038/s41598-022-15245-z · Cited by 168
Statistical Methods for Healthcare Regulation: Rating, Screening and Surveillance
Spiegelhalter, D. J., Sherlaw‐Johnson, C., Bardsley, M., Blunt, I., Wood, C., & Grigg, O. A. J. (2011) · Journal of the Royal Statistical Society Series A (Statistics in Society) · doi:10.1111/j.1467-985x.2011.01010.x · Cited by 165
Ensuring trial validity by data quality assurance and diversification of monitoring methods
Baigent, C., Harrell, F. E., Buyse, M., Emberson, J. R., & Altman, D. G. (2008) · Clinical Trials · doi:10.1177/1740774507087554 · Cited by 163
Hwang, T. J., Kesselheim, A. S., & Vokinger, K. N. (2019) · JAMA · doi:10.1001/jama.2019.16842 · Cited by 156
An Overview of Risk-Adjusted Charts
Grigg, O. A. J., & Farewell, V. T. (2004) · Journal of the Royal Statistical Society Series A (Statistics in Society) · doi:10.1111/j.1467-985x.2004.0apm2.x · Cited by 138
A framework for the oversight and local deployment of safe and high-quality prediction models
Bedoya, A. D., Economou-Zavlanos, N. J., Goldstein, B. A., Young, A., Jelovsek, J. E., O’Brien, C., Parrish, A. B., Elengold, S., Lytle, K., Balu, S., et al. (2022) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocac078 · Cited by 119
Brady, A. P., Allen, B., Chong, J., Kotter, E., Kottler, N. E., Mongan, J., Oakden‐Rayner, L., dos Santos, D. P., Tang, A., Wald, C., et al. (2024) · Insights into Imaging · doi:10.1186/s13244-023-01541-3 · Cited by 119
A Nationwide Network of Health AI Assurance Laboratories
Shah, N. H., Halamka, J. D., Saria, S., Pencina, M., Tazbaz, T., Tripathi, M., Callahan, A., Hildahl, H., & Anderson, B. (2024) · JAMA · doi:10.1001/jama.2023.26930 · Cited by 114
Adaptive Design Methods in Clinical Trials
Chow, S.-C., & Chang, M. (2006) · Chapman & Hall/CRC eBooks · doi:10.1201/9781584887775 · Cited by 108
Use of the false discovery rate when comparing multiple health care providers
Jones, H. E., Ohlssen, D. I., & Spiegelhalter, D. J. (2007) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2007.04.017 · Cited by 95
Prognostic models will be victims of their own success, unless…
Lenert, M. C., Matheny, M. E., & Walsh, C. G. (2019) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocz145 · Cited by 92
Oversight of clinical investigations- A risk based approach to monitoring
Obied, T. (2014) · Pharmaceutical Regulatory Affairs Open Access · Cited by 88
Statistical Issues in the Prospective Monitoring of Health Outcomes Across Multiple Units
Marshall, C., Best, N., Bottle, A., & Aylin, P. P. (2004) · Journal of the Royal Statistical Society Series A (Statistics in Society) · doi:10.1111/j.1467-985x.2004.apm10.x · Cited by 87
Developing predictive models using electronic medical records: challenges and pitfalls.
Paxton, C., Niculescu-Mizil, A., & Saria, S. (2013) · Europe PMC (PubMed Central) · Cited by 86
A Review and perspective on surveillance of Bernoulli processes
Szarka, J. L., & Woodall, W. H. (2011) · Quality and Reliability Engineering International · doi:10.1002/qre.1256 · Cited by 85
The Structured Process to Identify Fit‐For‐Purpose Data: A Data Feasibility Assessment Framework
Gatto, N. M., Campbell, U. B., Rubinstein, E. R., Jaksa, A. A., Mattox, P. W., Mo, J., & Reynolds, R. F. (2021) · Clinical Pharmacology & Therapeutics · doi:10.1002/cpt.2466 · Cited by 85
Red-Teaming for Generative AI: Silver Bullet or Security Theater?
Feffer, M., Sinha, A., Deng, W. H., Lipton, Z. C., & Heidari, H. (2024) · Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1609/aies.v7i1.31647 · Cited by 75
International Council for Harmonization
Dhandia, D., Jain, S., Shukla, V. K., Sharma, N., Yadav, P., & Arora, S. (2025) · Understanding Pharmaceutical Standards and Regulations · doi:10.4324/9781003500971-15 · Cited by 75
Ho, M., van der Laan, M. J., Lee, H., Chen, J., Lee, K., Fang, Y., He, W., Irony, T., Jiang, Q., Lin, X., et al. (2021) · Statistics in Biopharmaceutical Research · doi:10.1080/19466315.2021.1883475 · Cited by 59
Clinical deployment environments: Five pillars of translational machine learning for health
Harris, S. K., Bonnici, T., Keen, T., Lilaonitkul, W., White, M. J., & Swanepoel, N. (2022) · Frontiers in Digital Health · doi:10.3389/fdgth.2022.939292 · Cited by 54
Corbin, C. K., Maclay, R., Acharya, A., Mony, S., Punnathanam, S., Thapa, R., Kotecha, N., Shah, N. H., & Chen, J. H. (2023) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocad114 · Cited by 50
Harmon, D. M., Mangold, K., Baez Suarez, A., Scott, C. G., Murphree, D. H., Malik, A., Attia, Z. I., Lopez-Jimenez, F., Friedman, P. A., Dispenzieri, A., et al. (2023) · JACC: Advances · doi:10.1016/j.jacadv.2023.100612 · Cited by 47
Ristić, D. (2013) · Safety Engineering · doi:10.7562/se2013.3.03.03 · Cited by 45
Ross, J., Hammouche, S., Chen, Y., Rockall, A., Alabed, S., Chen, M., Dwivedi, K., Fascia, D., Greenhalgh, R., Hall, M., et al. (2024) · Clinical Radiology · doi:10.1016/j.crad.2024.01.026 · Cited by 35
Open questions and research gaps for monitoring and updating AI-enabled tools in clinical settings
Davis, S. E., Walsh, C. G., & Matheny, M. E. (2022) · Frontiers in Digital Health · doi:10.3389/fdgth.2022.958284 · Cited by 33
Simulation Practices for Adaptive Trial Designs in Drug and Device Development
Mayer, C., Perevozskaya, I. T., Leonov, S. L., Dragalin, V., Pritchett, Y., Bedding, A. W., Hartford, A. H., Fardipour, P., & Cicconetti, G. (2019) · Statistics in Biopharmaceutical Research · doi:10.1080/19466315.2018.1560359 · Cited by 29
Considerations for Quality Control Monitoring of Machine Learning Models in Clinical Practice
Faust, L. J., Wilson, P. M., Asai, S. W., Fu, S., Liu, H., Ruan, X., & Storlie, C. B. (2024) · JMIR Medical Informatics · doi:10.2196/50437 · Cited by 25
Detecting changes in the performance of a clinical machine learning tool over time
Schinkel, M., Boerman, A. W., Paranjape, K., Wiersinga, W. J., & Nanayakkara, P. W. B. (2023) · EBioMedicine · doi:10.1016/j.ebiom.2023.104823 · Cited by 21
Using Rapid Randomized Trials to Improve Health Care Systems
Horwitz, L. I., & Krelle, H. (2022) · Annual Review of Public Health · doi:10.1146/annurev-publhealth-071521-025758 · Cited by 17
Han, L., Arfè, A., & Trippa, L. (2023) · The American Statistician · doi:10.1080/00031305.2023.2216253 · Cited by 12
Sensitivity analysis for causality in observational studies for regulatory science
Diaz, I. M., Lee, H., Kıcıman, E., Schenck, E. J., Akacha, M., Follman, D., & Ghosh, D. (2023) · Journal of Clinical and Translational Science · doi:10.1017/cts.2023.688 · Cited by 9
Dashboarding to Monitor Machine-Learning-Based Clinical Decision Support Interventions
Hekman, D. J., Barton, H. J., Maru, A., Wills, G. J., Cochran, A. L., Fritsch, C., Wiegmann, D. A., Liao, F., & Patterson, B. W. (2023) · Applied Clinical Informatics · doi:10.1055/a-2219-5175 · Cited by 6
Research works printed but not linked: 10
- Van Belle G, Fisher LD, Heagerty PJ, Lumley T. Biostatistics: a methodology for the health sciences. Hoboken, NJ: John Wiley & Sons, Inc., 2004.
- Perdomo J Zrnic T Mendler-Dünner C Hardt M. Performative prediction. In: Daumè H III Singh A eds. Proceedings of the 37th International Conference on Machine Learning Virtual Meeting. PMLR 2020;119:7599-7609. (https://proceedings.mlr.press/v119/perdomo20a/perdomo20a.pdf)
- Liley J, Emerson S, Mateen B, Vallejos C, Aslett L, Vollmer S. Model updating after interventions paradoxically introduces bias. In: Banerjee A, Fukumizu K, eds. Proceedings of the 24th International Conference on Artificial Intelligence and Statistics, PMLR 2021;130:3916-3924 (https://proceedings.mlr.press/v130/liley21a.html).
- Pearl J. Causality. Cambridge: Cambridge University Press, 2009.
- Pyzdek T, Keller P. The six sigma handbook, sixth edition: a complete guide for green belts, black belts, and managers at all levels. New York, NY: McGraw-Hill Education, 2023.
- Montgomery DC. Statistical quality control. 7th ed. Nashville, TN: John Wiley & Sons, 2013.
- Feng J, Subbaswamy A, Gossmann A, et al. Designing monitoring strategies for deployed machine learning algorithms: navigating performativity through a causal lens. In: Locatello F, Didelez V, eds. Proceedings of the 3rd Conference on Causal Learning and Reasoning, PMLR 2024;236:587-608 (https://proceedings.mlr.press/v236/feng24a.html).
- Feng J, Gossmann A, Pennello GA, Petrick N, Sahiner B, Pirracchio R. Monitoring machine learning-based risk prediction algorithms in the presence of performativity. Int Conf Artif Intell Stat 2024;238:919-927 (https://proceedings.mlr.press/v238/feng24b.html).
- Boeken PA Zoeter O Mooij JM. Evaluating and correcting performative effects of decision support systems via causal domain shift. In: Locatello F Didelez V eds. Proceedings of the Third Conference on Causal Learning and Reasoning PMLR 2024:236:551-569 (https://proceedings.mlr.press/v236/boeken24a.html)
- Rotnitzky A. Inverse probability weighted methods. In: Fitzmaurice G, Davidian M, Verbeke G, Molendberghs G, eds. Longitudinal data analysis. Boca Raton, FL: CRC Press, 2009;453-476.
Cited by
Toward an Evaluation Science for Generative AI SystemsPreprint
Weidinger, L., Raji, I. D., Wallach, H., Mitchell, M., Wang, A., Salaudeen, O., Bommasani, R., Ganguli, D., Koyejo, S., & Isaac, W. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2503.05336 · Cited by 108
Bodien, Y. G., Busl, K. M., Chang, C. W. J., Claassen, J., Gaspard, N., Gosseries, O., Helbok, R., Massimini, M., Naccache, L., Newcombe, V., et al. (2025) · Intensive Care Medicine · doi:10.1007/s00134-025-08224-1 · Cited by 37
Toward governance of artificial intelligence in pediatric healthcare
Richter, F., Holmes, E., Richter, F., Guttmann, K., Duong, S. Q., Gangadharan, S., Schadt, E., Salmasian, H., Gelb, B. D., & Glicksberg, B. S. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-02000-7 · Cited by 33
WATCH: Adaptive Monitoring for AI Deployments via Weighted-Conformal MartingalesPreprint
Prinster, D., Han, X., Liu, A., & Saria, S. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2505.04608 · Cited by 25
Owens, K., Griffen, Z., & Damaraju, L. (2025) · BMC Health Services Research · doi:10.1186/s12913-025-13388-z · Cited by 19
Metadata
- Authors
- Feng, J., Xia, F., Singh, K., & Pirracchio, R.
- Affiliations
- University of California, San Francisco (US)
- University of California San Diego (US)
- Venue
- NEJM AI
- Publisher
- Massachusetts Medical Society
- Published
- 2025-01-23
- Volume
- 2
- Issue
- 2
- Pages
- AIra2400657-AIra2400657
- ISSN
- 2836-9386 (ISSN-L)
- OpenAlex
- W4406740986
- Cited by
- 28
- References
- 61
- Citations by year
- 2026: 16 · 2025: 12
- FWCI
- 6.78
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- NEJM AI (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Health Sciences › Medicine › Surgery › Healthcare Technology and Patient Monitoring
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Keywords
- artificial intelligence in healthcare, clinical decision support, AI evaluation, clinical implementation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.