Why and How to Monitor Deployed AI Systems in Health Care
Abstract
Postdeployment monitoring of artificial intelligence (AI) systems in health care is essential to ensure their safety, quality, and sustained benefit - and to support governance decisions about which systems to update, modify, or decommission. Motivated by these needs, the authors developed a framework for monitoring deployed AI systems organized around three complementary principles: system integrity, performance, and impact. System integrity monitoring focuses on maximizing system uptime, detecting runtime errors, and identifying when changes to the surrounding information technology ecosystem have unintended effects. Performance monitoring focuses on maintaining accurate and equitable system behavior in the face of changing health care practices (and thus input data) over time. Impact monitoring assesses whether a deployed system continues to have value in the form of benefit to clinicians, staff, and patients. Drawing on examples of deployed AI systems at their academic medical center, the authors provide practical guidance for creating monitoring plans based on these principles that specify which metrics to measure and at what cadence, who is responsible for acting when metrics change, and what concrete follow-up actions should be taken - for both traditional and generative AI. They also discuss challenges in implementing this framework, including the effort of monitoring for health systems with limited resources, and the difficulty of incorporating data-driven monitoring practices into complex organizations where conflicting priorities and definitions of success often coexist. This framework offers a starting point for health systems seeking to ensure that AI deployments remain safe and effective over time.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Cites
Wong, A., Otles, E., Donnelly, J. P., Krumm, A., McCullough, J., DeTroyer-Cooley, O., Pestrue, J., Phillips, M., Konye, J., Penoza, C., et al. (2021) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2021.2626 · Cited by 995 · see citing works
The Clinician and Dataset Shift in Artificial Intelligence
Finlayson, S. G., Subbaswamy, A., Singh, K., Bowers, J., Kupke, A., Zittrain, J., Kohane, I. S., & Saria, S. (2021) · New England Journal of Medicine · doi:10.1056/nejmc2104626 · Cited by 855 · see citing works
Feng, J., Phillips, R. V., Malenica, I., Bishara, A., Hubbard, A. E., Celi, L. A., & Pirracchio, R. (2022) · npj Digital Medicine · doi:10.1038/s41746-022-00611-y · Cited by 423 · see citing works
A History of the Joint Commission on Accreditation of Hospitals
Roberts, J. S. (1987) · JAMA · doi:10.1001/jama.1987.03400070074038 · Cited by 181 · see citing works
The rise of agentic AI teammates in medicine
Zou, J., & Topol, E. J. (2025) · The Lancet · doi:10.1016/s0140-6736(25)00202-8 · Cited by 138 · see citing works
A framework for the oversight and local deployment of safe and high-quality prediction models
Bedoya, A. D., Economou-Zavlanos, N. J., Goldstein, B. A., Young, A., Jelovsek, J. E., O’Brien, C., Parrish, A. B., Elengold, S., Lytle, K., Balu, S., et al. (2022) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocac078 · Cited by 119 · see citing works
Self-citationChen, J. H., Alagappan, M., Goldstein, M. K., Asch, S. M., & Altman, R. B. (2017) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2017.03.006 · Cited by 101 · see citing works
Self-citationBuilding Institutional Capacity for Diversity and Inclusion in Academic Medicine
Smith, D. G. (2012) · Academic Medicine · doi:10.1097/acm.0b013e31826d30d5 · Cited by 73 · see citing works
Singh, V., Cheng, S. C., Kwan, A. C., & Ebinger, J. E. (2025) · Mayo Clinic Proceedings Digital Health · doi:10.1016/j.mcpdig.2025.100231 · Cited by 70 · see citing works
Callahan, A., McElfresh, D., Banda, J. M., Bunney, G., Char, D., Chen, J., Corbin, C. K., Dash, D., Downing, N. L., Jain, S. S., et al. (2024) · NEJM Catalyst · doi:10.1056/cat.24.0131 · Cited by 61 · see citing works
Self-citationEconomou-Zavlanos, N. J., Bessias, S., Cary, M. P., Bedoya, A. D., Goldstein, B. A., Jelovsek, J. E., O’Brien, C. L., Walden, N., Elmore, M., Parrish, A. B., et al. (2024) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocad221 · Cited by 57 · see citing works
Self-citationIdeal algorithms in healthcare: Explainable, dynamic, precise, autonomous, fair, and reproducible
Loftus, T. J., Tighe, P. J., Ozrazgat‐Baslanti, T., Davis, J. P., Ruppert, M. M., Ren, Y., Shickel, B., Kamaleswaran, R., Hogan, W. R., Moorman, J. R., et al. (2022) · PLOS Digital Health · doi:10.1371/journal.pdig.0000006 · Cited by 57 · see citing works
Updating HIPAA for the electronic medical record era
Rosenbloom, S. T., Smith, J. R. L., Bowen, R. K., Burns, J., Riplinger, L. E., & Payne, T. H. (2019) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocz090 · Cited by 53 · see citing works
Machine Learning Operations in Health Care: A Scoping Review
Rajagopal, A., Ayanian, S., Ryu, A. J., Qian, R., Legler, S. R., Peeler, E. A., Issa, M., Coons, T. J., & Kawamoto, K. (2024) · Mayo Clinic Proceedings: Digital Health · doi:10.1016/j.mcpdig.2024.06.009 · Cited by 52 · see citing works
Experiences of Leaders in Diversity, Equity, and Inclusion in US Academic Health Centers
Esparza, C. J., Simon, M., London, M. R., Bath, E. P. J., & Ko, M. (2024) · JAMA Network Open · doi:10.1001/jamanetworkopen.2024.15401 · Cited by 36 · see citing works
Not All Clinical AI Monitoring Systems Are Created Equal: Review and Recommendations
Feng, J., Xia, F., Singh, K., & Pirracchio, R. (2025) · NEJM AI · doi:10.1056/aira2400657 · Cited by 28 · see citing works
Evaluation of AI Solutions in Health Care Organizations — The OPTICA Tool
Dagan, N., Devons-Sberro, S., Paz, Z., Zoller, L., Sommer, A., Shaham, G., Shahar, N., Ohana, R., Weinstein, O., Netzer, D., et al. (2024) · NEJM AI · doi:10.1056/aics2300269 · Cited by 27 · see citing works
Croxford, E., Gao, Y., First, E., Pellegrino, N., Schnier, M., Caskey, J., Oguss, M., Wills, G., Chen, G., Dligach, D., et al. (2025) · doi:10.1101/2025.04.22.25326219 · Cited by 25 · see citing works
Assessing the net benefit of machine learning models in the presence of resource constraints
Singh, K., Shah, N. H., & Vickers, A. J. (2023) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocad006 · Cited by 21 · see citing works
Self-citationConsiderations for Quality Control Monitoring of Machine Learning Models in Clinical Practice
Faust, L. J., Wilson, P. M., Asai, S. W., Fu, S., Liu, H., Ruan, X., & Storlie, C. B. (2024) · JMIR Medical Informatics · doi:10.2196/50437 · Cited by 18 · see citing works
Fleisher, L. A., & Economou-Zavlanos, N. (2024) · JAMA Health Forum · doi:10.1001/jamahealthforum.2024.1369 · Cited by 18 · see citing works
MedHELM: Holistic Evaluation of Large Language Models for Medical TasksPreprint
Bedi, S., Cui, H., Fuentes, M., et al. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2505.23802 · Cited by 11 · see citing works
Self-citationWong, L., Sexton, K. W., & Sanford, J. A. (2022) · ACI Open · doi:10.1055/s-0042-1757156 · Cited by 2 · see citing works
Adoption and Use of LLMs at an Academic Medical CenterPreprint
Shah, N. H., Ambers, N., Pandya, A., Keyes, T., Banda, J. M., Nallan, S., Lugtu, C., Trotsyuk, A. A., Bedi, S., Unell, A., et al. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2602.00074 · Cited by 2 · see citing works
Self-citationChen, W., Haredasht, F. N., Black, K. C., Grolleau, F., Alsentzer, E., Chen, J. H., & Ma, S. P. (2025) · Pacific Symposium on Biocomputing (PSB) · doi:10.1142/9789819824755_0014 · Cited by 1 · see citing works
Self-citationThe Optimization Paradox in Clinical AI Multi-Agent SystemsPreprint
Bedi, S., Mlauzi, I., Shin, D., Koyejo, S., & Shah, N. H. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2506.06574 · Cited by 1 · see citing works
Self-citationStatistically Valid Post-Deployment Monitoring Should Be Standard for AI-Based Digital Health
Dolin, P., Li, W., Dasarathy, G., & Berisha, V. (2025) · arXiv · doi:10.48550/arxiv.2506.05701
Grolleau, F., Alsentzer, E., Keyes, T., Chung, P., Swaminathan, A., Aali, A., Hom, J., Huynh, T., Lew, T., Liang, A. S., et al. (2025) · arXiv · doi:10.48550/arxiv.2509.05878
Self-citation
Flagged first, then by citations
Research works printed but not linked: 3
- Winden TJ, Chen ES, Monsen KA, Wang Y, Melton GB. Evaluation of flowsheet documentation in the electronic health record for residence, living situation, and living conditions. AMIA Jt Summits Transl Sci Proc 2018;2018:236.
- Lazaridou A, Kuncoro A, Gribovskaya E, et al. Mind the gap: assessing temporal generalization in neural language models. Adv Neural Inf Process Syst 2021;34:29348-29363.
- Microsoft Corporation, Databricks. Azure Databricks. 2025.
Cited by
Still being checked.
- Authors
- Keyes, T., Callahan, A., Pandya, A. S., Ambers, N., Banda, J. M., Fuentes, M., Lugtu, C., Masariya, P., Nallan, S., O’Brien, C., Wang, T., Alsentzer, E., Chen, J. H., Dash, D., Eisenberg, M. A., Garcia, P., Kotecha, N., Revri, A., Pfeffer, M. A., Shah, N. H., & Jain, S. S.
- Affiliations
- Stanford Health Care (US)
- Stanford University (US)
- Data Scientist, Stanford Health Care, Palo Alto, CA, USA
- Stanford Medicine (US)
- Associate Chief Medical Information Officer, Stanford Health Care, Palo Alto, CA, USA
- ORCID
- Venue
- NEJM Catalyst
- Publisher
- Massachusetts Medical Society
- Published
- 2026-05-20
- Volume
- 7
- Issue
- 6
- Pages
- CAT250372-CAT250372
- ISSN
- 2642-0007 (ISSN-L)
- OpenAlex
- W7161850328
- Cited by
- 10
- References
- 28
- Citations by year
- 2026: 10
- FWCI
- 19.45
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- NEJM Catalyst (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Keywords
- post-deployment monitoring, artificial intelligence in healthcare, AI governance, performance monitoring, impact monitoring, generative AI, AI safety, AI equity, monitoring indicators, health system resources
- MeSH terms
- Intelligent Systems, Artificial Intelligence, Delivery of Health Care, Humans
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
