CoChat

Not All Clinical AI Monitoring Systems Are Created Equal: Review and Recommendations

The DOI matches its Crossref record; 63 of the 63 references it cites were found in a registry or an index.

Abstract

AbstractWhile practices for the initial evaluation of clinical artificial intelligence (AI) algorithms are well established, there is little consensus on how to design effective monitoring systems for the post-deployment setting. In real-world case studies, design choices are often driven by practical constraints rather than their impact on the performance of the monitoring system. This narrative review critically examines the key decisions that shape the performance of AI monitoring systems, including the selection of monitoring criteria, the choice of data sources, and the statistical procedures employed. Our findings reveal significant variation in the designed systems, often with little transparency regarding how their design choices affect monitoring performance. To provide a more structured approach to designing monitoring systems that are both effective and practical, we introduce a road map for navigating the many options available. We bootstrap efforts in clinical AI monitoring by highlighting tools from three related fields that face similarly complex challenges: quality management, clinical trials, and real-world evidence generation.

Overview

Integrity signals

Registry

Crossref · 10.1056/aira2400657

Cites

63 of 63 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

Show all 63 references

Research works printed but not linked: 10

  1. Van Belle G, Fisher LD, Heagerty PJ, Lumley T. Biostatistics: a methodology for the health sciences. Hoboken, NJ: John Wiley & Sons, Inc., 2004.
  2. Perdomo J Zrnic T Mendler-Dünner C Hardt M. Performative prediction. In: Daumè H III Singh A eds. Proceedings of the 37th International Conference on Machine Learning Virtual Meeting. PMLR 2020;119:7599-7609. (https://proceedings.mlr.press/v119/perdomo20a/perdomo20a.pdf)
  3. Liley J, Emerson S, Mateen B, Vallejos C, Aslett L, Vollmer S. Model updating after interventions paradoxically introduces bias. In: Banerjee A, Fukumizu K, eds. Proceedings of the 24th International Conference on Artificial Intelligence and Statistics, PMLR 2021;130:3916-3924 (https://proceedings.mlr.press/v130/liley21a.html).
  4. Pearl J. Causality. Cambridge: Cambridge University Press, 2009.
  5. Pyzdek T, Keller P. The six sigma handbook, sixth edition: a complete guide for green belts, black belts, and managers at all levels. New York, NY: McGraw-Hill Education, 2023.
  6. Montgomery DC. Statistical quality control. 7th ed. Nashville, TN: John Wiley & Sons, 2013.
  7. Feng J, Subbaswamy A, Gossmann A, et al. Designing monitoring strategies for deployed machine learning algorithms: navigating performativity through a causal lens. In: Locatello F, Didelez V, eds. Proceedings of the 3rd Conference on Causal Learning and Reasoning, PMLR 2024;236:587-608 (https://proceedings.mlr.press/v236/feng24a.html).
  8. Feng J, Gossmann A, Pennello GA, Petrick N, Sahiner B, Pirracchio R. Monitoring machine learning-based risk prediction algorithms in the presence of performativity. Int Conf Artif Intell Stat 2024;238:919-927 (https://proceedings.mlr.press/v238/feng24b.html).
  9. Boeken PA Zoeter O Mooij JM. Evaluating and correcting performative effects of decision support systems via causal domain shift. In: Locatello F Didelez V eds. Proceedings of the Third Conference on Causal Learning and Reasoning PMLR 2024:236:551-569 (https://proceedings.mlr.press/v236/boeken24a.html)
  10. Rotnitzky A. Inverse probability weighted methods. In: Fitzmaurice G, Davidian M, Verbeke G, Molendberghs G, eds. Longitudinal data analysis. Boca Raton, FL: CRC Press, 2009;453-476.

Cited by

See all 43

Metadata

Authors
Feng, J., Xia, F., Singh, K., & Pirracchio, R.
Affiliations
  • University of California, San Francisco (US)
  • University of California San Diego (US)
Venue
NEJM AI
Published
2025-01-23
Volume
2
Issue
2
Pages
AIra2400657-AIra2400657
ISSN
2836-9386 (ISSN-L)
OpenAlex
W4406740986
Cited by
28
References
61
Citations by year
2026: 16 · 2025: 12
FWCI
6.78
Type
journal article
Language
en
Open access
closed
Locations
  • NEJM AI (publishedVersion)
  • PubMed (publishedVersion)
Topics
  • Health Sciences › Medicine › Surgery › Healthcare Technology and Patient Monitoring
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
Keywords
artificial intelligence in healthcare, clinical decision support, AI evaluation, clinical implementation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Not All Clinical AI Monitoring Systems Are Created Equal: Review and Recommendations. https://app.cochat.ai/papers/10.1056/aira2400657
The paper
Feng, J., Xia, F., Singh, K., & Pirracchio, R. (2025). Not All Clinical AI Monitoring Systems Are Created Equal: Review and Recommendations. NEJM AI. https://doi.org/10.1056/aira2400657

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.