Distribution shift detection for the postmarket surveillance of medical AI algorithms: a retrospective simulation study
Open accessThe DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index.
Abstract
Distribution shifts remain a problem for the safe application of regulated medical AI systems, and may impact their real-world performance if undetected. Postmarket shifts can occur for example if algorithms developed on data from various acquisition settings and a heterogeneous population are predominantly applied in hospitals with lower quality data acquisition or other centre-specific acquisition factors, or where some ethnicities are over-represented. Therefore, distribution shift detection could be important for monitoring AI-based medical products during postmarket surveillance. We implemented and evaluated three deep-learning based shift detection techniques (classifier-based, deep kernel, and multiple univariate kolmogorov-smirnov tests) on simulated shifts in a dataset of 130’486 retinal images. We trained a deep learning classifier for diabetic retinopathy grading. We then simulated population shifts by changing the prevalence of patients’ sex, ethnicity, and co-morbidities, and example acquisition shifts by changes in image quality. We observed classification subgroup performance disparities w.r.t. image quality, patient sex, ethnicity and co-morbidity presence. The sensitivity at detecting referable diabetic retinopathy ranged from 0.50 to 0.79 for different ethnicities. This motivates the need for detecting shifts after deployment. Classifier-based tests performed best overall, with perfect detection rates for quality and co-morbidity subgroup shifts at a sample size of 1000. It was the only method to detect shifts in patient sex, but required large sample sizes ( $$> 30^{\prime} 000$$ > 3 0 ′ 000 ). All methods identified easier-to-detect out-of-distribution shifts with small (≤300) sample sizes. We conclude that effective tools exist for detecting clinically relevant distribution shifts. In particular classifier-based tests can be easily implemented components in the post-market surveillance strategy of medical device manufacturers.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures






Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“agents trained with data from specific institutions or populations may exhibit unexpected failures when deployed in different clinical contexts [35].”
From Medical AI Agents for Clinical Decision Support: Viewpoint Using the Planning, Action, Reflection, and Memory (PARM) Analytical Lens (2026) · cites this paper as [35]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 1×
“We used a ResNet-50 neural network architecture 31 and a cross-entropy loss.”
MethodInternational Conference on Learning Representations (ICLR 2013)
松尾, 豊. (2013) · 人工知能学会誌 = Journal of Japanese Society for Artificial Intelligence · Cited by 6,286
Class-Balanced Loss Based on Effective Number of Samples
Cui, Y., Jia, M., Lin, T.-Y., Song, Y., & Belongie, S. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00949 · Cited by 3,094
Liu, X., Faes, L., Kale, A. U., Wagner, S. K., Fu, D. J., Bruynseels, A., Mahendiran, T., Moraes, G., Shamdas, M., Kern, C., et al. (2019) · The Lancet Digital Health · doi:10.1016/s2589-7500(19)30123-2 · Cited by 1,883
Improving predictive inference under covariate shift by weighting the log-likelihood function
Shimodaira, H. (2000) · Journal of Statistical Planning and Inference · doi:10.1016/s0378-3758(00)00115-4 · Cited by 1,678
Benjamens, S., Dhunnoo, P., & Meskó, B. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-00324-0 · Cited by 1,215 · cited 1×
“Undertheseregulations,arti fi cial intelligence (AI) algorithms for the automated analysis of retinal fundus images were among the fi rst to be approved and to be commercially available 3 .”
MethodThe Clinician and Dataset Shift in Artificial Intelligence
Finlayson, S. G., Subbaswamy, A., Singh, K., Bowers, J., Kupke, A., Zittrain, J., Kohane, I. S., & Saria, S. (2021) · New England Journal of Medicine · doi:10.1056/nejmc2104626 · Cited by 855
WILDS: A Benchmark of in-the-Wild Distribution Shifts
Koh, P. W., Sagawa, S., Marklund, H., Xie, S. M., Zhang, M., Balsubramani, A., Hu, W., Yasunaga, M., Phillips, R. L., Gao, I., et al. (2020) · arXiv · doi:10.48550/arxiv.2012.07421 · Cited by 518
Generalized ODIN: Detecting Out-of-Distribution Image Without Learning From Out-of-Distribution Data
Hsu, Y.-C., Shen, Y., Jin, H., & Kira, Z. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01096 · Cited by 499
OpenOOD: Benchmarking Generalized Out-Of-Distribution Detection
Yang, J., Wang, P., Zou, D., Zhou, Z., Ding, K., Peng, W., Wang, H., Chen, G., Li, B., Sun, Y., et al. (2022) · neural information processing systems · doi:10.52202/068431-2362 · Cited by 483
Research works printed but not linked: 10
- European Parliament, Council of the European Union. Regulation (EU) 2017/745 of the European Parliament and of the Council of 5 april 2017 on medical devices, amending Directive 2001/83/EC, Regulation (EC) No 178/2002 and Regulation (EC) No 1223/2009 and repealing Council Directives 90/385/EEC and 93/42/EEC. Official Journal of the European Union (2017).
- Koch, L. M., Schürch, C. M., Gretton, A. & Berens, P. Hidden in plain sight: Subgroup shifts escape OOD detection. In Proc. Medical Imaging with Deep Learning (MIDL) (2022).
- Jain, S., Lawrence, H., Moitra, A. & Madry, A. Distilling model failures as directions in latent space. In Proc. International Conference on Learning Representations (ICLR) (2023).
- Liang, S., Li, Y. & Srikant, R. Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks. In Proc. International Conference on Learning Representations (ICLR) (2018).
- Hendrycks, D., Mazeika, M. & Dietterich, T. Deep anomaly detection with outlier exposure. In Proc. International Conference on Learning Representations (ICLR) (2019).
- Sagawa, S., Koh, P. W., Hashimoto, T. B. & Liang, P. Distributionally robust neural networks. In International Conference on Learning Representations (2020).
- Kübler, J. M., Stimper, V., Buchholz, S., Muandet, K. & Schölkopf, B. Automl two-sample test. In Proc. Advances in Neural Information Processing Systems (NeurIPS) (2022).
- Rabanser, S., Günnemann, S. & Lipton, Z. C. Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shift. In Proc. Advances in Neural Information Processing Systems (NeurIPS) (2019).
- Lopez-Paz, D. & Oquab, M. Revisiting classifier two-sample tests. In International Conference on Learning Representations (ICLR) (2017).
- Gretton, A., Borgwardt, K., Rasch, M. J., Schoelkopf, B. & Smola, A. J. Mach. Learn. Res. 13, 723–773 (2012).
Cited by
Evaluating transparency in AI/ML model characteristics for FDA-reviewed medical devices
Mehta, V., Komanduri, A., Bhadouriya, R. S., Mehta, V., Johnson, M. D., Shrestha, P., Nikolov, M. C., Jain, B., Shah, N. H., & Schulman, K. A. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-02052-9 · Cited by 61
Guan, H., Bates, D. W., & Zhou, L. N. (2025) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2025.3642706 · Cited by 60
“The study [64] applies this method to monitor an AI system developed for diabetic retinopathy.”
Jeon, K., Park, W. Y., Kahn, C. E., Nagy, P. G., You, S. C., & Yoon, S. H. (2024) · Investigative Radiology · doi:10.1097/rli.0000000000001106 · Cited by 45
Applications of Multimodal Artificial Intelligence in Non-Hodgkin Lymphoma B Cells
Isavand, P., Aghamiri, S. S., & Amin, R. (2024) · Biomedicines · doi:10.3390/biomedicines12081753 · Cited by 28
“Such datasets can be susceptible to bias introduced by institutional factors, including manufacturer preferences for imaging equipment, established protocols for data acquisition, and variations in medical staff training [118,119].”
BackgroundAutomatic Dataset Shift Identification to Support Safe Deployment of Medical Imaging AI
Roschewitz, M., Mehta, R., Jones, C., & Glocker, B. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-04981-0_7 · Cited by 27
“Significant progress has been made in this field where state-of-the-art methods can detect various types of real-world shifts [12,23,24,30].”
Metadata
- Authors
- Koch, L. M., Baumgartner, C. F., & Berens, P.
- Affiliations
- University of Tübingen (DE)
- University of Lucerne (CH)
- Tübingen AI Center (DE)
- Corresponding author
- Koch, Lisa M.
- Venue
- npj Digital Medicine
- Published
- 2024-05-09
- Volume
- 7
- Issue
- 1
- Pages
- 120-120
- ISSN
- 2398-6352 (ISSN-L)
- PMC
- PMC11082139
- OpenAlex
- W4396780322
- Cited by
- 53
- References
- 20
- Citations by year
- 2026: 37 · 2025: 12 · 2024: 4
- FWCI
- 4.95
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- npj Digital Medicine (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- PUBLISSO (German National Library of Medicine) (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Funders
- Eberhard Karls Universität Tübingen (390727645)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Retinal Imaging and Analysis
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Keywords
- distribution shift detection, postmarket surveillance, artificial intelligence in healthcare, Kolmogorov-Smirnov test, diabetic retinopathy grading, fundus images, out-of-distribution detection, image quality, patient sex, ethnicity, comorbidities, medical device regulation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Distribution shift detection for the postmarket surveillance of medical AI algorithms: a retrospective simulation study. https://app.cochat.ai/papers/10.1038/s41746-024-01085-w
- The paper
- Koch, L. M., Baumgartner, C. F., & Berens, P. (2024). Distribution shift detection for the postmarket surveillance of medical AI algorithms: a retrospective simulation study. npj Digital Medicine. https://doi.org/10.1038/s41746-024-01085-w
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.