CoChat

Distribution shift detection for the postmarket surveillance of medical AI algorithms: a retrospective simulation study

Open access
Koch, Lisa M., Baumgartner, Christian F., Berens, Philipp · 2024 · npj Digital Medicine

The DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index.

Abstract

Distribution shifts remain a problem for the safe application of regulated medical AI systems, and may impact their real-world performance if undetected. Postmarket shifts can occur for example if algorithms developed on data from various acquisition settings and a heterogeneous population are predominantly applied in hospitals with lower quality data acquisition or other centre-specific acquisition factors, or where some ethnicities are over-represented. Therefore, distribution shift detection could be important for monitoring AI-based medical products during postmarket surveillance. We implemented and evaluated three deep-learning based shift detection techniques (classifier-based, deep kernel, and multiple univariate kolmogorov-smirnov tests) on simulated shifts in a dataset of 130’486 retinal images. We trained a deep learning classifier for diabetic retinopathy grading. We then simulated population shifts by changing the prevalence of patients’ sex, ethnicity, and co-morbidities, and example acquisition shifts by changes in image quality. We observed classification subgroup performance disparities w.r.t. image quality, patient sex, ethnicity and co-morbidity presence. The sensitivity at detecting referable diabetic retinopathy ranged from 0.50 to 0.79 for different ethnicities. This motivates the need for detecting shifts after deployment. Classifier-based tests performed best overall, with perfect detection rates for quality and co-morbidity subgroup shifts at a sample size of 1000. It was the only method to detect shifts in patient sex, but required large sample sizes ( $$> 30^{\prime} 000$$ > 3 0 ′ 000 ). All methods identified easier-to-detect out-of-distribution shifts with small (≤300) sample sizes. We conclude that effective tools exist for detecting clinically relevant distribution shifts. In particular classifier-based tests can be easily implemented components in the post-market surveillance strategy of medical device manufacturers.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1038/s41746-024-01085-w

Cites

21 of 21 references verified0 citation errors · 0 not checkable

Statistics

Figures

Fig. 1 Overview of the distribution shift detection framework in the context of postmarket surveillance of a medical image-based ML device. a The medical device lifecycle including validation (blue) and postmarket (red) phases, (b) illustration of potential distribution shifts between the source distribution P (blue) and the target Q (red), where all images from Q are within the support of the source distribution and thus cannot be detected as outliers. c Subgroup and other distribution shifts can be detected using neural network-based hypothesis tests consisting of a deep feature extractor and a test statistic. d The feature extractors and their optimisation depend on the shift detection approach. e Finally, a hypothesis test is carried out on held-out test samples. If a shift is detected, its causes can be investigated and corrective action can be taken in the context of post-market activities of a medical device manufacturer.
Fig. 1 Overview of the distribution shift detection framework in the context of postmarket surveillance of a medical image-based ML device. a The medical device lifecycle including validation (blue) and postmarket (red) phases, (b) illustration of potential distribution shifts between the source distribution P (blue) and the target Q (red), where all images from Q are within the support of the source distribution and thus cannot be detected as outliers. c Subgroup and other distribution shifts can be detected using neural network-based hypothesis tests consisting of a deep feature extractor and a test statistic. d The feature extractors and their optimisation depend on the shift detection approach. e Finally, a hypothesis test is carried out on held-out test samples. If a shift is detected, its causes can be investigated and corrective action can be taken in the context of post-market activities of a medical device manufacturer.CC BY · PubMed Central
Fig. 2 Shift detection rate for subgroup shifts. Panels show subgroup shifts with respect to (a) image quality, (b) the presence of co-morbidities, (c) patient sex. d Shows the shift detection rate for an out-of-distribution shift to images of insufficient quality. The top row (a–d) shows the shift detection rate for a fixed test sample size (m = 500) when varying the size of the training set. In the bottom row (e–h), the training dataset size was fixed (see dashed line in the respective plot above), and the test sample size varied between m = 10 and m = 500.
Fig. 2 Shift detection rate for subgroup shifts. Panels show subgroup shifts with respect to (a) image quality, (b) the presence of co-morbidities, (c) patient sex. d Shows the shift detection rate for an out-of-distribution shift to images of insufficient quality. The top row (a–d) shows the shift detection rate for a fixed test sample size (m = 500) when varying the size of the training set. In the bottom row (e–h), the training dataset size was fixed (see dashed line in the respective plot above), and the test sample size varied between m = 10 and m = 500.CC BY · PubMed Central
Fig. 3 Type I error for all shift detection methods, averaged over all shift types. The error bar depicts the standard deviation.
Fig. 3 Type I error for all shift detection methods, averaged over all shift types. The error bar depicts the standard deviation.CC BY · PubMed Central
Fig. 4 Shift detection rate for subgroup shifts towards different ethnicites. The top row (a–d) shows the shift detection rate for a fixed test sample size (m = 500) when varying the size of the training set. In the bottom row (e–h), the training dataset size was fixed (see dashed line in the respective plot above), and the test sample size varied between m = 10 and m = 500.
Fig. 4 Shift detection rate for subgroup shifts towards different ethnicites. The top row (a–d) shows the shift detection rate for a fixed test sample size (m = 500) when varying the size of the training set. In the bottom row (e–h), the training dataset size was fixed (see dashed line in the respective plot above), and the test sample size varied between m = 10 and m = 500.CC BY · PubMed Central
Fig. 5 Influence of varying quality subgroup shift strength on test power of the different shift detection techniques. Panels show (a) C2ST, (b) MMDD, and (c) MUKS tests.
Fig. 5 Influence of varying quality subgroup shift strength on test power of the different shift detection techniques. Panels show (a) C2ST, (b) MMDD, and (c) MUKS tests.CC BY · PubMed Central
Fig. 6 Example retinal fundus images from the Eyepacs dataset. Panel (a) shows a random selection. Magnified on the right are example (b) images of excellent, (c) adequate and (d) poor quality, (e) images with co-morbidities, and example images of (f) female and (g) male patients. For better visibility, the example images were contrast enhanced using contrast limited adaptive histogram equalisation (CLAHE).
Fig. 6 Example retinal fundus images from the Eyepacs dataset. Panel (a) shows a random selection. Magnified on the right are example (b) images of excellent, (c) adequate and (d) poor quality, (e) images with co-morbidities, and example images of (f) female and (g) male patients. For better visibility, the example images were contrast enhanced using contrast limited adaptive histogram equalisation (CLAHE).CC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 21 references

Research works printed but not linked: 10

  1. European Parliament, Council of the European Union. Regulation (EU) 2017/745 of the European Parliament and of the Council of 5 april 2017 on medical devices, amending Directive 2001/83/EC, Regulation (EC) No 178/2002 and Regulation (EC) No 1223/2009 and repealing Council Directives 90/385/EEC and 93/42/EEC. Official Journal of the European Union (2017).
  2. Koch, L. M., Schürch, C. M., Gretton, A. & Berens, P. Hidden in plain sight: Subgroup shifts escape OOD detection. In Proc. Medical Imaging with Deep Learning (MIDL) (2022).
  3. Jain, S., Lawrence, H., Moitra, A. & Madry, A. Distilling model failures as directions in latent space. In Proc. International Conference on Learning Representations (ICLR) (2023).
  4. Liang, S., Li, Y. & Srikant, R. Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks. In Proc. International Conference on Learning Representations (ICLR) (2018).
  5. Hendrycks, D., Mazeika, M. & Dietterich, T. Deep anomaly detection with outlier exposure. In Proc. International Conference on Learning Representations (ICLR) (2019).
  6. Sagawa, S., Koh, P. W., Hashimoto, T. B. & Liang, P. Distributionally robust neural networks. In International Conference on Learning Representations (2020).
  7. Kübler, J. M., Stimper, V., Buchholz, S., Muandet, K. & Schölkopf, B. Automl two-sample test. In Proc. Advances in Neural Information Processing Systems (NeurIPS) (2022).
  8. Rabanser, S., Günnemann, S. & Lipton, Z. C. Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shift. In Proc. Advances in Neural Information Processing Systems (NeurIPS) (2019).
  9. Lopez-Paz, D. & Oquab, M. Revisiting classifier two-sample tests. In International Conference on Learning Representations (ICLR) (2017).
  10. Gretton, A., Borgwardt, K., Rasch, M. J., Schoelkopf, B. & Smola, A. J. Mach. Learn. Res. 13, 723–773 (2012).

Cited by

See all 74

Metadata

Authors
Koch, L. M., Baumgartner, C. F., & Berens, P.
Affiliations
  • University of Tübingen (DE)
  • University of Lucerne (CH)
  • Tübingen AI Center (DE)
Corresponding author
Koch, Lisa M.
Venue
npj Digital Medicine
Published
2024-05-09
Volume
7
Issue
1
Pages
120-120
ISSN
2398-6352 (ISSN-L)
OpenAlex
W4396780322
Cited by
53
References
20
Citations by year
2026: 37 · 2025: 12 · 2024: 4
FWCI
4.95
Type
journal article
Language
en
Open access
Licence
Locations
  • npj Digital Medicine (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • PUBLISSO (German National Library of Medicine) (submittedVersion, cc-by)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
Funders
Eberhard Karls Universität Tübingen (390727645)
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Retinal Imaging and Analysis
  • Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
Keywords
distribution shift detection, postmarket surveillance, artificial intelligence in healthcare, Kolmogorov-Smirnov test, diabetic retinopathy grading, fundus images, out-of-distribution detection, image quality, patient sex, ethnicity, comorbidities, medical device regulation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Distribution shift detection for the postmarket surveillance of medical AI algorithms: a retrospective simulation study. https://app.cochat.ai/papers/10.1038/s41746-024-01085-w
The paper
Koch, L. M., Baumgartner, C. F., & Berens, P. (2024). Distribution shift detection for the postmarket surveillance of medical AI algorithms: a retrospective simulation study. npj Digital Medicine. https://doi.org/10.1038/s41746-024-01085-w

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.