Empirical data drift detection experiments on real-world medical imaging data
Open accessThe DOI matches its Crossref record; 63 of the 63 references it cites were found in a registry or an index; 13 of the 28 sources its claims cite were checked for support.
Abstract
While it is common to monitor deployed clinical artificial intelligence (AI) models for performance degradation, it is less common for the input data to be monitored for data drift – systemic changes to input distributions. However, when real-time evaluation may not be practical (eg., labeling costs) or when gold-labels are automatically generated, we argue that tracking data drift becomes a vital addition for AI deployments. In this work, we perform empirical experiments on real-world medical imaging to evaluate three data drift detection methods’ ability to detect data drift caused (a) naturally (emergence of COVID-19 in X-rays) and (b) synthetically. We find that monitoring performance alone is not a good proxy for detecting data drift and that drift-detection heavily depends on sample size and patient features. Our work discusses the need and utility of data drift detection in various scenarios and highlights gaps in knowledge for the practical application of existing methods.
Overview
CoChat Summary
TL;DR
This study empirically evaluates data drift detection on real-world chest X-ray data, including natural drift from COVID-19 and synthetic demographic/pathology shifts. It finds that monitoring model performance alone is a poor proxy for data drift, while data-based drift detection is sensitive to sample size and the specific feature being enriched.
Key findings
- Tracking aggregate model performance failed to capture clinically obvious COVID-19 drift: AUROC remained relatively stable across the first wave, whereas TAE + BBSD detected drift (with a delay). Aggregate scores changed less than 3% in the synthetic hospital-site shift shown in Fig. 4.
- Drift-detection sensitivity depended strongly on the enriched feature: a 5% increase in patients aged 18–35 was detected by TAE + BBSD, but a 30% increase in inpatients or patients aged 65+ was needed for significance. The authors note patients 18–35 were 6% of the population, patients 65+ were 62%, inpatients were 57%, and outpatients were 19%.
- Sample size strongly affected sensitivity: with 500 images the p-value never dropped below 0.1, while with 4000 images the p-value dropped below 0.05 at a 40% increase in male patients.
- TAE + BBSD was generally more sensitive than TAE alone, and BBSD was nearly as sensitive as TAE + BBSD.
Methods
- Retrospective study using 239,235 frontal chest radiographs from 78,542 patients over age 18 at Trillium Health Partners (Mississauga, Ontario, Canada), January 2016–December 2020; labels extracted with CheXpert NLP for 14 pathologies.
- Compared four approaches: model-performance tracking, image data-based drift detection (TorchXRayVision AutoEncoder, TAE), model output-based drift detection (Black Box Shift Detection, BBSD), and combined TAE + BBSD. Drift was tested with a two-sided multivariate maximum mean discrepancy (MMD) statistical test.
- Natural drift used COVID-19 emergence with rolling 30-day buckets and 500 images per source/target sample; synthetic categorical drifts enriched target datasets from 5% to 50% across sex, institution, ICU status, age, and patient class, repeated 10 times, generally using 4000 images per source and target unless testing sample size.
Limitations
- The paper states findings are limited to computer vision algorithms on chest radiographs and should be evaluated on other data types; synthetic drift was limited to common patient attributes, and ethnicity/race were not explored because those data are not collected at Trillium Health Partners.
- The paper states only a small selection of metrics was tested, with a focus on discriminative performance; calibration methods and other drift-detection methods, dimensionality reductions, and two-sample tests were not studied. It also notes no trustworthy automated way to explain what causes drift, and reports a hint of inequity because detection sensitivity differed across patient demographics.
- Evident from the text: the study is retrospective and from a single three-site hospital system, so the natural COVID-19 drift and synthetic shifts may not generalize to other institutions, populations, or deployment contexts.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures



![Fig. 4 Comparison of aggregate vs non-aggregate metrics for synthetic categorical shift for a target category of [Hospital Site 1] (4A) and [Hospital Site 2] (4B). The plots on the left show the absolute change in score for aggregate metrics (i.e., AUROC, Precision, Recall, F1 score, and Brier Score averaged all classes). The plots on the right show the breakdown across pathologies for the F1 score. Aggregate scores change less than 3%. p-values were calculated using a two-sided multivariate maximum mean discrepancy (MMD) statistical test with no correction for multiple comparison. Data are presented as mean values +/− 1 standard deviation. Eng. Card. stands for enlarged cardiomediastinum. AUROC area under the curve – receiver operating characteristic.](https://figures.cochat.ai/paper-figures/4af94a4e6f41474683a32ff1c0764d76/empirical-data-drift-detection-experiments-on-real-world-medical-imaging-data-figure-4.jpg)


Verification
Claims the paper makes
10 Valid · 1 Not valid · 6 Not checked · 47 Own results and interpretations
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“For example, it is widely recognized that algorithms tend to underperform when applied to populations that differ from those they are trained on [4][5][6] .”
What the sources say
- Not checked — no full text of the sourcecites [4] Dexter, G. P., Grannis, S. J., Dixon, B. E. & Kasthurirathne, S. N. Generalization of machine learning approaches to identify notifiable conditions from a statewide health information exchange. AMIA Summits Transl. Sci. Proc. 2020, 152 (2020).
- Not checked — no full text of the sourcecites [5] Matheny, M. E., Whicher, D. & Israni, S. T. Artificial intelligence in health care: a report from the National Academy of Medicine. Jama 323, 509–510 (2020). source
- Supportedcites [6] Ahluwalia, M. et al. The subgroup imperative: chest radiograph classifier generalization gaps in patient, setting, and pathology subgroups. Radiol.: Artif. Intell. (2023). source
The Abstract states that decreased classifier performance in new clinical settings and on previously unseen data is a well-established generalization gap. The Discussion reports that external performance varies across patient subgroups, supporting the citing sentence’s broad, qualified claim. (LLM, conf=0.91)
boostedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“because of public health policy guidelines during the pandemic in [Institution Location] 47 , patient encounters for non-COVIDrelated diseases decreased 48 .”
§ Empirical data drift detection experiments on real-world medical imaging data > Natural data drift (COVID-19).
What the sources say
- Partly supportedcites [47] McCoy, L. G. et al. Characterizing early Canadian federal, provincial, territorial and municipal nonpharmaceutical interventions in response to COVID-19: a descriptive analysis. Can. Med. Assoc. Open Access J. 8, E545–E553 (2020). sourceExperimental — accuracy not yet measured
Our copy supports only the policy-guideline half of the sentence: the Table 2 / Research section catalogs Canadian federal, provincial, territorial and municipal NPIs, including 'Health care facility restrictions' (75 entries, e.g., Whitehorse General Hospital suspending nonurgent surgery from Mar. 23, 2020), and the Introduction describes traditional public health interventions across Canadian jurisdictions. It does not report any change in patient encounters for non-COVID-related diseases, and the Limitations section states explicitly that the authors 'do not aim to make any causal or associative conclusions around the impact of these NPIs,' so neither the decrease nor the causal 'because of' framing is attributable to this source. The location is also unspecified in the citing sentence ('[Institution Location]'), whereas our copy covers only Canadian jurisdictions up to Apr. 19, 2020. (LLM, conf=0.70)
- Not matchedcites [48] Canadian Institute for Health Information. COVID-19’s impact on emergency departments. https://www.cihi.ca/en/covid-19-resources/impact-of-covid-19-on-canadas-health-care-systems/emergency-departments; (2021).
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“To establish ground truth labels for each imaging study, we extracted 14 pathology labels from the associated report for each image using the CheXpert NLP algorithm 31 , an approach we validated in previous work on our dataset 6 .”
§ Empirical data drift detection experiments on real-world medical imaging data > Dataset
What the sources say
- Not checked — no full text of the sourcecites [31] Irvin, J. et al. Chexpert: a large chest radiograph dataset with uncertainty labels and expert comparison. In Proceedings of the AAAI Conference on Artificial Intelligence 590–597 (2019). source
- Supportedcites [6] Ahluwalia, M. et al. The subgroup imperative: chest radiograph classifier generalization gaps in patient, setting, and pathology subgroups. Radiol.: Artif. Intell. (2023). source
The paper’s NLP validation analysis compares CheXpert labels with manual annotations of 502 radiology reports from its institutional dataset. The NLP Algorithm Performance section reports 94% accuracy and says CheXpert was selected to generate ground-truth labels for the full image dataset. This supports the validation attributed to citation 6; the separate citation 31 accompanies the CheXpert algorithm itself. (LLM, conf=0.92)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Every source the paper cites for this claim was checked and supports it.
In the paper
“The pre-trained version of the TorchXRayVision used in this study was pre-trained on the MIMIC-CXR dataset 55 .”
§ Empirical data drift detection experiments on real-world medical imaging data > Models and training
What the sources say
- Supportedcites [55] Johnson, A. E. et al. MIMIC-CXR, a de-identified publicly available database of chest radiographs with free-text reports. Sci Data 6, 317 (2019). source
The Background & Summary identifies MIMIC-CXR as a publicly available chest-radiograph dataset intended to support image-understanding research. The citation identifies the dataset; the statement that the citing authors’ classifier was pre-trained on it is their account of their own study, not a finding attributed to the dataset paper. (LLM, conf=0.88)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In Supplementary Figs.345678910, we plot other metrics (F1 score, precision, recall, and Brier score) to demonstrate the consistency of our finding.For these metrics we observe similar results: all are less sensitive than data-based drift detection approaches.”
- Also stated in: § Aggregate performance metrics are very poor for detecting data drift · boosted
“Aggregate performance metrics are very poor for detecting data drift”
§ Empirical data drift detection experiments on real-world medical imaging data > Overview of task, data, and approach
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
- Also stated in: § Aggregate performance metrics are very poor for detecting data drift · boosted
21 more claims
Create a free account to see all 64 claims and to verify the 4 references still open.
- Probably not a substantial claim. 25
- It may state two separate claims as one. 19
- The quote only partly states this claim. 1
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“If performance for the Male class is 90% accurate (83 correct alerts, 7 incorrect alerts)”
§ Empirical data drift detection experiments on real-world medical imaging data > Scenario 2a: Performance is observed to change.
Reported 90 · recomputed 92.222222
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“the total percentage of false positives is 9.8%.”
§ Empirical data drift detection experiments on real-world medical imaging data > Scenario 2a: Performance is observed to change.
Reported 9.8 · recomputed 9
Citations
Cites
Advances in neural information processing systems 7
Cios, K., & Shields, M. (1997) · Neurocomputing · doi:10.1016/s0925-2312(97)90018-7 · Cited by 22,059
Lippa, N. C., & Sanderson, S. C. (2013) · Human Heredity · doi:10.1159/000353712 · Cited by 6,078
Canadian Institute for Health Information.
Lalonde, A. (1994) · PubMed · Cited by 4,739
Wang, X., Peng, Y., Lu, L., Lu, Z., Bagheri, M., & Summers, R. M. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.369 · Cited by 4,616 · cited 1×
“The pre-trained TorchXRayVision autoencoder was trained on numerous chest radiograph datasets (PadChest 56 , Chestx-ray8 57 , CheXpert 31 , and MIMIC-CXR 55 ).”
(2012) · MPG.PuRe (Max Planck Society) · doi:10.5555/2188385.2188410 · Cited by 3,603
On calibration of modern neural networks
Guo, C., Pleiss, G., Sun, Y., & Weinberger, K. Q. (2017) · arXiv · doi:10.48550/arxiv.1706.04599 · Cited by 3,435 · cited 1×
“This could be because the underlying predictive model, while highly discriminative, is not well calibrated on the underlying dataset — something expected with deep learning classi fi ers 44,45 .”
The Stigma of Obesity: A Review and Update
Puhl, R. M., & Heuer, C. A. (2009) · Obesity · doi:10.1038/oby.2008.636 · Cited by 3,323 · cited 2×
“…and affecting the validity of performance monitoring (e.g., past work has shown increased negativity present in the clinical notes of obese patients 22,23 , a trend which may negatively impact the accuracy of gold labels created by neural language models if there is an increase in the proportion…”
Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating
Steyerberg, E. W. (2019) · Statistics for Biology and Health · doi:10.1007/978-3-030-16399-0 · Cited by 3,281
Calculating the sample size required for developing a clinical prediction model
Riley, R. D., Ensor, J., Snell, K. I. E., Harrell, F. E., Martin, G. P., Reitsma, J. B., Moons, K. G. M., Collins, G., & van Smeden, M. (2020) · BMJ · doi:10.1136/bmj.m441 · Cited by 2,917 · cited 1×
“Alternatively, the level of discrimination between different groups (e.g., the variance of groups) may require a larger number of samples to detect differences 30 .”
MIMIC-CXR, a de-identified publicly available database of chest radiographs with free-text reports
Johnson, A. E. W., Pollard, T. J., Berkowitz, S. J., Greenbaum, N. R., Lungren, M. P., Deng, C.-Y., Mark, R. G., & Horng, S. (2019) · Scientific Data · doi:10.1038/s41597-019-0322-0 · Cited by 1,816
Cited by
Towards Trustworthy AI: A Review of Ethical and Robust Large Language Models
Ferdaus, M. M., Abdelguerfi, M., Ioup, E., Niles, K. N., Pathak, K., & Sloan, S. (2024) · arXiv · doi:10.48550/arxiv.2407.13934 · Cited by 80
Framework for bias evaluation in large language models in healthcare settings
Templin, T., Fort, S., Padmanabham, P. S., Seshadri, P., Rimal, R., Oliva, J. B., Lich, K. H., Sylvia, S., & Sinnott-Armstrong, N. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01786-w · Cited by 79
Evaluating transparency in AI/ML model characteristics for FDA-reviewed medical devices
Mehta, V., Komanduri, A., Bhadouriya, R. S., Mehta, V., Johnson, M. D., Shrestha, P., Nikolov, M. C., Jain, B., Shah, N. H., & Schulman, K. A. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-02052-9 · Cited by 61
Towards Trustworthy AI: A Review of Ethical and Robust Large Language Models
Ferdaus, M. M., Abdelguerfi, M., Ioup, E., Niles, K. N., Pathak, K., & Sloan, S. D. (2025) · ACM Computing Surveys · doi:10.1145/3777382 · Cited by 137
Assudani, P. J., Bhurgy, A. S., Kollem, S., Bhurgy, B. S., Ahmad, M. O., Kulkarni, M. B., & Bhaiyya, M. L. (2025) · Microchemical Journal · doi:10.1016/j.microc.2025.115802 · Cited by 43
Metadata
- Authors
- Kore, A., Abbasi Bavil, E., Subasri, V., Abdalla, M., Fine, B., Dolatabadi, E., & Abdalla, M.
- Affiliations
- Vector Institute (CA)
- University Health Network (CA)
- Harvard University (US)
- University of Toronto (CA)
- York University (CA)
- Corresponding author
- Abdalla, Moustafa
- Abdalla, Mohamed
- Venue
- Nature Communications
- Published
- 2024-02-29
- Volume
- 15
- Issue
- 1
- Pages
- 1887-1887
- ISSN
- 2041-1723 (ISSN-L)
- PMC
- PMC10904813
- OpenAlex
- W4392283037
- Cited by
- 124
- References
- 50
- Citations by year
- 2026: 66 · 2025: 52 · 2024: 6
- FWCI
- 18.22
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- Nature Communications (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Data Stream Mining Techniques
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- data drift detection, diagnostic imaging, clinical AI, X-rays, sample size, field deployment
- MeSH terms
- COVID-19, Artificial Intelligence, Diagnostic Imaging, Humans, Radiography
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-09, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-09 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol and qwen3.8-omni-flash).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Empirical data drift detection experiments on real-world medical imaging data. https://app.cochat.ai/papers/10.1038/s41467-024-46142-w
- The paper
- Kore, A., Abbasi Bavil, E., Subasri, V., Abdalla, M., Fine, B., Dolatabadi, E., & Abdalla, M. (2024). Empirical data drift detection experiments on real-world medical imaging data. Nature Communications. https://doi.org/10.1038/s41467-024-46142-w
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.