The DOI matches its DataCite record; 55 of the 58 references it cites were found in a registry or an index.
Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shift
Open accessAbstract
We might hope that when faced with unexpected inputs, well-designed software systems would fire off warnings. Machine learning (ML) systems, however, which depend strongly on properties of their inputs (e.g. the i.i.d. assumption), tend to fail silently. This paper explores the problem of building ML systems that fail loudly, investigating methods for detecting dataset shift, identifying exemplars that most typify the shift, and quantifying shift malignancy. We focus on several datasets and various perturbations to both covariates and label distributions with varying magnitudes and fractions of data affected. Interestingly, we show that across the dataset shifts that we explore, a two-sample-testing-based approach, using pre-trained classifiers for dimensionality reduction, performs best. Moreover, we demonstrate that domain-discriminating approaches tend to be helpful for characterizing shifts qualitatively and determining if they are harmful.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“As demonstrated by Rabanser et al. 46 , the p-values are calculated by computing the maximum discrepancy between two distributions and comparing via permutation tests to a null-hypothesis distribution.”
From Empirical data drift detection experiments on real-world medical imaging data (2024) · cites this paper as [46]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“When implementing our experiments, we followed state-of-theart methodologies found in previous work 46 .”
From Empirical data drift detection experiments on real-world medical imaging data (2024) · cites this paper as [46]
The cited source says what the claim attributes to it.
In the citing paper
“Model output-based drift detection is a method to detect drift using only the output of the classifier without any gold labels 46 .”
From Empirical data drift detection experiments on real-world medical imaging data (2024) · cites this paper as [46]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Scikit-learn: Machine Learning in Python
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Müller, A., Nothman, J., Louppe, G., et al. (2012) · arXiv · doi:10.48550/arxiv.1201.0490 · Cited by 71,016
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805
Learning Multiple Layers of Features from Tiny Images
Krizhevsky, A. (2024) · doi:10.57702/zp44cu3g · Cited by 23,889
Sequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766
Chandola, V., Banerjee, A., & Kumar, V. (2009) · ACM Computing Surveys · doi:10.1145/1541880.1541882 · Cited by 11,687
Speech recognition with deep recurrent neural networks
Graves, A., Mohamed, A.-R., & Hinton, G. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6638947 · Cited by 8,893
Liu, F. T., Ting, K. M., & Zhou, Z.-H. (2008) · 2008 Eighth IEEE International Conference on Data Mining · doi:10.1109/icdm.2008.17 · Cited by 6,601
Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
Xiao, H., Rasul, K., & Vollgraf, R. (2017) · arXiv · doi:10.48550/arxiv.1708.07747 · Cited by 6,108
LOF: identifying density-based local outliers
Breunig, M. M., Kriegel, H.-P., Ng, R. T., & Sander, J. (2000) · ACM SIGMOD Record · doi:10.1145/335191.335388 · Cited by 5,207
Research works printed but not linked: 3
- [13][13] Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and Harnessing Adversarial Examples. In International Conference on Learning Representations (ICLR) , 2014. -
- [29][29] Zachary C Lipton, Yu-Xiang Wang, and Alex Smola. Detecting and Correcting for Label Shift with Black Box Predictors. In International Conference on Machine Learning (ICML) , 2018. -
- [51][51] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing Properties of Neural Networks. In International Conference on Learning Representations (ICLR) , 2014. -
Cited by
Sorted by citations
Items 201–208 of 208
Quantum-Inspired Adaptive Intelligence Framework for Next-Generation Predictive Systems
Kukkadapu, S. (2025) · International Journal of Science and Research (IJSR) · doi:10.21275/sr25413081739
Responsible Data Repurposing: A Conceptual FoundationPreprint
Ogunseye, S., & Parsons, J. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5255086
Using Drift Planning to Improve Safety of Visual Navigation in Unmanned Aerial Vehicles
Hansen, J. P., Echeverría, S., Pons, L., Zhan, L., Moreno, G. A., & Lewis, G. A. (2025) · doi:10.1109/raie66699.2025.00007
PdistMap: A Kernel Density Overlap Metric for Distributional Similarity and Cluster MatchingPreprint
Guha, R., & Nag, K. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5358971
Mitigating covariate shift in non-colocated data with learned parameter priors
Khan, B., Syed, T., Mirza, B., & Durrani, M. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.131656
Iriarte, F. J., Azoubel, B., Carrizo-Pérez, A., Linares, A. C., Unzueta, L., & Arganda‐Carreras, I. (2026) · Open Research Europe · doi:10.12688/openreseurope.22116.1 · Cited by 1
Probabilistic Runtime Verification, Evaluation, and Risk Assessment of Visual Deep Learning Systems
Torpmann-Hagen, B., Halvorsen, P., Riegler, M. A., & Johansen, D. (2026) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3795523
Multi-Output Drift Detection and Model Adaptation in Semiconductor Manufacturing
Mević, A., Ghoneim, M., & Krivić, S. (2026) · Lecture notes in networks and systems · doi:10.1007/978-3-032-30113-0_12
Metadata
- Authors
- Rabanser, S., Günnemann, S., & Lipton, Z. C.
- Affiliations
- Amazon (United States) (US)
- Technical University of Munich (DE)
- Carnegie Mellon University (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2018
- Volume
- 32
- Pages
- 1396-1408
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1810.11953
- OpenAlex
- W2898158860
- Cited by
- 204
- References
- 58
- Citations by year
- 2026: 4 · 2025: 23 · 2024: 36 · 2023: 44 · 2022: 29 · 2021: 44 · 2020: 22 · 2019: 2
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Data Stream Mining Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Keywords
- distribution shift, shift detection, fault detection, silent failures, covariate shift, label shift, two-sample test, dimensionality reduction, i.i.d. assumption
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
