The DOI matches its DataCite record; 55 of the 58 references it cites were found in a registry or an index.
Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shift
Open accessAbstract
We might hope that when faced with unexpected inputs, well-designed software systems would fire off warnings. Machine learning (ML) systems, however, which depend strongly on properties of their inputs (e.g. the i.i.d. assumption), tend to fail silently. This paper explores the problem of building ML systems that fail loudly, investigating methods for detecting dataset shift, identifying exemplars that most typify the shift, and quantifying shift malignancy. We focus on several datasets and various perturbations to both covariates and label distributions with varying magnitudes and fractions of data affected. Interestingly, we show that across the dataset shifts that we explore, a two-sample-testing-based approach, using pre-trained classifiers for dimensionality reduction, performs best. Moreover, we demonstrate that domain-discriminating approaches tend to be helpful for characterizing shifts qualitatively and determining if they are harmful.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“As demonstrated by Rabanser et al. 46 , the p-values are calculated by computing the maximum discrepancy between two distributions and comparing via permutation tests to a null-hypothesis distribution.”
From Empirical data drift detection experiments on real-world medical imaging data (2024) · cites this paper as [46]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“When implementing our experiments, we followed state-of-theart methodologies found in previous work 46 .”
From Empirical data drift detection experiments on real-world medical imaging data (2024) · cites this paper as [46]
The cited source says what the claim attributes to it.
In the citing paper
“Model output-based drift detection is a method to detect drift using only the output of the classifier without any gold labels 46 .”
From Empirical data drift detection experiments on real-world medical imaging data (2024) · cites this paper as [46]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Scikit-learn: Machine Learning in Python
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Müller, A., Nothman, J., Louppe, G., et al. (2012) · arXiv · doi:10.48550/arxiv.1201.0490 · Cited by 71,016
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805
Learning Multiple Layers of Features from Tiny Images
Krizhevsky, A. (2024) · doi:10.57702/zp44cu3g · Cited by 23,889
Sequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766
Chandola, V., Banerjee, A., & Kumar, V. (2009) · ACM Computing Surveys · doi:10.1145/1541880.1541882 · Cited by 11,687
Speech recognition with deep recurrent neural networks
Graves, A., Mohamed, A.-R., & Hinton, G. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6638947 · Cited by 8,893
Liu, F. T., Ting, K. M., & Zhou, Z.-H. (2008) · 2008 Eighth IEEE International Conference on Data Mining · doi:10.1109/icdm.2008.17 · Cited by 6,601
Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
Xiao, H., Rasul, K., & Vollgraf, R. (2017) · arXiv · doi:10.48550/arxiv.1708.07747 · Cited by 6,108
LOF: identifying density-based local outliers
Breunig, M. M., Kriegel, H.-P., Ng, R. T., & Sander, J. (2000) · ACM SIGMOD Record · doi:10.1145/335191.335388 · Cited by 5,207
Research works printed but not linked: 3
- [13][13] Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and Harnessing Adversarial Examples. In International Conference on Learning Representations (ICLR) , 2014. -
- [29][29] Zachary C Lipton, Yu-Xiang Wang, and Alex Smola. Detecting and Correcting for Label Shift with Black Box Predictors. In International Conference on Machine Learning (ICML) , 2018. -
- [51][51] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing Properties of Neural Networks. In International Conference on Learning Representations (ICLR) , 2014. -
Cited by
Sorted by citations
Items 101–200 of 208
Confidence-Aware Paced-Curriculum Learning by Label Smoothing for Surgical Scene Understanding
Xu, M., Islam, M., Glocker, B., & Ren, H. (2023) · IEEE Transactions on Automation Science and Engineering · doi:10.1109/tase.2023.3276361 · Cited by 8
TCR: topologically consistent reweighting for XGBoost in regression tasks
Zühlke, M.-M., & Kudenko⋆, D. (2025) · Machine Learning · doi:10.1007/s10994-024-06704-x · Cited by 7
The Problem with Real-World Novelty Detection - Issues in Multivariate Probabilistic Models
Gruhl, C., Hannan, A., Huang, Z., Nivarthi, C. P., & Vogt, S. (2021) · IEEE International Conference on Autonomic Computing and Self-Organizing Systems Companion (ACSOS-C) · doi:10.1109/acsos-c52956.2021.00055 · Cited by 7
Endogenous Macrodynamics in Algorithmic Recourse
Altmeyer, P., Angela, G., Buszydlik, A., Dobiczek, K., Arie Van Deursen, A., & Liem, C. C. S. (2023) · IEEE Conference on Secure and Trustworthy Machine Learning (SaTML) · doi:10.1109/satml54575.2023.00036 · Cited by 9
Dreves, M., Huang, G., Peng, Z., Polyzotis, N., Rosen, E., & Suganthan, G. C. P. (2020) · doi:10.1145/3399579.3399868 · Cited by 7
Tracking the risk of a deployed model and detecting harmful distribution shiftsPreprint
Podkopaev, A., & Ramdas, A. (2021) · International Conference on Learning Representations · doi:10.48550/arxiv.2110.06177 · Cited by 57
Fitzke, M., Stack, C., Dourson, A., Santana, R. M. B., Wilson, D., Ziemer, L. S., Soin, A., Lungren, M. P., Fisher, P. E., & Parkinson, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.08165 · Cited by 12
Dynamic robustness evaluation for automated model selection in operation
Zhang, J., Li, J., & Yang, Z. (2024) · Information and Software Technology · doi:10.1016/j.infsof.2024.107603 · Cited by 6
Confidence-based Estimators for Predictive Performance in Model Monitoring
Kivimäki, J., Nurminen, J. K., Białek, J., & Kuberski, W. (2025) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.16709 · Cited by 6
Anzaku, E. T., Wang, H., Babalola, A. R., Messem, A. V., & Neve, W. D. (2025) · Machine Learning · doi:10.1007/s10994-024-06693-x · Cited by 8
A Shape-Based Method for Concept Drift Detection and Signal Denoising
Hinder, F., Brinkrolf, J., Vaquet, V., & Hammer, B. (2021) · IEEE Symposium Series on Computational Intelligence (SSCI) · doi:10.1109/ssci50451.2021.9660111 · Cited by 6
Feature Selection for Concept Drift Detection
Hinder, F., & Hammer, B. (2023) · European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning (ESANN) · doi:10.14428/esann/2023.es2023-55 · Cited by 6
Vantilborgh, V., Staessens, T., Groote, W. D., & Crevecoeur, G. (2023) · Control Engineering Practice · doi:10.1016/j.conengprac.2023.105783 · Cited by 6
Sardana, P., & Wereley, S. T. (2023) · Measurement Science and Technology · doi:10.1088/1361-6501/ad108b · Cited by 6
Naveed, H. (2023) · ACM/IEEE International Conference on Model Driven Engineering Languages and Systems Companion (MODELS-C) · doi:10.1109/models-c59198.2023.00040 · Cited by 5
A Robust Framework for Distributional Shift Detection Under Sample-Bias
Torpmann-Hagen, B., Riegler, M. A., Halvorsen, P., & Johansen, D. (2024) · IEEE Access · doi:10.1109/access.2024.3393296 · Cited by 6
Automated Provenance-Based Screening of ML Data Preparation Pipelines
Schelter, S., Guha, S., & Grafberger, S. (2024) · Datenbank-Spektrum · doi:10.1007/s13222-024-00483-4 · Cited by 5
ClassTer: Mobile Shift-Robust Personalized Federated Learning via Class-Wise Clustering
Li, X., Liu, S., Zhou, Z., Yuan, X., Guo, B., & Yu, Z. (2024) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2024.3487294 · Cited by 5
Two-Sample Tests for Validating the UL-DL Conjecture in FDD Systems
Rizzello, V., Turan, N., Joham, M., & Utschick, W. (2021) · International Symposium on Wireless Communication Systems · doi:10.1109/iswcs49558.2021.9562147 · Cited by 5
Towards More Reliable Confidence Estimation
Qu, H., Foo, L. G., Li, Y., & Liu, J. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3291676 · Cited by 5
Towards robust production machine learning systems
Abdelkader, H. (2020) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3324884.3415281 · Cited by 5
InFlow: Robust outlier detection utilizing Normalizing FlowsPreprint
Kumar, N., V, Hanfeld, P., Hecht, M., Bußmann, M., Gumhold, S., & Hoffmann, N. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.12894 · Cited by 5
ClimateBench: A benchmark dataset for data-driven climate projectionsPreprint
Watson‐Parris, D., Rao, Y. D., Olivié, D., Seland, Ø., Nowack, P. J., Camps‐Valls, G., Stier, P., Bouabid, S., Dewey, M., Fons, E., et al. (2022) · doi:10.1002/essoar.10509765.2 · Cited by 5
Stoddard, G., Fitzpatrick, D., & Ludwig, J. (2024) · National Bureau of Economic Research · doi:10.3386/w32432 · Cited by 4
Hoang, V. T., Nguyen, Q., Nguyen, Q. C., Nguyen, X. P., & Nguyen, H. (2023) · PHM Society Asia-Pacific Conference · doi:10.36001/phmap.2023.v4i1.3604 · Cited by 4
Enhancing PM2.5 prediction by mitigating annual data drift using wrapped loss and neural networks
Hossen, M. K., Peng, Y., & Chen, M. C. (2025) · PLoS ONE · doi:10.1371/journal.pone.0314327 · Cited by 4
Friesacher, H. R., Svensson, E., Winiwarter, S., Mervin, L., Arany, Á., & Engkvist, O. (2025) · Artificial Intelligence in the Life Sciences · doi:10.1016/j.ailsci.2025.100132 · Cited by 8
Wang, Z., & Fan, W. D. (2025) · International Journal of Injury Control and Safety Promotion · doi:10.1080/17457300.2025.2541664 · Cited by 4
Pulling Up by the Causal Bootstraps
Gowda, S. C. M., Joshi, S. D., Zhang, H., & Ghassemi, M. (2021) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3459637.3482380 · Cited by 8
Deep meta-learning for the selection of accurate ultrasound based breast mass classifier
Byra, M., Karwat, P., Ryzhankow, I., Komorowski, P., Klimonda, Z., Fura, Ł., Pawłowska, A., Żołek, N., & Litniewski, J. (2022) · IEEE International Ultrasonics Symposium · doi:10.1109/ius54386.2022.9957191 · Cited by 5
Parameter Space Factorization for Zero-Shot Learning across Tasks and Languages
Ponti, E. M., Vulić, I., Cotterell, R., Parović, M., Reichart, R., & Korhonen, A. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00374 · Cited by 11
Modeling Non-Cooperative Dialogue: Theoretical and Empirical Insights
Sicilia, A., Maidment, T. D., Healy, P., & Alikhani, M. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00507 · Cited by 4
Elhalwagy, A., & Kalganova, T. (2023) · IEEE Access · doi:10.1109/access.2023.3326725 · Cited by 4
Online Distribution Shift Detection via Recency Prediction
Luo, R., Sinha, R., Sun, Y., Hindy, A., Zhao, S., Savarese, S., Schmerling, E., & Pavone, M. (2024) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra57147.2024.10611114 · Cited by 4
A novel machine learning-based method for identifying Industrial Symbiosis opportunities
Lan, Z., Sun, Y., & Xiao, G. (2025) · Journal of Cleaner Production · doi:10.1016/j.jclepro.2025.145607 · Cited by 4
Diffusing Gaussian Mixtures for Generating Categorical Data
Regol, F., & Coates, M. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i8.26145 · Cited by 3
Li, X., & Groth, P. (2024) · Language Resources and Evaluation · doi:10.1007/s10579-024-09754-8 · Cited by 5
slidSHAPs – sliding Shapley Values for correlation-based change detection in time series
Balestra, C., Li, B., & Müller, E. (2023) · IEEE International Conference on Data Science and Advanced Analytics (DSAA) · doi:10.1109/dsaa60987.2023.10302636 · Cited by 3
UDDT: An Unsupervised Drift Detection Method for Industrial Time Series Data
Maduskar, D., Sharma, D., Chandrika, K. R., Borrison, R., Manca, G., & Dix, M. (2023) · IEEE Industrial Electronics Society Annual On-Line Conference (ONCON) · doi:10.1109/oncon60463.2023.10431133 · Cited by 3
An Analysis of Radio Frequency Transfer Learning Behavior
Wong, L. J., Muller, B. P., McPherson, S., & Michaels, A. J. (2024) · Machine Learning and Knowledge Extraction · doi:10.3390/make6020057 · Cited by 3
Preventing Failures by Dataset Shift Detection in Safety-Critical Graph Applications
Song, H., Thiagarajan, J. J., & Kailkhura, B. (2021) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2021.589632 · Cited by 3
Towards Automated ML Model Monitoring: Measure, Improve and Quantify Data Quality
Rukat, T., Lange, D., Schelter, S., & Bießmann, F. (2020) · Cited by 3
Deploying clinical machine learning? Consider the following...Preprint
Lu, C., Chang, K., Singh, P., Pomerantz, S. R., Doyle, S., Kakarmath, S. S., Bridge, C. P., & Kalpathy–Cramer, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.06919 · Cited by 3
Out-of-distribution detection for SAR imagery using ATR systems
Hill, C. J. (2024) · Algorithms for synthetic aperture radar imagery. Conference · doi:10.1117/12.3013850 · Cited by 2
Concept drift detection based on radial distance
Shang, D., Zhang, G., & Jie Lu, B. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.131190 · Cited by 2
Board-level governance and risk oversight framework for Artificial Intelligence systems
Kumar, K., & Jaya, J. (2026) · EDPACS · doi:10.1080/07366981.2026.2645658 · Cited by 2
OPORP: One Permutation + One Random Projection
Li, P., & Li, X. (2023) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3580305.3599457 · Cited by 9
Robustness Attributes to Safeguard Machine Learning Models in Production
Abdelkader, H., Abdelrazek, M., Schneider, J.-G., Rani, P., & Vasa, R. (2023) · IEEE Engineering Informatics · doi:10.1109/ieeeconf58110.2023.10520555 · Cited by 3
Detecting and Adapting to CSI Feedback Drift in Data-Driven Compression for MIMO Systems
Wang, X., Mestav, K. R., Nuzman, C. J., & Saniee, I. (2024) · IEEE International Conference on Machine Learning for Communication and Networking (ICMLCN) · doi:10.1109/icmlcn59089.2024.10624762 · Cited by 2
Adversarial Validation in Image Classification Datasets by Means of Cumulative Spectral Gradient
Renza, D., Moya-Albor, E., & Chavarro, A. (2024) · Algorithms · doi:10.3390/a17110531 · Cited by 2
Omar, R., Bogner, J., Leest, J., Stoico, V., Lago, P., & Muccini, H. (2024) · International Conference on ICT for Sustainability (ICT4S) · doi:10.1109/ict4s64576.2024.00026 · Cited by 2
Pulling Up by the Causal Bootstraps: Causal Data Augmentation for Pre-training Debiasing
Gowda, S. C. M., Joshi, S. D., Zhang, H., & Ghassemi, M. (2021) · DSpace@MIT (Massachusetts Institute of Technology) · doi:10.48550/arxiv.2108.12510 · Cited by 2
Active Learning with Data Distribution Shift Detection for Updating Localization Systems
Barrows, J., Radu, V., Hill, M., & Ciravegna, F. (2021) · International Conference on Indoor Positioning and Indoor Navigation · doi:10.1109/ipin51156.2021.9662543 · Cited by 2
Negative Transfer in Cross Project Defect Prediction: Effect of Domain Divergence
Omondiagbe, O. P., Licorish, S. A., & MacDonell, S. G. (2022) · Euromicro Conference on Software Engineering and Advanced Applications (SEAA) · doi:10.1109/seaa56994.2022.00010 · Cited by 2
Feature Attribution Explanation to Detect Harmful Dataset Shift
Wang, Z., Huang, C., & Yao, X. (2023) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn54540.2023.10191221 · Cited by 3
Failing Conceptually: Concept-Based Explanations of Dataset ShiftPreprint
Wijaya, M. A., Kazhdan, D., Dimanov, B., & Jamnik, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.08952 · Cited by 2
Viehmann, T. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.12893 · Cited by 5
Two-sample KS test with approxQuantile in Apache Spark®
Eck, B. J., Kabakci-Zorlu, D., & Ba, A. (2023) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata59044.2023.10386638 · Cited by 1
Jin, M., Rajesh, M. D., Xing, W., Schniertshauer, J., Neuhüttler, J., & Zimmermann, F. (2023) · Hawaii International Conference on System Sciences (HICSS) · doi:10.24251/hicss.2023.807 · Cited by 1
Distribution-based detection of radiographic changes in pneumonia patterns: A COVID-19 case study
Pereira, S. C., Rocha, J., Campilho, A., & Mendonça, A. M. (2024) · Heliyon · doi:10.1016/j.heliyon.2024.e35677 · Cited by 1
Runtime Verification for Visual Deep Learning Systems With Loss Prediction
Torpmann-Hagen, B., Riegler, M. A., Halvorsen, P., & Johansen, D. (2025) · IEEE Access · doi:10.1109/access.2025.3552157 · Cited by 2
From Logs to Intelligence: Leveraging Data Science for Service Account Monitoring
Iyer, K. I. (2025) · Computer Fraud & Security · doi:10.52710/cfs.638 · Cited by 1
The Influence of Faulty Labels in Data Sets on Human Pose Estimation
Schwarz, A., Hernadi, L., Bießmann, F., & Hildebrand, K. (2025) · Computer Science Research Notes · doi:10.24132/csrn.2025-2 · Cited by 1
Mitigating Bias in Bayesian Optimized Data While Designing MacPherson Suspension Architecture
Thomas, S. S., Lamine, G., Palandri, J., Lakehal-Ayat, M., Chakravarty, P., Wolf, F., & Blaschko, M. B. (2023) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2023.3274100 · Cited by 1
Li, J., Arefan, D., Pease, M. W., Liu, C., Okonkwo, D. O., & Wu, S. (2024) · SPIE Medical Imaging: Imaging Informatics for Healthcare, Research, and Applications · doi:10.1117/12.3009348 · Cited by 1
Drift Detection for Black-Box Deep Learning Models
Piano, L., Garcea, F., Cavallone, A., Vazquez, I., Morra, L., & Lamberti, F. (2024) · IT Professional · doi:10.1109/mitp.2023.3338007 · Cited by 1
On Sample Selection for Continual Learning: A Video Streaming Case Study
Dietmüller, A., Jacob, R., & Vanbever, L. (2024) · ACM SIGCOMM Computer Communication Review · doi:10.1145/3687234.3687237 · Cited by 3
A Self-Organizing Clustering System for Unsupervised Distribution Shift Detection
Basterrech, S., Clemmensen, L. K. H., & Gerardo Rubino, C. (2024) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn60899.2024.10650314 · Cited by 1
Image Drift: Introducing Gaussian Split Detector
Fuccellaro, M., Simon, L., & Zemmari, A. (2024) · International Conference on Image Processing Theory, Tools and Applications (IPTA) · doi:10.1109/ipta62886.2024.10755938 · Cited by 2
Leiner, K., Peter, J., & Huber, M. F. (2024) · Procedia CIRP · doi:10.1016/j.procir.2024.10.066 · Cited by 1
Self-Healing AI Pipelines: Enforcing Data Integrity through Quality-First Learning Models
Chinthalapally, A. R. (2025) · International Journal of Information and Electronics Engineering · doi:10.48047/ijiee.2025.13.4.9 · Cited by 1
Time-Aware Data Profiling With Decision Tree Pattern Mining
Grote, A., Badewitz, W., Knierim, M. T., & Weinhardt, C. (2023) · IEEE Conference on Business Informatics · doi:10.1109/cbi58679.2023.10187535 · Cited by 1
Test for non-negligible adverse shiftsPreprint
Kamulete, V. M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.02990 · Cited by 1
ANOVA exemplars for understanding data drift.Preprint
Williamson, S. A. (2020) · arXiv (Cornell University)
Sequential changepoint detection for label shift in classificationPreprint
Evans, C., & G’Sell, M. (2020) · arXiv (Cornell University)
Using Distribution Divergence to Predict Changes in the Performance of Clinical Predictive Models
Tajgardoon, M., & Visweswaran, S. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-77211-6_14
Sample-free white-box out-of-distribution detection for deep learning
Begon, J.-M., & Geurts, P. (2021) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw53098.2021.00367
Making Table Understanding Work in PracticePreprint
Hulsebos, M., Gathani, S., Gale, J., Dillig, I., Groth, P., & Demiralp, Ç. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.05173 · Cited by 2
Domain generalization, invariance and the Time Robust Forest
dos Santos, L. G. M. (2021) · doi:10.11606/d.45.2021.tde-16112021-193305
Detecting Distributional Shift Responsible for Predictive Model's Failure
Mukherjee, D. S., Bhandari, D., & Yeri, N. G. (2022) · International Conference for Advancement in Technology (ICONAT) · doi:10.1109/iconat53423.2022.9726015
Performance monitoring of deep learning vision systems during deployment
Rahman, Q. M. (2022) · Queensland University of Technology · doi:10.5204/thesis.eprints.229733
Detecting Distributional Shift Responsible for Predictive Model’s Failure*Preprint
Mukherjee, D. S., Bhandari, D., & Yeri, N. G. (2021) · doi:10.36227/techrxiv.16656901
Detecting Distributional Shift Responsible for Predictive Model’s Failure*Preprint
Mukherjee, D. S., Bhandari, D., & Yeri, N. G. (2021) · doi:10.36227/techrxiv.16656901.v1
Detecting Distributional Shift Responsible for Predictive Model’s Failure*Preprint
Mukherjee, D. S., Bhandari, D., & Yeri, N. G. (2021) · doi:10.36227/techrxiv.16656901.v2
popmon: Analysis Package for Dataset Shift Detection
Brugman, S., Sostak, T., Patil, P., & Baak, M. (2022) · Proceedings of the Python in Science Conferences · doi:10.25080/majora-212e5952-01d
Towards Explaining Image-Based Distribution Shifts
Kulinski, S., & Inouye, D. I. (2022) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw56347.2022.00525
Banf, M., & Steinhagen, G. (2022) · Proceedings of the World Congress on Electrical Engineering and Computer Systems and Science · doi:10.11159/mhci22.104 · Cited by 1
MAGDiff: Covariate Data Set Shift Detection via Activation Graphs of Deep Neural Networks
Arnal, C., Hensel, F., Carrière, M., Lacombe, T., Kurihara, H., Ike, Y., & Chazal, F. (2023) · HAL · doi:10.48550/arxiv.2305.13271 · Cited by 2
Incremental Template Learning for Occlusion-Aware Visual Object TrackingPreprint
Kiran, M., Nguyen-Meidine, L. T., Cruz, R. M. O., Blais-Morin, L.-A., & Granger, É. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4530457
Dynamic Quantification With Constrained Error Under Unknown General Dataset Shift
Denham, B., Lai, E. M.-K., Sinha, R., & Naeem, M. A. (2024) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2023.3349286
Mitigating Catastrophic Forgetting in Continual Learning-Based Image Classification
Borde, S. (2023) · IEEE International Conference on Service Operations and Logistics, and Informatics (SOLI) · doi:10.1109/soli60636.2023.10425549 · Cited by 1
Predicting Police MisconductPreprint
Stoddard, G., Fitzpatrick, D., & Ludwig, J. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4826039
Predicting Police MisconductPreprint
Stoddard, G., Fitzpatrick, D., & Ludwig, J. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4829604
Data Drift for Automatic FAIR-compliant Dataset Versioning in Large Repositories
González–Cebrián, A., Ciolacu, I., Bradford, M., Dobre, C., & González–Vélez, H. (2024) · IEEE International Conference on eScience · doi:10.1109/e-science62913.2024.10678667
An Empirical Analysis of Data Drift Detection Techniques in Machine Learning Systems
Helfstein, L., & Braghetto, K. R. (2024) · Simpósio Brasileiro de Bancos de Dados (SBBD) · doi:10.5753/sbbd.2024.240606
Handling shift and irregularities in data through sequential ellipsoidal partitioning
Niranjan, R., & Rao, S. (2024) · Data-Centric Engineering · doi:10.1017/dce.2024.41
Out-of-Distribution Detection Using Maximum Entropy Coding and Generative Networks
Abolfazli, M., Zaeri-Amirani, M., Høst-Madsen, A., Zhang, J., & Bratincsák, A. (2024) · International Symposium on Information Theory and its Applications · doi:10.23919/isita60732.2024.10858233
Deep Learning-Based Data Drift Detection of Image Rotation in Surveillance Videos
Kim, Y., Kim, B., & Jeon, B. (2025) · International Conference on Ubiquitous Information Management and Communication (IMCOM) · doi:10.1109/imcom64595.2025.10857537
Metadata
- Authors
- Rabanser, S., Günnemann, S., & Lipton, Z. C.
- Affiliations
- Amazon (United States) (US)
- Technical University of Munich (DE)
- Carnegie Mellon University (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2018
- Volume
- 32
- Pages
- 1396-1408
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1810.11953
- OpenAlex
- W2898158860
- Cited by
- 204
- References
- 58
- Citations by year
- 2026: 4 · 2025: 23 · 2024: 36 · 2023: 44 · 2022: 29 · 2021: 44 · 2020: 22 · 2019: 2
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Data Stream Mining Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Keywords
- distribution shift, shift detection, fault detection, silent failures, covariate shift, label shift, two-sample test, dimensionality reduction, i.i.d. assumption
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
