The DOI matches its DataCite record; 55 of the 58 references it cites were found in a registry or an index.
Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shift
Open accessAbstract
We might hope that when faced with unexpected inputs, well-designed software systems would fire off warnings. Machine learning (ML) systems, however, which depend strongly on properties of their inputs (e.g. the i.i.d. assumption), tend to fail silently. This paper explores the problem of building ML systems that fail loudly, investigating methods for detecting dataset shift, identifying exemplars that most typify the shift, and quantifying shift malignancy. We focus on several datasets and various perturbations to both covariates and label distributions with varying magnitudes and fractions of data affected. Interestingly, we show that across the dataset shifts that we explore, a two-sample-testing-based approach, using pre-trained classifiers for dimensionality reduction, performs best. Moreover, we demonstrate that domain-discriminating approaches tend to be helpful for characterizing shifts qualitatively and determining if they are harmful.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“As demonstrated by Rabanser et al. 46 , the p-values are calculated by computing the maximum discrepancy between two distributions and comparing via permutation tests to a null-hypothesis distribution.”
From Empirical data drift detection experiments on real-world medical imaging data (2024) · cites this paper as [46]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“When implementing our experiments, we followed state-of-theart methodologies found in previous work 46 .”
From Empirical data drift detection experiments on real-world medical imaging data (2024) · cites this paper as [46]
The cited source says what the claim attributes to it.
In the citing paper
“Model output-based drift detection is a method to detect drift using only the output of the classifier without any gold labels 46 .”
From Empirical data drift detection experiments on real-world medical imaging data (2024) · cites this paper as [46]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Scikit-learn: Machine Learning in Python
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Müller, A., Nothman, J., Louppe, G., et al. (2012) · arXiv · doi:10.48550/arxiv.1201.0490 · Cited by 71,016
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805
Learning Multiple Layers of Features from Tiny Images
Krizhevsky, A. (2024) · doi:10.57702/zp44cu3g · Cited by 23,889
Sequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766
Chandola, V., Banerjee, A., & Kumar, V. (2009) · ACM Computing Surveys · doi:10.1145/1541880.1541882 · Cited by 11,687
Speech recognition with deep recurrent neural networks
Graves, A., Mohamed, A.-R., & Hinton, G. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6638947 · Cited by 8,893
Liu, F. T., Ting, K. M., & Zhou, Z.-H. (2008) · 2008 Eighth IEEE International Conference on Data Mining · doi:10.1109/icdm.2008.17 · Cited by 6,601
Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
Xiao, H., Rasul, K., & Vollgraf, R. (2017) · arXiv · doi:10.48550/arxiv.1708.07747 · Cited by 6,108
LOF: identifying density-based local outliers
Breunig, M. M., Kriegel, H.-P., Ng, R. T., & Sander, J. (2000) · ACM SIGMOD Record · doi:10.1145/335191.335388 · Cited by 5,207
Research works printed but not linked: 3
- [13][13] Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and Harnessing Adversarial Examples. In International Conference on Learning Representations (ICLR) , 2014. -
- [29][29] Zachary C Lipton, Yu-Xiang Wang, and Alex Smola. Detecting and Correcting for Label Shift with Black Box Predictors. In International Conference on Machine Learning (ICML) , 2018. -
- [51][51] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing Properties of Neural Networks. In International Conference on Learning Representations (ICLR) , 2014. -
Cited by
Sorted by citations
Toward Causal Representation Learning
Scholkopf, B., Locatello, F., Bauer, S., Ke, N. R., Kalchbrenner, N., Goyal, A., & Bengio, Y. (2021) · Proceedings of the IEEE · doi:10.1109/jproc.2021.3058954 · Cited by 1,170
Algorithmic Fairness: Choices, Assumptions, and Definitions
Mitchell, S., Potash, E., Barocas, S., D'Amour, A., & Lum, K. (2021) · Annual Review of Statistics and Its Application · doi:10.1146/annurev-statistics-042720-125902 · Cited by 687
Machine learning for medical imaging: methodological failures and recommendations for the future
Varoquaux, G. P., & Cheplygina, V. (2022) · npj Digital Medicine · doi:10.1038/s41746-022-00592-y · Cited by 815
Trustworthy AI: From Principles to Practices
Li, B., Qi, P., Liu, B., Di, S., Liu, J., Pei, J., Yi, J., & Zhou, B. (2023) · ACM Computing Surveys · doi:10.1145/3555803 · Cited by 612
“Data drift monitoring [268] provides important means to maintain the generalization of an AI system under concept change [394] caused by dynamic environment such as market change [289].”
BackgroundEthical Machine Learning in Healthcare
Chen, I. Y., Pierson, E., Rose, S., Joshi, S., Ferryman, K., & Ghassemi, M. (2021) · Annual Review of Biomedical Data Science · doi:10.1146/annurev-biodatasci-092820-114757 · Cited by 645
Causality for Machine Learning
Schölkopf, B. (2022) · ACM eBooks · doi:10.1145/3501714.3501755 · Cited by 548
Navigating the pitfalls of applying machine learning in genomics
Whalen, S., Schreiber, J. M., Noble, W. S., & Pollard, K. S. (2021) · Nature Reviews Genetics · doi:10.1038/s41576-021-00434-9 · Cited by 291
Assuring the Machine Learning Lifecycle: Desiderata, Methods, and Challenges
Ashmore, R., Calinescu, R., & Paterson, C. (2022) · ACM Computing Surveys · doi:10.1145/3453444 · Cited by 256
Aguiar, G., Krawczyk, B., & Cano, A. (2023) · Machine Learning · doi:10.1007/s10994-023-06353-6 · Cited by 170
Cabitza, F., Campagner, A., Ferrari, D., Resta, C. D., Ceriotti, D., Sabetta, E., Colombini, A., Vecchi, E. D., Banfi, G., Locatelli, M., et al. (2020) · Clinical Chemistry and Laboratory Medicine (CCLM) · doi:10.1515/cclm-2020-1294 · Cited by 174
Challenges with developing and deploying AI models and applications in industrial systems
Sinha, S. R., & Lee, Y. M. (2024) · Discover Artificial Intelligence · doi:10.1007/s44163-024-00151-2 · Cited by 192
Empirical data drift detection experiments on real-world medical imaging data
Kore, A., Abbasi Bavil, E., Subasri, V., Abdalla, M., Fine, B., Dolatabadi, E., & Abdalla, M. (2024) · Nature Communications · doi:10.1038/s41467-024-46142-w · Cited by 124
“Model output-based drift detection is a method to detect drift using only the output of the classi fi er without any gold labels 46 .”
InfluentialTowards risk-aware artificial intelligence and machine learning systems: An overview
Zhang, X., Chan, F. T. S., Yan, C., & Bose, I. (2022) · Decision Support Systems · doi:10.1016/j.dss.2022.113800 · Cited by 137
Duckworth, C., Chmiel, F. P., Burns, D. K., Zlatev, Z. D., White, N. M., Daniels, T. W. V., Kiuber, M., & Boniface, M. J. (2021) · Scientific Reports · doi:10.1038/s41598-021-02481-y · Cited by 109
ClimateBench v1.0: A Benchmark for Data‐Driven Climate Projections
Watson‐Parris, D., Rao, Y. D., Olivié, D., Seland, Ø., Nowack, P. J., Camps‐Valls, G., Stier, P., Bouabid, S., Dewey, M., Fons, E., et al. (2022) · Journal of Advances in Modeling Earth Systems · doi:10.1029/2021ms002954 · Cited by 124
Hinder, F., Vaquet, V., & Hammer, B. (2024) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2024.1330257 · Cited by 146
Assuring the Machine Learning Lifecycle: Desiderata, Methods, and ChallengesPreprint
Ashmore, R., Călinescu, R., & Paterson, C. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.04223 · Cited by 86
MultiBench: Multiscale Benchmarks for Multimodal Representation Learning
Liang, P. P., Lyu, Y., Fan, X., Wu, Z., Cheng, Y., Wu, J., Chen, L., Wu, P., Lee, M. A., Zhu, Y., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.07502 · Cited by 85
Look Before You Leap: An Exploratory Study of Uncertainty Analysis for Large Language Models
Huang, Y., Song, J., Wang, Z., Zhao, S., Chen, H., Juefei-Xu, F., & Ma, L. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3519464 · Cited by 275
Run-Time Monitoring of Machine Learning for Robotic Perception: A Survey of Emerging Trends
Rahman, Q. M., Corke, P., & Dayoub, F. (2021) · IEEE Access · doi:10.1109/access.2021.3055015 · Cited by 65
Corbin, C. K., Maclay, R., Acharya, A., Mony, S., Punnathanam, S., Thapa, R., Kotecha, N., Shah, N. H., & Chen, J. H. (2023) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocad114 · Cited by 64
Anatomy of Catastrophic Forgetting: Hidden Representations and Task Semantics
Ramasesh, V., Dyer, E., & Raghu, M. (2021) · International Conference on Learning Representations · Cited by 53
Bhatkalkar, B. J., Reddy, D. R., Prabhu, S., & Bhandary, S. V. (2020) · IEEE Access · doi:10.1109/access.2020.2972318 · Cited by 58
Representativeness in Statistics, Politics, and Machine Learning
Chasalow, K., & Levy, K. E. C. (2021) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3442188.3445872 · Cited by 61
Analysis of Continual Learning Models for Intrusion Detection System
Prasath, S., Sethi, K., Mohanty, D. K., Bera, P., & Samantaray, S. R. (2022) · IEEE Access · doi:10.1109/access.2022.3222715 · Cited by 50
GitTables: A Large-Scale Corpus of Relational Tables
Hulsebos, M., Demiralp, Ç., & Groth, P. (2023) · Proceedings of the ACM on Management of Data · doi:10.1145/3588710 · Cited by 89
Lascio, E. D., Gerebtzoff, G., & Rodríguez-Pérez, R. (2023) · Molecular Pharmaceutics · doi:10.1021/acs.molpharmaceut.2c00962 · Cited by 48
Suriyakumar, V. M., Papernot, N., Goldenberg, A., & Ghassemi, M. (2021) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3442188.3445934 · Cited by 62
Neural Mean Discrepancy for Efficient Out-of-Distribution Detection
Dong, X., Guo, J., Li, A., Ting, W.-T., Liu, C., & Kung, H. T. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01862 · Cited by 50
TensorFlow Data Validation: Data Analysis and Validation in Continuous ML Pipelines
Caveness, E., Paul Suganthan, G. C., Peng, Z., Polyzotis, N., Roy, S., & Zinkevich, M. (2020) · ACM SIGMOD International Conference on Management of Data (SIGMOD) · doi:10.1145/3318464.3384707 · Cited by 45
Towards Robust and Reliable Algorithmic RecoursePreprint
Upadhyay, S., Joshi, S. D., & Lakkaraju, H. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.13620 · Cited by 42
Diagnosis and Prognosis of COVID-19 Disease Using Routine Blood Values and LogNNet Neural Network
Huyut, M. T., & Velichko, A. (2022) · Sensors · doi:10.3390/s22134820 · Cited by 41
Feature Space Singularity for Out-of-Distribution DetectionPreprint
Huang, H., Li, Z., Wang, L., Chen, S., Dong, B., & Zhou, X. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.14654 · Cited by 52
A Validity Perspective on Evaluating the Justified Use of Data-driven Decision-making Algorithms
Coston, A., Kawakami, A., Zhu, H., Holstein, K., & Heidari, H. (2023) · IEEE Conference on Secure and Trustworthy Machine Learning (SaTML) · doi:10.1109/satml54575.2023.00050 · Cited by 57
Fogelberg, K., Chamarthi, S., Maron, R. C., Niebling, J., & Brinker, T. J. (2023) · New Biotechnology · doi:10.1016/j.nbt.2023.04.006 · Cited by 42
Learning to Validate the Predictions of Black Box Classifiers on Unseen Data
Schelter, S., Rukat, T., & Bießmann, F. (2020) · ACM SIGMOD International Conference on Management of Data (SIGMOD) · doi:10.1145/3318464.3380604 · Cited by 33
Monitoring and explainability of models in productionPreprint
Klaise, J., Looveren, A. V., Cox, C., Vacanti, G., & Coca, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.06299 · Cited by 81
Seedat, N., Imrie, F., & van der Schaar, M. (2023) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2023.3345805 · Cited by 49
Curriculum-Based Augmented Fourier Domain Adaptation for Robust Medical Image Segmentation
Wang, A., Islam, M., Xu, M., & Ren, H. (2023) · IEEE Transactions on Automation Science and Engineering · doi:10.1109/tase.2023.3295600 · Cited by 32
Towards Observability for Production Machine Learning Pipelines
Shankar, S., & Parameswaran, A. (2022) · Proceedings of the VLDB Endowment · doi:10.14778/3565838.3565853 · Cited by 60
Enabling interpretable machine learning for biological data with reliability scores
Ahlquist, K. D., Sugden, L. A., & Ramachandran, S. (2023) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1011175 · Cited by 28
Automated data validation in machine learning systems
Bießmann, F., Gołębiowski, J., Rukat, T., Lange, D., & Schmidt, P. S. (2021) · Cited by 22
Low-Overhead Clustered Federated Learning for Personalized Stress Monitoring
Jiang, S., Firouzi, F., & Chakrabarty, K. (2023) · IEEE Internet of Things Journal · doi:10.1109/jiot.2023.3299736 · Cited by 26
Out of distribution learning in bioinformatics: advancements and challenges
Shi, Y., Xu, W., & Hu, P. (2025) · Briefings in Bioinformatics · doi:10.1093/bib/bbaf294 · Cited by 25
Anatomy of Catastrophic Forgetting: Hidden Representations and Task SemanticsPreprint
Ramasesh, V., Dyer, E., & Raghu, M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.07400 · Cited by 20
CycleSense: Detecting near miss incidents in bicycle traffic from mobile motion sensors
Karakaya, A.-S., Ritter, T., Bießmann, F., & Bermbach, D. (2023) · Pervasive and Mobile Computing · doi:10.1016/j.pmcj.2023.101779 · Cited by 21
A unified framework for dataset shift diagnostics
Polo, F. M., Izbicki, R., Lacerda, E. G., Jimenez, J. P. I., & Vicente, R. (2023) · Information Sciences · doi:10.1016/j.ins.2023.119612 · Cited by 24
A monitoring framework for deployed machine learning models with supply chain examples
Eck, B. J., Kabakci-Zorlu, D., Chen, Y., Savard, F., & Bao, X. (2022) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata55660.2022.10020394 · Cited by 18
Advancing accountability in AI
OECD (2023) · OECD digital economy papers · doi:10.1787/2448f04b-en · Cited by 42
Dataset shift assessment measures in monitoring predictive models
Becker, A., & Becker, J. (2021) · Procedia Computer Science · doi:10.1016/j.procs.2021.09.112 · Cited by 17
van der Vorst, J. P., Smit, J. M., van de Sande, D., van der Ster, B. J. P., Daams, F. D., Schasfoort, R. A., Gommers, D. A. M. P. J., Verhoef, C. K., Grünhagen, D. J., van Genderen, M. E., et al. (2025) · BMJ Digital Health & AI · doi:10.1136/bmjdhai-2025-000046 · Cited by 16
Influence of Concept Drift on Metrological Performance of Low-Cost NO 2 Sensors
D’Elia, G., Ferro, M., Sommella, P., Vito, S. D., Ferlito, S., D’Auria, P., & Francia, G. D. (2022) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2022.3188028 · Cited by 16
Everything is varied: The surprising impact of instantial variation on ML reliability
Campagner, A., Famiglini, L., Carobene, A., & Cabitza, F. (2023) · Applied Soft Computing · doi:10.1016/j.asoc.2023.110644 · Cited by 17
Raca, D., Zahran, A. H., Sreenan, C. J., Sinha, R. K., Halepovic, E., & Gopalakrishnan, V. (2024) · IEEE Transactions on Machine Learning in Communications and Networking · doi:10.1109/tmlcn.2024.3352541 · Cited by 16
JENGA - A Framework to Study the Impact of Data Errors on the Predictions of Machine Learning Models
Schelter, S., Rukat, T., & Bießmann, F. (2021) · Movebank · doi:10.5441/002/edbt.2021.63 · Cited by 16
Hell, F., Hinz, G., Liu, F., Goyal, S., Pei, K., Lytvynenko, T., Knoll, A., & Chen, Y. (2021) · ACM Computer Science in Cars Symposium (CSCS) · doi:10.1145/3488904.3493382 · Cited by 15
Feature-based analyses of concept drift
Hinder, F., Vaquet, V., & Hammer, B. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.127968 · Cited by 14
Detecting Drift in Deep Learning: A Methodology Primer
Piano, L., Garcea, F., Gatteschi, V., Lamberti, F., & Morra, L. (2022) · IT Professional · doi:10.1109/mitp.2022.3191318 · Cited by 14
Mixture Proportion Estimation and PU Learning: A Modern ApproachPreprint
Garg, S., Wu, Y., Smola, A. J., Balakrishnan, S., & Lipton, Z. C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.00980 · Cited by 16
Self-citationTowards a Framework for Adapting Machine Learning Components
Casimiro, M., Romano, P., Garlan, D., & Rodrigues, L. (2022) · IEEE International Conference on Autonomic Computing and Self-Organizing Systems (ACSOS) · doi:10.1109/acsos55765.2022.00031 · Cited by 13
An Explainable and Actionable Mistrust Scoring Framework for Model Monitoring
Bhaskhar, N., Rubin, D. L., & Lee‐Messer, C. (2023) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2023.3272876 · Cited by 19
Ibrahim, S., Ponomareva, N., & Mazumder, R. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-26387-3_42 · Cited by 15
Adversarial Detection and Correction by Matching Prediction DistributionsPreprint
Vacanti, G., & Looveren, A. V. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.09364 · Cited by 13
Domain Generalization via Semi-supervised Meta LearningPreprint
Sharifi-Noghabi, H., Asghari, H., Mehrasa, N., & Ester, M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.12658 · Cited by 13
Can I Still Trust You?: Understanding the Impact of Distribution Shifts on Algorithmic Recourses.Preprint
Rawal, K., Kamar, E. S., & Lakkaraju, H. (2020) · arXiv (Cornell University) · Cited by 13
Conditional Generative Models for Counterfactual ExplanationsPreprint
Looveren, A. V., Klaise, J., Vacanti, G., & Cobb, O. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.10123 · Cited by 45
Delaney, E. D., Greene, D., & Keane, M. T. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.09734 · Cited by 13
Self-adaptive Machine Learning Systems: Research Challenges and Opportunities
Casimiro, M., Romano, P., Garlan, D., Moreno, G. A., Kang, E., & Klein, M. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-15116-3_7 · Cited by 13
MedShift: Automated Identification of Shift Data for Medical Image Dataset Curation
Guo, X., Gichoya, J. W., Trivedi, H., Purkayastha, S., & Banerjee, I. (2023) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2023.3275104 · Cited by 12
Monitoring Model Deterioration with Explainable Uncertainty Estimation via Non-parametric Bootstrap
Mougan, C., & Nielsen, D. S. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i12.26755 · Cited by 15
MLHOps: Machine Learning Health Operations
Khattak, F. K., Subasri, V., Krishnan, A., Pou-Prom, C., Koçak, S. A., Dolatabadi, E., Pandya, D., Seyyed-Kalantari, L., & Rudzicz, F. (2024) · IEEE Access · doi:10.1109/access.2024.3521279 · Cited by 31
Self-Adaptation for Machine Learning Based Systems.
Casimiro, M., Romano, P., Garlan, D., Moreno, G. A., Kang, E., & Klein, M. (2021) · Cited by 12
Frouros: An open-source Python library for drift detection in machine learning systems
Sisniega, J. C., & García, Á. L. (2024) · SoftwareX · doi:10.1016/j.softx.2024.101733 · Cited by 12
The Change that Matters in Discourse Parsing: Estimating the Impact of Domain Shift on Parser Error
Atwell, K., Sicilia, A., Hwang, S. J., & Alikhani, M. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.68 · Cited by 13
Sisniega, J. C., Rodríguez, V., Moltó, G., & García, Á. L. (2024) · Future Generation Computer Systems · doi:10.1016/j.future.2024.07.010 · Cited by 11
Safety Monitoring of Machine Learning Perception Functions: A Survey
Ferreira, R., Guerin, J. M., Delmas, K., Guiochet, J., & Waeselynck, H. (2025) · Computational Intelligence · doi:10.1111/coin.70032 · Cited by 23
Impact of Physiological Sensor Variance on Machine Learning Algorithms
Simons, A., Doyle, T. E., Musson, D. M., & Reilly, J. P. (2020) · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/smc42975.2020.9282912 · Cited by 11
Model-agnostic and Scalable Counterfactual Explanations via Reinforcement LearningPreprint
Samoilescu, R.-F., Looveren, A. V., & Klaise, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.02597 · Cited by 30
Online Drift Detection with Maximum Concept Discrepancy
Wan, K., Liang, Y., & Yoon, S. (2024) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3637528.3672016 · Cited by 10
Resilient Neural Forecasting Systems
Bohlke‐Schneider, M., Kapoor, S., & Januschowski, T. (2020) · doi:10.1145/3399579.3399869 · Cited by 10
Greco, S., & Cerquitelli, T. (2021) · IEEE ... International Conference on Data Mining workshops · doi:10.1109/icdmw53433.2021.00049 · Cited by 12
Benchmarking Scalable Predictive Uncertainty in Text Classification
Landeghem, J. V., Blaschko, M. B., Anckaert, B., & Moens, M. (2022) · IEEE Access · doi:10.1109/access.2022.3168734 · Cited by 13
A Review of Off-Line Mode Dataset Shifts
Takahashi, C. C., & Braga, A. P. (2020) · IEEE Computational Intelligence Magazine · doi:10.1109/mci.2020.2998231 · Cited by 10
Drift Detection in Text Data with Document Embeddings
Feldhans, R., Wilke, A., Heindorf, S., Shaker, M., Hammer, B., Ngomo, A.-C. N., & Hüllermeier, E. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-91608-4_11 · Cited by 12
MLDemon: Deployment Monitoring for Machine Learning SystemsPreprint
Ginart, A. A., Zhang, M. J., & Zou, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.13621 · Cited by 10
Imputation Strategies Under Clinical Presence: Impact on Algorithmic FairnessPreprint
Jeanselme, V., De‐Arteaga, M., Zhang, Z., Tom, B. D. M., & Barrett, J. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4496874 · Cited by 10
Perspectives on incorporating expert feedback into model updates
Chen, V., Bhatt, U., Heidari, H., Weller, A., & Talwalkar, A. (2023) · Patterns · doi:10.1016/j.patter.2023.100780 · Cited by 15
State-transition-aware anomaly detection under concept drifts
Li, B., Gupta, S., & Müller, E. (2024) · Data & Knowledge Engineering · doi:10.1016/j.datak.2024.102365 · Cited by 14
Politi, E., Davalas, C., Chronis, C., Dimitrakopoulos, G. J., Michail, D., & Varlamis, I. (2025) · IEEE Access · doi:10.1109/access.2025.3536524 · Cited by 9
Li, D., Hu, L., Peng, X., Xiao, N., Zhao, H., Liu, G., Liu, H., Li, K., Ai, B., Xia, H., et al. (2022) · iScience · doi:10.1016/j.isci.2022.103961 · Cited by 10
Learning to Validate the Predictions of Black Box Machine Learning Models on Unseen Data
Redyuk, S., Schelter, S., Rukat, T., Markl, V., & Bießmann, F. (2019) · Workshop on Human-In-the-Loop Data Analytics (HILDA) · doi:10.1145/3328519.3329126 · Cited by 9
Chasing Your Long Tails: Differentially Private Prediction in Health Care SettingsPreprint
Suriyakumar, V. M., Papernot, N., Goldenberg, A., & Ghassemi, M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.06667 · Cited by 9
Algorithmic Recourse in the Wild: Understanding the Impact of Data and Model ShiftsPreprint
Rawal, K., Kamar, E. S., & Lakkaraju, H. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.11788 · Cited by 9
MMD Aggregated Two-Sample TestPreprint
Schrab, A., Kim, I., Albert, M., Laurent, B., Guedj, B., & Gretton, A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.15073 · Cited by 9
Your Best Guess When You Know Nothing: Identification and Mitigation of Selection Bias
Dost, K., Taškova, K., Riddle, P., & Wicker, J. S. (2020) · Proceedings · doi:10.1109/icdm50108.2020.00115 · Cited by 9
Concept Drift Detection via Boundary Shrinking
Okawa, Y., & KOBAYASHI, K. P. (2021) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn52387.2021.9533334 · Cited by 8
DeepLens: Interactive Out-of-distribution Data Detection in NLP Models
Song, D., Wang, Z., Huang, Y., Ma, L., & Zhang, T. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580741 · Cited by 9
CyclOps: Cyclical development towards operationalizing ML models for healthPreprint
Krishnan, A., Subasri, V., McKeen, K., Kore, A., Ogidi, F., Alinoori, M., Lalani, N., Dhalla, A., Verma, A. A., Razak, F. A., et al. (2022) · medRxiv · doi:10.1101/2022.12.02.22283021 · Cited by 10
Reconstruction-based unsupervised drift detection over multivariate streaming data
Kaminskyi, D., Li, B., & Müller, E. (2022) · IEEE ... International Conference on Data Mining workshops · doi:10.1109/icdmw58026.2022.00109 · Cited by 7
Metadata
- Authors
- Rabanser, S., Günnemann, S., & Lipton, Z. C.
- Affiliations
- Amazon (United States) (US)
- Technical University of Munich (DE)
- Carnegie Mellon University (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2018
- Volume
- 32
- Pages
- 1396-1408
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1810.11953
- OpenAlex
- W2898158860
- Cited by
- 204
- References
- 58
- Citations by year
- 2026: 4 · 2025: 23 · 2024: 36 · 2023: 44 · 2022: 29 · 2021: 44 · 2020: 22 · 2019: 2
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Data Stream Mining Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Keywords
- distribution shift, shift detection, fault detection, silent failures, covariate shift, label shift, two-sample test, dimensionality reduction, i.i.d. assumption
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
