Machine learning with a reject option: a survey

Untitled work

(1996) · Cited by 20,179

Advances in neural information processing systems 7

(1996) · Computers & Mathematics with Applications · doi:10.1016/s0898-1221(96)90463-0 · Cited by 14,759

Advances in Neural Information Processing Systems 25

Zhang, Y., Sutton, C. A., Storkey, A., & Zoubin Ghahramani, A. (2012) · Cited by 12,653

Isolation Forest

Liu, F. T., Ting, K. M., & Zhou, Z.-H. (2008) · 2008 Eighth IEEE International Conference on Data Mining · doi:10.1109/icdm.2008.17 · Cited by 6,601

Untitled work

(2009) · Minds at UW (University of Wisconsin) · Cited by 4,791

Methods of combining multiple classifiers and their applications to handwriting recognition

Xu, L., Krzyżak, A., & Suen, C. Y. (1992) · IEEE Transactions on Systems Man and Cybernetics · doi:10.1109/21.155943 · Cited by 2,192

Management of validation of HPLC method for determination of acetylsalicylic acid impurities in a new pharmaceutical product

Kowalska, M. K., Woźniak, M., Kijek, M. M., Mitrosz, P., Szakiel, J., & Turek, P. (2022) · Scientific Reports · doi:10.1038/s41598-021-99269-x · Cited by 2,121

Support Vector Method for Novelty Detection

Schölkopf, B., Williamson, R. C., Smola, A. J., Shawe-Taylor, J. S., & Platt, J. C. (1999) · Cited by 2,038

Efficient algorithms for mining outliers from large data sets

Ramaswamy, S., Rastogi, R. R., & Shim, K. (2000) · ACM SIGMOD Record · doi:10.1145/335191.335437 · Cited by 1,908

A review of novelty detection

Pimentel, M. A. F., Clifton, D. A., Clifton, L., & Tarassenko, L. (2014) · Signal Processing · doi:10.1016/j.sigpro.2013.12.026 · Cited by 1,649

Novelty detection: a review—part 1: statistical approaches

Markou, M., & Singh, S. (2003) · Signal Processing · doi:10.1016/j.sigpro.2003.07.018 · Cited by 1,472

IEEE Transactions on Pattern Analysis and Machine Intelligence

(1978) · IEEE Transactions on Aerospace and Electronic Systems · doi:10.1109/taes.1978.308564 · Cited by 1,328

On optimum recognition error and reject tradeoff

Chow, C.-K. (1970) · IEEE Transactions on Information Theory · doi:10.1109/tit.1970.1054406 · Cited by 1,303

Weighted extreme learning machine for imbalance learning

Zong, W., Huang, G.-B., & Chen, Y. (2012) · Neurocomputing · doi:10.1016/j.neucom.2012.08.010 · Cited by 750

An Introduction to Conditional Random Fields

Sutton, C. A., & McCallum, A. (2012) · now publishers, Inc. eBooks · doi:10.1561/2200000013 · Cited by 734

A method of combining multiple experts for the recognition of unconstrained handwritten numerals

Huang, Y., & Suen, C. Y. (1995) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.368145 · Cited by 565

Second opinion needed: Communicating uncertainty in medical machine learning

Kompa, B., Snoek, J., & Beam, A. L. (2021) · npj Digital Medicine · doi:10.1038/s41746-020-00367-3 · Cited by 514

Generalized ODIN: Detecting Out-of-Distribution Image Without Learning From Out-of-Distribution Data

Hsu, Y.-C., Shen, Y., Jin, H., & Kira, Z. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01096 · Cited by 499

A comprehensive survey and analysis of generative models in machine learning

Harshvardhan, G. M., Gourisaria, M. K., Pandey, M., & Rautaray, S. S. (2020) · Computer Science Review · doi:10.1016/j.cosrev.2020.100285 · Cited by 477

A survey on instance selection for active learning

Fu, Y., Xingquan Zhu, A., & Li, B. (2012) · Knowledge and Information Systems · doi:10.1007/s10115-012-0507-8 · Cited by 377

To Trust Or Not To Trust A ClassifierPreprint

Jiang, H., Kim, B., Guan, M. Y., & Gupta, M. R. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.11783 · Cited by 340

Classification with a Reject Option using a Hinge Loss

Bartlett, P. L., & Wegkamp, M. H. (2008) · QUT ePrints (Queensland University of Technology) · doi:10.5555/1390681.1442792 · Cited by 326

Democracy in neural nets: Voting schemes for classification

Battiti, R., & Colla, A. M. (1994) · Neural Networks · doi:10.1016/0893-6080(94)90046-9 · Cited by 308

Optimal combinations of pattern classifiers

Lam, L., & Suen, C. Y. (1995) · Pattern Recognition Letters · doi:10.1016/0167-8655(95)00050-q · Cited by 307

MetaFraud: A Meta-Learning Framework for Detecting Financial Fraud1

Abbasi, A., Albrecht, C., Vance, A., & Hansen, J. (2012) · MIS Quarterly · doi:10.2307/41703508 · Cited by 298

Metalearning

Brazdil, P., Giraud-Carrier, C. G., Soares, C., & Vilalta, R. (2008) · Cognitive technologies · doi:10.1007/978-3-540-73263-1 · Cited by 268

Concrete DropoutPreprint

Gal, Y., Hron, J., & Kendall, A. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.07832 · Cited by 242

To reject or not to reject: that is the question-an answer in case of neural classifiers

Stefano, C. D., Sansone, C., & Vento, M. (2000) · IEEE Transactions on Systems Man and Cybernetics Part C (Applications and Reviews) · doi:10.1109/5326.827457 · Cited by 226

Selective Classification for Deep Neural Networks

Geifman, Y., & El-Yaniv, R. (2017) · arXiv · doi:10.48550/arxiv.1705.08500 · Cited by 220

The Nearest Neighbor Classification Rule with a Reject Option

Hellman, M. E. (1970) · IEEE Transactions on Systems Science and Cybernetics · doi:10.1109/tssc.1970.300339 · Cited by 219

Classification with reject option

Herbei, R., & Wegkamp, M. H. (2006) · Canadian Journal of Statistics · doi:10.1002/cjs.5550340410 · Cited by 203

Handwritten zip code recognition with multilayer networks

LeCun, Y., Matan, O., Boser, B. E., Denker, J. S., Henderson, D., Howard, R. E., Hubbard, W., Jacket, L. D., & Baird, H. S. (2002) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr.1990.119325 · Cited by 198

On the error-reject trade-off in biometric verification systems

Golfarelli, M., Maio, D., & Malton, D. (1997) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.598237 · Cited by 193

A statistical decision rule with incomplete knowledge about classes

Dubuisson, B., & Masson, M. (1993) · Pattern Recognition · doi:10.1016/0031-3203(93)90097-g · Cited by 183

On the Foundations of Noise-free Selective Classification

El‐Yaniv, R., & Wiener, Y. (2010) · Journal of Machine Learning Research · Cited by 170

Learning to Classify Ordinal Data: The Data Replication Method

Cardoso, J. S., & da Costa, J. P. (2007) · doi:10.5555/1314498.1314546 · Cited by 162

Learning with Rejection

Cortes, C., DeSalvo, G., & Mohri, M. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46379-7_5 · Cited by 162

SelectiveNet: A Deep Neural Network with an Integrated Reject OptionPreprint

Geifman, Y., & El‐Yaniv, R. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.09192 · Cited by 157

Moderating the outputs of support vector machine classifiers

Kwok, J. T.-Y. (1999) · IEEE Transactions on Neural Networks · doi:10.1109/72.788642 · Cited by 155

Support Vector Machines with Embedded Reject Option

Fumera, G., & Roli, F. (2002) · Lecture notes in computer science · doi:10.1007/3-540-45665-1_6 · Cited by 151

Reliable classification: Learning classifiers that distinguish aleatoric and epistemic uncertainty

Senge, R., Bösner, S., Dembczyński, K., Haasenritter, J., Hirsch, O., Donner‐Banzhoff, N., & Hüllermeier, E. (2013) · Information Sciences · doi:10.1016/j.ins.2013.07.030 · Cited by 146

Gaussian Process Regression: Active Data Selection and Test Point Rejection

Seo, S., Wallat, M., Graepel, T., & Obermayer, K. (2000) · Informatik aktuell · doi:10.1007/978-3-642-59802-9_4 · Cited by 145

Reliable Classification of Vehicle Types Based on Cascade Classifier Ensembles

Zhang, B. (2012) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2012.2213814 · Cited by 135

Self-Adaptive Training: beyond Empirical Risk MinimizationPreprint

Huang, L., Zhang, C., & Change Institutions to: University of Waterloo (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.10319 · Cited by 132

Classification with confidence

Lei, J. (2014) · Biometrika · doi:10.1093/biomet/asu038 · Cited by 131

Blocking Transferability of Adversarial Examples in Black-Box Learning SystemsPreprint

Hosseini, H., Chen, Y., Kannan, S., Zhang, B., & Poovendran, R. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.04318 · Cited by 123

An approach to novelty detection applied to the classification of image regions

Singh, S., & Markou, M. (2004) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2004.1269665 · Cited by 117

Growing a multi-class classifier with a reject option

Tax, D. M. J., & Duin, R. P. W. (2008) · Pattern Recognition Letters · doi:10.1016/j.patrec.2008.03.010 · Cited by 117

A method for improving classification reliability of multilayer perceptrons

Cordella, L. P., Stefano, C. D., Tortorella, F., & Vento, M. (1995) · IEEE Transactions on Neural Networks · doi:10.1109/72.410358 · Cited by 112

Combating Label Noise in Deep Learning Using AbstentionPreprint

Thulasidasan, S., Tanmoy Bhattacharya, A., Bilmes, J., Chennupati, G., & Mohd-Yusof, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.10964 · Cited by 111

Support Vector Machines with a Reject Option

Grandvalet, Y., Rakotomamonjy, A., Keshet, J., & Canu, S. (2008) · Infoscience (Ecole Polytechnique Fédérale de Lausanne) · Cited by 109

Evidence-based uncertainty sampling for active learning

Sharma, M., & Bilgic, M. (2016) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-016-0460-3 · Cited by 108

Online ensemble learning with abstaining classifiers for drifting and noisy data streams

Krawczyk, B., & Cano, A. (2017) · Applied Soft Computing · doi:10.1016/j.asoc.2017.12.008 · Cited by 107

Gaussian process regression: active data selection and test point rejection

Seo, S., Wallat, M., Graepel, T., & Obermayer, K. (2000) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2000.861310 · Cited by 96

Classification Methods with Reject Option Based on Convex Risk Minimization

Yuan, M., & Wegkamp, M. H. (2010) · Cited by 96

The optimum class-selective rejection rule

Ha, T. M. (1997) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.601248 · Cited by 94

Rotation-blended CNNs on a New Open Dataset for Tropical Cyclone Image-to-intensity Regression

Chen, B., Chen, B., & Lin, H.-T. (2018) · ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD) · doi:10.1145/3219819.3219926 · Cited by 92

Predict Responsibly: Improving Fairness and Accuracy by Learning to Defer

Madras, D., Pitassi, T., & Zemel, R. (2017) · arXiv · doi:10.48550/arxiv.1711.06664 · Cited by 90

The interaction between classification and reject performance for distance-based reject-option classifiers

Landgrebe, T. C. W., Tax, D. M. J., Paclı́k, P., & Duin, R. P. W. (2006) · Pattern Recognition Letters · doi:10.1016/j.patrec.2005.10.015 · Cited by 83

Semi-Supervised Anomaly Detection with an Application to Water Analytics

Vercruyssen, V., Meert, W., Verbruggen, G., Maes, K., Bäumer, R., & Davis, J. J. (2018) · Proceedings · doi:10.1109/icdm.2018.00068 · Cited by 78

Delegating classifiers

Ferri, C., Flach, P. A., & Hernández‐Orallo, J. (2004) · International Conference on Machine Learning · doi:10.1145/1015330.1015395 · Cited by 78

Classification with reject option in gene expression data

Hanczar, B., & Dougherty, E. R. (2008) · Bioinformatics · doi:10.1093/bioinformatics/btn349 · Cited by 76

Diagnostic of Pathology on the Vertebral Column with Embedded Reject Option

Neto, A. R. R., Sousa, R., Barreto, G. A., & Cardoso, J. S. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-21257-4_73 · Cited by 76

Accuracy-Rejection Curves (ARCs) for Comparing Classification Methods with a Reject Option

Nadeem, M. S. A., Zucker, J., & Hanczar, B. (2009) · Cited by 76

Multi-label classification with a reject option

Pillai, I., Fumera, G., & Roli, F. (2013) · Pattern Recognition · doi:10.1016/j.patcog.2013.01.035 · Cited by 75

Sleep Versus Wake Classification From Heart Rate Variability Using Computational Intelligence: Consideration of Rejection in Classification Models

Lewicke, A., Sazonov, E. E., Corwin, M. J., Neuman, M. R., & Schuckers, S. (2008) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2007.900558 · Cited by 74

Multistage pattern recognition with reject option

Pudil, P., Novovičová, J., Bláha, S., & Kittler, J. (2003) · IAPR International Conference on Pattern Recognition (ICPR) · doi:10.1109/icpr.1992.201729 · Cited by 73

Highly Accurate Recognition of Human Postures and Activities Through Classification With Rejection

Tang, W., & Sazonov, E. E. (2014) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2013.2287400 · Cited by 71

Deep neural rejection against adversarial examples

Sotgiu, A., Demontis, A., Melis, M., Biggio, B., Fumera, G., Feng, X., & Roli, F. (2020) · EURASIP Journal on Information Security · doi:10.1186/s13635-020-00105-y · Cited by 68

Confidence Estimation via Auxiliary Models

Corbière, C., Thome, N., Saporta, A., Vu, T.-H., Cord, M., & Pérez, P. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3085983 · Cited by 62

Confidence-based classifier design

Li, M., & Sethi, I. K. (2006) · Pattern Recognition · doi:10.1016/j.patcog.2006.01.010 · Cited by 61

Consistent Estimators for Learning to Defer to an Expert

Mozannar, H., & Sontag, D. (2020) · arXiv · doi:10.48550/arxiv.2006.01862 · Cited by 61

Untitled work

(2019) · Cited by 60

Support vector machines with a reject optionPreprint

Wegkamp, M. H., & Yuan, M. (2011) · Cited by 58

An Optimal Reject Rule for Binary Classifiers

Tortorella, F. (2000) · Lecture notes in computer science · doi:10.1007/3-540-44522-6_63 · Cited by 56

On the use of ROC analysis for the optimization of abstaining classifiers

Pietraszek, T. (2007) · Machine Learning · doi:10.1007/s10994-007-5013-y · Cited by 56

Beyond Disagreement-based Agnostic Active LearningPreprint

Zhang, C., & Chaudhuri, K. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1407.2657 · Cited by 53

Optimizing abstaining classifiers using ROC analysis

Pietraszek, T. (2005) · International Conference on Machine Learning · doi:10.1145/1102351.1102435 · Cited by 52

Classifiers with a reject option for early time-series classification

Hatami, N., & Chira, C. (2013) · doi:10.1109/ciel.2013.6613134 · Cited by 51

A support vector machines-based rejection technique for speech recognition

Ma, C., Randolph, M. A., & Drish, J. (2002) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2001.940847 · Cited by 51

Deep Gamblers: Learning to Abstain with Portfolio TheoryPreprint

Ziyin, L., Wang, Z., Liang, P. P., Salakhutdinov, R. R., Morency, L., & Ueda, M. (2019) · arXiv (Cornell University) · Cited by 49

AN EVALUATION OF MULTI-EXPERT CONFIGURATIONS FOR THE RECOGNITION OF HANDWRITTEN NUMERALS

Rahman, F., & Fairhurst, M. C. (1998) · Pattern Recognition · doi:10.1016/s0031-3203(97)00161-1 · Cited by 49

Deep Gamblers: Learning to Abstain with Portfolio TheoryPreprint

Ziyin, L., Wang, Z., Liang, P. P., Salakhutdinov, R., Morency, L., & Ueda, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1907.00208 · Cited by 48

Epistemic Uncertainty Sampling

Nguyen, V.-L., Destercke, S., & Hüllermeier, E. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-33778-0_7 · Cited by 48

Analysis of error-reject trade-off in linearly combined multiple classifiers

Fumera, G., & Roli, F. (2004) · Pattern Recognition · doi:10.1016/j.patcog.2003.12.005 · Cited by 46

Classification with reject option for software defect prediction

Mesquita, D. P. P., Rocha, L. S., Gomes, J. P. P., & Neto, A. R. R. (2016) · Applied Soft Computing · doi:10.1016/j.asoc.2016.06.023 · Cited by 45

On optimal reject rules and ROC curves

Pereira, C. S., & Pires, A. M. (2004) · Pattern Recognition Letters · doi:10.1016/j.patrec.2004.09.042 · Cited by 43

Investigating feedforward neural networks with respect to the rejection of spurious patterns

Vasconcelos, G. C., Fairhurst, M. C., & Bisset, D. L. (1995) · Pattern Recognition Letters · doi:10.1016/0167-8655(94)00092-h · Cited by 42

Datasets meta-feature description for recommending feature selection algorithm

Filchenkov, A. A., & Pendryak, A. (2015) · Artificial Intelligence and Natural Language · doi:10.1109/ainl-ismw-fruct.2015.7382962 · Cited by 42

Boosting with Abstention

Cortes, C., DeSalvo, G., & Mohri, M. (2016) · neural information processing systems · Cited by 39

A ROC-based reject rule for dichotomizers

Tortorella, F. (2004) · Pattern Recognition Letters · doi:10.1016/j.patrec.2004.09.004 · Cited by 39

Optimal local rejection for classifiers

Fischer, L., Hammer, B., & Wersing, H. (2016) · Neurocomputing · doi:10.1016/j.neucom.2016.06.038 · Cited by 39

Classifier design with incomplete knowledge

Muzzolini, R., Yang, Y., & Pierson, R. A. (1998) · Pattern Recognition · doi:10.1016/s0031-3203(97)00056-3 · Cited by 38

Design of reject rules for ECOC classification systems

Simeone, P., Marrocco, C., & Tortorella, F. (2011) · Pattern Recognition · doi:10.1016/j.patcog.2011.08.001 · Cited by 37

Performance measures for classification systems with rejection

Condessa, F., Bioucas‐Dias, J. M., & Kovačević, J. (2016) · Pattern Recognition · doi:10.1016/j.patcog.2016.10.011 · Cited by 37

An Ensemble of Rejecting Classifiers for Anomaly Detection of Audio Events

Conte, D., Foggia, P., Percannella, G., Saggese, A., & Vento, M. (2012) · IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS) · doi:10.1109/avss.2012.9 · Cited by 37

Active Learning with Abstaining Classifiers for Imbalanced Drifting Data Streams

Korycki, Ł., Cano, A., & Krawczyk, B. (2019) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata47090.2019.9006453 · Cited by 36

Multiclassification: reject criteria for the Bayesian combiner

Foggia, P., Sansone, C., Tortorella, F., & Vento, M. (1999) · Pattern Recognition · doi:10.1016/s0031-3203(98)00169-1 · Cited by 35

Agnostic Selective Classification

Wiener, Y., & El‐Yaniv, R. (2011) · neural information processing systems · Cited by 34

Reject option for VQ-based Bayesian classification

Vailaya, A., & Jain, A. (2002) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr.2000.906016 · Cited by 34

Rejection strategies and confidence measures for a k-NN classifier in an OCR task

Arlandis, J., Pérez-Cortés, J.-C., & Cano, J. (2003) · doi:10.1109/icpr.2002.1044806 · Cited by 33

Efficient rejection strategies for prototype-based classification

Fischer, L., Hammer, B., & Wersing, H. (2015) · Neurocomputing · doi:10.1016/j.neucom.2014.10.092 · Cited by 33

Classification with reject option in text categorisation systems

Fumera, G., Pillai, I., & Roli, F. (2004) · International Conference on Image Analysis and Processing · doi:10.1109/iciap.2003.1234113 · Cited by 33

Machine Learning with Abstention for Automated Liver Disease Diagnosis

Hamid, K., Asif, A., Abbasi, W. A., Sabih, D., & Minhas, F. (2017) · International Conference on Frontiers of Information Technology (FIT) · doi:10.1109/fit.2017.00070 · Cited by 33

Consistent algorithms for multiclass classification with an abstain option

Ramaswamy, H. G., Tewari, A., & Agarwal, S. (2018) · Electronic Journal of Statistics · doi:10.1214/17-ejs1388 · Cited by 32

Multiple Reject Thresholds for Improving Classification Reliability

Fumera, G., Roli, F., & Giacinto, G. (2000) · Lecture notes in computer science · doi:10.1007/3-540-44522-6_89 · Cited by 32

Reducing the classification cost of support vector classifiers through an ROC-based reject rule

Tortorella, F. (2004) · Pattern Analysis and Applications · doi:10.1007/s10044-004-0209-2 · Cited by 32

Application of optimum error-reject functions (Corresp.)

Fukunaga, K., & Kessell, D. (1972) · IEEE Transactions on Information Theory · doi:10.1109/tit.1972.1054919 · Cited by 32

Uncertainty-Based Rejection in Machine Learning: Implications for Model Development and Interpretability

Barandas, M., Folgado, D., Santos, R., Simão, R., & Gamboa, H. F. S. (2022) · Electronics · doi:10.3390/electronics11030396 · Cited by 31

Siamese neural network based similarity metric for inertial gesture classification and rejection

Berlemont, S., Lefebvre, G., Duffner, S., & García, C. (2015) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg.2015.7163112 · Cited by 31

Uncertainty-Aware Attention for Reliable Interpretation and PredictionPreprint

Heo, J., Lee, H., Kim, S., Lee, J., Kim, K. J., Yang, E., & Hwang, S. J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.09653 · Cited by 31

Uncertainty-Based Rejection Wrappers for Black-Box Classifiers

Mena, J., Pujol, O., & Vitria, J. M. (2020) · IEEE Access · doi:10.1109/access.2020.2996495 · Cited by 27

Improvement of reliability in banknote classification using reject option and local PCA

Ahmadi, A., Omatu, S., Fujinaka, T., & Kosaka, T. (2004) · Information Sciences · doi:10.1016/j.ins.2004.02.018 · Cited by 27

Lasso type classifiers with a reject option

Wegkamp, M. H. (2007) · Electronic Journal of Statistics · doi:10.1214/07-ejs058 · Cited by 26

CLASS-SELECTIVE REJECTION RULE TO MINIMIZE THE MAXIMUM DISTANCE BETWEEN SELECTED CLASSES

Horiuchi, T. (1998) · Pattern Recognition · doi:10.1016/s0031-3203(97)00136-2 · Cited by 26

Linear Classifier with Reject Option for the Detection of Vocal Fold Paralysis and Vocal Fold Edema

Kotropoulos, C. L., & Arce, G. R. (2009) · EURASIP Journal on Advances in Signal Processing · doi:10.1155/2009/203790 · Cited by 26

Online Learning with AbstentionPreprint

Cortes, C., DeSalvo, G., Gentile, C., Mohri, M., & Yang, S. C. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.03478 · Cited by 24

Two Coupled Rejection Metrics Can Tell Adversarial Examples Apart

Pang, T., Zhang, H., He, D., Dong, Y., Su, H., Chen, W. R., Zhu, J., & Liu, T. Y. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01479 · Cited by 24

RO-SVM: Support Vector Machine with Reject Option for Image Categorization

Zhang, R., & Metaxas, D. (2006) · British Machine Vision Conference (BMVC) · doi:10.5244/c.20.123 · Cited by 23

Robust classification with reject option using the self-organizing map

Sousa, R., Neto, A. R. R., Cardoso, J. S., & Barreto, G. A. (2015) · Neural Computing and Applications · doi:10.1007/s00521-015-1822-2 · Cited by 23

Performance visualization spaces for classification with rejection option

Hanczar, B. (2019) · Pattern Recognition · doi:10.1016/j.patcog.2019.106984 · Cited by 23

Quantifying the Confidence of Anomaly Detectors in Their Example-Wise Predictions

Perini, L., Vercruyssen, V., & Davis, J. J. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-67664-3_14 · Cited by 22

Know Your Limits: Uncertainty Estimation with ReLU Classifiers Fails at Reliable OOD DetectionPreprint

Ulmer, D., & Cinà, G. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.05329 · Cited by 22

Consistency of plug-in confidence sets for classification in semi-supervised learning

Denis, C., & Hebiri, M. (2019) · Journal of nonparametric statistics · doi:10.1080/10485252.2019.1689241 · Cited by 22

Gradient descent learning of nearest neighbor classifiers with outlier rejection

Urahama, K., & Furukawa, Y. (1995) · Pattern Recognition · doi:10.1016/0031-3203(94)00142-9 · Cited by 22

On the Calibration of Multiclass Classification with RejectionPreprint

Ni, C., Charoenphakdee, N., Honda, J., & Sugiyama, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.10655 · Cited by 21

Classification with Rejection Based on Cost-sensitive ClassificationPreprint

Charoenphakdee, N., Cui, Z., Zhang, Y., & Sugiyama, M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.11748 · Cited by 21

Reject-Optional LVQ-Based Two-Level Classifier to Improve Reliability in Footstep Identification

Suutala, J., Pirttikangas, S., Riekki, J. P., & Röning, J. (2004) · Lecture notes in computer science · doi:10.1007/978-3-540-24646-6_12 · Cited by 20

A Classification Approach with a Reject Option for Multi-label Problems

Pillai, I., Fumera, G., & Roli, F. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-24085-0_11 · Cited by 19

Interpretable machine learning with reject option

Brinkrolf, J., & Hammer, B. (2018) · at - Automatisierungstechnik · doi:10.1515/auto-2017-0123 · Cited by 18

An Ordinal Data Method for the Classification with Reject Option

Sousa, R., Mora, B., & Cardoso, J. S. (2009) · International Conference on Machine Learning and Applications · doi:10.1109/icmla.2009.11 · Cited by 18

Self-Adjusting Reject Options in Prototype Based Classification

Villmann, T., Kaden, M., Bohnsack, A., Villmann, J.-M., Drogies, T., Saralajew, S., & Hammer, B. (2016) · Advances in intelligent systems and computing · doi:10.1007/978-3-319-28518-4_24 · Cited by 18

A Model-Agnostic Heuristics for Selective Classification

Pugnana, A., & Ruggieri, S. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i8.26133 · Cited by 18

Reject Strategies Driven Combination of Pattern Classifiers

Mascarilla, L., & Frélicot, C. (2002) · Pattern Analysis and Applications · doi:10.1007/s100440200021 · Cited by 17

Multi-level Classification of Emphysema in HRCT Lung Images Using Delegated Classifiers

Prasad, M. N., & Sowmya, A. (2008) · Lecture notes in computer science · doi:10.1007/978-3-540-85988-8_8 · Cited by 17

Multiple classifier combination using reject options and markov fusion networks

Glodek, M., Schels, M., Palm, G., & Schwenker, F. (2012) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/2388676.2388778 · Cited by 17

On Reject and Refine Options in Multicategory Classification

Zhang, C., Wang, W., & Qiao, X. (2017) · Journal of the American Statistical Association · doi:10.1080/01621459.2017.1282372 · Cited by 17

Bayesian Active Meta-Learning for Reliable and Efficient AI-Based Demodulation

Cohen, K., Park, S., Simeone, O., & Shamai, S. (2021) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2022.3220035 · Cited by 16

Rejection Strategies for Learning Vector Quantization – A Comparison of Probabilistic and Deterministic Approaches

Fischer, L., Nebel, D., Villmann, T., Hammer, B., & Wersing, H. (2014) · Advances in intelligent systems and computing · doi:10.1007/978-3-319-07695-9_10 · Cited by 16

Classification with rejection based on various SVM techniques

Homenda, W., Luckner, M., & Pedrycz, W. (2014) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2014.6889655 · Cited by 16

Mitigating Concept Drift via Rejection

Göpfert, J. P., Hammer, B., & Wersing, H. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01418-6_45 · Cited by 16

Hyperparameter Optimization

Gridin, I. (2022) · Apress eBooks · doi:10.1007/978-1-4842-8149-9_2 · Cited by 16

Pattern rejection strategies for the design of self-paced EEG-based Brain-Computer Interfaces

Lotte, F., Mouchère, H., & Lécuyer, A. (2008) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr.2008.4761454 · Cited by 15

Classification with reject option using contextual information

Condessa, F., Bioucas‐Dias, J. M., Castro, C. A., Ozolek, J. A., & Kovačević, J. (2013) · IEEE International Symposium on Biomedical Imaging (ISBI) · doi:10.1109/isbi.2013.6556780 · Cited by 15

Transferring the Contamination Factor between Anomaly Detection Domains by Shape Similarity

Perini, L., Vercruyssen, V., & Davis, J. J. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i4.20331 · Cited by 15

Background Check: A General Technique to Build More Reliable and Versatile Classifiers

Perelló-Nieto, M., Filho, T. M. S., Kull, M., & Flach, P. A. (2016) · Proceedings · doi:10.1109/icdm.2016.0150 · Cited by 14

HEARTBEAT CLASSIFICATION USING SUPPORT VECTOR MACHINES (SVMs) WITH AN EMBEDDED REJECT OPTION

ZIDELMAL, Z., AMIROU, A., & Belouchrani, A. (2012) · International Journal of Pattern Recognition and Artificial Intelligence · doi:10.1142/s0218001412500012 · Cited by 14

A Kernel Based Rejection Method For Supervised Classification

Bounsiar, A., Grall‐Maës, E., & Beauseroy, P. (2007) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.1083044 · Cited by 14

Trading off Mistakes and Don’t-Know Predictions

Sayedi, A., Zadimoghaddam, M., & Blum, A. L. (2018) · Figshare · doi:10.1184/r1/6611060 · Cited by 14

A risk bound for ensemble classification with a reject option

Varshney, K. R. (2011) · IEEE Statistical Signal Processing Workshop (SSP) · doi:10.1109/ssp.2011.5967817 · Cited by 14

Supervised Hyperspectral Image Classification With Rejection

Condessa, F., Bioucas‐Dias, J. M., & Kovacevic, J. (2016) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2015.2510032 · Cited by 14

An Optimum Character Recognition System Using Decision Functions

Flores, I. (1958) · IEEE Transactions on Electronic Computers · doi:10.1109/tec.1958.5222530 · Cited by 14

Twin SVM with a reject option through ROC curve

Lin, D., Сун, Л., Toh, K., Zhang, J. B., & Lin, Z. (2017) · Journal of the Franklin Institute · doi:10.1016/j.jfranklin.2017.05.003 · Cited by 14

Reliable Multilabel Classification: Prediction with Partial Abstention

Nguyen, V.-L., & Hüllermeier, E. (2020) · AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) · doi:10.1609/aaai.v34i04.5972 · Cited by 14

COBRASTS: A New Approach to Semi-supervised Clustering of Time Series

Craenendonck, T. V., Meert, W., Dumančić, S., & Blockeel, H. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01771-2_12 · Cited by 14

Trading off coverage for accuracy in forecasts: Applications to clinical data analysis

Pazzani, M. J., Murphy, P. M., Ali, K., & Schulenburg, D. (1994) · Cited by 13

An optimum class-selective rejection rule for pattern recognition

HA, T. M. (1996) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr.1996.546727 · Cited by 13

Reject Options and Confidence Measures for kNN Classifiers

Dalitz, C. (2009) · Cited by 13

Nonparametric Uncertainty Quantification for Single Deterministic Neural Network

Kotelevskii, N., Artemenkov, A., Fedyanin, K., Noskov, F., Fishkov, A., Shelmanov, A., Vazhentsev, A., Petiushko, A., & Panov, M. (2022) · neural information processing systems · doi:10.52202/068431-2631 · Cited by 13

A family of measures for best top-n class-selective decision rules

Capitaine, H. L., & Frélicot, C. (2011) · Pattern Recognition · doi:10.1016/j.patcog.2011.04.030 · Cited by 12

Local Rejection Strategies for Learning Vector Quantization

Fischer, L., Hammer, B., & Wersing, H. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-11179-7_71 · Cited by 12

The data replication method for the classification with reject option

Sousa, R., & Cardoso, J. S. (2013) · AI Communications · doi:10.3233/aic-130566 · Cited by 12

Combination of One-Class Support Vector Machines for Classification with Reject Option

Hanczar, B., & Sébag, M. (2014) · Lecture notes in computer science · doi:10.1007/978-3-662-44848-9_35 · Cited by 12

A perspective on artificial intelligence: Learning to learn

Bock, P. (1988) · Annals of Operations Research · doi:10.1007/bf02283734 · Cited by 12

A CLASSIFICATION RELIABILITY DRIVEN REJECT RULE FOR MULTI-EXPERT SYSTEMS

Sansone, C., Tortorella, F., & Vento, M. (2001) · International Journal of Pattern Recognition and Artificial Intelligence · doi:10.1142/s0218001401001210 · Cited by 12

General solution and learning method for binary classification with performance constraints

Bounsiar, A., Beauseroy, P., & Grall‐Maës, E. (2008) · Pattern Recognition Letters · doi:10.1016/j.patrec.2008.02.025 · Cited by 12

Reliable prediction of anti-diabetic drug failure using a reject option

Kang, S., Cho, S., Rhee, S., & Yu, K. (2016) · Pattern Analysis and Applications · doi:10.1007/s10044-016-0585-4 · Cited by 12

Estimating the Contamination Factor's Distribution in Unsupervised Anomaly DetectionPreprint

Perini, L., Buerkner, P., & Arto Klami, A. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2210.10487 · Cited by 12

Bounded-abstaining classification for breast tumors in imbalanced ultrasound images

Guan, H., Zhang, Y., Cheng, H.-D., & Tang, X. (2020) · International Journal of Applied Mathematics and Computer Science · doi:10.34768/amcs-2020-0025 · Cited by 11

Probability of default estimation, with a reject option

Coenen, L., Abdullah, A. K. A., & Guns, T. (2020) · IEEE International Conference on Data Science and Advanced Analytics (DSAA) · doi:10.1109/dsaa49011.2020.00058 · Cited by 11

Can We Trust Fair-AI?

Ruggieri, S., Álvarez, J. M., Pugnana, A., State, L., & Turini, F. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i13.26798 · Cited by 11

A Two-Stage Classifier with Reject Option for Text Categorisation

Fumera, G., Pillai, I., & Roli, F. (2004) · Lecture notes in computer science · doi:10.1007/978-3-540-27868-9_84 · Cited by 11

On unifying probabilistic/fuzzy and possibilistic rejection-based classifiers

Frélicot, C. (1998) · Lecture notes in computer science · doi:10.1007/bfb0033298 · Cited by 11

Rejection Criteria and Pairwise Discrimination of Handwritten Numerals Based on Structural Features

Lou, Z., Liu, K., Yang, J., & Suen, C. Y. (1999) · Pattern Analysis and Applications · doi:10.1007/s100440050031 · Cited by 11

SafePredict: A Meta-Algorithm for Machine Learning That Uses Refusals to Guarantee Correctness

Kocak, M., Ramirez, D., Erkip, E., & Shasha, D. E. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2019.2932415 · Cited by 11

On discriminative learning of prediction uncertainty

Franc, V., & Průša, D. (2019) · International Conference on Machine Learning · Cited by 11

Reliable Classifiers in ROC Space

Vanderlooy, S., Sprinkhuizen-Kuyper, I. G., & Smirnov, E. (2006) · Cited by 11

Optimal strategies for reject option classifiersPreprint

Franc, V., Prusa, D., & Voràček, V. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.12523 · Cited by 11

Classification Under Human Assistance

De, A., Okati, N., Zarezade, A., & Gomez-Rodriguez, M. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i7.16738 · Cited by 11

A meta-learning BCI for estimating decision confidence

Tremmel, C., Fernández-Vargas, J., Stamos, D., Cinel, C., Pontil, M., Citi, L., & Poli, R. (2022) · Journal of Neural Engineering · doi:10.1088/1741-2552/ac7ba8 · Cited by 11

Optimum tradeoff between class-selective rejection error and average number of classes

HA, T. M. (1997) · Engineering Applications of Artificial Intelligence · doi:10.1016/s0952-1976(97)00034-1 · Cited by 10

Classification with a reject option under Concept Drift: The Droplets algorithm

Loeffel, P.-X., Marsala, C., & Detyniecki, M. (2015) · IEEE International Conference on Data Science and Advanced Analytics (DSAA) · doi:10.1109/dsaa.2015.7344808 · Cited by 10

An Optimum Class-Rejective Decision Rule and Its Evaluation

Capitaine, H. L., & Frélicot, C. (2010) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr.2010.810 · Cited by 10

Multilabel Classification with Partial Abstention: Bayes-Optimal Prediction under Label Independence

Nguyen, V.-L., & Hüllermeier, E. (2021) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.12610 · Cited by 10

An adaptive reject option for LVQ classifiers

Cordella, L. P., Stefano, C. D., Sansone, C., & Vento, M. (1995) · Lecture notes in computer science · doi:10.1007/3-540-60298-4_238 · Cited by 9

A Novel Classification-Rejection Sphere SVMs for Multi-class Classification Problems

Wu, Q., Jia, C., & Chen, W. (2007) · International Conference on Natural Computation (ICNC) · doi:10.1109/icnc.2007.109 · Cited by 9

A Ranking Stability Measure for Quantifying the Robustness of Anomaly Detection Methods

Perini, L., Galvin, C., & Vercruyssen, V. (2020) · Communications in computer and information science · doi:10.1007/978-3-030-65965-3_27 · Cited by 9

An Analysis of Reliable Classifiers through ROC Isometrics

Vanderlooy, S., Sprinkhuizen-Kuyper, I. G., & Smirnov, E. (2006) · Radboud Repository (Radboud University) · Cited by 9

Versatile Verification of Tree EnsemblesPreprint

Devos, L., Meert, W., & Davis, J. J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.13880 · Cited by 9

An Empirical Comparison of Ideal and Empirical ROC-Based Reject Rules

Marrocco, C., Molinara, M., & Tortorella, F. (2007) · Lecture notes in computer science · doi:10.1007/978-3-540-73499-4_5 · Cited by 8

Cost-sensitive Multi-class SVM with Reject Option: A Method for Steam Turbine Generator Fault Diagnosis

Zou, C., Zheng, E., Xu, H., & Chen, L. (2011) · International Journal of Computer Theory and Engineering · doi:10.7763/ijcte.2011.v3.286 · Cited by 8

Optimizing the error/reject trade-off for a multi-expert system using the Bayesian combining rule

Cordella, L. P., Foggia, P., Sansone, C., Tortorella, F., & Vento, M. (1998) · Lecture notes in computer science · doi:10.1007/bfb0033296 · Cited by 8

Enhanced reliability of multilayer perceptron networks through controlled pattern rejection

Vasconcelos, G. C., Fairhurst, M. C., & Bisset, D. L. (1993) · Electronics Letters · doi:10.1049/el:19930179 · Cited by 8

Bootstrap Methods for Reject Rules of Fisher LDA

Xie, J., Qiu, Z., & Wu, J. (2006) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr.2006.338 · Cited by 8

Accuracy Rejection Normalized-Cost Curves (ARNCCs): A Novel 3-Dimensional Framework for Robust Classification

Abbas, M. R., Nadeem, M. S. A., Shaheen, A., Alshdadi, A. A., Alharbey, R., Shim, S., & Aziz, W. (2019) · IEEE Access · doi:10.1109/access.2019.2950244 · Cited by 8

A Novel Meta Learning Framework for Feature Selection using Data Synthesis and Fuzzy Similarity

Shen, Z., Chen, X., & Garibaldi, J. M. (2020) · IEEE International Conference on Fuzzy Systems (FUZZ-IEEE) · doi:10.1109/fuzz48607.2020.9177769 · Cited by 8

Fast and Accurate $k$-means++ via Rejection SamplingPreprint

Cohen-Addad, V., Lattanzi, S., Norouzi-Fard, A., Sohler, C., & Svensson, O. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.11891 · Cited by 8

Combining Base-Learners into Ensembles

Giraud-Carrier, C. G. (2022) · Cognitive technologies · doi:10.1007/978-3-030-67024-5_9 · Cited by 8

ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings

Lee, I., Ko, H., & Han, D. K. (2002) · Cited by 8

Adversarial Training with Rectified RejectionPreprint

Pang, T., Zhang, H., He, D., Dong, Y., Su, H., Chen, W. R., Zhu, J., & Liu, T. (2021) · arXiv (Cornell University) · Cited by 8

Policy advice and best practices on bias and fairness in AI

Álvarez, J. M., Colmenarejo, A. B., Elobaid, A., Fabbrizzi, S., Fahimi, M., Ferrara, A., Ghodsi, S., Mougan, C., Papageorgiou, I. T., Lobo, P. R., et al. (2024) · Ethics and Information Technology · doi:10.1007/s10676-024-09746-w · Cited by 147

Methodology and evaluation in sports analytics: challenges, approaches, and lessons learned

Davis, J., Bransen, L., Devos, L., Jaspers, A., Meert, W., Robberechts, P., Haaren, J. V., & Roy, M. V. (2024) · Machine-mediated learning · doi:10.1007/s10994-024-06585-0 · Cited by 94

Towards Human-AI Teaming to Mitigate Alert Fatigue in Security Operations Centres

Chhetri, M. B., Tariq, S., Singh, R., Jalalvand, F., Paris, C. L., & Nepal‬, ‪. (2024) · ACM Transactions on Internet Technology · doi:10.1145/3670009 · Cited by 70

Bayesian autoencoders with uncertainty quantification: Towards trustworthy anomaly detection

Yong, B. X., & Brintrup, A. M. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.118196 · Cited by 55

A.I. Robustness: a Human-Centered Perspective on Technological Challenges and Opportunities

Tocchetti, A., Corti, L., Balayn, A., Yurrita, M., Lippmann, P., Brambilla, M., & Yang, J. (2024) · ACM Computing Surveys · doi:10.1145/3665926 · Cited by 77

The Science of Rejection: A Research Area for Human Computation

Sayin, B., Passerini, A., Casati, F., & Yang, J. (2021) · Institutional Research Information System (Università degli Studi di Trento)

Classification with Reject Option: Distribution-free Error Guarantees via Conformal Prediction

Szabadváry, J. H., Löfström, T., Johansson, U., Sönströd, C., Ahlberg, E., & Carlsson, L. (2025) · Machine Learning with Applications · doi:10.1016/j.mlwa.2025.100664 · Cited by 12

Reject Options for Incremental Regression Scenarios

Jakob, J., Hasenjäger, M., & Hammer, B. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-15937-4_21 · Cited by 1

Meta-Learning and Novelty Detection for Machine Learning with Reject Option

Drapal, P., Filho, T. M. S., & Prudêncio, R. B. C. (2024) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn60899.2024.10650424 · Cited by 3

Things Machine Learning Models Know That They Don’t Know

Ruggieri, S., & Pugnana, A. (2025) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v39i27.35094 · Cited by 2