Selective Classification for Deep Neural Networks
Open accessAbstract
Selective classification techniques (also known as reject option) have not yet been considered in the context of deep neural networks (DNNs). These techniques can potentially significantly improve DNNs prediction performance by trading-off coverage. In this paper we propose a method to construct a selective classifier given a trained neural network. Our method allows a user to set a desired risk level. At test time, the classifier rejects instances as needed, to grant the desired risk (with high probability). Empirical results over CIFAR and ImageNet convincingly demonstrate the viability of our method, which opens up possibilities to operate DNNs in mission-critical applications. For example, using our method an unprecedented 2% error in top-5 ImageNet classification can be guaranteed with probability 99.9%, and almost 60% test coverage.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Within the coding domain, the MEMRM gate transfers to memory strategies that did not appear in training (sliding-window, observation-masking, and structured memories), reaching AUROC ≈ 0.71 at ∼27% coverage on the cohorts retained by a pre-declared per-axis selection rule based on selective classification [12].
“Within the coding domain, the gate transfers to memory strategies that did not appear in training (sliding-window, observation-masking, and structured memories), reaching AUROC ≈ 0 . 71 at ∼ 27% coverage on the cohorts retained by a pre-declared per-axis selection rule based on selective classification [12].”
From MemGym: a Long-Horizon Memory Environment for LLM Agents (2026) · cites this paper as [12]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Zagoruyko, S., & Komodakis, N. (2016) · Procedings of the British Machine Vision Conference 2016 · doi:10.5244/c.30.87 · Cited by 7,244 · see citing works
Flagged first, then by citations
Research works printed but not linked: 17
- [1][1] Chao K Chow. An optimum character recognition system using decision functions. IRE Transactions on Electronic Computers , (4):247-254, 1957. -
- [2][2] Luigi Pietro Cordella, Claudio De Stefano, Francesco Tortorella, and Mario Vento. A method for improving classification reliability of multilayer perceptrons. IEEE Transactions on Neural Networks , 6(5):1140-1147, 1995. -
- [3][3] Corinna Cortes, Giulia DeSalvo, and Mehryar Mohri. Boosting with abstention. In Advances in Neural Information Processing Systems , pages 1660-1668, 2016. -
- [4][4] Claudio De Stefano, Carlo Sansone, and Mario Vento. To reject or not to reject: that is the question-an answer in case of neural classifiers. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews) , 30(1):84-94, 2000. -
- [5][5] R. El-Yaniv and Y. Wiener. On the foundations of noise-free selective classification. Journal of Machine Learning Research , 11:1605-1641, 2010. -
- [6][6] Ran El-Yaniv and Yair Wiener. Active learning via perfect selective classification. Journal of Machine Learning Research (JMLR) , 13(Feb):255-279, 2012. -
- [7][7] Yoav Freund, Yishay Mansour, and Robert E Schapire. Generalization bounds for averaged classifiers. Annals of Statistics , pages 1698-1722, 2004. -
- [8][8] Giorgio Fumera and Fabio Roli. Support vector machines with embedded reject option. In Pattern recognition with support vector machines , pages 68-82. Springer, 2002. -
- [9][9] Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: representing model uncertainty in deep learning. In Proceedings of The 33rd International Conference on Machine Learning , pages 1050-1059, 2016. -
- [10][10] O. Gascuel and G. Caraux. Distribution-free performance bounds with the resubstitution error estimate. Pattern Recognition Letters , 13:757-764, 1992. -
- [11][11] R. Gelbhart and R. El-Yaniv. The Relationship Between Agnostic Selective Classification and Active. ArXiv e-prints , January 2017. -
- [12][12] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 770-778, 2016. -
- [13][13] Martin E Hellman. The nearest neighbor classification rule with a reject option. IEEE Transactions on Systems Science and Cybernetics , 6(3):179-185, 1970. -
- [15][15] Shuying Liu and Weihong Deng. Very deep convolutional neural network based image classification using small training sample size. In Pattern Recognition (ACPR), 2015 3rd IAPR Asian Conference on , pages 730-734. IEEE, 2015. -
- [16][16] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li FeiFei. ImageNet large scale visual recognition challenge. International Journal of Computer Vision (IJCV) , 115(3):211-252, 2015. -
- [17][17] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 , 2014. - · arXiv
- [18][18] Kush R Varshney. A risk bound for ensemble classification with a reject option. In Statistical Signal Processing Workshop (SSP), 2011 IEEE , pages 769-772. IEEE, 2011. -
Cited by
Sorted by citations
A Survey on Evaluation of Large Language Models
Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., et al. (2024) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3641289 · Cited by 2,842 · see citing works
“By summarizing and analyzing existing efforts on LLMs evaluation, we may identify the current success and failure cases of LLMs, derive new trends for evaluation protocols, and most importantly, propose new challenges and opportunities for future research.”
BackgroundCan You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift
Ovadia, Y., Fertig, E., Ren, J., Nado, Z., Sculley, D., Nowozin, S., Dillon, J. V., Lakshminarayanan, B., & Snoek, J. (2019) · arXiv · doi:10.48550/arxiv.1906.02530 · Cited by 989 · see citing works
Second opinion needed: Communicating uncertainty in medical machine learning
Kompa, B., Snoek, J., & Beam, A. L. (2021) · npj Digital Medicine · doi:10.1038/s41746-020-00367-3 · Cited by 514 · see citing works
Generalized ODIN: Detecting Out-of-Distribution Image Without Learning From Out-of-Distribution Data
Hsu, Y.-C., Shen, Y., Jin, H., & Kira, Z. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01096 · Cited by 499 · see citing works
WILDS: A Benchmark of in-the-Wild Distribution Shifts
Koh, P. W., Sagawa, S., Marklund, H., Xie, S. M., Zhang, M., Balsubramani, A., Hu, W., Yasunaga, M., Phillips, R. L., Gao, I., et al. (2020) · arXiv · doi:10.48550/arxiv.2012.07421 · Cited by 518 · see citing works
Machine learning with a reject option: A survey
Hendrickx, K., Perini, L., Van der Plas, D., Meert, W., & Davis, J. (2024) · Machine Learning · doi:10.1007/s10994-024-06534-x · Cited by 134 · see citing works
Tian, K., Mitchell, E., Zhou, A., Sharma, A., Rafailov, R., Yao, H., Finn, C., & Manning, C. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.330 · Cited by 123 · see citing works
Adaptation with Self-Evaluation to Improve Selective Prediction in LLMs
Chen, J., Yoon, J., Ebrahimi, S., Arik, S., Pfister, T., & Jha, S. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.345 · Cited by 13 · see citing works
MemGym: a Long-Horizon Memory Environment for LLM Agents
Xu, W., Wang, Y., Mei, K., Liang, K., Wang, Z., Jin, M., Zhang, H., Zhang, S.-X., Hua, W., Sahu, S., et al. (2026) · arXiv · doi:10.48550/arxiv.2605.20833
“71 at ∼ 27% coverage on the cohorts retained by a pre-declared per-axis selection rule based on selective classification [12].”
Cross-Model Agreement as a Deployment-Time Reliability Signal for Automatic Polyp Segmentation
Gupta, S., & Singla, J. (2026) · arXiv · doi:10.48550/arxiv.2609.10495
“Beyond binary reliability estimation, selective classification with a reject option [12, 13, 14] formalizes the risk–coverage trade-off between prediction coverage and error rate; RBQE’s continuous agreement score is evaluated under this same framework in Section 5.6.”
InfluentialEvaluating and Guarding Citation Faithfulness in Agentic Scientific Synthesis
Goo, T., Kim, J., Han, K., Jo, G., Kim, J.-S., & Kim, T.-H. (2026) · arXiv · doi:10.48550/arxiv.2607.20527
Decision Models vs. LLMs as Rubric Judges: Cheaper, Faster, and Wrong in the Same Places
Rao, D., & Callison-Burch, C. (2026) · arXiv · doi:10.48550/arxiv.2609.29769
- Authors
- Geifman, Y., & El-Yaniv, R.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2017
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1705.08500
- OpenAlex
- W2618169590
- Cited by
- 220
- References
- 1
- Citations by year
- 2026: 9 · 2025: 15 · 2024: 30 · 2023: 49 · 2022: 27 · 2021: 39 · 2020: 32 · 2019: 14 · 2018: 4 · 2017: 1
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Keywords
- selective prediction, reject option, deep neural network, test coverage, risk level, uncertainty estimation, image classification, CIFAR, ImageNet
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
