Cited by 135Machine Learning and Data ClassificationMachine Learning and AlgorithmsAdversarial Robustness in Machine Learning

Machine learning with a reject option: A survey

Open access
Hendrickx, Kilian, Perini, Lorenzo, Van der Plas, Dries, Meert, Wannes, Davis, Jesse·2024·Machine Learning

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1007/s10994-024-06534-x

Cites

245 of 245 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • We include an additional multiple choice answer 'Insufficient information to answer the question' to assess models' ability to refrain from answering when uncertain.

    “In order to assess models' ability to refrain from answering a question when uncertain [14], we also include an additional multiple choice answer for each question:”

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

7 of the 252 references couldn’t be matched to a known paper.

Research works printed but not linked: 14

  1. Alhoniemi, E., Himberg, J., and Vesanto, J. (1999). Probabilistic measures for responses of self-organizing map units. In H. Bothe and E. Oja and E. Massad and C. Haefke, editor, Proceedings of the International ICSC Congress on Computational Intelligence Methods and Applications. ICSC Academic Press, Zurich, Switzerland, pages 286-290.
  2. Cao, Y., Cai, T., and An, B. (2022). Generalizing Consistent Multi-Class Classification with Rejection to be Compatible with Arbitrary Losses. (NeurIPS):1-14.
  3. Ferri, C. and Hernández-Orallo, J. (2004). Cautious Classifiers. Proceedings of ROC Analysis in Artificial Intelligence, 1st International Workshop (ROCAI-2004), pages 27-36.
  4. Frélicot, C. (1997). Learning rejection thresholds for a class of fuzzy classifiers from possibilistic clustered noisy data. In 7th International Fuzzy Systems Association World Congress, IFSA, volume 3, pages 111-116.
  5. Friedel, C. C., Rückert, U., and Kramer, S. (2006). Cost Curves for Abstaining Classifiers. In ICML 2006 -3th workshop ROC Analysis in ML, Pittsburgh, Pennsylvania.
  6. Hansen, L. K., Liisberg, C., and Salamon, P. (1997). The Error-Reject Tradeoff. Open Systems & Information Dynamics, 4(2):159-184.
  7. Landgrebe, T. C., Tax, D. M., Paclík, P., Duin, R. P., and Andrew, C. (2004). A combining strategy for ill-defined problems. In Proc. of the 15th Annual Symposium of the Pattern Recognition Association of South Africa, pages 57-62.
  8. Lee, J. K., Bu, Y., Rajan, D., Sattigeri, P., Panda, R., Das, S., and Wornell, G. W. (2021). Fair Selective Classification Via Sufficiency. Proceedings of the 38th International Conference on Machine Learning, 139:6076-6086.
  9. Liu, J., Gallego, B., and Barbieri, S. (2022). Incorporating uncertainty in learning to defer algorithms for safe computer-aided diagnosis. Scientific Reports, 12(1):1-9.
  10. Nalisnick, E., Matsukawa, A., Whye, Y., Dilan, T., and Balaji, G. (2019). Hybrid Models with Deep and Invertible Features (ICML, 2019).
  11. Prasad, N. R., Almanza-Garcia, S., and Lu, T. T. (2009). Anomaly detection. Computers, Materials and Continua, 14(1):1-22.
  12. Van der Plas, D., Meert, W., Verbraecken, J., and Davis, J. (2021). A reject option for automated sleep stage scoring. In ICML Workshop on Interpretable Machine Learning in Healthcare, Virtual event.
  13. Vanschoren, J. (2018). Meta-Learning: A Survey. pages 1-29.
  14. Zhou, L., Martínez-Plumed, F., Hernández-Orallo, J., Ferri, C., and Schellaert, W. (2022). Reject Before You Run: Small Assessors Anticipate Big Language Models. In CEUR Workshop Proceedings, volume 3169.

Cited by

Sorted by citations

Authors
Hendrickx, K., Perini, L., Van der Plas, D., Meert, W., & Davis, J.
Affiliations
  • Siemens (Belgium) (BE)
  • KU Leuven (BE)
Corresponding author
Hendrickx, Kilian
Venue
Machine Learning
Published
2024-03-29
Volume
113
Issue
5
Pages
3073-3110
ISSN
  • 0885-6125 (ISSN-L)
  • 1573-0565
OpenAlex
W3183516990
Cited by
134
References
252
Citations by year
2026: 59 · 2025: 31 · 2024: 23 · 2023: 15 · 2022: 6
FWCI
16.68
Type
journal article
Language
en
Open access
Locations
  • Machine Learning (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • Lirias (KU Leuven) (submittedVersion, other-oa)
  • arXiv (Cornell University) (cc-by)
Funders
  • agentschap innoveren en ondernemen (HBC.2017.0226, HBC.2019.2615)
  • onderzoeksraad, ku leuven (C14/17/070)
  • vlaamse overheid (Onderzoeksprogramma Artificiële Intelligentie (AI) Vlaanderen)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
Keywords
abstention, decision support

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review