Generalized Out-of-Distribution Detection: A Survey
Open accessThe DOI matches its Crossref record; 332 of the 332 references it cites were found in a registry or an index.
Abstract
Out-of-distribution (OOD) detection is critical to ensuring the reliability and safety of machine learning systems. For instance, in autonomous driving, we would like the driving system to issue an alert and hand over the control to humans when it detects unusual scenes or objects that it has never seen during training time and cannot make a safe decision. The term, OOD detection, first emerged in 2017 and since then has received increasing attention from the research community, leading to a plethora of methods developed, ranging from classification-based to density-based to distance-based ones. Meanwhile, several other problems, including anomaly detection (AD), novelty detection (ND), open set recognition (OSR), and outlier detection (OD), are closely related to OOD detection in terms of motivation and methodology. Despite common goals, these topics develop in isolation, and their subtle differences in definition and problem setting often confuse readers and practitioners. In this survey, we first present a unified framework called generalized OOD detection, which encompasses the five aforementioned problems, i.e.,AD, ND, OSR, OOD detection, and OD. Under our framework, these five problems can be seen as special cases or sub-tasks, and are easier to distinguish. Despite comprehensive surveys of related fields, the summarization of OOD detection methods remains incomplete and requires further advancement. This paper specifically addresses the gap in recent technical developm…
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Cortes, C., & Vapnik, V. (1995) · Machine Learning · doi:10.1007/bf00994018 · Cited by 59,421
Fischler, M. A., & Bolles, R. C. (1981) · Communications of the ACM · doi:10.1145/358669.358692 · Cited by 25,893
Learning Multiple Layers of Features from Tiny Images
Krizhevsky, A. (2024) · doi:10.57702/zp44cu3g · Cited by 23,395
An introduction to ROC analysis
Fawcett, T. (2006) · Pattern Recognition Letters · doi:10.1016/j.patrec.2005.10.010 · Cited by 21,879
(1996) · Cited by 20,179
Auto-Encoding Variational Bayes
Kingma, D. P., & Welling, M. (2013) · arXiv · doi:10.48550/arxiv.1312.6114 · Cited by 20,067
Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification
He, K., Zhang, X., Ren, S., & Sun, J. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.123 · Cited by 19,741
Learning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293
Research works printed but not linked: 53
- Angelopoulos, A. N., & Bates, S. (2021). A gentle introduction to conformal prediction and distribution-free uncertainty quantification, arXiv preprint arXiv:2107.07511 · arXiv
- Cao, K., Brbic, M., & Leskovec, J. (2021). Open-world semi-supervised learning, arXiv preprint arXiv:2102.03526 · arXiv
- Chen, J., Li, Y., Wu, X., Liang, Y., & Jha, S. (2020b). Robust out-of-distribution detection for neural networks, arXiv preprint arXiv:2003.09711 · arXiv
- Choi, H., Jang, E., & Alemi, A. A. (2018). Waic, but why? generative ensembles for robust anomaly detection, arXiv preprint arXiv:1810.01392 · arXiv
- Denouden, T., Salay, R., Czarnecki, K., Abdelzad, V., Phan, B., & Vernekar, S. (2018). Improving reconstruction autoencoder out-of-distribution detection with Mahalanobis distance, arXiv preprint arXiv:1812.02765 · arXiv
- DeVries, T., & Taylor, G. W. (2017). Improved regularization of convolutional neural networks with cutout, arXiv preprint arXiv:1708.04552 · arXiv
- DeVries, T., & Taylor, G. W. (2018). Learning confidence for out-of-distribution detection in neural networks, arXiv preprint arXiv:1802.04865 · arXiv
- Dolhansky, B., Howes, R., Pflaum, B., Baram, N., & Ferrer, C. C. (2019). The deepfake detection challenge (dfdc) preview dataset, arXiv preprint arXiv:1910.08854 · arXiv
- Dong, J., Gao, Y., Zhou, H., Cen, J., Yao, Y., Yoon, S., & Sun, P. D. (2023). Towards few-shot out-of-distribution detection, arXiv preprint arXiv:2311.12076 · arXiv
- Dong, X., Guo, J., Li, A., Ting, W.-T., Liu, C., & Kung, H. (2022a). Neural mean discrepancy for efficient out-of-distribution detection. In CVPR. · DOI
- Drummond, N., & Shearer, R. (2006). The open world assumption. In eSI workshop.
- Fang, Z., Li, Y., Lu, J., Dong, J., Han, B., & Liu, F. (2022). Is out-of-distribution detection learnable? In NeurIPS.
- Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., & Bengio, Y. (2014). Generative adversarial nets. In NIPS.
- Han, X., Chen, X., & Liu, L.-P. (2020). Gan ensemble for anomaly detection, arXiv preprint arXiv:2012.07988 · arXiv
- Hendrycks, D., Basart, S., Mazeika, M., Mostajabi, M., Steinhardt, J., & Song, D. (2022a) Scaling out-of-distribution detection for real-world settings. In ICML.
- Hendrycks, D., & Mazeika, M. (2022). X-risk analysis for AI research. arXiv preprint, arXiv:2206.05862 · arXiv
- Hendrycks, D., Mu, N., Cubuk, E. D., Zoph, B., Gilmer, J., & Lakshminarayanan, B. (2019c). Augmix: A simple data processing method to improve robustness and uncertainty. arXiv preprint arXiv:1912.02781 · arXiv
- Huang, H., Li, Z., Wang, L., Chen, S., Dong, B., & Zhou, X. (2020a). Feature space singularity for out-of-distribution detection, arXiv preprint arXiv:2011.14654 · arXiv
- Igoe, C., Chung, Y., Char, I., & Schneider, J. (2022). How useful are gradients for ood detection really? arXiv preprint arXiv:2205.10439 · arXiv
- Jang, J., & Kim, C. O. (2020). One-vs-rest network-based deep probability model for open set recognition, arXiv preprint arXiv:2004.08067 · arXiv
- Jiang, D., Sun, S., & Yu, Y. (2021a). Revisiting flow generative models for out-of-distribution detection. In International conference on learning representations.
- Jiang, L., Guo, Z., Wu, W., Liu, Z., Liu, Z., Loy, C.C., Yang, S., Xiong, Y., Xia, W., Chen, B., Zhuang, P., Li, S., Chen, S., Yao, T., Ding, S., Li, J., Huang, F., Cao, L., Ji, R., Lu, C., & Tan, G. (2021c). DeeperForensics Challenge 2020 on real-world face forgery detection: Methods and results, arXiv preprint arXiv:2102.09471 · arXiv
- Jiang, X., Liu, F., Fang, Z., Chen, H., Liu, T., Zheng, F., & Han, B. (2023b). Detecting out-of-distribution data through in-distribution class prior. In International conference on machine learning (pp. 15067–15088). PMLR.
- Kingma, D. P., & Dhariwal, P. (2018). Glow: Generative flow with invertible 1x1 convolutions. In NeurIPS.
- Kirichenko, P., Izmailov, P., & Wilson, A. G. (2020). Why normalizing flows fail to detect out-of-distribution data. in NeurIPS.
- Krizhevsky, A., Nair, V., & Hinton, G. (2009). Cifar-10 and cifar-100 datasets (vol. 6, (no. 1), p. 1). https://www.cs.toronto.edu/kriz/cifar.html
- Li, B., Zhang, Y., Chen, L., Wang, J., Yang, J., & Liu, Z. (2023a). Otter: A multi-modal model with in-context instruction tuning, arXiv preprintarXiv:2305.03726
- Liu, H., Li, C., Wu, Q., & Lee, Y. J. (2023). Visual instruction tuning, arXiv preprint arXiv:2304.08485 · arXiv
- Loureiro, A., Torgo, L., & Soares, C. (2004). Outlier detection using clustering methods: A data cleaning application. In Proceedings of KDNet symposium on knowledge-based systems.
- Lu, F., Zhu, K., Zheng, K., Zhai, W., & Cao, Y. (2023). Likelihood-aware semantic alignment for full-spectrum out-of-distribution detection, arXiv preprint arXiv:2312.01732 · arXiv
- Madry, A., Makelov, A., Schmidt, L., Tsipras, D., & Vladu, A. (2018). Towards deep learning models resistant to adversarial attacks, ICLR.
- Meinke, A., & Hein, M. (2019). Towards neural networks that provably know when they don’t know, arXiv preprint arXiv:1909.12180 · arXiv
- Miljković, D. (2010). Review of novelty detection methods. In MIPRO.
- Ming, Y., Sun, Y., Dia, O., & Li, Y. (2023). Cider: Exploiting hyperspherical embeddings for out-of-distribution detection. In ICLR.
- Miyai, A., Yang, J., Zhang, J., Ming, Y., Yu, Q., Irie, G., Li, Y., Li, H., Liu, Z., & Aizawa, K. (2024). Unsolvable problem detection: Evaluating trustworthiness of vision language models. arXiv preprint, arXiv:2403.20331 · arXiv
- Miyai, A., Yu, Q., Irie, G., & Aizawa, K. (2023a). Can pre-trained networks detect familiar out-of-distribution data? arXiv preprint arXiv:2310.00847 · arXiv
- Miyai, A., Yu, Q., Irie, G., & Aizawa, K. (2023b). Locoop: Few-shot out-of-distribution detection via prompt learning, arXiv preprint arXiv:2306.01293 · arXiv
- Mohseni, S., Wang, H., Yu, Z., Xiao, C., Wang, Z., & Yadawa, J. (2021). Practical machine learning safety: A survey and primer. arXiv preprint, arXiv:2106.04823 · arXiv
- Münz, G., Li, S., & Carle, G. (2007). Traffic anomaly detection using k-means clustering. In GI/ITG workshop MMBnet.
- Nalisnick, E., Matsukawa, A., Teh, Y. W., Gorur, D., & Lakshminarayanan, B. (2018). Do deep generative models know what they don’t know? In NeurIPS.
- Nguyen, V. D. (2022). Out-of-distribution detection for lidar-based 3d object detection, Master’s thesis, University of Waterloo.
- Ren, J., Fort, S., Liu, J., Roy, A. G., Padhy, S., & Lakshminarayanan, B. (2021). A simple fix to Mahalanobis distance for improving near-ood detection, arXiv preprint arXiv:2106.09022 · arXiv
- Salehi, M., Mirzaei, H., Hendrycks, D., Li, Y., Rohban, M. H., & Sabokrou, M. (2021). A unified survey on anomaly, novelty, open-set, and out-of-distribution detection: Solutions and future challenges, arXiv preprint arXiv:2110.14051 · arXiv
- Sun, X., Ding, H., Zhang, C., Lin, G., & Ling, K.-V. (2021a). M2iosr: Maximal mutual information open set recognition, arXiv preprint arXiv:2108.02373 · arXiv
- Van Oord, A., Kalchbrenner, N., & Kavukcuoglu, K. (2016). Pixel recurrent neural networks. In ICML.
- Vinyals, O., Ewalds, T., Bartunov, S., Georgiev, P., Vezhnevets, A. S., Yeo, M., Makhzani, A., Küttler, H., Agapiou, J., Schrittwieser, J., & Quan, J. (2017). Starcraft II: A new challenge for reinforcement learning, arXiv preprint arXiv:1708.04782 · arXiv
- Wang, H., Liu, W., Bocchieri, A., & Li, Y. (2021). Can multi-label classification networks know what they don’t know? NeurIPS, 34, 29074–29087.
- Wenzel, F., Roth, K., Veeling, B. S., Światkowski, J., Tran, L., Mandt, S., Snoek, J., Salimans, T., Jenatton, R., & Nowozin, S. (2020). How good is the Bayes posterior in deep neural networks really? In ICML.
- Yang, J., Zhou, K., & Liu, Z. (2022b). Full-spectrum out-of-distribution detection, arXiv preprint arXiv:2204.05306 · arXiv
- Yang, Z., Li, L., Lin, K., Wang, J., Lin, C.-C., Liu, Z., & Wang, L. (2023). The dawn of lmms Preliminary explorations with gpt-4v (ision). arXiv preprint, arXiv:2309.17421 · arXiv
- Zhang, J., Fu, Q., Chen, X., Du, L., Li, Z., Wang, G., Han, S., & Zhang, D. (2023a). Out-of-distribution detection based on in-distribution data patterns memorization with modern Hopfield energy. In ICLR.
- Zhang, J., Yang, J., Wang, P., Wang, H., Lin, Y., Zhang, H., Sun, Y., Du, X., Zhou, K., Zhang, W., Li, Y., Liu, Z., Chen, Y., & Li, H. (2023c). Openood v1.5: Enhanced benchmark for out-of-distribution detection. arXiv preprint, arXiv:2306.09301 · arXiv
- Zong, B., Song, Q., Min, M. R., Cheng, W., Lumezanu, C., Cho, D., & Chen, H. (2018). Deep autoencoding Gaussian mixture model for unsupervised anomaly detection. In ICLR.
Cited by
Domain Generalization: A Survey
Zhou, K., Liu, Z., Qiao, Y., Xiang, T., & Loy, C. C. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3195549 · Cited by 1,315
Self-citationTransformers in medical imaging: A survey
Shamshad, F., Khan, S., Zamir, S. W., Khan, M. H., Hayat, M., Khan, F. S., & Fu, H. (2023) · Medical Image Analysis · doi:10.1016/j.media.2023.102802 · Cited by 1,296
Generalizing to unseen domains: A survey on domain generalization
Wang, J., Lan, C., Liu, C., Ouyang, Y., Qin, T., Lu, W., Chen, Y., Zeng, W., & Yu, P. (2022) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2022.3178128 · Cited by 1,046
Díaz-Rodríguez, N., Del Ser, J., Coeckelbergh, M., López de Prado, M., Herrera-Viedma, E., & Herrera, F. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101896 · Cited by 800
ReAct: Out-of-distribution Detection With Rectified ActivationsPreprint
Sun, Y., Guo, C., & Li, Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.12797 · Cited by 484
Self-citation
Metadata
- Authors
- Yang, J., Zhou, K., Li, Y., & Liu, Z.
- Affiliations
- Nanyang Technological University (SG)
- University of Wisconsin–Madison (US)
- Corresponding author
- Yang, Jingkang
- Zhou, Kaiyang
- Li, Yixuan
- Liu, Ziwei
- Venue
- International Journal of Computer Vision
- Published
- 2024-06-23
- Volume
- 132
- Issue
- 12
- Pages
- 5635-5662
- ISSN
- 0920-5691 (ISSN-L)
- 1573-1405
- OpenAlex
- W4399929809
- Cited by
- 884
- References
- 319
- Citations by year
- 2026: 217 · 2025: 283 · 2024: 173
- FWCI
- 99.11
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- www.springernature.com/gp/researchers/text-and-data-mining (tdm)
- www.springernature.com/gp/researchers/text-and-data-mining (vor)
- Locations
- International Journal of Computer Vision (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- DR-NTU (Nanyang Technological University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Wisconsin Alumni Research Foundation
- Ministry of Education - Singapore (AcRF‐Tier2, MOET2EP20221-0012, Tier 2, RIE2020)
- Ministry of Education, India
- Division of Graduate Education
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Anomaly Detection Techniques and Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- out-of-distribution detection, anomaly detection, novelty detection, open-set recognition, outlier detection, density-based methods, distance-based methods, autonomous driving safety, unified framework
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Generalized Out-of-Distribution Detection: A Survey. https://app.cochat.ai/papers/10.1007/s11263-024-02117-4
- The paper
- Yang, J., Zhou, K., Li, Y., & Liu, Z. (2024). Generalized Out-of-Distribution Detection: A Survey. International Journal of Computer Vision. https://doi.org/10.1007/s11263-024-02117-4
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.