CoChat
Sign Up
PreprintCited by 388Anomaly Detection Techniques and ApplicationsAdversarial Robustness in Machine LearningSoftware System Performance and Reliability

Deep Semi-Supervised Anomaly Detection

Open access
Ruff, Lukas, Vandermeulen, Robert A., Görnitz, Nico, Binder, Alexander, Müller, Emmanuel, Müller, Klaus-Robert, Kloft, Marius·2019·arXiv
DOI 10.48550/arxiv.1906.02694arXiv 1906.02694
Open this paper in CoChat — free accountOpen PDF

Abstract

Deep approaches to anomaly detection have recently shown promising results over shallow methods on large and complex datasets. Typically anomaly detection is treated as an unsupervised learning problem. In practice however, one may have---in addition to a large set of unlabeled samples---access to a small pool of labeled samples, e.g. a subset verified by some domain expert as being normal or anomalous. Semi-supervised approaches to anomaly detection aim to utilize such labeled samples, but most proposed methods are limited to merely including labeled normal samples. Only a few methods take advantage of labeled anomalies, with existing deep approaches being domain-specific. In this work we present Deep SAD, an end-to-end deep methodology for general semi-supervised anomaly detection. We further introduce an information-theoretic framework for deep anomaly detection based on the idea that the entropy of the latent distribution for normal data should be lower than the entropy of the anomalous distribution, which can serve as a theoretical interpretation for our method. In extensive experiments on MNIST, Fashion-MNIST, and CIFAR-10, along with other anomaly detection benchmark datasets, we demonstrate that our method is on par or outperforms shallow, hybrid, and deep competitors, yielding appreciable performance improvements even when provided with only little labeled data.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1906.02694

Cites

7 of 7 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Supervised anomaly detection typically requires extensive labelled anomaly examples, which are often unavailable or prohibitively expensive to obtain.

    “supervised methods typically require extensive anomaly labelling, which is often impractical.”

    From AnomalyMatch: Discovering Rare Objects of Interest with Semi-supervised and Active Learning (2025) · cites this paper as [3]

    The anomaly score section (p. 2) describes a real-world setting with many unlabeled samples but only a few verified labels, potentially supplied by a domain expert. Section 2 (p. 3) says supervised classification recognizes only anomalies similar to those seen in training. These passages support the concern about relying on labeled anomalies, but do not state that supervised methods typically require extensive labeling or that obtaining it is impractical. (LLM, conf=0.78)

    Experimental — accuracy not yet measured

  • Semi-supervised learning approaches, using unlabelled data along with limited labelled instances, offer a practical balance, leveraging minimal supervision to improve performance significantly over fully unsupervised approaches.

    “Semi-supervised learning (SSL) approaches, which use unlabelled data along with some limited labelled instances, offer a practical balance, leveraging minimal supervision to improve performance significantly over fully unsupervised approaches [3,5].”

    From AnomalyMatch: Discovering Rare Objects of Interest with Semi-supervised and Active Learning (2025) · cites this paper as [3]

    The anomaly-score discussion (p. 2) describes semi-supervised anomaly detection as combining mostly unlabeled data with a few labeled examples and striking a balance between one-class learning and classification. The Results (Section 4.2, p. 7) report benefits for Deep SAD, especially on CIFAR-10, but do not establish that semi-supervised learning approaches generally improve performance significantly over fully unsupervised approaches. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 76

  1. Alessandro Achille and Stefano Soatto. Emergence of invariance and disentanglement in deep representations. Journal of Machine Learning Research , 19(1):1947-1980, 2018.
  2. Samet Akcay, Amir Atapour-Abarghouei, and Toby P Breckon. GANomaly: Semi-supervised anomaly detection via adversarial training. In ACCV , pp. 622-637, 2018.
  3. Alexander A Alemi, Ian Fischer, Joshua V Dillon, and Kevin Murphy. Deep variational information bottleneck. In ICLR , 2017.
  4. Alexander A Alemi, Ben Poole, Ian Fischer, Joshua V Dillon, Rif A Saurous, and Kevin Murphy. Fixing a broken ELBO. In ICML , volume 80, pp. 159-168, 2018.
  5. Jerone T A Andrews, Edward J Morton, and Lewis D Griffin. Detecting Anomalous Data Using Auto-Encoders. IJMLC , 6(1):21, 2016.
  6. Sanjeev Arora, Nadav Cohen, and Elad Hazan. On the optimization of deep networks: Implicit acceleration by overparameterization. In ICML , volume 80, pp. 244-253, 2018.
  7. Mikhail Belkin, Siyuan Ma, and Soumik Mandal. To understand deep learning we need to understand kernel learning. In ICML , pp. 540-548, 2018.
  8. Gilles Blanchard, Gyemin Lee, and Clayton Scott. Semi-supervised novelty detection. Journal of Machine Learning Research , 11(Nov):2973-3009, 2010.
  9. Raghavendra Chalapathy and Sanjay Chawla. Deep learning for anomaly detection: A survey. arXiv:1901.03407 , 2019. · arXiv
  10. Raghavendra Chalapathy, Aditya Krishna Menon, and Sanjay Chawla. Anomaly detection using one-class neural networks. arXiv:1802.06360 , 2018. · arXiv
  11. Varun Chandola, Arindam Banerjee, and Vipin Kumar. Anomaly Detection: A Survey. ACM Computing Surveys , 41(3):1-58, 2009.
  12. Olivier Chapelle, Bernhard Schölkopf, and Alexander Zien. Semi-supervised learning. IEEE Transactions on Neural Networks , 20(3):542-542, 2009.
  13. Jinghui Chen, Saket Sathe, Charu C Aggarwal, and Deepak S Turaga. Outlier Detection with Autoencoder Ensembles. In SDM , pp. 90-98, 2017.
  14. Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. InfoGAN: Interpretable representation learning by information maximizing generative adversarial nets. In NIPS , pp. 2172-2180, 2016.
  15. Nadav Cohen, Or Sharir, and Amnon Shashua. On the expressive power of deep learning: A tensor analysis. In COLT , volume 49, pp. 698-728, 2016.
  16. Thomas M Cover and Joy A Thomas. Elements of Information Theory . John Wiley & Sons, 2012.
  17. Zihang Dai, Zhilin Yang, Fan Yang, William W Cohen, and Ruslan R Salakhutdinov. Good semisupervised learning that requires a bad gan. In NIPS , pp. 6510-6520, 2017.
  18. Lucas Deecke, Robert A Vandermeulen, Lukas Ruff, Stephan Mandt, and Marius Kloft. Image anomaly detection with generative adversarial networks. In ECML-PKDD , 2018.
  19. François Denis. PAC learning from positive statistical queries. In International Conference on Algorithmic Learning Theory , pp. 112-126, 1998.
  20. Ronen Eldan and Ohad Shamir. The power of depth for feedforward neural networks. In COLT , volume 49, pp. 907-940, 2016.
  21. Tolga Ergen, Ali Hassan Mirza, and Suleyman Serdar Kozat. Unsupervised and semi-supervised anomaly detection with LSTM neural networks. arXiv:1710.09207 , 2017. · arXiv
  22. Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In AISTATS , pp. 249-256, 2010.
  23. Izhak Golan and Ran El-Yaniv. Deep anomaly detection using geometric transformations. In NeurIPS , pp. 9758-9769, 2018.
  24. Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning . MIT Press, 2016. http: //www.deeplearningbook.org .
  25. Nico Görnitz, Marius Kloft, Konrad Rieck, and Ulf Brefeld. Toward supervised anomaly detection. Journal of Artificial Intelligence Research , 46:235-262, 2013.
  26. Simon Hawkins, Hongxing He, Graham Williams, and Rohan Baxter. Outlier Detection Using Replicator Neural Networks. In DaWaK , volume 2454, pp. 170-180, 2002.
  27. Dan Hendrycks, Mantas Mazeika, and Thomas G Dietterich. Deep anomaly detection with outlier exposure. In ICLR , 2019a.
  28. Dan Hendrycks, Mantas Mazeika, Saurav Kadavath, and Dawn Song. Using self-supervised learning can improve model robustness and uncertainty. In NeurIPS , pp. 15637-15648, 2019b.
  29. Geoffrey E Hinton and Ruslan R Salakhutdinov. Reducing the Dimensionality of Data with Neural Networks. Science , 313(5786):504-507, 2006.
  30. R Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. In ICLR , 2019.
  31. Matthew D Hoffman and Matthew J Johnson. ELBO surgery: yet another way to carve up the variational evidence lower bound. In NIPS Workshop in Advances in Approximate Bayesian Inference , 2016.
  32. Xu Ji, Joao F Henriques, and Andrea Vedaldi. Invariant information distillation for unsupervised image segmentation and clustering. arXiv:1807.06653 , 2018. · arXiv
  33. JooSeuk Kim and Clayton D Scott. Robust kernel density estimation. Journal of Machine Learning Research , 13(Sep):2529-2565, 2012.
  34. Diederik P Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. In ICLR , 2015.
  35. Diederik P Kingma and Max Welling. Auto-encoding variational bayes. In ICLR , 2013.
  36. Diederik P Kingma, Shakir Mohamed, Danilo Jimenez Rezende, and Max Welling. Semi-supervised learning with deep generative models. In NIPS , pp. 3581-3589, 2014.
  37. B Kiran, Dilip Thomas, and Ranjith Parakkal. An overview of deep learning based methods for unsupervised and semi-supervised anomaly detection in videos. Journal of Imaging , 4(2):36, 2018.
  38. Sebastian Lapuschkin, Stephan Wäldchen, Alexander Binder, Grégoire Montavon, Wojciech Samek, and Klaus-Robert Müller. Unmasking clever hans predictors and assessing what machines really learn. Nature communications , 10(1):1096, 2019.
  39. Ralph Linsker. Self-organization in a perceptual network. IEEE Computer , 21(3):105-117, 1988.
  40. Yi Liu and Yuan F Zheng. Minimum enclosing and maximum excluding machine for pattern description and discrimination. In ICPR , pp. 129-132, 2006.
  41. Alireza Makhzani and Brendan Frey. K-sparse autoencoders. In ICLR , 2014.
  42. Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly, Ian Goodfellow, and Brendan Frey. Adversarial autoencoders. arXiv:1511.05644 , 2015. · arXiv
  43. Erxue Min, Jun Long, Qiang Liu, Jianjing Cui, Zhiping Cai, and Junbo Ma. SU-IDS: A semisupervised and unsupervised framework for network intrusion detection. In International Conference on Cloud Computing and Security , pp. 322-334, 2018.
  44. Grégoire Montavon, Mikio L Braun, and Klaus-Robert Müller. Kernel analysis of deep networks. Journal of Machine Learning Research , 12(Sep):2563-2581, 2011.
  45. Mary M Moya, Mark W Koch, and Larry D Hostetler. One-class classifier networks for target recognition applications. In Proceedings World Congress on Neural Networks , pp. 797-801, 1993.
  46. Jordi Muñoz-Marí, Francesca Bovolo, Luis Gómez-Chova, Lorenzo Bruzzone, and Gustavo CampValls. Semi-Supervised One-Class Support Vector Machines for Classification of Remote Sensing Sata. IEEE Transactions on Geoscience and Remote Sensing , 48(8):3188-3197, 2010.
  47. Behnam Neyshabur, Srinadh Bhojanapalli, David McAllester, and Nati Srebro. Exploring generalization in deep learning. In NIPS , pp. 5947-5956, 2017.
  48. Miguel Nicolau, James McDermott, et al. A hybrid autoencoder and density estimation model for anomaly detection. In International Conference on Parallel Problem Solving from Nature , pp. 717-726, 2016.
  49. Augustus Odena. Semi-supervised learning with generative adversarial networks. arXiv:1606.01583 , 2016. · arXiv
  50. Avital Oliver, Augustus Odena, Colin Raffel, Ekin D Cubuk, and Ian J Goodfellow. Realistic evaluation of deep semi-supervised learning algorithms. In NeurIPS , pp. 3235-3246, 2018.
  51. Guansong Pang, Chunhua Shen, and Anton van den Hengel. Deep anomaly detection with deviation networks. In KDD , pp. 353-362, 2019.
  52. E. Parzen. On Estimation of a Probability Density Function and Mode. The annals of mathematical statistics , 33(3):1065-1076, 1962.
  53. Maithra Raghu, Ben Poole, Jon Kleinberg, Surya Ganguli, and Jascha Sohl Dickstein. On the expressive power of deep neural networks. In ICML , volume 70, pp. 2847-2854, 2017.
  54. Antti Rasmus, Mathias Berglund, Mikko Honkala, Harri Valpola, and Tapani Raiko. Semisupervised learning with ladder networks. In NIPS , pp. 3546-3554, 2015.
  55. Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic Backpropagation and Approximate Inference in Deep Generative Models. In ICML , volume 32, pp. 1278-1286, 2014.
  56. Lukas Ruff, Robert A Vandermeulen, Nico Görnitz, Lucas Deecke, Shoaib A Siddiqui, Alexander Binder, Emmanuel Müller, and Marius Kloft. Deep one-class classification. In ICML , volume 80, pp. 4390-4399, 2018.
  57. Lukas Ruff, Yury Zemlyanskiy, Robert Vandermeulen, Thomas Schnake, and Marius Kloft. Selfattentive, multi-context one-class classification for unsupervised anomaly detection on text. In ACL , pp. 4061-4071, 2019.
  58. David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. Learning internal representations by error propagation. In Parallel Distributed Processing - Explorations in the Microstructure of Cognition , chapter 8, pp. 318-362. MIT Press, 1986.
  59. Mayu Sakurada and Takehisa Yairi. Anomaly detection using autoencoders with nonlinear dimensionality reduction. In Proceedings of the 2nd MLSDA Workshop , pp. 4, 2014.
  60. Andrew Michael Saxe, Yamini Bansal, Joel Dapello, Madhu Advani, Artemy Kolchinsky, Brendan Daniel Tracey, and David Daniel Cox. On the information bottleneck theory of deep learning. In ICLR , 2018.
  61. Bernhard Schölkopf and Alex J Smola. Learning with Kernels . MIT press, 2002.
  62. Bernhard Schölkopf, John C Platt, John Shawe-Taylor, Alex J Smola, and Robert C Williamson. Estimating the Support of a High-Dimensional Distribution. Neural computation , 13(7):14431471, 2001.
  63. Clayton D Scott and Robert D Nowak. Learning minimum volume sets. Journal of Machine Learning Research , 7(Apr):665-704, 2006.
  64. Ravid Shwartz-Ziv and Naftali Tishby. Opening the black box of deep neural networks via information. arXiv:1703.00810 , 2017. · arXiv
  65. Noam Slonim, Gurinder Singh Atwal, Gašper Tkaˇ cik, and William Bialek. Information-based clustering. Proceedings of the National Academy of Sciences , 102(51):18297-18302, 2005.
  66. Ingo Steinwart, Don Hush, and Clint Scovel. A classification framework for anomaly detection. Journal of Machine Learning Research , 6(Feb):211-232, 2005.
  67. David M J Tax and Robert P W Duin. Support Vector Data Description. Machine learning , 54(1): 45-66, 2004.
  68. Naftali Tishby and Noga Zaslavsky. Deep learning and the information bottleneck principle. In 2015 IEEE Information Theory Workshop (ITW) , pp. 1-5, 2015.
  69. Robert A Vandermeulen and Clayton Scott. Consistency of robust kernel density estimators. In COLT , pp. 568-591, 2013.
  70. Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. In ICML , pp. 1096-1103, 2008.
  71. Jigang Wang, Predrag Neskovic, and Leon N Cooper. Pattern classification via single spheres. In International Conference on Discovery Science , pp. 241-252. Springer, 2005.
  72. Thomas Wiatowski and Helmut Bölcskei. A mathematical theory of deep convolutional neural networks for feature extraction. IEEE Transactions on Information Theory , 64(3):1845-1866, 2018.
  73. Frank Wilcoxon. Individual comparisons by ranking methods. Biometrics Bulletin , 1(6):80-83, 1945.
  74. Bangzuo Zhang and Wanli Zuo. Learning from positive and unlabeled examples: A survey. In Proceedings of the IEEE International Symposium on Information Processing , pp. 650-654, 2008.
  75. Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. In ICLR , 2017.
  76. Shengjia Zhao, Jiaming Song, and Stefano Ermon. InfoVAE: Information maximizing variational autoencoders. arXiv:1706.02262 , 2017. · arXiv

Cited by

Sorted by citations

Authors
Ruff, L., Vandermeulen, R. A., Görnitz, N., Binder, A., Müller, E., Müller, K.-R., & Kloft, M.
Affiliations
  • Technische Universität Berlin (DE)
  • University of Kaiserslautern (DE)
  • Singapore University of Technology and Design (SG)
  • University of Bonn (DE)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2019
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1906.02694
arXiv
1906.02694
OpenAlex
W2948079128
Cited by
388
References
7
Citations by year
2026: 3 · 2025: 37 · 2024: 77 · 2023: 99 · 2022: 72 · 2021: 71 · 2020: 27 · 2019: 2
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • Technische Universität Kaiserslautern
  • National Research Foundation
  • National Research Foundation Singapore
  • Ministry of Education - Singapore (MOE2016-T2-2-154)
  • Deutsche Forschungsgemeinschaft (KL 2698/2-1, EXC-2046, 01GQ0850, EXC‐2046/1, 2698/2, 01GQ1115, 01IS18037I, 01IS14013A-E, 2046/1, 390685689, 2698/2-1)
  • Bundesministerium für Bildung und Forschung (01 IS 14013 A, 01IS18051A, EXC-2046/1, 01IS14013A-E, 031B0770E, 01GQ0850, 01IS18037A, 01GQ1115, 031L0023A, 01IS18025A, 390685689, 01IS18037I, 01IS18049B)
  • Singapore University of Technology and Design
  • Institute for Information and Communications Technology Promotion (2017-0-00451, 2017‐0‐01779)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Anomaly Detection Techniques and Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
  • Physical Sciences › Computer Science › Computer Networks and Communications › Software System Performance and Reliability
Keywords
semi-supervised anomaly detection, deep anomaly detection, information-theoretic framework, unlabeled data, few labeled samples, generative adversarial networks

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review