CoChat
Sign Up
Cited by 1,032Anomaly Detection Techniques and ApplicationsSoftware System Performance and ReliabilityImbalanced Data Classification Techniques

A Comparative Evaluation of Unsupervised Anomaly Detection Algorithms for Multivariate Data

Open access
Goldstein, Markus, Uchida, Seiichi·2016·PLOS ONE
DOI 10.1371/journal.pone.0152173PMC PMC4836738
Open this paper in CoChat — free accountOpen PDF

Abstract

Anomaly detection is the process of identifying unexpected items or events in datasets, which differ from the norm. In contrast to standard classification tasks, anomaly detection is often applied on unlabeled data, taking only the internal structure of the dataset into account. This challenge is known as unsupervised anomaly detection and is addressed in many practical applications, for example in network intrusion detection, fraud detection as well as in the life science and medical domain. Dozens of algorithms have been proposed in this area, but unfortunately the research community still lacks a comparative universal evaluation as well as common publicly available datasets. These shortcomings are addressed in this study, where 19 different unsupervised anomaly detection algorithms are evaluated on 10 different datasets from multiple application domains. By publishing the source code and the datasets, this paper aims to be a new well-funded basis for unsupervised anomaly detection research. Additionally, this evaluation reveals the strengths and weaknesses of the different approaches for the first time. Besides the anomaly detection performance, computational effort, the impact of parameter settings as well as the global/local anomaly detection behavior is outlined. As a conclusion, we give an advise on algorithm selection for typical real-world tasks.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1371/journal.pone.0152173

Cites

2 of 2 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 62

  1. [1]Grubbs FE. Procedures for Detecting Outlying Observations in Samples. Technometrics. 1969; 11(1): 1– 21. 10.1080/00401706.1969.10490657 · DOI · Paper page
  2. [2]Portnoy L, Eskin E, Stolfo S. Intrusion Detection with Unlabeled Data Using Clustering. In: In Proceedings of ACM CSS Workshop on Data Mining Applied to Security (DMSA-2001); 2001. p. 5–8.
  3. [3]Garcia-Teodoro P, Diaz-Verdejo JE, Macia-Fernandez G, Vazquez E. Anomaly-based network intrusion detection: Techniques, systems and challenges. Computers and Security. 2009; 28: 18– 28. 10.1016/j.cose.2008.08.003 · DOI · Paper page
  4. [4]Yeung DY, Ding Y. Host-Based Intrusion Detection Using Dynamic and Static Behavioral Models. Pattern Recognition. 2003; 36: 229– 243. 10.1016/S0031-3203(02)00026-2 · DOI
  5. [6]Thiprungsri S, Vasarhelyi MA. Cluster Analysis for Anomaly Detection in Accounting Data: An Audit Approach. International Journal of Digital Accounting Research. 2011; 11 10.4192/1577-8517-v11_4 · DOI
  6. [7]Bolton RJ, Hand DJ. Unsupervised Profiling Methods for Fraud Detection. Statistical Science. 2002; 17(3): 235– 255.
  7. [8]Sigholm J, Raciti M. Best-Effort Data Leakage Prevention in Inter-Organizational Tactical MANETs. In: Proceedings of IEEE Military Communications Conference (MILCOM 2012). IEEE Computer Society Press; 2012.
  8. [9]Lin J, Keogh E, Fu A, Herle HV. Approximations to Magic: Finding unusual Medical Time Series. In: In 18th IEEE Symposium on Computer-Based Medical Systems (CBMS). IEEE Computer Society Press; 2005. p. 23–24.
  9. [10]Basharat A, Gritai A, Shah M. Learning Object Motion Patterns for Anomaly Detection and Improved Object Detection. In: Computer Vision and Pattern Recognition. (CVPR 2008). IEEE Conference on. IEEE Computer Society Press; 2008. p. 1–8.
  10. [11]Goldstein M, Uchida S. Behavior Analysis Using Unsupervised Anomaly Detection. In: The 10th Joint Workshop on Machine Perception and Robotics (MPR 2014). Online; 2014.
  11. [12]Pawling A, Chawla N, Madey G. Anomaly Detection in a Mobile Communication Network. Computational & Mathematical Organization Theory. 2007; 13: 407– 422. 10.1007/s10588-007-9018-7 · DOI
  12. [13]Gebhardt J, Goldstein M, Shafait F, Dengel A. Document Authentication using Printing Technique Features and Unsupervised Anomaly Detection. In: Proceedings of the 12th International Conference on Document Analysis and Recognition (ICDAR 2013). IEEE Computer Society Press; 2013. p. 479–483.
  13. [14]Martin RA, Schwabacher M, Oza NC, Srivastava AN. Comparison of Unsupervised Anomaly Detection Methods for Systems Health Management Using Space Shuttle. In: Proceedings of the Joint Army Navy NASA Air Force Conference on Propulsion; 2007.
  14. [15]Chandola V, Banerjee A, Kumar V. Anomaly Detection: A Survey. ACM Computing Surveys. 2009; 41(3): 1– 58. 10.1145/1541880.1541882 · DOI · Paper page
  15. [16]Hodge VJ, Austin J. A Survey of Outlier Detection Methodologies. Artificial Intelligence Review. 2004; 22(2): 85– 126. 10.1023/B:AIRE.0000045502.10941.a9 · DOI · Paper page
  16. [18]Markou M, Singh S. Novelty Detection: A Review—Part 1: Statistical Approaches. Signal Processing. 2003; 83(12): 2481– 2497. 10.1016/j.sigpro.2003.07.018 · DOI
  17. [19]Goldstein M, Asanger S, Reif M, Hutchinson A. Enhancing Security Event Management Systems with Unsupervised Anomaly Detection. In: Proceedings of the 2nd International Conference on Pattern Recognition Applications and Methods (ICPRAM 2013). INSTICC. SciTePress; 2013. p. 530–538.
  18. [20]Quinlan JR. C4.5: Programs for Machine Learning. San Francisco, CA, USA: Morgan Kaufmann Publishers Inc.; 1993.
  19. [21]Schölkopf B, Smola AJ. Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond Adaptive Computation and Machine Learning. MIT Press, Cambridge, MA; 2002.
  20. [22]Mehrotra K, Mohan CK, Ranka S. Elements of Artificial Neural Networks. Cambridge, MA, USA: MIT Press; 1997.
  21. [23]Moya MM, Hush DR. Network Constraints and Multi-objective Optimization for One-class Classification. Neural Networks. 1996; 9(3): 463– 474. 10.1016/0893-6080(95)00120-4 · DOI
  22. [24]Schölkopf B, Platt JC, Shawe-Taylor JC, Smola AJ, Williamson RC. Estimating the Support of a High-Dimensional Distribution. Neural Computation. 2001; 13(7): 1443– 1471. 10.1162/089976601750264965 11440593 · DOI · Paper page
  23. [25]Hawkins S, He H, Williams GJ, Baxter RA. Outlier Detection Using Replicator Neural Networks. In: Proceedings of the 4th International Conference on Data Warehousing and Knowledge Discovery (DaWaK 2000). London, UK: Springer-Verlag; 2000. p. 170–180.
  24. [26]Lindsay B. Mixture Models: Theory, Geometry, and Applications. NSF-CBMS Regional Conference Series in Probability and Statistics. Penn. State University: Institute of Mathematical Statistics; 1995.
  25. [27]Rosenblatt M. Remarks on Some Nonparametric Estimates of a Density Function. The Annals of Mathematical Statistics. 1956; 27(3): 832– 837. 10.1214/aoms/1177728190 · DOI
  26. [29]Goldstein M. Anomaly Detection in Large Datasets [PhD-Thesis]. University of Kaiserslautern. München, Germany; 2014.
  27. [30]Akoglu L, Tong H, Koutra D. Graph based Anomaly Detection and Description: A Survey. Data Mining and Knowledge Discovery. 2015; 29(3): 626– 688. 10.1007/s10618-014-0365-y · DOI · Paper page
  28. [31]Aggarwal CC. Outlier Analysis. Springer-Verlag New York; 2013.
  29. [32]Chhabra P, Scott C, Kolaczyk ED, Crovella M. Distributed Spatial Anomaly Detection. In: INFOCOM 2008. The 27th Conference on Computer Communications. IEEE; 2008.
  30. [33]Lazarevic A, Ertoz L, Kumar V, Ozgur A, Srivastava J. A Comparative Study of Anomaly Detection Schemes in Network Intrusion Detection. In: In Proceedings of the Third SIAM International Conference on Data Mining. vol. 3. Siam; 2003. p. 25–36.
  31. [34]Eskin E, Arnold A, Prerau M, Portnoy L, Stolfo S. A Geometric Framework for Unsupervised Anomaly Detection In: Barbara D, Jajodia S, editors. Applications of Data Mining in Computer Security. vol. 6 of Advances in Information Security. Springer US; 2002 p. 77– 101.
  32. [35]Auslander B, Gupta K, Aha DW. Comparative evaluation of anomaly detection algorithms for local maritime video surveillance In: Proc. SPIE 8019, Sensors, and Command, Control, Communications, and Intelligence (C3I) Technologies for Homeland Security and Homeland Defense X. Orlando, FL: SPIE; 2011.
  33. [36]Ding X, Li Y, Belatreche A, Maguire LP. An Experimental Evaluation of Novelty Detection Methods. Neurocomputing. 2014; 135: 313– 327. 10.1016/j.neucom.2013.12.002 · DOI
  34. [37]Carrasquilla U. Benchmarking Algorithms for Detecting Anomalies in Large Datasets. CMG Journal. 2011;.
  35. [38]Schubert E, Wojdanowski R, Zimek A, Kriegel HP. On Evaluation of Outlier Rankings and Outlier Scores. In: The 12th SIAM International Conference on Data Mining (SDM). SIAM / Omnipress; 2012. p. 1047–1058.
  36. [39]Zimek A, Campello RJGB, Sander J. Ensembles for Unsupervised Outlier Detection: Challenges and Research Questions a Position Paper. SIGKDD Explor Newsl. 2014; 15(1): 11– 22. 10.1145/2594473.2594476 · DOI
  37. [40]Amer M, Goldstein M, Abdennadher S. Enhancing One-class Support Vector Machines for Unsupervised Anomaly Detection. In: Proceedings of the ACM SIGKDD Workshop on Outlier Detection and Description (ODD’13). New York, NY, USA: ACM Press; 2013. p. 8–15.
  38. [41]Ramaswamy S, Rastogi R, Shim K. Efficient Algorithms for Mining Outliers from Large Data Sets. In: Proceedings of the 2000 ACM SIGMOD International Conference on Management of Data (SIGMOD’00). New York, NY, USA: ACM Press; 2000. p. 427–438.
  39. [42]Angiulli F, Pizzuti C. Fast Outlier Detection in High Dimensional Spaces In: Elomaa T, Mannila H, Toivonen H, editors. Principles of Data Mining and Knowledge Discovery. vol. 2431 of Lecture Notes in Computer Science. Springer Berlin / Heidelberg; 2002 p. 43– 78.
  40. [44]Tang J, Chen Z, Fu A, Cheung D. Enhancing Effectiveness of Outlier Detections for Low Density Patterns In: Chen MS, Yu P, Liu B, editors. Advances in Knowledge Discovery and Data Mining. vol. 2336 of Lecture Notes in Computer Science. Springer Berlin / Heidelberg; 2002 p. 535– 548.
  41. [45]Jin W, Tung A, Han J, Wang W. Ranking Outliers Using Symmetric Neighborhood Relationship In: Ng WK, Kitsuregawa M, Li J, Chang K, editors. Advances in Knowledge Discovery and Data Mining. vol. 3918 of Lecture Notes in Computer Science. Springer Berlin / Heidelberg; 2006 p. 577– 593.
  42. [46]Kriegel HP, Kröger P, Schubert E, Zimek A. LoOP: Local Outlier Probabilities. In: Proceeding of the 18th ACM Conference on Information and Knowledge Management (CIKM’09). New York, NY, USA: ACM Press; 2009. p. 1649–1652.
  43. [47]Papadimitriou S, Kitagawa H, Gibbons PB, Faloutsos C. LOCI: Fast Outlier Detection Using the Local Correlation Integral. In: Proceedings of the 19th International Conference on Data Engineering. Los Alamitos, CA, USA: IEEE Computer Society Press; 2003. p. 315–326.
  44. [49]He Z, Xu X, Deng S. Discovering Cluster-based Local Outliers. Pattern Recognition Letters. 2003; 24(9–10): 1641– 1650. 10.1016/S0167-8655(03)00003-5 · DOI
  45. [50]Amer M, Goldstein M. Nearest-Neighbor and Clustering based Anomaly Detection Algorithms for RapidMiner. In: Simon Fischer IM, editor. Proceedings of the 3rd RapidMiner Community Meeting and Conferernce (RCOMM 2012). Shaker Verlag GmbH; 2012. p. 1–12.
  46. [51]Mahalanobis PC. On the Generalised Distance in Statistics In: Proceedings National Institute of Science, India. vol. 2; 1936 p. 49– 55.
  47. [52]Friedman JH. Regularized Discriminant Analysis. Journal of the American Statistical Association. 1989; 84(405): 165– 175. 10.1080/01621459.1989.10478752 · DOI
  48. [53]Rousseeuw PJ. Least Median of Squares Regression. Journal of the American Statistical Association. 1984; 79(388): 871– 880. 10.1080/01621459.1984.10477105 · DOI
  49. [54]Rousseeuw PJ, Driessen KV. A Fast Algorithm for the Minimum Covariance Determinant Estimator. Technometrics. 1999; 41(3): 212– 223. 10.1080/00401706.1999.10485670 · DOI
  50. [56]Kim Y, Lau WC, Chuah MC, Chao HJ. Packetscore: Statistics-based Overload Control against Distributed Denial-of-Service Attacks. In: Proceedings of the Twenty-third Annual Joint Conference of the IEEE Computer and Communications Societies (INFOCOM 2004). vol. 4. IEEE Computer Society Press; 2004. p. 2594–2604.
  51. [57]Vert R, Vert JP. Consistency and Convergence Rates of One-Class SVMs and Related Algorithms. The Journal of Machine Learning Research. 2006; 7: 817– 854.
  52. [58]Kwitt R, Hofmann U. Unsupervised Anomaly Detection in Network Traffic by Means of Robust PCA. In: Proceedings of the International Multi-Conference on Computing in the Global Information Technology (ICCGI’07). Washington, DC, USA: IEEE Computer Society Press; 2007. p. 37.
  53. [59]ling Shyu M, ching Chen S, Sarinnapakorn K, Chang L. A novel anomaly detection scheme based on principal component classifier. In: in Proceedings of the IEEE Foundations and New Directions of Data Mining Workshop, in conjunction with the Third IEEE International Conference on Data Mining (ICDM’03; 2003. p. 171–179.
  54. [60]Mierswa I, Wurst M, Klinkenberg R, Scholz M, Euler T. YALE: Rapid Prototyping for Complex Data Mining Tasks. In: Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD 2006). New York, NY, USA: ACM Press; 2006. p. 935–940.
  55. [62]Micenkova B, McWilliams B, Assent I. Learning Outlier Ensembles: The Best of Both Worlds—Supervised and Unsupervised. In: Proceedings of the ACM SIGKDD 2014 Workshop on Outlier Detection and Description under Data Diversity (ODD2). New York, NY, USA; 2014. p. 51–54.
  56. [63]Geusebroek JM, Burghouts GJ, Smeulders AWM. The Amsterdam Library of Object Images. Int J Comput Vision. 2005; 61(1): 103– 112. 10.1023/B:VISI.0000042993.50813.60 · DOI
  57. [64]Mangasarian OL, Street WN, Wolberg WH. Breast Cancer Diagnosis and Prognosis via Linear Programming. SIAM News. 1990; 23(5): 1– 18.
  58. [65]Dehak N, Kenny P, Dehak R, Dumouchel P, Ouellet P. Front-End Factor Analysis for Speaker Verification. IEEE Transactions on Audio, Speech & Language Processing. 2011; 19: 788– 798. 10.1109/TASL.2010.2064307 · DOI · Paper page
  59. [67]Abe N, Zadrozny B, Langford J. Outlier Detection by Active Learning. In: Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD’06). New York, NY, USA: ACM Press; 2006. p. 504–509.
  60. [68]Reif M, Goldstein M, Stahl A, Breuel T. Anomaly Detection by Combining Decision Trees and Parametric Densities. In: 19th International Conference on Pattern Recognition (ICPR-2008). IEEE Computer Society Press; 2008.
  61. [69]Leung K, Leckie C. Unsupervised Anomaly Detection in Network Intrusion Detection using Clusters. In: Proceedings of the Twenty-eighth Australasian Conference on Computer Science (ACSC’05). Darlinghurst, Australia, Australia: Australian Computer Society, Inc.; 2005. p. 333–342.
  62. [70]Fawcett T. An Introduction to ROC Analysis. Pattern Recognition Letters. 2006; 27(8): 861– 874. 10.1016/j.patrec.2005.10.010 · DOI · Paper page

Cited by

Sorted by citations

Authors
Goldstein, M., & Uchida, S.
Affiliations
Kyushu University (JP)
Corresponding author
Goldstein, Markus
ORCID
Venue
PLOS ONE
Publisher
Public Library of Science (PLoS)
Published
2016-04-19
Volume
11
Issue
4
Pages
e0152173
ISSN
1932-6203 (ISSN-L)
DOI
10.1371/journal.pone.0152173
PMC
PMC4836738
OpenAlex
W2337344967
Cited by
1,032
References
2
Citations by year
2026: 83 · 2025: 102 · 2024: 104 · 2023: 118 · 2022: 137 · 2021: 143 · 2020: 129 · 2019: 121 · 2018: 60 · 2017: 31 · 2016: 3 · 2012: 1
FWCI
59.03
Type
journal article
Language
en
Open access
Licence
Locations
  • PLoS ONE (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • RePEc: Research Papers in Economics (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
Japan Science and Technology Agency
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Anomaly Detection Techniques and Applications
  • Physical Sciences › Computer Science › Computer Networks and Communications › Software System Performance and Reliability
  • Physical Sciences › Computer Science › Artificial Intelligence › Imbalanced Data Classification Techniques
Keywords
unsupervised anomaly detection, multivariate data, algorithm comparison, benchmark dataset, network intrusion detection, fraud detection, parameter sensitivity, computational effort, algorithm selection
MeSH terms
Algorithms, Humans, Statistics as Topic, Multivariate Analysis, Datasets as Topic

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review