HUMAN - Hierarchical Clustering for Unsupervised Anomaly Detection & Interpretation

Abstract

The automatic detection and interpretation of network traffic anomalies through machine learning is a well-known problem, for which no general solution is available. Both supervised and unsupervised (i.e., anomaly detection) approaches require prior knowledge on the monitoring data, either in terms of normal operation profiles or on the specific anomalies to detect. As a consequence, both approaches have clear limitations when it comes to detecting, and in particular interpreting, previously unseen events. We present HUMAN, a general hierarchical-clustering-based approach for unsupervised network traffic analysis, which can both detect and interpret anomalous behaviors in a completely black-box manner, without relying on ground-truth on the system under analysis. HUMAN can detect and interpret complex patterns in the analyzed data, using a structural approach which exploits hierarchical cluster relationships and correlation among features. We describe the building blocks of HUMAN and explain its functioning in detail, using as case study the detection and interpretation of performance issues in major cloud platforms, through the unsupervised analysis of distributed active cloud latency measurements. The HUMAN approach can be applied to the unsupervised analysis of any kind of nested or hierarchically structured multi-dimensional data, showing the potential of hierarchical clustering for general unsupervised data analytics.