The DOI matches its Crossref record; 55 of the 55 references it cites were found in a registry or an index.
Opprentice: Towards Practical and Automatic Anomaly Detection Through Machine Learning
Abstract
Closely monitoring service performance and detecting anomalies are critical for Internet-based services. However, even though dozens of anomaly detectors have been proposed over the years, deploying them to a given service remains a great challenge, requiring manually and iteratively tuning detector parameters and thresholds. This paper tackles this challenge through a novel approach based on supervised machine learning. With our proposed system, Opprentice (Operators' apprentice), operators' only manual work is to periodically label the anomalies in the performance data with a convenient tool. Multiple existing detectors are applied to the performance data in parallel to extract anomaly features. Then the features and the labels are used to train a random forest classifier to automatically select the appropriate detector-parameter combinations and the thresholds. For three different service KPIs in a top global search engine, Opprentice can automatically satisfy or approximate a reasonable accuracy preference (recall >= 0.66 and precision>= 0.66). More importantly, Opprentice allows operators to label data in only tens of minutes, while operators traditionally have to spend more than ten days selecting and tuning detectors, which may still turn out not to work in the end.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Breiman, L. (2001) · Machine Learning · doi:10.1023/a:1010933404324 · Cited by 133,538
Scikit-learn: Machine Learning in Python
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Müller, A., Nothman, J., Louppe, G., et al. (2012) · arXiv · doi:10.48550/arxiv.1201.0490 · Cited by 71,016
Classification and Regression Trees.
Gordon, A. D., Breiman, L., Friedman, J. H., Olshen, R. A., & Stone, C. J. (1984) · Biometrics · doi:10.2307/2530946 · Cited by 23,522
Classification and Regression Trees.
Ryzin, J. V., Breiman, L., Friedman, J. H., Olshen, R. A., & Stone, C. J. (1986) · Journal of the American Statistical Association · doi:10.2307/2288003 · Cited by 20,825
Time Series Analysis: Forecasting and Control
Geurts, M., Box, G. E. P., & Jenkins, G. M. (1977) · Journal of Marketing Research · doi:10.2307/3150485 · Cited by 19,186
Classification and Regression by randomForest
Liaw, A. I., & Wiener, M. C. (2007) · Cited by 18,594
Chandola, V., Banerjee, A., & Kumar, V. (2009) · ACM Computing Surveys · doi:10.1145/1541880.1541882 · Cited by 11,687 · cited 4×
“For example, anomalies can be rare, and new types of anomalies might appear in the future [16].”
BackgroundMethodInfluentialClassification And Regression Trees
Breiman, L., Friedman, J. H., Olshen, R. A., & Stone, C. J. (2017) · doi:10.1201/9781315139470 · Cited by 11,417
A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection
Kohavi, R. (1995) · Zenodo · doi:10.5281/zenodo.19712698 · Cited by 10,909
Peng, H., Long, F., & Ding, C. (2005) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2005.159 · Cited by 10,544
He, H., & Garcia, E. A. (2009) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2008.239 · Cited by 10,359
Wrappers for feature subset selection
Kohavi, R., & John, G. H. (1997) · Artificial Intelligence · doi:10.1016/s0004-3702(97)00043-x · Cited by 8,947
Scikit-learn: Machine Learning in Python
(2011) · Journal of Machine Learning Research · doi:10.5555/1953048.2078195 · Cited by 8,539
An introduction to variable and feature selection
GuyonIsabelle, & ElisseeffAndré (2003) · Journal of Machine Learning Research · doi:10.5555/944919.944968 · Cited by 7,715
The relationship between Precision-Recall and ROC curves
Davis, J., & Goadrich, M. (2006) · Proceedings of the 23rd international conference on Machine learning - ICML '06 · doi:10.1145/1143844.1143874 · Cited by 6,559
Time Series Analysis: Forecasting and Control.
Anderson, O. D., Box, G. E. P., & Jenkins, G. M. (1978) · The Statistician · doi:10.2307/2988198 · Cited by 5,298
A Short Introduction to Boosting
Freund, Y., & Schapire, R. E. (1999) · Cited by 3,140
Gene selection and classification of microarray data using random forest
Díaz-Uriarte, R., & Alvarez de Andrés, S. (2006) · BMC Bioinformatics · doi:10.1186/1471-2105-7-3 · Cited by 2,929
Time series analysis forecasting and control
Janacek, G. (2010) · Journal of Time Series Analysis · doi:10.1111/j.1467-9892.2009.00643.x · Cited by 2,584
Perkins, N. J., & Schisterman, E. F. (2006) · American Journal of Epidemiology · doi:10.1093/aje/kwj063 · Cited by 1,759
Feature extraction methods for character recognition-A survey
Due Trier, Ø., Jain, A. K., & Taxt, T. (1996) · Pattern Recognition · doi:10.1016/0031-3203(95)00118-2 · Cited by 1,264
Mining anomalies using traffic feature distributions
Lakhina, A., Crovella, M., & Diot, C. (2005) · Proceedings of the 2005 conference on Applications, technologies, architectures, and protocols for computer communications · doi:10.1145/1080091.1080118 · Cited by 1,054
Diagnosing network-wide traffic anomalies
Lakhina, A., Crovella, M., & Diot, C. (2004) · Proceedings of the 2004 conference on Applications, technologies, architectures, and protocols for computer communications · doi:10.1145/1015467.1015492 · Cited by 1,039
Time Series Analysis Forecasting and Control.
Chakravarti, I. M., Box, G. E. P., & Jenkins, G. M. (1973) · Journal of the American Statistical Association · doi:10.2307/2284112 · Cited by 972
The problem of concept drift: definitions and related work
Tsymbal, A. (2004) · Cited by 929
A signal analysis of network traffic anomalies
Barford, P., Kline, J., Plonka, D., & Ron, A. (2002) · Proceedings of the second ACM SIGCOMM Workshop on Internet measurment - IMW '02 · doi:10.1145/637201.637210 · Cited by 787
Sketch-based change detection: methods, evaluation, and applications
Krishnamurthy, B., Sen, S., Zhang, Y., & Chen, Y. (2003) · Proceedings of the 2003 ACM SIGCOMM conference on Internet measurement - IMC '03 · doi:10.1145/948205.948236 · Cited by 528
A hybrid machine learning approach to network anomaly detection
Shon, T., & Moon, J. (2007) · Information Sciences · doi:10.1016/j.ins.2007.03.025 · Cited by 440
Area under the Precision-Recall Curve: Point Estimates and Confidence Intervals
Boyd, K., Eng, K. H., & Page, C. D. (2013) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-40994-3_29 · Cited by 381
Developing a predictive model of quality of experience for internet video
Balachandran, A., Sekar, V., Akella, A., Seshan, S., Stoica, I., & Zhang, H. (2013) · Proceedings of the ACM SIGCOMM 2013 conference on SIGCOMM · doi:10.1145/2486001.2486025 · Cited by 349
ANTIDOTE: understanding and defending against poisoning of anomaly detectors
Rubinstein, B. I. P., Nelson, B., Huang, L., Joseph, A. D., Lau, S.-H., Rao, S., Taft, N., & Tygar, J. D. (2009) · Proceedings of the 9th ACM SIGCOMM conference on Internet measurement · doi:10.1145/1644893.1644895 · Cited by 331
Aberrant Behavior Detection in Time Series for Network Monitoring
Brutlag, J. D. (2000) · USENIX Large Installation Systems Administration Conference · Cited by 296
Fontugne, R., Borgnat, P., Abry, P., & Fukuda, K. (2010) · Proceedings of the 6th International COnference · doi:10.1145/1921168.1921179 · Cited by 295 · cited 3×
“We also compare random forests with two static combination methods: the normalization schema [21] and the majority-vote [8].”
BackgroundMethodCombining Filtering and Statistical Methods for Anomaly Detection
Soule, A., Salamatian, K., & Taft, N. (2005) · Cited by 258
Detection and identification of network anomalies using sketch subspaces
Li, X., Bian, F., Crovella, M., Diot, C., Govindan, R., Iannaccone, G., & Lakhina, A. (2006) · Proceedings of the 6th ACM SIGCOMM conference on Internet measurement · doi:10.1145/1177080.1177099 · Cited by 232
Zhang⋆, Y., Ge, Z., Greenberg, A. G., & Roughan, M. (2005) · doi:10.5555/1251086.1251116 · Cited by 209
Impact of response latency on user behavior in web search
Arapakis, I., Bai, X., & Cambazoglu, B. B. (2014) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/2600428.2609627 · Cited by 164
Erratum: Area under the Precision-Recall Curve: Point Estimates and Confidence Intervals
Boyd, K., Eng, K. H., & Page, C. D. (2013) · Lecture notes in computer science · doi:10.1007/978-3-642-40994-3_55 · Cited by 159
Crowdsourcing service-level network event monitoring
Choffnes, D., Bustamante, F. E., & Ge, Z. (2010) · ACM SIGCOMM Conference (SIGCOMM) · doi:10.1145/1851182.1851228 · Cited by 98
Rapid detection of maintenance induced changes in service performance
Mahimkar, A. A., Ge, Z., Wang, J., Yates, J. M., Zhang⋆, Y., Emmons, J., Huntley, B. J., & Stockert, M. (2011) · ACM International Conference on emerging Networking EXperiments and Technologies (CoNEXT) · doi:10.1145/2079296.2079309 · Cited by 91
Argus: End-to-end service anomaly detection and localization from an ISP's point of view
Yan, H., Flavel, A., Ge, Z., Gerber, A., Dan Massey, G., Papadopoulos, C., Shah, H., & Yates, J. M. (2012) · IEEE Conference on Computer Communications (INFOCOM) · doi:10.1109/infcom.2012.6195694 · Cited by 84
G-RCA: a generic root cause analysis platform for service quality management in large IP networks
Yan, H., Breslau, L., Ge, Z., Massey, D., Pei, D., & Yates, J. (2012) · IEEE/ACM Transactions on Networking · doi:10.1109/tnet.2012.2188837 · Cited by 72
Self-citationDetecting the performance impact of upgrades in large operational networks
Mahimkar, A. A., Song, H. H., Ge, Z., Shaikh, A., Wang, J., Yates, J. M., Zhang⋆, Y., & Emmons, J. (2010) · ACM SIGCOMM Conference (SIGCOMM) · doi:10.1145/1851182.1851219 · Cited by 72
A provider-side view of web search response time
Chen, Y., Mahajan, R., Sridharan, B., & Zhang, Z.-L. (2013) · ACM SIGCOMM Conference (SIGCOMM) · doi:10.1145/2486001.2486035 · Cited by 70
Silveira, F., Diot, C., Taft, N., & Govindan, R. (2010) · ACM SIGCOMM Conference (SIGCOMM) · doi:10.1145/1851182.1851215 · Cited by 68
URCA: Pulling out Anomalies by their Root Causes
Silveira, F., & Diot, C. (2010) · IEEE Conference on Computer Communications (INFOCOM) · doi:10.1109/infcom.2010.5462151 · Cited by 65
Calders, T., Esposito, F., Hüllermeier, E., & Meo, R. (2014) · Lecture notes in computer science · doi:10.1007/978-3-662-44845-8 · Cited by 65
Anomaly extraction in backbone networks using association rules
Brauckhoff, D., Dimitropoulos, X. A., Wagner, A., & Salamatian, K. (2009) · ACM Internet Measurement Conference (IMC) · doi:10.1145/1644893.1644897 · Cited by 57
Accurate anomaly detection through parallelism
Shanbhag, S., & Wolf, T. (2009) · IEEE Network · doi:10.1109/mnet.2009.4804320 · Cited by 52
Change-point detection with feature selection in high-dimensional time-series data
Yamada, M., Kimura, A., Naya, F., & Sawada, H. (2013) · International Joint Conference on Artificial Intelligence · Cited by 51
Threshold compression for 3G scalable monitoring
Lee, S.-B., Pei, D., Hajiaghayi, M., Pefkianakis, I., Lu, S., Yan, H., Ge, Z., Yates, J. M., & Kosseifi, M. (2012) · IEEE Conference on Computer Communications (INFOCOM) · doi:10.1109/infcom.2012.6195498 · Cited by 29
Self-citationYan, H., Breslau, L., Ge, Z., Dan Massey, G., Pei, D., & Yates, J. M. (2010) · Proceedings of the 6th International Conference · doi:10.1145/1921168.1921175 · Cited by 25
Self-citationRingberg, H., Soule, A., & Rexford, J. (2008) · ACM SIGCOMM Computer Communication Review · doi:10.1145/1341431.1341437 · Cited by 24
An Automatic and Dynamic Parameter Tuning of a Statistics-Based Anomaly Detection Algorithm
Himura, Y., Fukuda, K., Cho, K., & Esaki, H. (2009) · International Conference on Communications · doi:10.1109/icc.2009.5198722 · Cited by 23
An Information-Theoretic Combining Method for Multi-Classifier Anomaly Detection Systems
Ashfaq, A. B., Javed, M., Khayam, S. A., & Radha, H. (2010) · International Conference on Communications · doi:10.1109/icc.2010.5501984 · Cited by 21
Cited by
Unsupervised Anomaly Detection via Variational Auto-Encoder for Seasonal KPIs in Web Applications
Xu, H., Feng, Y., Chen, J., Wang, Z., Qiao, H., Chen, W., Zhao, N., Li, Z., Bu, J., Li, Z., et al. (2018) · Proceedings of the 2018 World Wide Web Conference on World Wide Web - WWW '18 · doi:10.1145/3178876.3185996 · Cited by 808
Self-citationMachine Learning for Anomaly Detection: A Systematic Review
Nassif, A. B., Talib, M. A., Nasir, Q., & Dakalbab, F. M. (2021) · IEEE Access · doi:10.1109/access.2021.3083060 · Cited by 748
Time-Series Anomaly Detection Service at Microsoft
Ren, H., Xu, B., Wang, Y., Yi, C., Huang, C., Kou, X., Xing, T., Yang, M., Tong, J., & Zhang, Q. (2019) · Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining · doi:10.1145/3292500.3330680 · Cited by 623
A comprehensive survey and taxonomy of the SVM-based intrusion detection systems
Mohammadi, M., Rashid, T. A., Karim, S. H. T., Aldalwie, A. H. M., Tho, Q. T., Bidaki, M., Rahmani, A. M., & Hosseinzadeh, M. (2021) · Journal of Network and Computer Applications · doi:10.1016/j.jnca.2021.102983 · Cited by 262
A survey of AIOps methods for failure management
Notaro, P., Cardoso, J., & Gerndt, M. (2021) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3483424 · Cited by 142
Metadata
- Authors
- Liu, D., Zhao, Y., Xu, H., Sun, Y., Pei, D., Luo, J., Jing, X., & Feng, M.
- Venue
- Proceedings of the 2015 Internet Measurement Conference
- Publisher
- ACM
- Published
- 2015-10-28
- Pages
- 211-224
- OpenAlex
- W2026453187
- Cited by
- 252
- References
- 55
- FWCI
- 7.49
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.