Robust log-based anomaly detection on unstable log data
The DOI matches its Crossref record; 47 of the 47 references it cites were found in a registry or an index.
Abstract
Logs are widely used by large and complex software-intensive systems for troubleshooting. There have been a lot of studies on log-based anomaly detection. To detect the anomalies, the existing methods mainly construct a detection model using log event data extracted from historical logs. However, we find that the existing methods do not work well in practice. These methods have the close-world assumption, which assumes that the log data is stable over time and the set of distinct log events is known. However, our empirical study shows that in practice, log data often contains previously unseen log events or log sequences. The instability of log data comes from two sources: 1) the evolution of logging statements, and 2) the processing noise in log data. In this paper, we propose a new log-based anomaly detection approach, called LogRobust. LogRobust extracts semantic information of log events and represents them as semantic vectors. It then detects anomalies by utilizing an attention-based Bi-LSTM model, which has the ability to capture the contextual information in the log sequences and automatically learn the importance of different log events. In this way, LogRobust is able to identify and handle unstable log events and sequences. We have evaluated LogRobust using logs collected from the Hadoop system and an actual online service system of Microsoft. The experimental results show that the proposed approach can well address the problem of log instability and achieve accurate and robust results on real-world, ever-changing log data.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Log2Vec [48] transforms raw log messages into semantic vectors, thus can avoid errors from log parsers.”
From Log-based Anomaly Detection Without Log Parsing (2021) · cites this paper as [48]
In the citing paper
“Since DeepLog represents log messages as the indexes of log templates, it cannot prevent semantic information loss and could produce many wrong detection results [47], [48].”
From Log-based Anomaly Detection Without Log Parsing (2021) · cites this paper as [48]
In the citing paper
“Log2Vec [48] accurately extracts the semantic and syntax information from log messages and leverages the Deeplog [5] model to improve the accuracy of anomaly detection.”
From Log-based Anomaly Detection Without Log Parsing (2021) · cites this paper as [48]
In the citing paper
“We do not compare with DeepLog [5] because previous studies already showed that Log2Vec outperforms DeepLog [48].”
From Log-based Anomaly Detection Without Log Parsing (2021) · cites this paper as [48]
In the citing paper
“Its paradigm has shifted from traditional rule matching to deep Natural Language Processing (NLP) models, represented by DeepLog [12] and LogRobust [13] .”
From A Survey of Anomaly Detection and Root Cause Analysis Techniques for Artificial Intelligence for IT Operations (AIOps) in Large-Scale Software Systems (2026) · cites this paper as [13]
2 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
Term-weighting approaches in automatic text retrieval
Salton, G., & Buckley, C. (1988) · Information Processing & Management · doi:10.1016/0306-4573(88)90021-0 · Cited by 9,740
Stochastic Estimation of the Maximum of a Regression Function
Kiefer, J., & Wolfowitz, J. (1952) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177729392 · Cited by 2,133 · cited 3×
“Our attention-based Bi-LSTM is trained using a back propagation algorithm [37] and thus it can naturally be updated incrementally through the gradient descent method [25] to fine-tune its parameters automatically.”
MethodDeepLog: Anomaly Detection and Diagnosis from System Logs through Deep Learning
Du, M., Li, F., Zheng, G., & Srikumar, V. (2017) · Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3133956.3134015 · Cited by 1,835
Detecting large-scale system problems by mining console logs
Xu, W., Huang, L., Fox, A., Patterson, D., & Jordan, M. I. (2009) · Proceedings of the ACM SIGOPS 22nd symposium on Operating systems principles · doi:10.1145/1629575.1629587 · Cited by 1,332 · cited 13×
“Over the years, many automated log-based approaches have been proposed to detect system anomalies [3, 10, 18, 29, 44].”
BackgroundMethodResultInfluentialDrain: an online log parsing approach with fixed depth tree
He, P., Zhu, J., Zheng, Z., & Lyu, M. R. (2017) · 2017 IEEE International Conference on Web Services (ICWS) · doi:10.1109/icws.2017.13 · Cited by 943 · cited 3×
“Drain is proposed in [19] and is characterized by its high parsing accuracy and efficiency.”
BackgroundMethodInfluentialContinuous Delivery: Reliable Software Releases Through Build, Test, and Deployment Automation
Humble, J., & Farley, D. R. (2010) · Cited by 911
FastText.zip: Compressing text classification models
Joulin, A., Grave, E., Bojanowski, P., Douze, M., Jégou, H., & Mikolov, T. (2016) · arXiv · doi:10.48550/arxiv.1612.03651 · Cited by 887 · cited 1×
“Specifically, LogRobust leverages off-the-shelf word vectors, which were pre-trained on Common Crawl Corpus dataset using the FastText algorithm [23], to extract the semantic information from the English vocabularies.”
Method(2009) · Advanced Topics in Science and Technology in China · doi:10.1007/978-3-540-73762-9_2 · Cited by 589
Tools and benchmarks for automated log parsing
Zhu, J., He, S., Liu, J., He, P., Xie, Q., Zheng, Z., & Lyu, M. R. (2019) · 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1109/icse-seip.2019.00021 · Cited by 574 · cited 3×
“In fact, it achieves the best performance compared with other related methods evaluated in [48].”
BackgroundMethodInfluentialAnomaly detection of web-based attacks
Kruegel, C., & Vigna, G. (2003) · Proceedings of the 10th ACM conference on Computer and communications security · doi:10.1145/948109.948144 · Cited by 551 · cited 1×
“There have been a lot of studies on log-based anomaly detection[2, 4, 7, 11, 12, 14, 15, 20, 26, 28, 29, 33–35, 38, 41].”
Execution Anomaly Detection in Distributed Systems through Unstructured Log Analysis
Fu, Q., Lou, J.-G., Wang, Y., & Li, J. (2009) · 2009 Ninth IEEE International Conference on Data Mining · doi:10.1109/icdm.2009.60 · Cited by 542 · cited 2×
“There have been a lot of studies on log-based anomaly detection[2, 4, 7, 11, 12, 14, 15, 20, 26, 28, 29, 33–35, 38, 41].”
MethodSelf-citationA data clustering algorithm for mining patterns from event logs
Vaarandi, R. (2004) · Proceedings of the 3rd IEEE Workshop on IP Operations & Management (IPOM 2003) (IEEE Cat. No.03EX764) · doi:10.1109/ipom.2003.1251233 · Cited by 453 · cited 1×
“He, et al. [17] evaluated four commonly used log parsers, including SLCT [40], IPLoM [32], LKE [12] and LogSig [39].”
MethodFailure diagnosis using decision trees
Chen, M., Zheng, A. X., Lloyd, J., Jordan, M. I., & Brewer, E. (2004) · International Conference on Autonomic Computing, 2004. Proceedings. · doi:10.1109/icac.2004.1301345 · Cited by 433
Log clustering based problem identification for online service systems
Lin, Q., Zhang, H., Lou, J.-G., Zhang, Y., & Chen, X. (2016) · Proceedings of the 38th International Conference on Software Engineering Companion · doi:10.1145/2889160.2889232 · Cited by 427
Self-citationMining invariants from console logs for system problem detection
Lou, J., Fu, Q., Yang, S., Xu, Y., & Li, J. (2010) · Cited by 302
Self-citationFailure Prediction in IBM BlueGene/L Event Logs
Liang, Y., Zhang, Y., Xiong, H., & Sahoo, R. (2007) · Seventh IEEE International Conference on Data Mining (ICDM 2007) · doi:10.1109/icdm.2007.46 · Cited by 292 · cited 10×
“Over the years, many automated log-based approaches have been proposed to detect system anomalies [3, 10, 18, 29, 44].”
BackgroundMethodResultInfluentialStructured comparative analysis of systems logs to diagnose performance problems
Nagaraj, K., Killian, C. E., & Neville, J. (2012) · Purdue e-Pubs (Purdue University System) · Cited by 289
Continuous Delivery: Huge Benefits, but Challenges Too
Chen, L. (2015) · IEEE Software · doi:10.1109/ms.2015.27 · Cited by 261 · cited 1×
“This is especially important for organizations that adopt the Continue Delivery/Deployment approaches [6, 22].”
BackgroundFingerprinting the datacenter: automated classification of performance crises
Bodik, P., Goldszmidt, M., Fox, A., Woodard, D. B., & Andersen, H. (2010) · Proceedings of the 5th European conference on Computer systems · doi:10.1145/1755913.1755926 · Cited by 250
LogSig: generating system events from raw textual logs
Tang, L., Li, T., & Perng, C.-S. (2011) · Proceedings of the 20th ACM international conference on Information and knowledge management · doi:10.1145/2063576.2063690 · Cited by 244
Where do developers log? an empirical study on logging practices in industry
Fu, Q., Zhu, J., Hu, W., Lou, J.-G., Ding, R., Lin, Q., Zhang, D., & Xie, T. (2014) · Companion Proceedings of the 36th International Conference on Software Engineering · doi:10.1145/2591062.2591175 · Cited by 230 · cited 2×
“Recently, much research has been devoted to the logging practice and log data quality [13, 16, 17, 45].”
BackgroundMethodSelf-citationAn Evaluation Study on Log Parsing and Its Use in Log Mining
He, P., Zhu, J., He, S., Li, J., & Lyu, M. R. (2016) · 2016 46th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN) · doi:10.1109/dsn.2016.66 · Cited by 230 · cited 8×
“They observed that many existing log parsers are not sufficiently accurate or efficient, which make them ineligible for log parsing in modern systems [17].”
BackgroundMethodInfluentialTowards Automated Log Parsing for Large-Scale Log Data Analysis
He, P., Zhu, J., He, S., Li, J., & Lyu, M. R. (2018) · IEEE Transactions on Dependable and Secure Computing · doi:10.1109/tdsc.2017.2762673 · Cited by 230 · cited 8×
“Over the years, many automated log-based approaches have been proposed to detect system anomalies [3, 10, 18, 29, 44].”
BackgroundMethodInfluentialLogMine: Fast Pattern Recognition for Log Analytics
Hamooni, H., Debnath, B., Xu, J., Zhang, H., Jiang, G., & Mueen, A. (2016) · Proceedings of the 25th ACM International on Conference on Information and Knowledge Management · doi:10.1145/2983323.2983358 · Cited by 227
Self-citationLearning to Log: Helping Developers Make Informed Logging Decisions
Zhu, J., He, P., Fu, Q., Zhang, H., Lyu, M. R., & Zhang, D. (2015) · 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering · doi:10.1109/icse.2015.60 · Cited by 212 · cited 1×
“For example, an empirical study [47] on two Microsoft systems and two open source projects shows that logging is commonly used.”
BackgroundSelf-citationIdentifying impactful service system problems via log analysis
He, S., Lin, Q., Lou, J.-G., Zhang, H., Lyu, M. R., & Zhang, D. (2018) · Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3236024.3236083 · Cited by 201 · cited 2×
“There have been a lot of studies on log-based anomaly detection[2, 4, 7, 11, 12, 14, 15, 20, 26, 28, 29, 33–35, 38, 41].”
Self-citationRecurrent Neural Network Attention Mechanisms for Interpretable System Log Anomaly Detection
Brown, A., Tuor, A., Hutchinson, B., & Nichols, N. (2018) · Proceedings of the First Workshop on Machine Learning for Computing Systems · doi:10.1145/3217871.3217872 · Cited by 198 · cited 1×
“[5] combined various attention-based models and word vector to detect anomalous log events.”
BackgroundDetection of Early-Stage Enterprise Infection by Mining Large-Scale Log Data
Oprea, A., Li, Z., Yen, T.-F., Chin, S. H., & Alrwais, S. (2015) · 2015 45th Annual IEEE/IFIP International Conference on Dependable Systems and Networks · doi:10.1109/dsn.2015.14 · Cited by 174 · cited 1×
“There have been a lot of studies on log-based anomaly detection[2, 4, 7, 11, 12, 14, 15, 20, 26, 28, 29, 33–35, 38, 41].”
Self-citationAutomated IT system failure prediction: A deep learning approach
Zhang, K., Xu, J., Min, M. R., Jiang, G., Pelechrinis, K., & Zhang, H. (2016) · 2016 IEEE International Conference on Big Data (Big Data) · doi:10.1109/bigdata.2016.7840733 · Cited by 171
Self-citationCharacterizing Logging Practices in Open-Source Software
Yuan, D., Park, S., & Zhou, Y. (2012) · Cited by 148
A Lightweight Algorithm for Message Type Extraction in System Application Logs
Makanju, A., Zincir-Heywood, A. N., & Milios, E. E. (2012) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2011.138 · Cited by 145 · cited 1×
“He, et al. [17] evaluated four commonly used log parsers, including SLCT [40], IPLoM [32], LKE [12] and LogSig [39].”
MethodExperience Report: Log Mining Using Natural Language Processing and Application to Anomaly Detection
Bertero, C., Roy, M., Sauvanaud, C., & Tredan, G. (2017) · 2017 IEEE 28th International Symposium on Software Reliability Engineering (ISSRE) · doi:10.1109/issre.2017.43 · Cited by 139 · cited 1×
“[1] used word2vec and traditional classifiers, like SVM and Random Forest, to check whether a log event is an anomaly or not.”
MethodAutomated Identification of Failure Causes in System Logs
Mariani, L., & Pastore, F. (2008) · 2008 19th International Symposium on Software Reliability Engineering (ISSRE) · doi:10.1109/issre.2008.48 · Cited by 127 · cited 1×
“There have been a lot of studies on log-based anomaly detection[2, 4, 7, 11, 12, 14, 15, 20, 26, 28, 29, 33–35, 38, 41].”
Examining the stability of logging statements
Kabinna, S., Bezemer, C., Shang, W., Syer, M. D., & Hassan, A. E. (2017) · Empirical Software Engineering · doi:10.1007/s10664-017-9518-0 · Cited by 119 · cited 1×
“As Kabinna, et al. [24] observed, around 20% ∼ 45% of logging statements in their studied projects changed throughout their lifetime.”
BackgroundFarshchi, M., Schneider, J.-G., Weber, I., & Grundy, J. (2015) · 2015 IEEE 26th International Symposium on Software Reliability Engineering (ISSRE) · doi:10.1109/issre.2015.7381796 · Cited by 104 · cited 2×
“There have been a lot of studies on log-based anomaly detection[2, 4, 7, 11, 12, 14, 15, 20, 26, 28, 29, 33–35, 38, 41].”
MethodSALSA: analyzing logs as state machines
Tan, J., Pan, X., Kavulya, S. P., Gandhi, R., & Narasimhan, P. (2008) · Cited by 95
Characterizing the natural language descriptions in software logging statements
He, P., Chen, Z., He, S., & Lyu, M. R. (2018) · Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering · doi:10.1145/3238147.3238193 · Cited by 91 · cited 1×
“Recently, much research has been devoted to the logging practice and log data quality [13, 16, 17, 45].”
BackgroundCan Better Identifier Splitting Techniques Help Feature Location?
Dit, B., Guerrouj, L., Poshyvanyk, D., & Antoniol, G. (2011) · 2011 IEEE 19th International Conference on Program Comprehension · doi:10.1109/icpc.2011.47 · Cited by 89 · cited 1×
“LogRobust splits these composite tokens into individual tokens according to Camel Case [9].”
Anomaly Detection from Log Files Using Data Mining Techniques
Breier, J., & Branišová, J. (2015) · Lecture notes in electrical engineering · doi:10.1007/978-3-662-46578-3_53 · Cited by 85 · cited 8×
“Over the years, many automated log-based approaches have been proposed to detect system anomalies [3, 10, 18, 29, 44].”
BackgroundMethodResultInfluentialStudying and detecting log-related issues
Hassani, M., Shang, W., Shihab, E., & Tsantalis, N. (2018) · Empirical Software Engineering · doi:10.1007/s10664-018-9603-z · Cited by 79
Breier, J., & Branišová, J. (2015) · Wireless Personal Communications · doi:10.1007/s11277-015-3128-1 · Cited by 52 · cited 1×
“There have been a lot of studies on log-based anomaly detection[2, 4, 7, 11, 12, 14, 15, 20, 26, 28, 29, 33–35, 38, 41].”
FLAP: An End-to-End Event Log Analysis Platform for System Management
Li, T., Jiang, Y., Zeng, C., Xia, B., Liu, Z., Zhou, W., Zhu, X., Wang, W., Zhang, L., Wu, J., et al. (2017) · Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/3097983.3098022 · Cited by 45
System problem detection by mining console logs
Patterson, D. A., & Xu, W. (2010) · Cited by 37
Information Science and Applications
Kim, K. J., & Kim, H.-Y. (2019) · Lecture notes in electrical engineering · doi:10.1007/978-981-15-1465-4 · Cited by 32
Ferraudo, A. S. (2014) · CABI eBooks · doi:10.1079/9781780644110.0088 · Cited by 18
(2015) · doi:10.4230/oasics.slate.2021.8 · Cited by 1 · cited 3×
“Bi-LSTM splits the hidden neurons layer of a standard LSTM into two directions: the forward pass and the backward pass [21], in order to capture sufficient information of input log sequences in both directions.”
MethodInfluential
Research works printed but not linked: 5
- 10.5555/2228298.2228334 · doi:10.5555/2228298.2228334
- R Vinayakumar , KP Soman , and Prabaharan Poornachandran . 2017 . Long shortterm memory based operation log anomaly detection. In Advances in Computing , Communications and Informatics (ICACCI), 2017 International Conference on. IEEE, 236–242 . R Vinayakumar, KP Soman, and Prabaharan Poornachandran. 2017. Long shortterm memory based operation log anomaly detection. In Advances in Computing, Communications and Informatics (ICACCI), 2017 International Conference on. IEEE, 236–242.
- 10.1145/1629575.1629587 · DOI · Paper page
- 10.5555/2337223.2337236 · doi:10.5555/2337223.2337236
- 10.5555/2818754.2818807 · doi:10.5555/2818754.2818807
Cited by
LogBERT: Log Anomaly Detection via BERT
Guo, H., Yuan, S., & Wu, X. (2021) · 2021 International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn52387.2021.9534113 · Cited by 454
“Recently, many deep learning-based log anomaly detection approaches are proposed for log anomaly detection [21,2,22,23,8,17].”
Log-based Anomaly Detection Without Log Parsing
Le, V.-H., & Zhang, H. (2021) · 2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase51524.2021.9678773 · Cited by 361
“In this section, we evaluate the impact of log parsing errors on two representative anomaly detection methods SVM-based method [30], and LogRobust [8].”
BackgroundMethodResultInfluentialSelf-citationA Survey on Automated Log Analysis for Reliability Engineering
He, S., He, P., Chen, Z., Yang, T., Su, Y., & Lyu, M. R. (2022) · ACM Computing Surveys · doi:10.1145/3460345 · Cited by 334
“Poorly written static text and inconsistent dynamic content could affect the subsequent diagnosis and maintenance activities [86, 111, 136, 196].”
BackgroundMethodSemi-Supervised Log-Based Anomaly Detection via Probabilistic Label Estimation
Yang, L., Chen, J., Wang, Z., Wang, W., Jiang, J., Dong, X., & Zhang, W. (2021) · 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE) · doi:10.1109/icse43902.2021.00130 · Cited by 244
Self-citationDeep learning for anomaly detection in log data: A survey
Landauer, M., Onder, S., Skopik, F., & Wurzenberger, M. (2023) · Machine Learning with Applications · doi:10.1016/j.mlwa.2023.100470 · Cited by 202
“Experiments suggest that Bi-LSTM RNNs outperform LSTM RNNs [17], [34], [46], [50], [55], [57], [72], [75].”
BackgroundMethodInfluential
Metadata
- Authors
- Zhang, X., Xu, Y., Lin, Q., Qiao, B., Zhang, H., Dang, Y., Xie, C., Yang, X., Cheng, Q., Li, Z., Chen, J., He, X., Yao, R., Lou, J.-G., Chintalapati, M., Shen, F., & Zhang, D.
- Affiliations
- Nanjing University (CN)
- Microsoft Research Asia (China) (CN)
- University of Newcastle Australia (AU)
- Microsoft (United States) (US)
- Microsoft, USA
- Tianjin University (CN)
- ORCID
- Venue
- Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering
- Publisher
- ACM
- Published
- 2019-08-12
- Pages
- 807-817
- OpenAlex
- W2947815220
- Cited by
- 761
- References
- 46
- Citations by year
- 2026: 116 · 2025: 169 · 2024: 142 · 2023: 136 · 2022: 89 · 2021: 78 · 2020: 29 · 2019: 2
- FWCI
- 42.81
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) (publishedVersion)
- NOVA (University of Newcastle Australia) (submittedVersion)
- ACM Proceedings
- Topics
- Physical Sciences › Computer Science › Computer Networks and Communications › Software System Performance and Reliability
- Physical Sciences › Computer Science › Artificial Intelligence › Anomaly Detection Techniques and Applications
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Keywords
- log anomaly detection, online service systems
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.