Towards Automated Log Parsing for Large-Scale Log Data Analysis
The DOI matches its Crossref record; 41 of the 42 references it cites were found in a registry or an index.
Abstract
Logs are widely used in system management for dependability assurance because they are often the only data available that record detailed system runtime behaviors in production. Because the size of logs is constantly increasing, developers (and operators) intend to automate their analysis by applying data mining methods, therefore structured input data (e.g., matrices) are required. This triggers a number of studies on log parsing that aims to transform free-text log messages into structured events. However, due to the lack of open-source implementations of these log parsers and benchmarks for performance comparison, developers are unlikely to be aware of the effectiveness of existing log parsers and their limitations when applying them into practice. They must often reimplement or redesign one, which is time-consuming and redundant. In this paper, we first present a characterization study of the current state of the art log parsers and evaluate their efficacy on five real-world datasets with over ten million log messages. We determine that, although the overall accuracy of these parsers is high, they are not robust across all datasets. When logs grow to a large scale (e.g., 200 million log messages), which is common in practice, these parsers are not efficient enough to handle such data on a single computer. To address the above limitations, we design and implement a parallel log parser (namely POP) on top of Spark, a large-scale data processing platform. Comprehensive experiments have been conducted to evaluate POP on both synthetic and real-world datasets. The evaluation results demonstrate the capability of POP in terms of accuracy, efficiency, and effectiveness on subsequent log mining tasks.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Introduction to information retrieval
(2009) · Choice Reviews Online · doi:10.5860/choice.46-2715 · Cited by 12,278
Introduction to Information Retrieval
Manning, C. D., Raghavan, P., & Schütze, H. (2008) · doi:10.1017/cbo9780511809071 · Cited by 11,288
Everitt, B. J., Landau, S., & Leese, M. (1974) · Cited by 9,358
Resilient distributed datasets: a fault-tolerant abstraction for in-memory cluster computing
Zaharia, M., Chowdhury, M., Das, T., Dave, A., Ma, J. H., McCauley, M., Franklin, M. J. M., Shenker, S. J., & Stoica, I. G. (2012) · UC Berkeley · Cited by 4,891
Everitt, B. S., Landau, S., Leese, M., & Stahl, D. (2011) · Wiley Series in Probability and Statistics · doi:10.1002/9780470977811 · Cited by 2,907
Detecting large-scale system problems by mining console logs
Xu, W., Huang, L., Fox, A., Patterson, D., & Jordan, M. I. (2009) · Proceedings of the ACM SIGOPS 22nd symposium on Operating systems principles · doi:10.1145/1629575.1629587 · Cited by 1,332 · cited 10×
“Furthermore, modern systems often integrate open-source software components written by hundreds of developers [3].”
BackgroundMethodInfluentialMinimum Spanning Trees and Single Linkage Cluster Analysis
Gower, J. C., & Ross, G. J. S. (1969) · Applied Statistics · doi:10.2307/2346439 · Cited by 1,223 · cited 1×
“To address over-parsing and further improve parsing accuracy, in this step, POP employs hierarchical clustering [27] to cluster similar groups based on their log events.”
MethodDiagnosing network-wide traffic anomalies
Lakhina, A., Crovella, M., & Diot, C. (2004) · Proceedings of the 2004 conference on Applications, technologies, architectures, and protocols for computer communications · doi:10.1145/1015467.1015492 · Cited by 1,039 · cited 1×
“However, the anomaly detection task evaluated is an important log mining task widely studied [37], [38], which is presented in a paper [3] enjoying more than 300 citations.”
BackgroundWhat Supercomputers Say: A Study of Five System Logs
Oliner, A., & Stearley, J. (2007) · 37th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN'07) · doi:10.1109/dsn.2007.103 · Cited by 620 · cited 2×
“Besides, we thank those who release log data [3], [33], [34], which greatly facilitates our research.”
MethodThe Complexity of Some Problems on Subsequences and Supersequences
Maier, D. (1978) · Journal of the ACM · doi:10.1145/322063.322075 · Cited by 616 · cited 1×
“The groups in the same cluster will be merged, and a new log event will be generated by calculating the Longest Common Subsequence (LCS) [28] of the original log events.”
Method
Research works printed but not linked: 4
- salfner, 2004, Comprehensive logfiles for autonomic systems, Proc 18th Int Parallel Distrib Process Symp
- krause, 1987, Taxicab Geometry
- gu, 2015, Proc 45th Annu IEEE/IFIP Int Conf Dependable Syst Netw, 57
- lang, 2013, Using SEC, LOGIN The USENIX Magazine, 38, 38
Cited by
Robust log-based anomaly detection on unstable log data
Zhang, X., Xu, Y., Lin, Q., Qiao, B., Zhang, H., Dang, Y., Xie, C., Yang, X., Cheng, Q., Li, Z., et al. (2019) · Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3338906.3338931 · Cited by 761
“Over the years, many automated log-based approaches have been proposed to detect system anomalies [3, 10, 18, 29, 44].”
BackgroundMethodInfluentialTools and benchmarks for automated log parsing
Zhu, J., He, S., Liu, J., He, P., Xie, Q., Zheng, Z., & Lyu, M. R. (2019) · 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1109/icse-seip.2019.00021 · Cited by 574
“[53] have studied large-scale log parsing through the parallelization on Spark.”
MethodSelf-citationSarker, I. H. (2021) · SN Computer Science · doi:10.1007/s42979-021-00765-8 · Cited by 570
A Survey on Automated Log Analysis for Reliability Engineering
He, S., He, P., Chen, Z., Yang, T., Su, Y., & Lyu, M. R. (2022) · ACM Computing Surveys · doi:10.1145/3460345 · Cited by 334
“ch [82] in logging practice and Deeplog [42] in anomaly detection. Besides machine learning, parallelization has been employed in various recent papers, such as Logzip [90] in log compression and POP [59] in log parsing. These extensive studies on automated log analysis across multiple core directions have largely boo…”
BackgroundMethodInfluentialSelf-citationUllah, F., Ullah, S., Srivastava, G., & Lin, J. C.-W. (2024) · Digital Communications and Networks · doi:10.1016/j.dcan.2023.03.008 · Cited by 266
Semi-Supervised Log-Based Anomaly Detection via Probabilistic Label Estimation
Yang, L., Chen, J., Wang, Z., Wang, W., Jiang, J., Dong, X., & Zhang, W. (2021) · 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE) · doi:10.1109/icse43902.2021.00130 · Cited by 244
“1) Log Parsing: As shown in Figure 1, raw log messages are unstructured data and contain variable log parameters, which could hinder automated log analysis [42], [43].”
MethodHitAnomaly: Hierarchical Transformers for Anomaly Detection in System Log
Huang, S., Liu, Y., Fung, C., He, R., Zhao, Y., Yang, H., & Luan, Z. (2020) · IEEE Transactions on Network and Service Management · doi:10.1109/tnsm.2020.3034647 · Cited by 228
“Existing log-based anomaly detection methods can be broadly classified into two categories: log event indexes based approaches (e.g., PCA [5], Invariant Mining [6], SVM [7], DeepLog [8]) and log template semantics-based approaches (e.g., LogAnomaly [9], LogRobust [10]).”
MethodInfluentialLogram: Efficient Log Parsing Using n-Gram Dictionaries
Dai, H., Li, H., Chen, C. S., Shang, W., & Chen, T.-H. (2020) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2020.3007554 · Cited by 170
Robust Failure Diagnosis of Microservice System Through Multimodal Data
Zhang, S., Jin, P., Lin, Z., Sun, Y., Zhang, B., Xia, S., Li, Z., Zhong, Z., Ma, M., Jin, W., et al. (2023) · IEEE Transactions on Services Computing · doi:10.1109/tsc.2023.3290018 · Cited by 118
“To utilize logs more effectively, researchers have proposed various parsing techniques to extract templates and parameters, e.g., FT-Tree [23], Drain [22], POP [24], MoLFI [25], Spell [26], and Logram [27].”
MethodSystem log clustering approaches for cyber security applications: A survey
Landauer, M., Skopik, F., Wurzenberger, M., & Rauber, A. (2020) · Computers & Security · doi:10.1016/j.cose.2020.101739 · Cited by 108
LogEvent2vec: LogEvent-to-Vector Based Anomaly Detection for Large-Scale Logs in Internet of Things
Wang, J., Tang, Y., He, S., Zhao, C., Sharma, P. K., Alfarraj, O., & Tolba, A. (2020) · Sensors · doi:10.3390/s20092451 · Cited by 107
“He et al. [26] design and implement a parallel log parser (namely POP) on top of Spark, a large-scale data processing platform.”
Self-citationLightLog: A lightweight temporal convolutional network for log anomaly detection on the edge
Wang, Z., Tian, J., Fang, H., Chen, L., & Qin, J. (2021) · Computer Networks · doi:10.1016/j.comnet.2021.108616 · Cited by 116
Logzip: Extracting Hidden Structures via Iterative Clustering for Log Compression
Liu, J., Zhu, J., He, S., He, P., Zheng, Z., & Lyu, M. R. (2019) · 2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2019.00085 · Cited by 97
Self-citationCharacterizing the natural language descriptions in software logging statements
He, P., Chen, Z., He, S., & Lyu, M. R. (2018) · Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering · doi:10.1145/3238147.3238193 · Cited by 91
“To facilitate log analysis, researchers also focus on related log data preprocessing topics, including log collection [12, 17] and log parsing [25, 26, 42].”
MethodSelf-citationLLMParser: An Exploratory Study on Using Large Language Models for Log Parsing
Ma, Z., Chen, A. R., Kim, D. J., Chen, T.-H., & Wang, S. (2024) · Proceedings of the IEEE/ACM 46th International Conference on Software Engineering · doi:10.1145/3597503.3639150 · Cited by 92
BERT-Log: Anomaly Detection for System Logs Based on Pre-trained Language Model
Chen, S., & Liao, H. (2022) · Applied Artificial Intelligence · doi:10.1080/08839514.2022.2145642 · Cited by 87
“Word2Vec (Wang et al. 2021) is the up-to-date encoding method proposed in the HitAnomaly (He et al. 2018) to map each word in log template to a vector.”
MethodInfluentialZoppi, T., Ceccarelli, A., Puccetti, T., & Bondavalli, A. (2023) · Computers & Security · doi:10.1016/j.cose.2023.103107 · Cited by 104
“Recent trends ( Zhang et al., 2022 ; He et al., 2017 ; Chou and Jiang, 2021 ) show how data-driven detection may provide a means to detect intrusions, as opposed to traditional rule-and signature-based mechanisms.”
Zhang, T., Qiu, H., Castellano, G., Rifai, M., Chen, C. S., & Pianese, F. (2023) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2022.3222417 · Cited by 75
“For example, POP [130] provides two types of preprocessing functions for users to trim redundant fields”
BackgroundMethodInfluentialDivLog: Log Parsing with Prompt Enhanced In-Context Learning
Xu, J., Yang, R., Huo, Y., Zhang, C., & He, P. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3597503.3639155 · Cited by 164
Self-citationA systematic literature review on automated log abstraction techniques
El-Masri, D., Petrillo, F. D. S., Guéhéneuc, Y., Hamou‐Lhadj, A., & Bouziane, A. (2020) · Information and Software Technology · doi:10.1016/j.infsof.2020.106276 · Cited by 93
“The time complexity of SLCT is O ( n ) where n is the number of raw log messages [12, 14].”
BackgroundMethodInfluentialHeterogeneous Anomaly Detection for Software Systems via Semi-supervised Cross-modal Attention
Lee, C., Yang, T., Chen, Z., Su, Y., Yang, Y., & Lyu, M. R. (2023) · 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) · doi:10.1109/icse48619.2023.00148 · Cited by 68
“As aforementioned, raw log messages are unstructured and contain variables that can hinder log analysis [19].”
Self-citationAnomaly Detection in Operating System Logs with Deep Learning-Based Sentiment Analysis
Studiawan, H., Sohel, F. A., & Payne, C. (2020) · IEEE Transactions on Dependable and Secure Computing · doi:10.1109/tdsc.2020.3037903 · Cited by 73
PLELog: Semi-Supervised Log-Based Anomaly Detection via Probabilistic Label Estimation
Yang, L., Chen, J., Wang, Z., Wang, W., Jiang, J., Dong, X., & Zhang, W. (2021) · 2021 IEEE/ACM 43rd International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse-companion52605.2021.00106 · Cited by 66
Unsupervised Cross-system Log Anomaly Detection via Domain Adaptation
Han, X., & Yuan, S. (2021) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3459637.3482209 · Cited by 75
“In recent years, many machine learning-based log anomaly detection approaches are proposed to detect anomalous log sequences [2, 4, 5, 7–11 , 13– 15 ].”
BackgroundLogUAD: Log Unsupervised Anomaly Detection Based on Word2Vec
Wang, J., Zhao, C., He, S., Gu, Y., Alfarraj, O., & Abugabah, A. (2021) · Computer Systems Science and Engineering · doi:10.32604/csse.2022.022365 · Cited by 68
“After log parsing, the percent of OOV words can exceed 80% [21].”
Self-citationA Survey on Log Anomaly Detection using Deep Learning
Yadav, R. B., Kumar, P. S., & Dhavale, S. V. (2020) · 2020 8th International Conference on Reliability, Infocom Technologies and Optimization (Trends and Future Directions) (ICRITO) · doi:10.1109/icrito48877.2020.9197818 · Cited by 54
“Pinjia et al. [17] propose a parallel log parser(POP) and compare the accuracy of this with other log parsers.”
BackgroundAn automated approach to estimating code coverage measures via execution logs
Chen, B., Song, J., Xu, P., Hu, X., & Jiang, Z. M. (. (2018) · Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering · doi:10.1145/3238147.3238214 · Cited by 61
“, [25, 34]), which automatically recog-”
Understanding the Security Risks of Docker Hub
Liu, P., Ji, S., Fu, L., Lu, K., Zhang, X., Lee, W.-H., Lu, T., Chen, W., & Beyah, R. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58951-6_13 · Cited by 63
“Motivated by [21], we plan to include more automatic techniques to parse and analyzing logs as future work.”
MethodLi, B., Ma, S., Deng, R., Choo, K. R., & Yang, J. Q. (2022) · IEEE Transactions on Network and Service Management · doi:10.1109/tnsm.2022.3152620 · Cited by 68
“2, if the window size is set as 3 and the step size is set as 1, we can obtain three log sequences, i.e., [1, 2, 3], [2, 3, 4], and [3, 4, 5], from the log message series [1, 2, 3, 4, 5].”
BackgroundMethodBig data in cybersecurity: a survey of applications and future trends
Alani, M. M. (2021) · Journal of Reliable Intelligent Environments · doi:10.1007/s40860-020-00120-3 · Cited by 67
Zhang, C., Wang, X., Zhang, H., Zhang, H., & Han, P. (2021) · IEEE Transactions on Network and Service Management · doi:10.1109/tnsm.2021.3125967 · Cited by 49
SPINE: a scalable log parser with feedback guidance
Wang, X., Zhang, X., Li, L., He, S., Zhang, H., Liu, Y., Zheng, L., Kang, Y., Lin, Q., Dang, Y., et al. (2022) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3540250.3549176 · Cited by 63
“POP [11], as the distributed enhanced version of Drain, tried to make up for this deficiency.”
BackgroundSelf-citationBigdata logs analysis based on seq2seq networks for cognitive Internet of Things
Wu, P., Lu, Z., Zhou, Q., Lei, Z., Li, X., Qiu, M., & Hung, P. C. K. (2018) · Future Generation Computer Systems · doi:10.1016/j.future.2018.08.021 · Cited by 50
Knowledge-based anomaly detection: Survey, challenges, and future directions
Khan, A. Q., Jaouhari, S. E., Tamani, N., & Mroueh, L. (2024) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.108996 · Cited by 52
ALchemist: Fusing Application and Audit Logs for Precise Attack Provenance without Instrumentation
Yu, L., Ma, S., Zhang, Z., Tao, G., Zhang, X., Xu, D., Urias, V. E., Han, L., Ciocarlie, G. F., Yegneswaran, V., et al. (2021) · Network and Distributed System Security Symposium · doi:10.14722/ndss.2021.24445 · Cited by 71
“…is related to data provenance [65], [17], [62], [21], [15], [64], [37], [55], [30] [69], audit logging [57], [41], [40], [62], [18], [61], [36], [75], [74], log parsing [79], [70], [58], [34], [66], [26] and causality analysis [80], [42], [41], [40], [35], [45], [52], [53], [54], [43] [77], [78].”
An Approach to Cloud Execution Failure Diagnosis Based on Exception Logs in OpenStack
Yuan, Y., Shi, W., Liang, B., & Qin, B. (2019) · 2019 IEEE 12th International Conference on Cloud Computing (CLOUD) · doi:10.1109/cloud.2019.00031 · Cited by 44
LayerLog: Log sequence anomaly detection based on hierarchical semantics
Zhang, C., Wang, X., Zhang, H., Zhang, J., Zhang, H., Liu, C., & Han, P. (2022) · Applied Soft Computing · doi:10.1016/j.asoc.2022.109860 · Cited by 54
Temporal Logical Attention Network for Log-Based Anomaly Detection in Distributed Systems
Liu, Y., Ren, S., Wang, X., & Zhou, M. (2024) · Sensors · doi:10.3390/s24247949 · Cited by 110
“He et al. [18] employed Principal Component Analysis (PCA) to detect anomalies in system logs.”
Improving Performances of Log Mining for Anomaly Prediction Through NLP-Based Log Parsing
Aussel, N., Petetin, Y., & Chabridon, S. (2018) · Proceedings - International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems · doi:10.1109/mascots.2018.00031 · Cited by 56
“The last step presented in Table IV is the comparison of the best combination, LDA, bi-gram, with existing results from the studies [6], [8] and [9] on a public benchmark dataset from an HDFS cluster.”
BackgroundLog-based software monitoring: a systematic mapping study
Cândido, J., Aniche, M. F., & van Deursen, A. (2021) · PeerJ Computer Science · doi:10.7717/peerj-cs.489 · Cited by 86
“Indeed, as mentioned early, He et al. (2018b) leverages Spark for parallel parsing because of its fast in-memory processing.”
BackgroundLogNADS: Network anomaly detection scheme based on log semantics representation
Liu, X., Liu, W., Di, X., Li, J., Cai, B., Ren, W., & Yang, H. (2021) · Future Generation Computer Systems · doi:10.1016/j.future.2021.05.024 · Cited by 36
Self-citationLogan: A Distributed Online Log Parser
Agrawal, A., Karlupia, R., & Gupta, R. K. (2019) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde.2019.00211 · Cited by 37
Anomaly Detection via Mining Numerical Workflow Relations from Logs
Zhang, B., Zhang, H., Moscato, P., & Zhang, A. (2020) · IEEE Symposium on Reliable Distributed Systems (SRDS) · doi:10.1109/srds51746.2020.00027 · Cited by 46
Research on anomaly detection and real-time reliability evaluation with the log of cloud platform
Wang, B., Hua, Q., Zhang, H., Tan, X. L., Nan, Y., Chen, R., & Shu, X. (2022) · Alexandria Engineering Journal · doi:10.1016/j.aej.2021.12.061 · Cited by 46
Zhao, J., Liu, X., Yan, Q., Li, B., Shao, M., Peng, H. M., & Sun, L. (2020) · Computers & Security · doi:10.1016/j.cose.2020.102152 · Cited by 33
DeepTaskAPT: Insider APT detection using Task-tree based Deep Learning
Mamun, M., & Shi, K. (2021) · 2021 IEEE 20th International Conference on Trust, Security and Privacy in Computing and Communications (TrustCom) · doi:10.1109/trustcom53373.2021.00102 · Cited by 29
Studying Duplicate Logging Statements and Their Relationships With Code Clones
Li, Z., Chen, T.-H. P., Yang, J., & Shang, W. (2021) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2021.3060918 · Cited by 38
Bhanage, D. A., Pawar, A., & Kotecha, K. V. (2021) · IEEE Access · doi:10.1109/access.2021.3128283 · Cited by 55
“POP [39] operate on parallel processing; this uses distributed computing to speed up the parsing process of large scale logs.”
MethodA Directed Acyclic Graph Approach to Online Log ParsingPreprint
He, P., Zhu, J., Xu, P., Zheng, Z., & Lyu, M. R.-T. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.04356 · Cited by 42
“We focus on the first token and the last token in this layer, because according return false to our previous experience on log parsing [23], [28], [29], developers tend to either start or end a log message with a constant.”
BackgroundMethodInfluentialSelf-citationUsing log analytics and process mining to enable self-healing in the Internet of Things
Singh, P., Azari, M. S., Vitale, F., Flammini, F., Mazzocca, N., Caporuscio, M., & Thornadtsson, J. (2022) · Environment Systems & Decisions · doi:10.1007/s10669-022-09859-x · Cited by 32
“Other tedious methods, such as parsing Event Logs (He et al. 2017; Du and Li 2016) from each entity, may have to be performed, in case internal access is unavailable.”
MethodResearch on Log Anomaly Detection Based on Sentence-BERT
Hu, C., Xuekui, S., Dai, H., Zhang, H., & Liu, H. (2023) · Electronics · doi:10.3390/electronics12173580 · Cited by 29
“Supervised machine learning methods include Support Vector Classifier (SVM) [9,10], Linear Regression (LR) [11,12], Decision Tree (DT) [13], KNeighborhood Algorithm (KNN) [14], etc.”
MethodA Survey of Deep Anomaly Detection for System Logs
Zhao, X., Jiang, Z., & Ma, J. (2022) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn55064.2022.9892726 · Cited by 29
“POP [26] is a parallel log parser based on heuristic rules and iterative partitioning.”
MethodZhang, B., Gao, Y., Kuang, B., Yu, C., Fu, A., & Susilo, W. (2024) · ACM Computing Surveys · doi:10.1145/3700749 · Cited by 38
Anomaly Detection on Interleaved Log Data With Semantic Association Mining on Log-Entity Graph
Chu, G., Wang, J., Qi, Q., Sun, H., Zhuang, Z., He, B., Jing, Y., Zhang, L., & Liao, J. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3527856 · Cited by 28
“PCA [43] is a widely-used machine learning algorithm for dimension reduction.”
AllInfoLog: Robust Diverse Anomalies Detection Based on All Log Features
Xiao, R., Chen, H., Lu, J., Li, W., & Jin, S. (2022) · IEEE Transactions on Network and Service Management · doi:10.1109/tnsm.2022.3224974 · Cited by 28
“Several prior works [38], [39], [40], [41] studied log parsing methodologies.”
BackgroundAn online log template extraction method based on hierarchical clustering
Yang, R., Qu, D., Qian, Y., Dai, Y., & Zhu, S. (2019) · EURASIP Journal on Wireless Communications and Networking · doi:10.1186/s13638-019-1430-4 · Cited by 28
Asynchronous Real-Time Federated Learning for Anomaly Detection in Microservice Cloud Applications
Raeiszadeh, M., Ebrahimzadeh, A., Glitho, R., Eker, J., & Mini, R. A. F. (2025) · IEEE Transactions on Machine Learning in Communications and Networking · doi:10.1109/tmlcn.2025.3527919 · Cited by 27
Confidence guided anomaly detection model for anti-concept drift in dynamic logs
Xie, X., Jin, Z., Wang, J., Yang, L., Lu, Y., & Li, T. (2020) · Journal of Network and Computer Applications · doi:10.1016/j.jnca.2020.102659 · Cited by 22
Rosenberg, C. M., & Moonen, L. (2020) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1145/3382494.3410684 · Cited by 23
“in the events in a log [32, 33]. He et al. analyze the quality of these log parsers and conclude that many of them are not accurate or efficient enough for parsing the logs of modern software systems [34]. In contrast to these automated approaches, our study uses a handcrafted log abstracter developed by Cisco Norway.…”
MethodWang, Q., Zhang, X., Wang, X., & Cao, Z. (2021) · Entropy · doi:10.3390/e24010069 · Cited by 29
Interpretable Federated Transformer Log Learning for Cloud Threat Forensics
Parra, G. D. L. T., Selvera, L., Khoury, J. D., Irizarry, H., Bou‐Harb, E., & Rad, P. (2022) · Network and Distributed System Security Symposium · doi:10.14722/ndss.2022.23102 · Cited by 30
“Such methods include Principal Component Analysis (PCA) [56], Support Vector Machine (SVM) [20], Bi-LSTM [59], LSTM [16, 38], and Transformers [22, 53].”
MethodContext-Aware Learning for Anomaly Detection with Imbalanced Log Data
Sun, P., Yuepeng, E., Li, T., Wu, Y., Ge, J., You, J., & Wu, B. (2020) · IEEE International Conference on High Performance Computing and Communications; IEEE International Conference on Smart City; IEEE International Conference on Data Science and Systems (HPCC/SmartCity/DSS) · doi:10.1109/hpcc-smartcity-dss50907.2020.00055 · Cited by 26
“Raw logs belong to unstructured data and contain some parameters that hinder automatic log analysis [8].”
BackgroundMethodInfluentialMADMM: microservice system anomaly detection via multi-modal data and multi-feature extraction
Wang, P., Zhang, X., Cao, Z., & Chen, Z. (2024) · Neural Computing and Applications · doi:10.1007/s00521-024-09918-1 · Cited by 25
“He et al. [19] utilized statistical vectors of the events as inputs and employed support vector machine(SVM) to create a hyperplane for separating the high-dimensional space.”
A General and Efficient Querying Method for Learning to Hash
Li, J., Yan, X., Zhang, J., Xu, A. P., Cheng, J. S.-C., Liu, J., Ng, K. K. W., & Cheng, T.-C. (2018) · ACM SIGMOD International Conference on Management of Data (SIGMOD) · doi:10.1145/3183713.3183750 · Cited by 38
“The nearest neighbors search problem has been extensively studied [4, 6, 11, 12, 27, 37–39, 41, 42, 51].”
MethodSelf-citationQu, H., Yin, L., & Tang, X. (2020) · Applied Soft Computing · doi:10.1016/j.asoc.2020.106929 · Cited by 19
Leveraging RAG-Enhanced Large Language Model for Semi-Supervised Log Anomaly Detection
Zhang, W., Zhang, Q., Yu, E., Ren, Y., Meng, Y., Qiu, M., & Wang, J. (2024) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre62328.2024.00026 · Cited by 25
“The parameters contained in raw logs such as block id and ip address may hinder further log analysis [34], thus before implementing anomaly detection, we need to use log parsing technique to convert the raw logs into structured data.”
SSDLog: a semi-supervised dual branch model for log anomaly detection
Lu, S., Han, N., Wang, M., Wei, X., Lin, Z., & Wang, D. (2023) · World Wide Web · doi:10.1007/s11280-023-01174-y · Cited by 24
Incremental Analysis of Large-Scale System Logs for Anomaly Detection
Astekin, M., Ozcan, S., & Sözer, H. (2019) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata47090.2019.9006593 · Cited by 19
“Construct validity threats are mitigated by repeating the experiment with various samples of the benchmark dataset [1], [17], [18] that is commonly utilized in previous studies and well-known in the research domain.”
BackgroundLogOnline: A Semi-Supervised Log-Based Anomaly Detector Aided with Online Learning Mechanism
Wang, X., Song, J., Zhang, X., Tang, J., Gao, W., & Lin, Q. (2023) · 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase56229.2023.00043 · Cited by 18
“The parameter part of a log message contains much specific information such as blk id and IP address, that can hinder subsequent automated log analysis task [23].”
BackgroundHue: A User-Adaptive Parser for Hybrid Logs
Xu, J., Fu, Q., Zhu, Z., Cheng, Y., Li, Z., Ma, Y., & He, P. (2023) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3611643.3616260 · Cited by 25
“POP [11] improves Drain and provides a parallel implementation on Spark for distributed deployment.”
BackgroundSelf-citationDILAF: A framework for distributed analysis of large‐scale system logs for anomaly detection
Astekin, M., Zengin, H., & Sözer, H. (2018) · Software Practice and Experience · doi:10.1002/spe.2653 · Cited by 23
“POP and log parser of Xu et al complete this process in 100 seconds 23 and approximately 3 minutes 1 on the same dataset, respectively.”
BackgroundMethodInfluentialStrategies for evaluating visual analytics systems: A systematic review and new perspectives
Islam, M. R., Prome, S. A., Islam, L., Razzak, I., Wang, X., & Xu, G. (2023) · Information Visualization · doi:10.1177/14738716231212568 · Cited by 27
“He et al. 84 targeted automated log parsing for the large-scale log analysis of modern systems.”
BackgroundAnomalyAdapters: Parameter-Efficient Multi-Anomaly Task Detection
Ünal, U., & Dağ, H. (2022) · IEEE Access · doi:10.1109/access.2022.3141161 · Cited by 21
Log Sequence Anomaly Detection Based on Template and Parameter Parsing via BERT
Chai, X., Zhang, H., Zhang, J., Sun, Y., & Das, S. K. (2024) · IEEE Transactions on Dependable and Secure Computing · doi:10.1109/tdsc.2024.3428538 · Cited by 26
“Therefore, most methods use ready-made log parsers [6], [7], [8] to extract the constant part in the log for analysis before anomaly detection [9], [10], [11], [12].”
SKDLog: Self-Knowledge Distillation-based CNN for Abnormal Log Detection
Han, N., Lu, S., Wang, D., Wang, M., Tan, X., & Wei, X. (2022) · IEEE Smart World Congress (SWC) · doi:10.1109/smartworld-uic-atc-scalcom-digitaltwin-pricomp-metaverse56740.2022.00122 · Cited by 18
The Insider Threat Detection Method of University Website Clusters Based on Machine Learning
Li, Y., & Su, Y. (2023) · 2023 6th International Conference on Artificial Intelligence and Big Data (ICAIBD) · doi:10.1109/icaibd57115.2023.10206282 · Cited by 14
“Linear regression-based methods have been widely used to solve binary classification problems (Breier et al. [14] and He et al. [15]).”
MethodA secure and efficient log storage and query framework based on blockchain
Li, W., Feng, Y., Liu, N., Li, Y., Fu, X., & Yu, Y. (2024) · Computer Networks · doi:10.1016/j.comnet.2024.110683 · Cited by 26
Multi-source Anomaly Detection in Distributed IT Systems
Bogatinovski, J., & Nedelkoski, S. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-76352-7_22 · Cited by 15
LogAttn: Unsupervised Log Anomaly Detection with an AutoEncoder Based Attention Mechanism
Zhang, L., Li, W., Zhang, Z., Lu, Q., Hou, C., Hu, P. C., Gui, T., & Lu, S. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-82153-1_19 · Cited by 16
Gait analysis and recognition prediction of the human skeleton based on migration learning
Sun, C., Wang, C., & Lai, W. (2019) · Physica A Statistical Mechanics and its Applications · doi:10.1016/j.physa.2019.121812 · Cited by 15
Distributed system anomaly detection using deep learning‐based log analysis
Han, P., Li, H., Xue, G., & Zhang, C. (2023) · Computational Intelligence · doi:10.1111/coin.12573 · Cited by 27
Detecting abnormal behavior of automatic test equipment using autoencoder with event log data
Bae, Y.-M., Kim, Y.-G., Seo, J.-W., Kim, H.-A., Shin, C.-H., Son, J.-H., Lee, G., & Kim, K.-J. (2023) · Computers & Industrial Engineering · doi:10.1016/j.cie.2023.109547 · Cited by 16
Xie, X., Jian, S., Huang, C., Yu, F., & Deng, Y. (2023) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre59848.2023.00015 · Cited by 18
A robust Wide & Deep learning framework for log-based anomaly detection
Niu, W., Liao, X., Huang, S., Li, Y., Zhang, X., & Li, B. (2024) · Applied Soft Computing · doi:10.1016/j.asoc.2024.111314 · Cited by 17
Yuan, Y., Anu, H., Shi, W., Liang, B., & Qin, B. (2019) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/compsac.2019.00019 · Cited by 15
“Log parsing has been widely studied recently [7], [12], [16], [32].”
BackgroundQuickLogS: A Quick Log Parsing Algorithm based on Template Similarity
Fang, L., Di, X., Liu, X., Qin, Y., Ren, W., & Ding, Q. (2021) · IEEE International Conference on Trust, Security and Privacy in Computing and Communications (TrustCom) · doi:10.1109/trustcom53373.2021.00148 · Cited by 14
“Most log analysis techniques consist of the following three steps [12]: log parsing, matrix generation, and log mining (As shown in Fig.”
MethodGholamian, S., & Ward, P. A. S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.12489 · Cited by 41
Silpa, N., & Rao, V. V. R. M. (2021) · Indian Journal of Computer Science and Engineering · doi:10.21817/indjcse/2021/v12i5/211205050 · Cited by 15
“Particularly the authors [2, 5, 10, 14, 16, 17, 23] have paid attention on deriving functional web usage patterns from big web log data, still, the researchers expressed the need of more attention on data preparation stage in the overall process of big data analytics.”
BackgroundAnomaly Detectors for Self-Aware Edge and IoT Devices
Zoppi, T., Merlino, G., Ceccarelli, A., Puliafito, A., & Bondavalli, A. (2023) · IEEE International Conference on Software Quality, Reliability and Security (QRS) · doi:10.1109/qrs60937.2023.00013 · Cited by 13
Unsupervised Learning for Log Data Analysis Based on Behavior and Attribute Features
Wang, X., Wang, D., Zhang, Y., Jin, L., & Song, M. N. (2019) · International Conference on Artificial Intelligence and Computer Science (AICS) · doi:10.1145/3349341.3349460 · Cited by 13
“At present, the main ways to detect abnormal behavior by analyzing log information are data mining [5,20,9] and artificial intelligence [2, 15, 4, 18, 14].”
BackgroundBlockchain-driven anomaly detection framework on edge intelligence
Xie, X., Fang, Y., Jian, Z., Lu, Y., Li, T., & Wang, G. (2020) · CCF Transactions on Networking · doi:10.1007/s42045-020-00044-9 · Cited by 13
“…LogSig Tang et al. (2011), LogMine Ham-ooni et al. (2016), SHISO Mizutani (2013), and LenMa Shima (2016; 3) heuristic rules, such as AEL Jiang et al. (2008), IPLoM Makanju et al. (2012), Spell Min and Li (2017), Drain He et al. (2017), POP He et al. (2018), and MoLFI Messaoudi et al. (2018).”
BackgroundHong, A. E., Malinovsky, P. P., & Damodaran, S. K. (2024) · Digital Threats: Research and Practice · doi:10.1145/3604434 · Cited by 11
“Unsupervised deep learning approaches towards anomaly identification in time series have been employed previously towards identify deviations from expected behavior for example by applying CUSUM procedures to the residuals or identifying spikes in the loglikelihood [1, 33].”
MethodBatoun, M. A., Sayagh, M., Aghili, R., Ouni, A., & Li, H. (2024) · Empirical Software Engineering · doi:10.1007/s10664-024-10452-w · Cited by 31
Identifying Anomaly Detection Patterns from Log Files: A Dynamic Approach
Cavallaro, C., & Ronchieri, E. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-86960-1_36 · Cited by 15
Online Log Data Analysis With Efficient Machine Learning: A Review
Skopik, F., Landauer, M., & Wurzenberger, M. (2021) · IEEE Security & Privacy · doi:10.1109/msec.2021.3113275 · Cited by 14
New Algorithm for Predicting the States of a Computer Network Using Multivalued Dependencies
Sheluhin, O. I., Владимирович, О. А., & Rakovskiy, D. I. (2023) · Automatic Control and Computer Sciences · doi:10.3103/s0146411623010091 · Cited by 10
“Sum=0 for in : sum+=elem [4] // since elem is a quadruple of the form if sum> ): return 1 else: return 0”
BackgroundOptimizing Logging and Monitoring in Heterogeneous Cloud Environments for IoT and Edge Applications
Kim, C., & Kim, S. (2023) · IEEE Internet of Things Journal · doi:10.1109/jiot.2023.3304373 · Cited by 10
“He et al. [17] and Dai et al. [18] proposed parallel and n-gram dictionary-based log parsers, respectively, to improve the performance of parser and evaluated their scheme on widely used logging systems.”
MethodZhou, J., Ying, S., Wang, S., Zhao, D., Xiang, J., Liang, K., & Liu, P. (2025) · IEEE Transactions on Dependable and Secure Computing · doi:10.1109/tdsc.2025.3548050 · Cited by 17
“Moreover, the parsing process is system-specific, and usually requires writing suitable regular expressions for the log structure of a specific system, to ensure the quality of templates [24], [25], [26].”
Delog: A High-Performance Privacy Preserving Log Filtering Framework
Agrawal, A., Dixit, A., Shettar, N. A., Kapadia, D., Agrawal, V., Gupta, R. K., & Karlupia, R. (2019) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata47090.2019.9006218 · Cited by 15
“POP [5] and IPLoM [8] further split these initial clusters by identifying the tokens which occur most frequently at a given position and then splitting the clusters at these positions.”
BackgroundMethodInfluentialFine-Grained Access Control-Enabled Logging Method on ARM TrustZone
Lee, S.-H., Jo, H. J., Choi, W., Kim, H., Park, J. H., & Lee, D. H. (2020) · IEEE Access · doi:10.1109/access.2020.2991431 · Cited by 11
Metadata
- Authors
- He, P., Zhu, J., He, S., Li, J., & Lyu, M. R.
- Affiliations
- Chinese University of Hong Kong, Shenzhen (CN)
- Huawei Technologies (China) (CN)
- Venue
- IEEE Transactions on Dependable and Secure Computing
- Published
- 2018-11-01
- Volume
- 15
- Issue
- 6
- Pages
- 931-944
- ISSN
- 1545-5971 (ISSN-L)
- 1941-0018
- 2160-9209
- OpenAlex
- W2762028850
- Cited by
- 230
- References
- 42
- Citations by year
- 2026: 12 · 2025: 26 · 2024: 28 · 2023: 40 · 2022: 23 · 2021: 45 · 2020: 30 · 2019: 18 · 2018: 8
- FWCI
- 10.19
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- IEEE Transactions on Dependable and Secure Computing (publishedVersion)
- Funders
- National Natural Science Foundation of China (61332010)
- Research Grants Council of the Hong Kong Special Administrative Region, China (CUHK 14205214)
- 2015 Microsoft Research Asia Collaborative Research Program (FY16-RES-THEME-005)
- Topics
- Physical Sciences › Computer Science › Computer Networks and Communications › Software System Performance and Reliability
- Physical Sciences › Computer Science › Information Systems › Advanced Database Systems and Queries
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Keywords
- log parsing, log mining, Apache Spark, dependability assurance
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.