Towards Automated Log Parsing for Large-Scale Log Data Analysis
The DOI matches its Crossref record; 41 of the 42 references it cites were found in a registry or an index.
Abstract
Logs are widely used in system management for dependability assurance because they are often the only data available that record detailed system runtime behaviors in production. Because the size of logs is constantly increasing, developers (and operators) intend to automate their analysis by applying data mining methods, therefore structured input data (e.g., matrices) are required. This triggers a number of studies on log parsing that aims to transform free-text log messages into structured events. However, due to the lack of open-source implementations of these log parsers and benchmarks for performance comparison, developers are unlikely to be aware of the effectiveness of existing log parsers and their limitations when applying them into practice. They must often reimplement or redesign one, which is time-consuming and redundant. In this paper, we first present a characterization study of the current state of the art log parsers and evaluate their efficacy on five real-world datasets with over ten million log messages. We determine that, although the overall accuracy of these parsers is high, they are not robust across all datasets. When logs grow to a large scale (e.g., 200 million log messages), which is common in practice, these parsers are not efficient enough to handle such data on a single computer. To address the above limitations, we design and implement a parallel log parser (namely POP) on top of Spark, a large-scale data processing platform. Comprehensive experiments have been conducted to evaluate POP on both synthetic and real-world datasets. The evaluation results demonstrate the capability of POP in terms of accuracy, efficiency, and effectiveness on subsequent log mining tasks.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Introduction to information retrieval
(2009) · Choice Reviews Online · doi:10.5860/choice.46-2715 · Cited by 12,278
Introduction to Information Retrieval
Manning, C. D., Raghavan, P., & Schütze, H. (2008) · doi:10.1017/cbo9780511809071 · Cited by 11,288
Everitt, B. J., Landau, S., & Leese, M. (1974) · Cited by 9,363
Resilient distributed datasets: a fault-tolerant abstraction for in-memory cluster computing
Zaharia, M., Chowdhury, M., Das, T., Dave, A., Ma, J. H., McCauley, M., Franklin, M. J. M., Shenker, S. J., & Stoica, I. G. (2012) · UC Berkeley · Cited by 4,891
Everitt, B. S., Landau, S., Leese, M., & Stahl, D. (2011) · Wiley Series in Probability and Statistics · doi:10.1002/9780470977811 · Cited by 2,907
Detecting large-scale system problems by mining console logs
Xu, W., Huang, L., Fox, A., Patterson, D., & Jordan, M. I. (2009) · Proceedings of the ACM SIGOPS 22nd symposium on Operating systems principles · doi:10.1145/1629575.1629587 · Cited by 1,332 · cited 10×
“Furthermore, modern systems often integrate open-source software components written by hundreds of developers [3].”
BackgroundMethodInfluentialMinimum Spanning Trees and Single Linkage Cluster Analysis
Gower, J. C., & Ross, G. J. S. (1969) · Applied Statistics · doi:10.2307/2346439 · Cited by 1,223 · cited 1×
“To address over-parsing and further improve parsing accuracy, in this step, POP employs hierarchical clustering [27] to cluster similar groups based on their log events.”
MethodDiagnosing network-wide traffic anomalies
Lakhina, A., Crovella, M., & Diot, C. (2004) · Proceedings of the 2004 conference on Applications, technologies, architectures, and protocols for computer communications · doi:10.1145/1015467.1015492 · Cited by 1,039 · cited 1×
“However, the anomaly detection task evaluated is an important log mining task widely studied [37], [38], which is presented in a paper [3] enjoying more than 300 citations.”
BackgroundWhat Supercomputers Say: A Study of Five System Logs
Oliner, A., & Stearley, J. (2007) · 37th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN'07) · doi:10.1109/dsn.2007.103 · Cited by 620 · cited 2×
“Besides, we thank those who release log data [3], [33], [34], which greatly facilitates our research.”
MethodThe Complexity of Some Problems on Subsequences and Supersequences
Maier, D. (1978) · Journal of the ACM · doi:10.1145/322063.322075 · Cited by 616 · cited 1×
“The groups in the same cluster will be merged, and a new log event will be generated by calculating the Longest Common Subsequence (LCS) [28] of the original log events.”
Method
Research works printed but not linked: 4
- salfner, 2004, Comprehensive logfiles for autonomic systems, Proc 18th Int Parallel Distrib Process Symp
- krause, 1987, Taxicab Geometry
- gu, 2015, Proc 45th Annu IEEE/IFIP Int Conf Dependable Syst Netw, 57
- lang, 2013, Using SEC, LOGIN The USENIX Magazine, 38, 38
Cited by
Items 101–200 of 231
Han, D., Sun, M., Li, M., & Chen, Q. (2023) · Applied Sciences · doi:10.3390/app13137668 · Cited by 10
FSMFLog: Discovering Anomalous Logs Combining Full Semantic Information and Multifeature Fusion
Niu, W., Li, Z., He, Z., Wang, A., Li, B., & Zhang, X. (2023) · IEEE Internet of Things Journal · doi:10.1109/jiot.2023.3300690 · Cited by 11
“The first two strategies were the same as POP [19], and the third strategy was grouping according to the word mapping relationship.”
MethodLog Analytics in HPC: A Data-driven Reinforcement Learning Framework
Luo, Z., Hou, T., Nguyen, T. T., Zeng, H., & Lu, Z. (2020) · IEEE Conference on Computer Communications Workshops (INFOCOM WKSHPS) · doi:10.1109/infocomwkshps50562.2020.9162664 · Cited by 9
“General workflow of using logs to detect intruders [5], [9], [10], [12], [13].”
MethodNowhere to Hide Methodology: Application of Clustering Fault Diagnosis in the Nuclear Power Industry
Zong, C., Huang, S., Liu, E., Yao, Y., & Tang, S. (2019) · IEEE Access · doi:10.1109/access.2019.2957807 · Cited by 11
Anomaly Detection Model for Log Based on LSTM Network and Variational Autoencoder
Zhang, X., Chai, X., Yu, M., & Qiu, D. (2023) · International Conference on Information Science, Parallel and Distributed Systems (ISPDS) · doi:10.1109/ispds58840.2023.10235370 · Cited by 9
“According to whether labeled data is required or not, anomaly detection methods can be classified into two categories: supervised and unsupervised log anomaly detection including LOGROBUST [5], SVM [6], LR [7], decision tree [8], etc.”
Log Parsing Evaluation in the Era of Modern Software Systems
Petrescu, S., den Hengst, F., Uta, A., & Rellermeyer, J. S. (2023) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre59848.2023.00019 · Cited by 14
“Log parsing is formulated clearly as the task of mining the underlying log templates that generate runtime logs [13], [32]– [35].”
BackgroundDefense Against Advanced Persistent Threats in Smart Grids: A Reinforcement Learning Approach
Ning, B., & Xiao, L. (2021) · Chinese Control Conference (CCC) · doi:10.23919/ccc52363.2021.9549271 · Cited by 8
“Based on the intrusion detection system, e.g., Bro [20], the α ( k ) i over total b ( k ) CPUs are utilized to complete APT detection of DC i , such as log analysis [21], matching the strings of the meter data with known attack signatures, and so on.”
MethodCorbelle, C., Carneiro, V., & Cacheda, F. (2024) · Applied Sciences · doi:10.3390/app14135388 · Cited by 9
“• SVM [32]: this work represents log sequences as vectors and uses a Support Vector Machine (SVM) as its classification algorithm.”
MethodFrom Logs to Causal Inference: Diagnosing Large Systems
Markakis, M., Youngmann, B., Gao, T., Zhang, Z., Shahout, R., Chen, P. B., Liu, C., Sabek, I., & Cafarella, M. (2024) · Proceedings of the VLDB Endowment · doi:10.14778/3705829.3705836 · Cited by 14
Confidence ensembles: Tabular data classifiers on steroids
Zoppi, T., & Popov, P. T. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103126 · Cited by 11
Log Anomaly Detection Method Based on Transformer and Temporal Convolutional Networks
Liao, N., & Liu, Z. (2025) · IEEE Access · doi:10.1109/access.2025.3561669 · Cited by 13
“The prevailing log parsing methods can be classified into three principal categories: clustering-based parsing (e.g., Drain [23], Spell [24]), frequent itemset mining-based parsing (e.g., SLCT [25]), and heuristic-based parsing (e.g., POP [26]).”
Read the digital fingerprints: log analysis for digital forensics and security
Kara, İ. (2021) · Computer Fraud & Security · doi:10.1016/s1361-3723(21)00074-9 · Cited by 9
Sprelog: Log-Based Anomaly Detection with Self-matching Networks and Pre-trained Models
Yang, H., Zhao, X., Sun, D., Wang, Y., & Huang, W. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-91431-8_50 · Cited by 7
CLDTLog: System Log Anomaly Detection Method Based on Contrastive Learning and Dual Objective Tasks
Tian, G., Luktarhan, N., Wu, H., & Shi, Z. (2023) · Sensors · doi:10.3390/s23115042 · Cited by 15
TCMS: A Multi-Sequence Log Parsing Method Based on Token Conversion
Wei, M., Wen, J., He, S., Xie, K., Liang, W., Xie, G., Li, K., & Zhu, Z. (2024) · IEEE Transactions on Dependable and Secure Computing · doi:10.1109/tdsc.2024.3520628 · Cited by 8
“• The large datasets: Different from the first kind, following paper [40], we also use four large datasets [41]: HDFS, Zookeeper, BGL and HPC.”
Self-citationTransfer Log-based Anomaly Detection with Pseudo Labels
Huang, S., Liu, Y., Fung, C., He, R., Zhao, Y., Yang, H., & Luan, Z. (2020) · International Conference on Network and Service Management (CNSM) · doi:10.23919/cnsm50824.2020.9269069 · Cited by 7
“Many supervised log-based anomaly detection methods have been proposed, such as SVM [1], LR [2], LogRobust [3].”
MethodOLMPT: Research on Online Log Parsing Method Based on Prefix Tree
Wen, P., Zhang, Z., & Deng, B. (2020) · International Conference on Information Technologies and Electrical Engineering (ICITEE) · doi:10.1145/3452940.3452951 · Cited by 7
“Logs record user operation and system execute status, which can help maintainers to localize the fault [4].”
BackgroundOn the provenance extraction techniques from large scale log files
Tufek, A., & Aktaş, M. S. (2021) · Concurrency and Computation Practice and Experience · doi:10.1002/cpe.6559 · Cited by 7
Log Abstraction for Information Security: Heuristics and Reproducibility
Copstein, R., Schwartzentruber, J., Zincir-Heywood, N., & Heywood, M. I. (2021) · International Conference on Availability, Reliability and Security (ARES) · doi:10.1145/3465481.3470083 · Cited by 8
“These five algorithms are summarized as follows: POP (Parallel Log Parsing) [15] is built on the observation that most algorithms fail to complete an abstraction task in a reasonable time period, i.e. do not scale to log files over 200 million lines.”
MethodAn Open-Source Framework Unifying Stream and Batch Processing
Deshpande, K., & Rao, M. (2022) · Lecture notes in networks and systems · doi:10.1007/978-981-16-6723-7_45 · Cited by 9
Deep Learning-Based Log Parsing for Monitoring Industrial ICT Systems
Yang, Y., Wang, B., & Zhao, C. (2023) · Applied Sciences · doi:10.3390/app13063691 · Cited by 6
Substructure-Aware Log Anomaly Detection
Tang, Y., Zhang, Z., Zhao, K., Fang, L., Li, Z., & Chen, W. (2024) · Proceedings of the VLDB Endowment · doi:10.14778/3705829.3705840 · Cited by 11
“However, these manually designed metrics fail to capture the complex relationships within logs [10, 13, 14, 16, 32].”
Linnaeus: A highly reusable and adaptable ML based log classification pipeline
Catovic, A., Cartwright, C., Gebreyesus, Y. T., & Ferlin, S. (2021) · IEEE/ACM International Conference on AI Engineering – Software Engineering for AI (CAIN) · doi:10.1109/wain52551.2021.00008 · Cited by 6
Self Adjusting Log Observability for Cloud Native Applications
Pathak, D., Verma, M., Chakraborty, A., & Kumar, H. (2024) · IEEE ... International Conference on Cloud Computing · doi:10.1109/cloud62652.2024.00061 · Cited by 14
“To mitigate these issues and to reap the benefits of logs, researchers have proposed several approaches such as sampling of useful logs [24], parsing of unstructured logs into structured format [25]–[27], and compression of logs based on commonality and variability [28]–[30].”
BackgroundKobayashi, S., Yamashiro, Y., Otomo, K., & Fukuda, K. (2021) · International Journal of Network Management · doi:10.1002/nem.2195 · Cited by 9
“Past works have used disparate metrics such as the Rand Index, 7,29 pairwise F measure, 22,27,33,43,46,49 and parsing accuracy.”
MethodYu, S. (2022) · International Conference on Communications · doi:10.1109/icc45855.2022.9838858 · Cited by 6
“The control center analyzes the system log [11] and matches the data strings to complete the APT detection within the duration denoted by l ( k ) i .”
MethodFastLogSim: A Quick Log Pattern Parser Scheme Based on Text Similarity
Liu, W., Liu, X., Di, X., & Cai, B. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-55130-8_19 · Cited by 6
A Confidence-Guided Evaluation for Log Parsers Inner Quality
Xie, X., Wang, Z., Xiao, X., Lu, Y., Huang, S., & Li, T. (2020) · Mobile Networks and Applications · doi:10.1007/s11036-019-01501-6 · Cited by 6
“…characteristics of event size, character position, occurrences and longest common subsequence contained in a large number of log messages, the researchers designed many log parser algorithms based on heuristic rules (AEL [27, 28], IPLoM [29], Spell [30], Drain [31, 32], DrainV1 [22] and POP [33]).”
MethodAnomaly detection in virtual machine logs against irrelevant attribute interference
Zhang, H., Zhou, Y., Xu, H., Shi, J., Lin, X., & Gao, Y. (2025) · PLoS ONE · doi:10.1371/journal.pone.0315897 · Cited by 7
Towards robust log parsing using self-supervised learning for system security analysis
Cao, J., Di, X., Liu, X., Xu, R., Li, J., Ren, W., Qi, H., Hu, P., Zhang, K., & Li, B. (2024) · Intelligent Data Analysis · doi:10.3233/ida-230133 · Cited by 7
Self-citationAnomaly Detection Mechanism Based on Hierarchical Weights through Large-Scale Log Data
Xi, L., Yang, X. S., Luo, S., Shang, Y., & Tang, Q. (2021) · International Conference on Computer Communication and Artificial Intelligence (CCAI) · doi:10.1109/ccai50917.2021.9447458 · Cited by 5
“…into: (1) removing specific parameter information base on regular expressions, such as IP addresses, pure numbers etc. which are common parameters[19]; (2) word segmentation, that means dividing each log sequence into a list composed by multiple key words; (3) removing meaningless characters,…”
BackgroundReview on Log-Based Anomaly Detection Techniques
Raut, P., Mishra, A., Rao, S., Kawoor, S., Shelke, S., Deore, M. P., & Kumar, V. (2022) · Lecture notes in networks and systems · doi:10.1007/978-981-16-7657-4_72 · Cited by 5
DualAttlog: Context aware dual attention networks for log-based anomaly detection
Yang, H., Sun, D., & Huang, W. (2024) · Neural Networks · doi:10.1016/j.neunet.2024.106680 · Cited by 6
Unveiling anomalies: transformative insights from transformer-based autoencoder models
Dehkordi, S. B., Nasri, S., & Dami, S. (2024) · International Journal of Computers and Applications · doi:10.1080/1206212x.2024.2441147 · Cited by 9
“…data may be generally categorized into two groups: approaches that rely on log event indexes (such as PCA [26], Invariant Mining [27], SVM [28], DeepLog [29]) and approaches that rely on log template semantics (such as LogAnomaly [30] and LogRobust [31]).”
Log parsers' performance on raw logs from Android devices
Bessa, J. A., Filho, R. M., Souza, G., Barreto, R., & de Freitas, R. (2025) · Journal of Internet Services and Applications · doi:10.5753/jisa.2025.5049 · Cited by 4
A Hyperparameter Tuning Approach for an Online Log Parser
Marlaithong, T., Barroso, V. C., & Phunchongharn, P. (2021) · International Conference on Electrical Engineering/Electronics, Computer, Telecommunications and Information Technology (ECTI-CON) · doi:10.1109/ecti-con51831.2021.9454924 · Cited by 5
Extending the Space of Software Test Monitoring: Practical Experience
Lasynskyi, M., & Sosnowski, J. S. (2021) · IEEE Access · doi:10.1109/access.2021.3136138 · Cited by 4
“…[3, 22, 23, 38, 39] Test optimization and prioritization [18 – 23, 42, 43] Logging schemes [9, 28, 30] Log analysis and parsing [9, 30 - 32] Anomaly detection [9, 33, 34] Log statement optimization [9, 34] Issue and version control tracking [7, 25, 30, 37] Bug triaging [27] The created…”
BackgroundMethodInfluentialLPV: A Log Parsing Framework Based on Vectorization
Xiao, T. Y., Quan, Z., Wang, Z.-J., Zhao, K., Liao, X., Huang, H., Du, Y., & Li, K. (2023) · IEEE Transactions on Network and Service Management · doi:10.1109/tnsm.2023.3248124 · Cited by 5
“In typical automatic log analysis, the first and fundamental step is log parsing, which aims to parse unstructured logs into structured data [23], as logs are usually unstructured plain text output by the “ printf ” or similar functions in C language and the equivalents in other programming…”
MethodOnline log parsing using evolving research tree
Vervaet, A., Callau-Zori, M., Chabchoub, Y., & Chiky, R. (2023) · Knowledge and Information Systems · doi:10.1007/s10115-023-01953-z · Cited by 5
“He et al. [26] proposed a parallel log-parsing method (POP) using some heuristic rules and hierarchical clustering.”
Bouricha, H., Arwa, B., Hsairi, L., & Ahmadi, H. A. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.122122 · Cited by 6
Digging Evidence for Violation of Cloud Security Compliance with Knowledge Learned from Logs
Yuan, Y., Torgonshar, A., Shi, W., Liang, B., & Qin, B. (2019) · Communications in computer and information science · doi:10.1007/978-981-13-5913-2_20 · Cited by 6
Log Based Analysis of Software Application Operation
Obrębski, D., & Sosnowski, J. S. (2019) · Advances in intelligent systems and computing · doi:10.1007/978-3-030-19501-4_37 · Cited by 5
Anomaly Detection via Mining Numerical Workflow Relations from LogsPreprint
Zhang, B., Zhang, H., Moscato, P., & Zhang, A. (2020) · doi:10.36227/techrxiv.12570926 · Cited by 4
AI-Driven Log Analysis: Advances and Challenges
Wang, Y., Gan, W., & Yu, P. S. (2025) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata66926.2025.11401507 · Cited by 5
“These logs not only differ significantly in format but also vary widely in field definitions, timestamp formats, encoding methods, and even semantic expressions [83].”
Location and Time Aware Real Time Cloud Service Recommendation System Based on Multilayer Perceptron
Bose, S. K., & Sujatha, S. (2021) · International journal of intelligent engineering and systems · doi:10.22266/ijies2021.1031.35 · Cited by 3
“This is the freely available open-source dataset [26, 27].”
BackgroundLognroll: discovering accurate log templates by iterative filtering
Tak, B., & Han, W.-S. (2021) · International Middleware Conference · doi:10.1145/3464298.3493400 · Cited by 7
“Recently, there had been comparison studies of existing log template discovery techniques [13, 14, 40].”
BackgroundRobust Analysis of IT Infrastructure's Log Data with BERT Language Model
Bhanage, D. A., & Pawar, A. (2023) · International Journal of Advanced Computer Science and Applications · doi:10.14569/ijacsa.2023.0140675 · Cited by 7
“Various researchers have discussed miscellaneous log parsers such as POP [28], Spell [29], SLCT [30], etc. Drain [31] parser has been adapted in this research work to parse log datasets.”
MethodHCLPars: Α New Hierarchical Clustering Log Parsing Method
Binlashram, A., Hsairi, L., & Ahmadi, H. A. (2023) · Engineering Technology & Applied Science Research · doi:10.48084/etasr.6013 · Cited by 3
“POP is able to handle large datasets with high accuracy, as it was implemented on top of Spark, and it used Resilient Distributed Datasets (RDD) abstraction, which is ineffective with data of small size because it consumes more time, while using it is expensive when the size of the data increases [18].”
MethodInfluentialSimple, Fast, and Accurate Cybercrime Detection on E-Government with Elastic Stack SIEM
Yudhianto, I. (2023) · Jurnal Edukasi dan Penelitian Informatika (JEPIN) · doi:10.26418/jp.v9i2.64213 · Cited by 6
“Logs are textual data generated by logging statements within a system's source code, and they typically do not follow a specific structure [19], [21].”
BackgroundMethodMulti-objective Automatic Clustering with Gene Rearrangement and Cluster Merging
Qu, H., Yin, L., & Tang, X. (2021) · Studies in systems, decision and control · doi:10.1007/978-3-030-78124-8_5 · Cited by 3
The Runtime System Problem Identification Method Based on Log Analysis
Liu, Y., Lv, J., Ma, S., & Yao, W. (2018) · International Conference on Computer Communications and Networks (ICCCN) · doi:10.1109/icccn.2018.8487466 · Cited by 4
“The other researches propose data-driven log parsers such as SLCT (Simple Logfile Clustering Tool) [6], IPLoM (Iterative Partitioning Log Mining) [7], LogSig [8], POP (Parallel log Parsing) [9].”
InfluentialFibre Channel Switch Port State Machine Analysis Based on the Port Log Dump
Ivanova, D. L., Kutepov, S. V., & Dyumin, A. A. (2020) · IEEE Conference of Russian Young Researchers in Electrical and Electronic Engineering · doi:10.1109/eiconrus49466.2020.9038939 · Cited by 3
“There are also studies on log analysis, with the goals of converting free-text log entries into structured events and automating log analysis by applying data mining techniques [6].”
MethodSecurity of HPC Systems: From a Log-analyzing Perspective
Luo, Z., Qu, Z., Nguyen, T. T., Hui, Z., & Lu, Z. (2019) · ICST Transactions on Security and Safety · doi:10.4108/eai.19-8-2019.163134 · Cited by 9
“Usually we have four steps: log collection, log parsing, filtering/feature extraction and anomaly/intrusion detection, to conduct log file-based analysis in defending HPC systems against intrusions or other malicious behaviors [18, 23, 25, 26, 35]”
MethodEvoParser: An Evolutionary Approach to Log Parsing
Setinský, J., & Žádník, M. (2025) · IEEE/IFIP Network Operations and Management Symposium (NOMS) · doi:10.1109/noms57970.2025.11073703 · Cited by 2
“Building on Drain’s architecture, several enhanced parsers have emerged: POP [21] introduces parallel implementation on Spark for distributed deployment, and SPINE [6] incorporates a progressive clustering step for template correction with human feed-back.”
ALogSCAN: A Self-Supervised Dual Network for Adaptive and Timely Log Anomaly Detection in Clouds
Raeiszadeh, M., Estrada‐Solano, F., Glitho, R., Eker, J., & Mini, R. A. F. (2025) · IEEE Transactions on Machine Learning in Communications and Networking · doi:10.1109/tmlcn.2025.3594653 · Cited by 4
“Log parsing represents the initial phase of automated log analysis and transforms raw log messages into a structured format of events [17].”
Parse-LLM: A Prior-Free LLM Parser for Unknown System Logs
Song, C., Yang, L., Zheng, J., Liao, J., Yang, F., Ma, L., & Cai, F. (2025) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3746252.3761363 · Cited by 4
An Efficient Way to Parse Logs Automatically for Multiline Events
Yu, M., & Zhang, X. (2023) · Computer Systems Science and Engineering · doi:10.32604/csse.2023.037505 · Cited by 2
A Log Parsing Framework for ALICE O2 Facilities
Marlaithong, T., Barroso, V. C., & Phunchongharn, P. (2023) · IEEE Access · doi:10.1109/access.2023.3293406 · Cited by 2
“Previous log parsing studies [9], [10], [12], [19] have utilized these datasets.”
BackgroundMethodTowards Massive Log Anomaly Detection Based on an Enhanced Multi-dimensional Time-domain LSTM
Wang, Y., Zhang, C., & Yu, J. (2023) · IEEE International Conference on Big Data Computing Service and Applications (BigDataService) · doi:10.1109/bigdataservice58306.2023.00042 · Cited by 3
Análise de desempenho de Log Parsers da coleção Logpai em dados brutos de dispositivos Android
Bessa, J. A., Filho, R., Souza, G., Pessoa, L., Barreto, R., & de Freitas, R. (2024) · doi:10.5753/wperformance.2024.2423 · Cited by 3
A Data-Driven Approach for Adaptive Real-Time Log Parsing in Cloud Environments
Raeiszadeh, M., Estrada‐Solano, F., Glitho, R., Eker, J., & Mini, R. A. F. (2024) · IEEE International Mediterranean Conference on Communications and Networking (MeditCom) · doi:10.1109/meditcom61057.2024.10621416 · Cited by 4
“Log parsing represents the initial phase of automated log analysis and transforms raw log messages into sequences of structured events [5].”
BackgroundLogLSHD: Fast Log Parsing with Locality-Sensitive Hashing and Dynamic Time Warping
Huang, S.-W., Wu, X., & Li, H. (2025) · doi:10.1145/3727582.3728685 · Cited by 3
“Log parsing addresses these challenges by converting raw log data into structured formats represented by a limited number of log templates and their corresponding parameters (i.e., log variables), enabling automated log analysis [12, 14, 19, 46, 53, 54].”
A Methodology for the Dynamic Design of Adaptive Log Management Infrastructures
Anastopoulos, V., & Katsikas, S. K. (2019) · ICST Transactions on Security and Safety · doi:10.4108/eai.25-1-2019.159347 · Cited by 2
“In [17] the log management task of preprocessing is addressed.”
BackgroundTowards Self-Healing in the Internet of Things by Log Analytics and Process Mining
Singh, P., Flammini, F., Caporuscio, M., Azari, M. S., & Thornadtsson, J. (2020) · European Safety and Reliability Conference (ESREL) · doi:10.3850/978-981-14-8593-0_5634-cd · Cited by 3
ADCaDeM: A Novel Method of Calculating Attack Damage Based on Differential Manifolds
Liu, Z., Hu, C., Shan, C., & Yan, Z. (2022) · IEEE Transactions on Dependable and Secure Computing · doi:10.1109/tdsc.2022.3214809 · Cited by 2
“The attribute values in the simulations are derived from the content semantics obtained by analyzing log files [51].”
MethodAn Online Parsing Framework for Semistructured Streaming System Logs of Internet of Things Systems
Bhattacharya, S., Ray, B. R., Chugh, R., & Gordon, S. (2022) · IEEE Open Journal of Instrumentation and Measurement · doi:10.1109/ojim.2022.3232650 · Cited by 2
“Such models drastically reduced the processing time and error generation, making them superior to their predecessor models [23], [52].”
BackgroundMethodResultInfluentialAn Automatic Clustering Algorithm Using NSGA-II with Gene Rearrangement
Qu, H., & Yin, L. (2020) · IEEE International Conference on Intelligent Systems · doi:10.1109/is48319.2020.9199842 · Cited by 2
“It has been widely used in image segmentation [2], text analysis [3], gene analysis [4] and social networks [5].”
MethodSurvey on Log Clustering Approaches
Skopik, F., Wurzenberger, M., & Landauer, M. (2021) · Springer eBooks · doi:10.1007/978-3-030-74450-2_2 · Cited by 3
Black-box Attacks to Log-based Anomaly Detection
Huang, S., Liu, Y., Fung, C., Yang, H., & Luan, Z. (2022) · International Conference on Network and Service Management (CNSM) · doi:10.23919/cnsm55787.2022.9964935 · Cited by 3
“The first reason is that LogCluster, SVM and LR models use count vectors as their features and these features are order independent.”
BackgroundMethodInfluentialAnomaly based malware threat detection on Linux Systems
Ramamoorthy, J., Shashidhar, N. K., & Zhou, B. (2023) · IEEE International Conference on Trust, Security and Privacy in Computing and Communications (TrustCom) · doi:10.1109/trustcom60117.2023.00237 · Cited by 2
Wang, P., Zhang, X., & Cao, Z. (2025) · The Journal of Supercomputing · doi:10.1007/s11227-024-06850-y · Cited by 2
nDrain: A Robust Log Template Mining Algorithm
Yuan, J., Zhou, H., Wang, C., & Guan, B. (2024) · IEEE International Conference on Computer and Communications (ICCC) · doi:10.1109/iccc62609.2024.10941951 · Cited by 1
“The second category of work is based on frequent term mining methods, such as Logram [7], FT-Tree [8], Pop [9], etc.”
Empirical Analysis of LLMDPP: Advancing Log Parsing in the LLM Era
Wang, K., Zhang, S., Nan, H., Hou, X., Zou, J., & Miao, Z. (2025) · International Conference on Mobile Systems, Applications, and Services · doi:10.1145/3711875.3736683 · Cited by 2
“Log parsing aims to transform raw log messages into templates by replacing variables with wildcards [6, 7].”
A multi-device Android mobile raw log Dataset for intelligent analysis
Bessa, J. A., Filho, R. M., Barreto, R., & de Freitas, R. (2025) · Brazilian Symposium on Computing System Engineering (SBESC) · doi:10.1109/sbesc68008.2025.11288820 · Cited by 2
“The importance of the Log Parsing process is highlighted by [11], who describe Log Parsing as an initial step in Log mining to identify abnormal patterns and operational insights.”
STGraph: Spatio-Temporal Graph Mining for Anomaly Detection in Distributed System Logs
Li, T., Zhang, S., Feng, Y., Xu, J., Dang, Z., Liu, Y., & Ma, J. (2025) · International Symposium on Research in Attacks, Intrusions and Defenses (RAID) · doi:10.1109/raid67961.2025.00051 · Cited by 3
“In general, current log-based anomaly detection methods for distributed systems require further research and improvement to enhance their efficiency, accuracy, and robustness in handling complex log data, adapting to system changes, and minimizing false positives and negatives [17].”
LogExpertSolver: A Multi-Agent Framework for Domain-Specialized Log Parsing
Mao, C., Su, Y., & Li, D. (2025) · Asia-Pacific Software Engineering Conference (APSEC) · doi:10.1109/apsec66846.2025.00108 · Cited by 1
“Improvements based on Drain include: POP [43] enables distributed deployment, and SPINE [31] introduces human feedback mechanisms.”
NovaADLS: Semi-supervised Anomaly Detection in Log Series
Roqaya, N., Jabeen, H., & Papenbrock, T. (2026) · Lecture notes in networks and systems · doi:10.1007/978-3-032-32726-0_14 · Cited by 1
A Hypothesis Testing Approach to Sharing Logs with Confidence
Long, Y., Xu, L., & Gunter, C. A. (2020) · ACM Conference on Data and Application Security and Privacy (CODASPY) · doi:10.1145/3374664.3375743 · Cited by 2
“unstructured logs into structured sequential data [11, 17].”
Understanding Students' Identification and Use of Patterns While Writing SQL Queries
Hatami, Z., & Wolcott, P. (2020) · Conference on Information Technology Education · doi:10.1145/3368308.3415367 · Cited by 2
“Logs data are widely used as the main source of system operation status, analyzing users’ behavior, etc in different system management since the information about users’ behaviors could be recorded by details [5, 6].”
MethodLogCAD: An Efficient and Robust Model for Log-Based Conformal Anomaly Detection
Liu, C., Liang, M., Hou, J., Gu, Z., & Wang, Z. (2022) · Security and Communication Networks · doi:10.1155/2022/5822124 · Cited by 3
“In addition, although many algorithms, such as Decision Tree (DT [9]), Näıve Bayes (NB [10]), Support VectorMachine (SVM [11]), Logistic Regression (LR [12]), Random Forest (RF [13]), etc.”
MethodModelling Auto-scalable Big Data Enabled Log Analytic Framework
Deshpande, K., & Rao, M. (2022) · Lecture notes on data engineering and communications technologies · doi:10.1007/978-981-19-3035-5_64 · Cited by 1
Improved Insider Threat Detection Method of University Cluster System based on Log-Clustering
Li, Y. (2023) · International Conference on Big Data Technologies (ICBDT) · doi:10.1145/3627377.3627414 · Cited by 1
Cognition: Accurate and Consistent Linear Log Parsing Using Template Correction
Tian, R., Diao, Z., Jiang, H., & Xie, G. (2023) · Journal of Computer Science and Technology · doi:10.1007/s11390-021-1691-3 · Cited by 2
Introduction to Big Data Analytics
Darwish, D. A. M. (2024) · Advances in business information systems and analytics book series · doi:10.4018/979-8-3693-0413-6.ch001 · Cited by 3
Log Anomaly Detection Based on Semantic Features and Topic Features
Wang, P., Zhang, X., & Cao, Z. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-0808-6_24 · Cited by 1
A Review of Software Testing Process Log Parsing and Mining
Xue, K., Han, Q., Sheng, H., Shi, Z., & Qiao, Y. (2024) · IEEE International Conference on Software Services Engineering (SSE) · doi:10.1109/sse62657.2024.00055 · Cited by 4
“POP (Parallel Log Parsing) [1]effectively processes large-scale log data, overcoming the limitations of traditional parsing methods.”
BackgroundMethodSemantic Log Partitioning: Towards Automated Root Cause Analysis
Afshinpour, B., Amini, M.-R., & Groz, R. (2024) · IEEE International Conference on Software Quality, Reliability and Security (QRS) · doi:10.1109/qrs62785.2024.00069 · Cited by 1
“Different offline and online log parsing techniques have been proposed recently [21], [22], [23].”
Enhancing IoT Security: A Novel Attack Detection Model for IoT Logs with Machine Learning Techniques
Jacob, A. V., Issac, E., & krishna, A. A. (2024) · International Conference on Smart Computing and Communications (ICSCC) · doi:10.1109/icscc62041.2024.10690532 · Cited by 2
“In order to solve these issues, Pinjia He and Jieming Zhu,et al. [6] investigated log parsers in their research [6], emphasizing the necessity for more dependable systems to handle substantial log volumes.”
BackgroundUspenskij, M. B. (2020) · Izvestiya of Samara Scientific Center of the Russian Academy of Sciences · doi:10.37313/1990-5378-2020-22-1-107-113 · Cited by 1
Distributed Cloud Monitoring Platform Based on Log In-Sight
Haihong, E., Chen, Y., Song, M., & Sun, M. (2020) · Lecture notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering · doi:10.1007/978-3-030-48513-9_6 · Cited by 1
An Efficient Analytical Approach to Visualize Text-Based Event Logs for Semiconductor Equipment
Lee, G., & Jeong, J. (2021) · Applied Sciences · doi:10.3390/app11135944 · Cited by 1
“Many semiconductor equipment manufacturers store everything that occurs during the operation of the equipment in a log [1,2].”
BackgroundA Comprehensive Performance Evaluation of Novel Big Data Log Analytic Framework
Deshpande, K., & Rao, M. (2022) · Lecture notes in electrical engineering · doi:10.1007/978-981-19-2828-4_33 · Cited by 1
Deep learning based log anomaly detection with time differences
Sağında, B. (2020) · Cited by 1
Peer Review #1 of "Log-based software monitoring: a systematic mapping study (v0.1)"
(2021) · PeerJ Computer Science · doi:10.7287/peerj-cs.489v0.1/reviews/1 · Cited by 1
Explainable Anomaly Detection for System Logs in Distributed Environments
Gu, Z., Yue, W., & Liu, C. (2026) · Computers, Materials & Continua · doi:10.32604/cmc.2026.077388
Delog: A Privacy Preserving Log Filtering Framework for Online Compute PlatformsPreprint
Agrawal, A., Dixit, A., Shettar, N. A., Kapadia, D., Karlupia, R., Agrawal, V., & Gupta, R. K. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.04843
“On the other hand, LKE [7], POP [10] and IPLoM [17] attempt to segregate the log lines in various clusters such that all the lines in a cluster correspond to the same template.”
BackgroundMethodInfluentialPeer Review #2 of "Log-based software monitoring: a systematic mapping study (v0.1)"
(2021) · PeerJ Computer Science · doi:10.7287/peerj-cs.489v0.1/reviews/2
Peer Review #2 of "Log-based software monitoring: a systematic mapping study (v0.2)"
(2021) · PeerJ Computer Science · doi:10.7287/peerj-cs.489v0.2/reviews/2
Tejaswini, S., & Nasreen, A. (2021) · International Journal of Engineering and Advanced Technology · doi:10.35940/ijeat.e2816.0610521 · Cited by 1
“Real-time systems generate log files that are massive in size that find use in many fields such as security[37],[24], health monitoring[15],[13], failure handling[31],[25] and anomaly detection[22],[30].”
MethodCorrelating Time Series Signals and Event Logs in Embedded Systems
Krosman, K., & Sosnowski, J. S. (2021) · Sensors · doi:10.3390/s21217128 · Cited by 1
“This is supported with log parsing algorithms [6,22].”
Method
Metadata
- Authors
- He, P., Zhu, J., He, S., Li, J., & Lyu, M. R.
- Affiliations
- Chinese University of Hong Kong, Shenzhen (CN)
- Huawei Technologies (China) (CN)
- Venue
- IEEE Transactions on Dependable and Secure Computing
- Published
- 2018-11-01
- Volume
- 15
- Issue
- 6
- Pages
- 931-944
- ISSN
- 1545-5971 (ISSN-L)
- 1941-0018
- 2160-9209
- OpenAlex
- W2762028850
- Cited by
- 230
- References
- 42
- Citations by year
- 2026: 12 · 2025: 26 · 2024: 28 · 2023: 40 · 2022: 23 · 2021: 45 · 2020: 30 · 2019: 18 · 2018: 8
- FWCI
- 10.19
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- IEEE Transactions on Dependable and Secure Computing (publishedVersion)
- Funders
- National Natural Science Foundation of China (61332010)
- Research Grants Council of the Hong Kong Special Administrative Region, China (CUHK 14205214)
- 2015 Microsoft Research Asia Collaborative Research Program (FY16-RES-THEME-005)
- Topics
- Physical Sciences › Computer Science › Computer Networks and Communications › Software System Performance and Reliability
- Physical Sciences › Computer Science › Information Systems › Advanced Database Systems and Queries
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Keywords
- log parsing, log mining, Apache Spark, dependability assurance
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.