Abstracting log lines to log event types for mining software system logs
The DOI matches its Crossref record; 8 of the 8 references it cites were found in a registry or an index.
Abstract
Log files contain valuable information about the execution of a system. This information is often used for debugging, operational profiling, finding anomalies, detecting security threats, measuring performance etc. The log files are usually too big for extracting this valuable information manually, even though manual perusal is still one of the more widely used techniques. Recently a variety of data mining and machine learning algorithms are being used to analyze the information in the log files. A major road block for the efficient use of these algorithms is the inherent variability present in every log line of a log file. Each log line is a combination of a static message type field and a variable parameter field. Even though both these fields are required, the analyses algorithm often requires that these be separated out, in order to find correlations in the repeating log event types. This disentangling of the message and parameter fields to find the event types is called abstraction of log lines. Each log line is abstracted to a unique ID or event type and the dynamic parameter value is extracted to give an insight on the current state of the system. In this paper we present a technique based on a clustering technique used in the Simple Log file Clustering Tool for log file abstraction. This solution is especially useful when we don't have access to the source code of the application or when the lines in the log file do not conform to a rigid structure. We evaluated our implementation on log files from the Virtual Computing Lab, a cloud computer management system at North Carolina State University, and abstracted it to 727 unique event types.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“The reason we choose WordPiece is that it can effectively handle the OOV words and reduce the vocabulary size.”
From Log-based Anomaly Detection Without Log Parsing (2021) · cites this paper as [41]
Citations
Cites
A data clustering algorithm for mining patterns from event logs
Vaarandi, R. (2004) · Proceedings of the 3rd IEEE Workshop on IP Operations & Management (IPOM 2003) (IEEE Cat. No.03EX764) · doi:10.1109/ipom.2003.1251233 · Cited by 453 · cited 1×
“The log lines in the log files can then be matched to these regular expressions and the ID for the best match is used for each line in the log file.”
MethodAutomated Identification of Failure Causes in System Logs
Mariani, L., & Pastore, F. (2008) · 2008 19th International Symposium on Software Reliability Engineering (ISSRE) · doi:10.1109/issre.2008.48 · Cited by 127 · cited 2×
“These include but are not limited to, fault detection by monitoring, fault isolation [5], operational profiling [6] etc. Tools like Splunk [1], and Swatch [3], are used to monitor log files.”
BackgroundMethodInfluentialAutomated System Monitoring and Notification With Swatch
Hansen, S., & Atkins, E. T. (1993) · Cited by 125
Mining console logs for large-scale system problem detection
Xu, W., Huang, L., Fox, A., Patterson, D. A., & Jordan, M. I. (2008) · Cited by 105
An automated approach for abstracting execution logs to execution events
Jiang, Z. M., Hassan, A. E., Hamann, G., & Flora, P. (2008) · Journal of Software Maintenance and Evolution: Research and Practice · doi:10.1002/smr.374 · Cited by 83
Error log processing for accurate failure prediction
Salfner, F., & Tschirpke, S. (2008) · Cited by 63
Efficiently Extracting Operational Profiles from Execution Logs Using Suffix Arrays
Nagappan, M., Wu, K., & Vouk, M. A. (2009) · 2009 20th International Symposium on Software Reliability Engineering · doi:10.1109/issre.2009.23 · Cited by 63 · cited 3×
“These include but are not limited to, fault detection by monitoring, fault isolation [5], operational profiling [6] etc. Tools like Splunk [1], and Swatch [3], are used to monitor log files.”
BackgroundMethodSelf-citationAbstracting Execution Logs to Execution Events for Enterprise Applications
Jiang, Z. M., Hassan, A. E., Flora, P. K., & Hamann, G. (2008) · Cited by 18
Research works printed but not linked: 2
- 0, Virtual Computing Lab
- 0, Splunk
Cited by
Tools and benchmarks for automated log parsing
Zhu, J., He, S., Liu, J., He, P., Xie, Q., Zheng, Z., & Lyu, M. R. (2019) · 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1109/icse-seip.2019.00021 · Cited by 574
“1) Frequent Pattern Mining: Among all the log parsing methods, three of them belong to this category, i.e., SLCT [20], LFA [42], and LogCluster [21].”
BackgroundLog-based Anomaly Detection Without Log Parsing
Le, V.-H., & Zhang, H. (2021) · 2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase51524.2021.9678773 · Cited by 361
“There are many log parsing techniques, including frequent pattern mining [17], [18], [19], clustering [20], [21], [22], language modeling [23], heuristics [24], [25], [26], etc.”
BackgroundLoghub: a large collection of system log datasets for AI-driven log analytics
Zhu, J., He, S., He, P., Liu, J., & Lyu, M. R. (2023) · 2023 IEEE 34th International Symposium on Software Reliability Engineering (ISSRE) · doi:10.1109/issre59848.2023.00071 · Cited by 302
“Conversely, LogSig, LFA, MoLFI, and LKE have the lowest average accuracy.”
BackgroundMethodInfluentialLog-based anomaly detection with deep learning: how far are we?
Le, V.-H., & Zhang, H. (2022) · Proceedings of the 44th International Conference on Software Engineering · doi:10.1145/3510003.3510155 · Cited by 257
“There are many log parsing techniques, based on frequent pattern mining [10, 38, 45], clustering [16, 42, 44], and heuristics [17, 22, 34].”
MethodHitAnomaly: Hierarchical Transformers for Anomaly Detection in System Log
Huang, S., Liu, Y., Fung, C., He, R., Zhao, Y., Yang, H., & Luan, Z. (2020) · IEEE Transactions on Network and Service Management · doi:10.1109/tnsm.2020.3034647 · Cited by 228
“To further evaluate the impact of log parser on log anomaly detection tasks, we use LFA [28] to preprocess the OpenStack dataset.”
MethodLogram: Efficient Log Parsing Using n-Gram Dictionaries
Dai, H., Li, H., Chen, C. S., Shang, W., & Chen, T.-H. (2020) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2020.3007554 · Cited by 170
SwissLog: Robust and Unified Deep Learning Based Log Anomaly Detection for Diverse Faults
Li, X., Chen, P., Jing, L., He, Z., & Yu, G. (2020) · 2020 IEEE 31st International Symposium on Software Reliability Engineering (ISSRE) · doi:10.1109/issre5003.2020.00018 · Cited by 143
“SLCT [41], LFA [42] and LogCluster [43] firstly record frequency of item and then group them into multiple groups.”
BackgroundUniParser: A Unified Log Parser for Heterogeneous Log Data
Liu, Y., Zhang, X., He, S., Zhang, H., Li, L., Kang, Y., Xu, Y., Ma, M., Lin, Q., Dang, Y., et al. (2022) · Proceedings of the ACM Web Conference 2022 · doi:10.1145/3485447.3511993 · Cited by 125
“Typical approaches include SLCT [22], LFA [19], LogCluster [23] and Logram [3].”
MethodInfluentialLog Parsing with Prompt-based Few-shot Learning
Le, V.-H., & Zhang, H. (2023) · 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) · doi:10.1109/icse48619.2023.00204 · Cited by 129
“To achieve automated log parsing, many data-driven approaches [12], [13], [14], [15] have been proposed over the years to extract the common parts that constantly occur among log messages as templates and the dynamic parts that vary during runtime as parameters.”
MethodLILAC: Log Parsing using LLMs with Adaptive Parsing Cache
Jiang, Z., Liu, J., Chen, Z., Li, Y., Huang, J., Huo, Y., He, P., Gu, J., & Lyu, M. R. (2024) · Proceedings of the ACM on Software Engineering · doi:10.1145/3643733 · Cited by 123
“Syntax-based log parsers [9, 12, 18, 67] utilize specific features or heuristics ( e.g. , log length, word length and frequency) to extract the constant parts of log messages as templates.”
System log clustering approaches for cyber security applications: A survey
Landauer, M., Skopik, F., Wurzenberger, M., & Rauber, A. (2020) · Computers & Security · doi:10.1016/j.cose.2020.101739 · Cited by 108
Brain: Log Parsing With Bidirectional Parallel Tree
Yu, S., He, P., Chen, N., & Wu, Y. (2023) · IEEE Transactions on Services Computing · doi:10.1109/tsc.2023.3270566 · Cited by 126
Guidelines for assessing the accuracy of log message template identification techniques
Khan, Z. A., Shin, D., Bianculli, D., & Briand, L. (2022) · Proceedings of the 44th International Conference on Software Engineering · doi:10.1145/3510003.3510101 · Cited by 88
SwissLog: Robust Anomaly Detection and Localization for Interleaved Unstructured Logs
Li, X., Chen, P., Jing, L., He, Z., & Yu, G. (2022) · IEEE Transactions on Dependable and Secure Computing · doi:10.1109/tdsc.2022.3162857 · Cited by 96
“SLCT [53], LFA [54] and LogCluster [55] firstly record frequency of items and then group them into multiple groups.”
BackgroundZhang, T., Qiu, H., Castellano, G., Rifai, M., Chen, C. S., & Pianese, F. (2023) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2022.3222417 · Cited by 75
“We specifically measure the PA of 17 open-source log parsers, namely LogSig [91], LKE [41], SLCT [93], LFA [96], MoLFI [127], SHISO [113], LogClusterr [95], LogMine [7], AEL [155], Spell [104], LenMa [84], IPLoM [56], Drain [109], Logram [27], Paddy [131],”
MethodDivLog: Log Parsing with Prompt Enhanced In-Context Learning
Xu, J., Yang, R., Huo, Y., Zhang, C., & He, P. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3597503.3639155 · Cited by 164
Experience Report: Deep Learning-based System Log Analysis for Anomaly Detection
Chen, Z., Liu, J., Gu, W., Su, Y., & Lyu, M. R.-T. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.05908 · Cited by 60
“, Logram [6], LFA [32], SLCT [43]), clustering (e.”
Nedelkoski, S., Bogatinovski, J., Acker, A., Cardoso, J., & Kao, O. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-67667-4_8 · Cited by 118
Mining Historical Test Logs to Predict Bugs and Localize Faults in the Test Logs
Amar, A., & Rigby, P. C. (2019) · 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE) · doi:10.1109/icse.2019.00031 · Cited by 73
“These works have used statistical learning approaches to identify sequences in logs [20], [36], [51], find repeating and anomalous patterns [14], [18], [31], [46], [54], and clustering similar logs [19], [30], [36], [37], [49].”
MethodA systematic literature review on automated log abstraction techniques
El-Masri, D., Petrillo, F. D. S., Guéhéneuc, Y., Hamou‐Lhadj, A., & Bouziane, A. (2020) · Information and Software Technology · doi:10.1016/j.infsof.2020.106276 · Cited by 93
“For example, Na-gappan et al. [34] showed that with thresholds of 50%, 25%, 10%, 5% and even 1%, SLCT could not abstract all log-messages.”
BackgroundMethodInfluentialSemParser: A Semantic Parser for Log Analytics
Huo, Y., Su, Y., Lee, C., & Lyu, M. R. (2023) · 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) · doi:10.1109/icse48619.2023.00082 · Cited by 58
“For instance, SLCT [6] and LFA [7] constructed log templates by counting the number of historical frequently-appearing words while Logram [8] considered frequent n-gram patterns.”
BackgroundLogParser-LLM: Advancing Efficient Log Parsing with Large Language Models
Zhong, A., Mo, D., Liu, G., Liu, J., Lu, Q., Zhou, Q., Wu, J., Li, Q., & Wen, Q. (2024) · Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining · doi:10.1145/3637528.3671810 · Cited by 59
Log Parsing: How Far Can ChatGPT Go?
Le, V.-H., & Zhang, H. (2023) · 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase56229.2023.00206 · Cited by 51
“2) Frequent pattern mining: SLCT [25], LFA [16], and Logram [15] leverage frequent patterns of token position or n-gram information to extract log templates that appear constantly across log messages.”
BackgroundMethodDid We Miss Something Important? Studying and Exploring Variable-Aware Log Abstraction
Li, Z., Luo, C., Chen, T.-H. P., Shang, W., He, S., Lin, Q., & Zhang, D. (2023) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse48619.2023.00078 · Cited by 86
“, Logram [20] and LFA [25]), clustering algorithms (e.”
BackgroundInfluentialImpact of log parsing on deep learning-based anomaly detection
Khan, Z. A., Shin, D., Bianculli, D., & Briand, L. C. (2024) · Empirical Software Engineering · doi:10.1007/s10664-024-10533-w · Cited by 49
“…their log templates using different heuristics, have been proposed in the literature (e.g., AEL (Jiang et al. 2008), Drain (He et al. 2017), IPLoM (Makanju et al. 2009), LenMa (Shima 2016), LFA (Nagappan and Vouk 2010), LogCluster (Vaarandi and Pihelgas 2015), LogMine (Hamooni et al. 2016),…”
BackgroundMethodInfluentialSPINE: a scalable log parser with feedback guidance
Wang, X., Zhang, X., Li, L., He, S., Zhang, H., Liu, Y., Zheng, L., Kang, Y., Lin, Q., Dang, Y., et al. (2022) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3540250.3549176 · Cited by 63
“Similar to this method, LFA [28] considered the token frequency distribution.”
MethodAn empirical study of the impact of log parsers on the performance of log-based anomaly detection
Fu, Y., Yan, M. L., Xu, Z., Xia, X., Zhang, X., & Yang, D. (2022) · Empirical Software Engineering · doi:10.1007/s10664-022-10214-6 · Cited by 53
“The classic frequency-based log parser LFA (Nagappan and Vouk 2010) and the latest frequency-based log parser Logram (Dai et al.”
BackgroundMethodInfluentialPreLog: A Pre-trained Model for Log Analytics
Le, V.-H., & Zhang, H. (2024) · Proceedings of the ACM on Management of Data · doi:10.1145/3654966 · Cited by 46
“(1 ) Data-driven log parsing : many studies have been proposed to leverage data-driven approaches for log parsing, including frequent pattern mining [19, 72, 86], clustering [28, 59, 81], and heuristics [24, 36, 69].”
Paddy: An Event Log Parsing Approach using Dynamic Dictionary
Huang, S., Liu, Y., Fung, C., He, R., Zhao, Y., Yang, H., & Luan, Z. (2020) · IEEE/IFIP Network Operations and Management Symposium (NOMS) · doi:10.1109/noms47738.2020.9110435 · Cited by 38
“LFA [18] leverages the token frequency distribution in each log message to improve its robustness.”
BackgroundInvestigating and improving log parsing in practice
Fu, Y., Yan, M. L., Xu, J., Li, J., Liu, Z., Zhang, X., & Yang, D. (2022) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3540250.3558947 · Cited by 46
“Frequent-patternmining-based log parsers include SLCT [43], LFA [35], LogCluster [44], and Logram [6].”
MethodRobust and Transferable Log-based Anomaly Detection
Jia, P., Cai, S., Ooi, B. C., Wang, P., & Xiong, Y. (2023) · Proceedings of the ACM on Management of Data · doi:10.1145/3588918 · Cited by 47
“Frequent pattern mining techniques such as LFA [46], SLCT [59], and Log Cluster [60] extract log event templates that frequently occur in historical log messages.”
MethodAn Assessment of ChatGPT on Log Data
Mudgal, P., & Wouhaybi, R. H. (2023) · Communications in computer and information science · doi:10.1007/978-981-99-7587-7_13 · Cited by 56
Prefix-Graph: A Versatile Log Parsing Approach Merging Prefix Tree with Probabilistic Graph
Chu, G., Wang, J., Qi, Q., Sun, H., Tao, S., & Liao, J. (2021) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde51399.2021.00274 · Cited by 37
“LFA [19] combines tokens with similar frequency into log templates.”
MethodBhanage, D. A., Pawar, A., & Kotecha, K. V. (2021) · IEEE Access · doi:10.1109/access.2021.3128283 · Cited by 56
“Similarly, LFA [51] also works on the same clustering technique to abstract log lines and derive event types.”
USTEP: Unfixed Search Tree for Efficient Log Parsing
Vervaet, A., Chiky, R., & Callau-Zori, M. (2021) · Proceedings · doi:10.1109/icdm51629.2021.00077 · Cited by 31
Exploring Better Black-Box Test Case Prioritization via Log Analysis
Chen, Z., Chen, J., Wang, W., Zhou, J., Wang, M., Chen, X., Zhou, S., & Wang, J. (2022) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3569932 · Cited by 43
“Over the years, many log parsing methods have been proposed, including frequent-pattern-mining based methods (e.g., LFA [81] and Logram [28]), clustering-based methods (e.g., LKE [37] and LogCluster [110]), and heuristics-based methods (e.g., AEL [52] and Drain [43, 44]).”
MethodUsing large language models for template detection from security event logs
Vaarandi, R., & Bahşi, H. (2025) · International Journal of Information Security · doi:10.1007/s10207-025-01018-y · Cited by 37
“LFA [14] begins with identifying frequent words like SLCT and LogCluster and builds templates around frequent words that have similar occurrence times in event log messages.”
A Survey of Deep Anomaly Detection for System Logs
Zhao, X., Jiang, Z., & Ma, J. (2022) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn55064.2022.9892726 · Cited by 29
“LFA [17] conducts frequency traversal filter for each log and abstracts the clustering similar frequency words in each line into event template.”
BackgroundLogReducer: Identify and Reduce Log Hotspots in Kernel on the Fly
Yu, G., Chen, P., Li, P., Weng, T., Zheng, H., Deng, Y., & Zheng, Z. (2023) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse48619.2023.00151 · Cited by 35
“To overcome the above shortcoming, some data-driven log parsers [18], [26], [27], [29], [30], [55], [56] have been proposed.”
BackgroundMethodInfluentialYang, L., Chen, J., Gao, S., Gong, Z., Zhang, H., Kang, Y., & Li, H. H. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3644386 · Cited by 34
“Following these existing work, we also used them as the studied techniques in our study.”
MethodClusterLog: Clustering Logs for Effective Log-based Anomaly Detection
Egersdoerfer, C., Zhang, D., & Dai, D. (2022) · IEEE/ACM Workshop on Fault Tolerance for HPC at eXtreme Scale (FTXS) · doi:10.1109/ftxs56515.2022.00006 · Cited by 28
“Frequent pattern mining approaches extract frequently occurring patterns, with the ultimate goal of grouping the logs which have the highest degree of similarity [30], [31], [24].”
BackgroundSemi-supervised log anomaly detection based on bidirectional temporal convolution network
Yin, Z., Kong, X., & Yin, C. (2024) · Computers & Security · doi:10.1016/j.cose.2024.103808 · Cited by 25
No More Labelled Examples? An Unsupervised Log Parser with LLMs
Huang, J., Jiang, Z., Chen, Z., & Lyu, M. R.-T. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3729377 · Cited by 63
“Therefore, numerous efforts have been made to achieve accurate and efficient log parsing [7, 11, 45, 46, 48, 49, 56, 57, 59, 67].”
Towards multi-tenant and interoperable monitoring of virtual machines in cloud
Tovarňák, D., & Pitner, T. (2012) · International Symposium on Symbolic and Numeric Algorithms for Scientific Computing (SYNASC) · doi:10.1109/synasc.2012.55 · Cited by 21
“Foster et al. in [7] identify the multi-tenant access to monitoring data as a significant challenge for the Cloud computing since, depending on the service model, the users are not able to utilize their own monitoring infrastructure and the produced monitoring data may not provide necessary level…”
BackgroundConfidence guided anomaly detection model for anti-concept drift in dynamic logs
Xie, X., Jin, Z., Wang, J., Yang, L., Lu, Y., & Li, T. (2020) · Journal of Network and Computer Applications · doi:10.1016/j.jnca.2020.102659 · Cited by 22
Automatic Event Log Abstraction to Support Forensic Investigation
Studiawan, H., Sohel, F. A., & Payne, C. (2020) · Australasian Computer Science Week Multiconference (ACSW) · doi:10.1145/3373017.3373018 · Cited by 21
“Furthermore, abstraction can reduce the computation time as one abstraction represents many log lines for anomaly detection and analysis [6, 18].”
BackgroundDemonstration-Free: Towards More Practical Log Parsing with Large Language Models
Xiao, Y., Le, V.-H., & Zhang, H. (2024) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3691620.3694994 · Cited by 61
Self-supervised log parsing using semantic contribution difference
Yu, S., Chen, N., Wu, Y., & Dou, W. (2023) · Journal of Systems and Software · doi:10.1016/j.jss.2023.111646 · Cited by 22
Automatic Parsing and Utilization of System Log Features in Log Analysis: A Survey
Ma, J., Liu, Y., Wan, H., & Sun, G. (2023) · Applied Sciences · doi:10.3390/app13084930 · Cited by 33
LogGrep: Fast and Cheap Cloud Log Storage by Exploiting both Static and Runtime Patterns
Wei, J., Zhang, G., Chen, J., Wang, Y., Zheng, W., Sun, T., Wu, J., & Jiang, J. (2023) · European Conference on Computer Systems · doi:10.1145/3552326.3567484 · Cited by 32
“To achieve this goal, we leverage existing log parsing methods to struc-turize log entries into templates and variables [23, 30, 31, 45, 51, 66, 67, 69, 76], because values of the same variable are more likely to share common features [45, 69].”
MethodHue: A User-Adaptive Parser for Hybrid Logs
Xu, J., Fu, Q., Zhu, Z., Cheng, Y., Li, Z., Ma, Y., & He, P. (2023) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3611643.3616260 · Cited by 25
“For example, SLCT [30], LFA [25], LogCluster [31], and Logram [2] try to use different methods to extract frequent patterns in logs.”
MethodLogExtractor: Extracting digital evidence from android log messages via string and taint analysis
Cheng, C. C.-C., Chen, S., Gong, N. Z., & Guan, Y. (2021) · Forensic Science International Digital Investigation · doi:10.1016/j.fsidi.2021.301193 · Cited by 21
“Most of existing studies (Vaarandi, 2003; Vaarandi and Pihelgas, 2015; Fu et al., 2009; Du and Li, 2016; Makanju et al., 2009; Nagappan and Vouk, 2010; Messaoudi et al., 2018; He et al., 2017; Hamooni et al., 2016) summarize log message patterns by deploying data mining techniques on massive amount…”
A Survey On Log Research Of AIOps: Methods and Trends
Jiang, Z., Tong, L., Zhenguo, Z., Ge, J., Junling, Y., & Li, L. (2021) · Mobile Networks and Applications · doi:10.1007/s11036-021-01832-3 · Cited by 40
“Unlike SLCT and LogCluster’s, which look for frequent log file items, LFA looks for frequent items in a log entry and obtains the log template by traversing twice.”
BackgroundXDrain: Effective log parsing in log streams using fixed-depth forest
Liu, C., Tian, Y., Yu, S., Gao, D., Wu, Y., Huang, S., Hu, X., & Chen, N. (2024) · Information and Software Technology · doi:10.1016/j.infsof.2024.107546 · Cited by 18
FlexParser—The adaptive log file parser for continuous results in a changing world
Rücker, N., & Maier, A. (2022) · Journal of Software Evolution and Process · doi:10.1002/smr.2426 · Cited by 15
“Another Log File Abstraction (LFA) 19 method utilizes token frequencies within each log message and extracts log events for all log messages, independently.”
MethodAn Efficient Log Parsing Algorithm Based on Heuristic Rules
Zhang, L., Xie, X., Xie, K., Wang, Z., Lu, Y., & Zhang, Y. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-29611-7_10 · Cited by 15
A Theoretical Framework for Understanding the Relationship Between Log Parsing and Anomaly Detection
Shin, D., Khan, Z. A., Bianculli, D., & Briand, L. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-88494-9_16 · Cited by 22
“This is mainly because most of the existing log parsing approaches, including AEL [10], Drain [7], IPLoM [13], LenMa [18], LFA [17], LogCluster [20], LogMine [6], LogSig [19], MoLFI [14], SHISO [15], SLCT [21], Spell [4], and Logram [3], have been proposed as general-purpose approaches rather than specialized for anom…”
BackgroundLog Parsing with Generalization Ability under New Log Types
Yu, S., Wu, Y., Li, Z., He, P., Chen, N., & Liu, C. (2023) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3611643.3616355 · Cited by 22
“The main approaches of automated log parsing include heuristics [17], clustering [12, 15, 37, 39, 45], frequent item extraction [5, 32, 40] and deep learning [29] [33].”
BackgroundMethodMETING: A Robust Log Parser Based on Frequent n-Gram Mining
Coustié, O., Mothe, J., Teste, O., & Baril, X. (2020) · IEEE International Conference on Web Services (ICWS) · doi:10.1109/icws49710.2020.00018 · Cited by 16
“Finally, frequent pattern mining methods [14], [15], [16] offer a more text-driven representation.”
MethodQuickLogS: A Quick Log Parsing Algorithm based on Template Similarity
Fang, L., Di, X., Liu, X., Qin, Y., Ren, W., & Ding, Q. (2021) · IEEE International Conference on Trust, Security and Privacy in Computing and Communications (TrustCom) · doi:10.1109/trustcom53373.2021.00148 · Cited by 14
“SLCT [17], LFA [22] and LogCluster [23] to extract frequent tokens or token pairs into log templates by traversing the log data multiple times.”
MethodSymptom-based problem determination using log data abstraction
Huang, L. P., Ke, X., Wong, K., & Mankovskii, S. (2010) · IEEE International Conference on Collaborative Advances in Software and Computing (CASCON) · doi:10.1145/1923947.1923979 · Cited by 18
“[22] in which the relative word frequency rather than the number of appearance is applied to discover dynamic parts in log lines.”
BackgroundBlockchain-driven anomaly detection framework on edge intelligence
Xie, X., Fang, Y., Jian, Z., Lu, Y., Li, T., & Wang, G. (2020) · CCF Transactions on Networking · doi:10.1007/s42045-020-00044-9 · Cited by 13
“Logcluster, FLE, AEL, Drain, and LFA still keep lightweight on NVidia TK1.”
BackgroundMethodInfluentialBatoun, M. A., Sayagh, M., Aghili, R., Ouni, A., & Li, H. (2024) · Empirical Software Engineering · doi:10.1007/s10664-024-10452-w · Cited by 31
SemiRALD: A semi-supervised hybrid language model for robust Anomalous Log Detection
Sun, Y., Keung, J., Yang, Z., Liu, S., & Yu, H. K. (2025) · Information and Software Technology · doi:10.1016/j.infsof.2025.107743 · Cited by 18
MapReduce based log file analysis for system threats and problem identification
Vernekar, S., & Buchade, A. (2013) · IEEE International Advance Computing Conference · doi:10.1109/iadcc.2013.6514334 · Cited by 11
“A cluster based detection system is presented in [11].”
MethodLogBase: A Large-Scale Benchmark for Semantic Log Parsing
Zhang, C., Xu, W., Liu, J., Zhang, L., Liu, G., Guan, J., Zhou, Q., & Zhou, S. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3728969 · Cited by 16
“To answer RQ2, we evaluate fifteen log parsers on LogBase, including 11 unsupervised parsers: AEL [23], Drain [17], IPLoM [37], LenMa [45], LFA [40], LogCluster [51], LogMine [16], Logram [6], MoLFI [38], SHISO [39] and Spell [8], two supervised parsers: LogPPT [29] and VALB [32], and two LLM-based…”
SemiSMAC: A semi-supervised framework for log anomaly detection with automated hyperparameter tuning
Sun, Y., Keung, J., Yang, Z., Liu, S., & Liao, Y. (2025) · Information and Software Technology · doi:10.1016/j.infsof.2025.107869 · Cited by 13
ADAL-NN: Anomaly Detection and Localization Using Deep Relational Learning in Distributed Systems
Ahmed, K. M., Altaf, A., Jamail, N. S. M., Iqbal, F., & Latif, R. (2023) · Applied Sciences · doi:10.3390/app13127297 · Cited by 11
“SLCT [27], LFA [28], and LogCluster [29] first record the frequency of items before categorizing them into multiple groups.”
BackgroundMonitoring Event Logs within a Cluster System
Sosnowski, J. S., Kubacki, M., & Krawczyk, H. (2012) · Advances in intelligent and soft computing · doi:10.1007/978-3-642-30662-4_17 · Cited by 9
Log Parser with One-to-One Markup
Chunyong, Z., & Meng, X. (2020) · International Conference on Information and Computer Technologies (ICICT) · doi:10.1109/icict50521.2020.00045 · Cited by 10
“There are also some methods based on frequent pattern mining, such as SLCT [28], LFA[11]andLogcluster [ 29], template-based clustering methods, such as three offline methods LogSig[12], LogMine[13],LKE [31]and two online methods SHISO[15],LenMa [16].”
BackgroundMethodLTmatch: A Method to Abstract Pattern from Unstructured Log
Wang, X., Zhao, Y., Xiao, H., Wang, X., & Chi, X. (2021) · Applied Sciences · doi:10.3390/app11115302 · Cited by 9
“Nagappan [9] also propose a log pattern extraction method LFA based on frequent pattern mining.”
MethodLog Parsing Using LLMs with Self-Generated In-Context Learning and Self-Correction
Wu, Y., Yu, S., & Li, Y. (2025) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc66645.2025.00063 · Cited by 23
“(1) Frequency-based parsers: These log parsers [10], [12], [55], [56] utilize frequent patterns of token position or n-gram information to distinguish templates and variables in log messages.”
Reducing Events to Augment Log-based Anomaly Detection Models: An Empirical Study
Zhang, L., Jia, T., Wang, K., Jia, M., Yang, Y., & Li, Y. (2024) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1145/3674805.3695403 · Cited by 12
Efficient Online Log Parsing with Log Punctuations Signature
Zhang, S., & Wu, G. (2021) · Applied Sciences · doi:10.3390/app112411974 · Cited by 9
“(1) Frequent Pattern Mining: SLCT [19], LFA [20], and LogCluster [31] propose automated log parsers that parse log messages by mining the frequent tokens in log files.”
BackgroundMethodLog Parsing Evaluation in the Era of Modern Software Systems
Petrescu, S., den Hengst, F., Uta, A., & Rellermeyer, J. S. (2023) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre59848.2023.00019 · Cited by 14
“We attribute this result to the limited variability of logs that originate from a specific system: as methods have to parse the same amount of log data as in the previous experiment, but with fewer logs from the same distribution (system), it becomes harder for parsers to recognize variables and to generate quality tem…”
Creating a knowledge database on system dependability and resilience
Kubacki, M., & Sosnowski, J. S. (2013) · Control and Cybernetics · Cited by 8
Designing a Unified Cloud Log Analytics Platform
Sun, M., Convertino, G., & Detweiler, M. C. (2016) · International Conference on Collaboration Technologies and Systems (CTS) · doi:10.1109/cts.2016.0057 · Cited by 8
TCMS: A Multi-Sequence Log Parsing Method Based on Token Conversion
Wei, M., Wen, J., He, S., Xie, K., Liang, W., Xie, G., Li, K., & Zhu, Z. (2024) · IEEE Transactions on Dependable and Secure Computing · doi:10.1109/tdsc.2024.3520628 · Cited by 8
“…with the ever-changing and growing log volumes, automated log parsing techniques have been developed, utilizing various data mining techniques such as decision trees [19], frequent pattern mining [20], [21], [22], and clustering [23], [24] to extract common parts from log messages as log templates.”
OLMPT: Research on Online Log Parsing Method Based on Prefix Tree
Wen, P., Zhang, Z., & Deng, B. (2020) · International Conference on Information Technologies and Electrical Engineering (ICITEE) · doi:10.1145/3452940.3452951 · Cited by 7
“The template lengths in Figure 1 are [6, 7, 8, 5] respectively.”
BackgroundLog Abstraction for Information Security: Heuristics and Reproducibility
Copstein, R., Schwartzentruber, J., Zincir-Heywood, N., & Heywood, M. I. (2021) · International Conference on Availability, Reliability and Security (ARES) · doi:10.1145/3465481.3470083 · Cited by 8
“LFA (Log File Abstraction) [28] is built as an attempt to fix SLCT’s shortcomings.”
PVE: A log parsing method based on VAE using embedding vectors
Yuan, W., Ying, S., Duan, X., Cheng, H., Zhao, Y., & Shang, J. (2023) · Information Processing & Management · doi:10.1016/j.ipm.2023.103476 · Cited by 8
Revisiting Log Parsing: The Present, the Future, and the Uncertainties
Li, Z., Fu, Q., Huang, Z., Yu, J., Li, Y., Lai, Y., & Ma, Y. (2024) · IEEE Transactions on Reliability · doi:10.1109/tr.2023.3340020 · Cited by 11
“They are SLCT [14], AEL [15], IPLoM [16], LKE [17], LFA [18], LogSig [19], SHISO [20], LogCluster [21], LenMa [22], LogMine [23], Spell [24], Drain [3], and MoLFI [4].”
Structured and Interoperable Logging for the Cloud Computing Era: The Pitfalls and Benefits
Tovarňák, D., Vaekova, A., Novak, S., & Pitner, T. (2013) · IEEE/ACM International Conference on Utility and Cloud Computing (UCC) · doi:10.1109/ucc.2013.30 · Cited by 10
“Approximately 16% of modifications add new variables to log messages.”
BackgroundNagappan, M., & Robinson, B. (2011) · International Workshop on Traceability in Emerging Forms of Software Engineering (TEFSE) · doi:10.1145/1987856.1987869 · Cited by 7
“First pass through the N lines of the log file: Parse the log message in each log line of the log file as static event information (or event identifier) and dynamic parameter information [7].”
MethodSelf-citationLogBug: Generating Adversarial System Logs in Real Time
Sun, J., Liu, B., & Hong, Y. (2020) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3340531.3412165 · Cited by 9
Intelligent Clustering Scheme for Log Data Streams
Joshi, B., Bista, U., & Ghimire, M. (2014) · Lecture notes in computer science · doi:10.1007/978-3-642-54903-8_38 · Cited by 7
“All log lines that don’t satisfy this condition are considered as outliers [8].”
BackgroundQin, Q., Aghili, R., Li, H., & Merlo, E. (2025) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner64311.2025.00036 · Cited by 13
“Unlike semantic-based parsers, statistic-based parsers such as LFA [15] do not require the labor-demanding data labeling and model training but analyze log messages with information such as token frequencies.”
PosParser: A Heuristic Online Log Parsing Method Based on Part-of-Speech Tagging
Jiang, J., Fu, Y., & Xu, J. (2024) · IEEE Transactions on Big Data · doi:10.1109/tbdata.2024.3453756 · Cited by 6
“LFA [13] is similar to SLCT, in which it counts the word occurrences with position, and then determines within each log message whether the word is a constant or a variable based on its frequency.”
Normalization of Unstructured Log Data into Streams of Structured Event Objects
Tovarňák, D., & Pitner, T. (2019) · Immunotechnology · Cited by 6
Threat Hunting in Windows Using Big Security Log Data
Fatemi, M. R., & Ghorbani, A. A. (2019) · Advances in information security, privacy, and ethics book series · doi:10.4018/978-1-5225-9742-1.ch007 · Cited by 6
A Confidence-Guided Evaluation for Log Parsers Inner Quality
Xie, X., Wang, Z., Xiao, X., Lu, Y., Huang, S., & Li, T. (2020) · Mobile Networks and Applications · doi:10.1007/s11036-019-01501-6 · Cited by 6
“Frequent pattern mining and its extension have been used in offline log parsing methods such as SLCT [20], LFA [21] and LogCluster [22].”
MethodAS-Parser: Log Parsing Based on Adaptive Segmentation
Chen, X., Wang, P., Chen, J., & Wang, W. (2023) · Proceedings of the ACM on Management of Data · doi:10.1145/3626719 · Cited by 6
LLM-powered Zero-shot Online Log Parsing
Zhi, C., Cheng, L., Liu, M., Zhao, X., Xu, Y., & Deng, S. (2024) · IEEE International Conference on Web Services (ICWS) · doi:10.1109/icws62655.2024.00106 · Cited by 13
LogGenius: An Unsupervised Log Parsing Framework with Zero-shot Prompt Engineering
Yu, X., Nong, S., He, D., Zheng, W., Ma, T., Liu, N., Li, J., & Xie, G. (2024) · IEEE International Conference on Web Services (ICWS) · doi:10.1109/icws62655.2024.00159 · Cited by 13
“These approaches use statistical methods to identify common words or tokens in logs, assuming that these are likely to contain constants [16], [17].”
MethodStudying and Complementing the Use of Identifiers in Logs
Zhao, J., Tang, Y., Sunil, S., & Shang, W. (2023) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner56733.2023.00019 · Cited by 10
“[11] propose an approach to abstract log lines and assign each log line a number in accordance with abstraction, in which the number is similar to the log event ID mentioned in the aforementioned papers.”
MethodTowards robust log parsing using self-supervised learning for system security analysis
Cao, J., Di, X., Liu, X., Xu, R., Li, J., Ren, W., Qi, H., Hu, P., Zhang, K., & Li, B. (2024) · Intelligent Data Analysis · doi:10.3233/ida-230133 · Cited by 7
“The log file abstraction (LFA) [13] extracted log events based on the frequency of the token in a particular location.”
Hu, J., Long, L., Sui, H., Gu, Z., & Zheng, G. (2025) · Applied Sciences · doi:10.3390/app15041740 · Cited by 6
Analyzing Logs of the University Data Repository
Sosnowski, J. S., Gawkowski, P., Cabaj, K., & Kubacki, M. (2014) · Studies in computational intelligence · doi:10.1007/978-3-319-04714-0_9 · Cited by 5
“1 Distribution of IDF parameter for Neptun server (x-axis [0,1184] words, y-axis [0–5]) 146 J.”
ResultSchema Independent Reduction of Streaming Log Data
Kalamatianos, T., & Kontogiannis, K. A. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-07881-6_27 · Cited by 5
Adaptive and Efficient Log Parsing as a Cloud Service
Li, Z., Song, J., Zhang, T., Yang, T., Ou, X., Ye, Y., Duan, P., Lin, M., & Chen, J. (2025) · ACM SIGMOD International Conference on Management of Data (SIGMOD) · doi:10.1145/3722212.3724427 · Cited by 9
“Frequent pattern mining methods [11, 25, 29] are sensitive to parameter tuning and preprocessing, which limits their robustness.”
Metadata
- Authors
- Nagappan, M., & Vouk, M. A.
- Affiliations
- North Carolina State University (US)
- ORCID
- Nagappan, Meiyappan
- Venue
- 2010 7th IEEE Working Conference on Mining Software Repositories (MSR 2010)
- Publisher
- IEEE
- Published
- 2010-05
- Pages
- 114-117
- OpenAlex
- W1994625872
- Cited by
- 171
- References
- 8
- Citations by year
- 2026: 12 · 2025: 20 · 2024: 27 · 2023: 29 · 2022: 14 · 2021: 19 · 2020: 17 · 2019: 9 · 2018: 2 · 2016: 2 · 2015: 2 · 2014: 4 · 2013: 6 · 2012: 4
- Type
- conference abstract
- Language
- en
- Open access
- closed
- Locations
- International Conference on Mining Software Repositories (MSR) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Networks and Communications › Software System Performance and Reliability
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Software › Advanced Software Engineering Methodologies
- Keywords
- log mining
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.