Towards Automated Log Parsing for Large-Scale Log Data Analysis
The DOI matches its Crossref record; 41 of the 42 references it cites were found in a registry or an index.
Abstract
Logs are widely used in system management for dependability assurance because they are often the only data available that record detailed system runtime behaviors in production. Because the size of logs is constantly increasing, developers (and operators) intend to automate their analysis by applying data mining methods, therefore structured input data (e.g., matrices) are required. This triggers a number of studies on log parsing that aims to transform free-text log messages into structured events. However, due to the lack of open-source implementations of these log parsers and benchmarks for performance comparison, developers are unlikely to be aware of the effectiveness of existing log parsers and their limitations when applying them into practice. They must often reimplement or redesign one, which is time-consuming and redundant. In this paper, we first present a characterization study of the current state of the art log parsers and evaluate their efficacy on five real-world datasets with over ten million log messages. We determine that, although the overall accuracy of these parsers is high, they are not robust across all datasets. When logs grow to a large scale (e.g., 200 million log messages), which is common in practice, these parsers are not efficient enough to handle such data on a single computer. To address the above limitations, we design and implement a parallel log parser (namely POP) on top of Spark, a large-scale data processing platform. Comprehensive experiments have been conducted to evaluate POP on both synthetic and real-world datasets. The evaluation results demonstrate the capability of POP in terms of accuracy, efficiency, and effectiveness on subsequent log mining tasks.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Introduction to information retrieval
(2009) · Choice Reviews Online · doi:10.5860/choice.46-2715 · Cited by 12,278
Introduction to Information Retrieval
Manning, C. D., Raghavan, P., & Schütze, H. (2008) · doi:10.1017/cbo9780511809071 · Cited by 11,288
Everitt, B. J., Landau, S., & Leese, M. (1974) · Cited by 9,358
Resilient distributed datasets: a fault-tolerant abstraction for in-memory cluster computing
Zaharia, M., Chowdhury, M., Das, T., Dave, A., Ma, J. H., McCauley, M., Franklin, M. J. M., Shenker, S. J., & Stoica, I. G. (2012) · UC Berkeley · Cited by 4,891
Everitt, B. S., Landau, S., Leese, M., & Stahl, D. (2011) · Wiley Series in Probability and Statistics · doi:10.1002/9780470977811 · Cited by 2,907
Detecting large-scale system problems by mining console logs
Xu, W., Huang, L., Fox, A., Patterson, D., & Jordan, M. I. (2009) · Proceedings of the ACM SIGOPS 22nd symposium on Operating systems principles · doi:10.1145/1629575.1629587 · Cited by 1,332 · cited 10×
“Furthermore, modern systems often integrate open-source software components written by hundreds of developers [3].”
BackgroundMethodInfluentialMinimum Spanning Trees and Single Linkage Cluster Analysis
Gower, J. C., & Ross, G. J. S. (1969) · Applied Statistics · doi:10.2307/2346439 · Cited by 1,223 · cited 1×
“To address over-parsing and further improve parsing accuracy, in this step, POP employs hierarchical clustering [27] to cluster similar groups based on their log events.”
MethodDiagnosing network-wide traffic anomalies
Lakhina, A., Crovella, M., & Diot, C. (2004) · Proceedings of the 2004 conference on Applications, technologies, architectures, and protocols for computer communications · doi:10.1145/1015467.1015492 · Cited by 1,039 · cited 1×
“However, the anomaly detection task evaluated is an important log mining task widely studied [37], [38], which is presented in a paper [3] enjoying more than 300 citations.”
BackgroundWhat Supercomputers Say: A Study of Five System Logs
Oliner, A., & Stearley, J. (2007) · 37th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN'07) · doi:10.1109/dsn.2007.103 · Cited by 620 · cited 2×
“Besides, we thank those who release log data [3], [33], [34], which greatly facilitates our research.”
MethodThe Complexity of Some Problems on Subsequences and Supersequences
Maier, D. (1978) · Journal of the ACM · doi:10.1145/322063.322075 · Cited by 616 · cited 1×
“The groups in the same cluster will be merged, and a new log event will be generated by calculating the Longest Common Subsequence (LCS) [28] of the original log events.”
MethodExperience Report: System Log Analysis for Anomaly Detection
He, S., Zhu, J., He, P., & Lyu, M. R. (2016) · 2016 IEEE 27th International Symposium on Software Reliability Engineering (ISSRE) · doi:10.1109/issre.2016.21 · Cited by 593 · cited 1×
“Typical examples of log analysis techniques on dependability assurance include anomaly detection [3], [4], [5], program verification [6], [7], problem diagnosis [8], [9], and security assurance [10], [11].”
BackgroundSelf-citationExecution Anomaly Detection in Distributed Systems through Unstructured Log Analysis
Fu, Q., Lou, J.-G., Wang, Y., & Li, J. (2009) · 2009 Ninth IEEE International Conference on Data Mining · doi:10.1109/icdm.2009.60 · Cited by 542 · cited 6×
“Recent work proposes data-driven log parsers (e.g., SLCT [16], IPLoM [23], LKE [4], LogSig [18]), in which data mining techniques are employed.”
BackgroundMethodResultInfluentialA data clustering algorithm for mining patterns from event logs
Vaarandi, R. (2004) · Proceedings of the 3rd IEEE Workshop on IP Operations & Management (IPOM 2003) (IEEE Cat. No.03EX764) · doi:10.1109/ipom.2003.1251233 · Cited by 453 · cited 6×
“Recent work proposes data-driven log parsers (e.g., SLCT [16], IPLoM [23], LKE [4], LogSig [18]), in which data mining techniques are employed.”
MethodResultInfluentialSensitivity of PCA for traffic anomaly detection
Ringberg, H., Soule, A., Rexford, J., & Diot, C. (2007) · Proceedings of the 2007 ACM SIGMETRICS international conference on Measurement and modeling of computer systems · doi:10.1145/1254882.1254895 · Cited by 353 · cited 1×
“[38], which is presented in a paper [3] enjoying more than 300 citations.”
BackgroundClustering event logs using iterative partitioning
Makanju, A. A. O., Zincir-Heywood, A. N., & Milios, E. E. (2009) · Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/1557019.1557154 · Cited by 338 · cited 3×
“More specifically, we study SLCT [16], IPLoM [17], LKE [4], and LogSig [18], which are widely used log parsers in log analysis.”
MethodResultStructured comparative analysis of systems logs to diagnose performance problems
Nagaraj, K., Killian, C. E., & Neville, J. (2012) · Purdue e-Pubs (Purdue University System) · Cited by 289
LogSig: generating system events from raw textual logs
Tang, L., Li, T., & Perng, C.-S. (2011) · Proceedings of the 20th ACM international conference on Information and knowledge management · doi:10.1145/2063576.2063690 · Cited by 244 · cited 6×
“Similar to the existing work [18], we randomly sample 2k log messages from each data-set in our evaluation, LKE and LogSig cannot parse large log datasets in reasonable time (e.g., LogSig requires 1 day to parse the entire BGL data).”
MethodResultInfluentialWhere do developers log? an empirical study on logging practices in industry
Fu, Q., Zhu, J., Hu, W., Lou, J.-G., Ding, R., Lin, Q., Zhang, D., & Xie, T. (2014) · Companion Proceedings of the 36th International Conference on Software Engineering · doi:10.1145/2591062.2591175 · Cited by 230 · cited 1×
“Besides, the logging practices in industry has been studied in some recent work [53], [54].”
BackgroundSelf-citationAn Evaluation Study on Log Parsing and Its Use in Log Mining
He, P., Zhu, J., He, S., Li, J., & Lyu, M. R. (2016) · 2016 46th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN) · doi:10.1109/dsn.2016.66 · Cited by 230 · cited 4×
“The source code of POP has been release [21] for reuse.”
BackgroundMethodInfluentialSelf-citationLogMine: Fast Pattern Recognition for Log Analytics
Hamooni, H., Debnath, B., Xu, J., Zhang, H., Jiang, G., & Mueen, A. (2016) · Proceedings of the 25th ACM International on Conference on Information and Knowledge Management · doi:10.1145/2983323.2983358 · Cited by 227 · cited 1×
“Many researchers (e.g., [6], [9], [48], [49]) and practitioners (as revealed in StackOverflow questions [50], [51]) in this field have to implement their own log parsers to deal with their log data.”
BackgroundLeveraging existing instrumentation to automatically infer invariant-constrained models
Beschastnikh, I., Brun, Y., Schneider, S., Sloan, M., & Ernst, M. D. (2011) · Proceedings of the 19th ACM SIGSOFT symposium and the 13th European conference on Foundations of software engineering · doi:10.1145/2025113.2025151 · Cited by 216 · cited 4×
“Typical examples of log analysis techniques on dependability assurance include anomaly detection [3], [4], [5], program verification [6], [7], problem diagnosis [8], [9], and security assurance [10], [11].”
BackgroundMethodInfluentialLearning to Log: Helping Developers Make Informed Logging Decisions
Zhu, J., He, P., Fu, Q., Zhang, H., Lyu, M. R., & Zhang, D. (2015) · 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering · doi:10.1109/icse.2015.60 · Cited by 212 · cited 1×
“This problem is compounded by the fact that the log printing statements in modern systems update frequently [14] (e.g., hundreds of new logging statements every month [15]); consequently, developers must regularly review the updated log printing statements of various system components for the…”
BackgroundSelf-citationCharacterizing logging practices in open-source software
Yuan, D., Park, S., & Zhou, Y. (2012) · 2012 34th International Conference on Software Engineering (ICSE) · doi:10.1109/icse.2012.6227202 · Cited by 193 · cited 1×
“In particular, Yuan et al. [8], [52] conduct an empirical study on the logging practices in open-source systems.”
BackgroundMi, H., Wang, H., Zhou, Y., Lyu, M. R.-T., & Cai, H. (2013) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2013.21 · Cited by 176
Self-citationDetection of Early-Stage Enterprise Infection by Mining Large-Scale Log Data
Oprea, A., Li, Z., Yen, T.-F., Chin, S. H., & Alrwais, S. (2015) · 2015 45th Annual IEEE/IFIP International Conference on Dependable Systems and Networks · doi:10.1109/dsn.2015.14 · Cited by 174 · cited 2×
“Typical examples of log analysis techniques on dependability assurance include anomaly detection [3], [4], [5], program verification [6], [7], problem diagnosis [8], [9], and security assurance [10], [11].”
BackgroundCharacterizing Logging Practices in Open-Source Software
Yuan, D., Park, S., & Zhou, Y. (2012) · Cited by 148
A Lightweight Algorithm for Message Type Extraction in System Application Logs
Makanju, A., Zincir-Heywood, A. N., & Milios, E. E. (2012) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2011.138 · Cited by 145 · cited 3×
“Recent work proposes data-driven log parsers (e.g., SLCT [16], IPLoM [23], LKE [4], LogSig [18]), in which data mining techniques are employed.”
MethodBe conservative: enhancing failure diagnosis with proactive logging
Yuan, D., Park, S., Huang, P., Liu, Y., Lee, M. M., Tang, X., Zhou, Y., & Savage, S. (2013) · Cited by 143
Assisting developers of Big Data Analytics Applications when deploying on Hadoop clouds
Shang, W., Jiang, Z. M., Hemmati, H., Adams, B., Hassan, A. E., & Martin, P. (2013) · 2013 35th International Conference on Software Engineering (ICSE) · doi:10.1109/icse.2013.6606586 · Cited by 118 · cited 3×
“Typical examples of log analysis techniques on dependability assurance include anomaly detection [3], [4], [5], program verification [6], [7], problem diagnosis [8], [9], and security assurance [10], [11].”
BackgroundTowards Building Forensics Enabled Cloud Through Secure Logging-as-a-Service
Zawoad, S., Dutta, A. K., & Hasan, R. (2015) · IEEE Transactions on Dependable and Secure Computing · doi:10.1109/tdsc.2015.2482484 · Cited by 107 · cited 1×
“Zawoad et al. [41] propose a scheme to reveal cloud users’ logs for forensics investigation while preserving their confidentiality.”
BackgroundIndustry Practices and Event Logging: Assessment of a Critical Software Development Process
Pecchia, A., Cinque, M., Carrozza, G., & Cotroneo, D. (2015) · 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering · doi:10.1109/icse.2015.145 · Cited by 78
Collaborative Search Log Sanitization: Toward Differential Privacy and Boosted Utility
Hong, Y., Vaidya, J., Lu, H., Karras, P., & Goel, S. (2015) · IEEE Transactions on Dependable and Secure Computing · doi:10.1109/tdsc.2014.2369034 · Cited by 75 · cited 1×
“Hong et al. [40] design a framework to sanitize search logs with strong privacy guarantee and sufficiently retained utility.”
BackgroundLog-Based Reliability Analysis of Software as a Service (SaaS)
Banerjee, S. K., Srikanth, H., & Čukić, B. (2010) · IEEE International Symposium on Software Reliability Engineering (ISSRE) · doi:10.1109/issre.2010.46 · Cited by 52 · cited 1×
“Many researchers (e.g., [6], [9], [48], [49]) and practitioners (as revealed in StackOverflow questions [50], [51]) in this field have to implement their own log parsers to deal with their log data.”
BackgroundELT: Efficient Log-based Troubleshooting System for Cloud Computing Infrastructures
Kc, K., & Gu, X. (2011) · 2011 IEEE 30th International Symposium on Reliable Distributed Systems · doi:10.1109/srds.2011.11 · Cited by 47 · cited 1×
“Kc et al. [45] detect anomalies by using both coarse-grained and fine-grained log features.”
MethodComprehensive log .les for autonomic systems
Salfne, F., Tschirpke, S., & Malek, M. (2004) · IEEE International Parallel and Distributed Processing Symposium (IPDPS) · doi:10.1109/ipdps.2004.1303243 · Cited by 40
LEAPS: Detecting Camouflaged Attacks with Statistical Learning Guided by Program Analysis
Gu, Z., Pei, K., Wang, Q., Si, L., Zhang, X., & Xu, D. (2015) · International Conference on Dependable Systems & Networks/Proceedings · doi:10.1109/dsn.2015.34 · Cited by 39 · cited 2×
“Typical examples of log analysis techniques on dependability assurance include anomaly detection [3], [4], [5], program verification [6], [7], problem diagnosis [8], [9], and security assurance [10], [11].”
BackgroundAssessing time coalescence techniques for the analysis of supercomputer logs
Martino, C. D., Cinque, M., & Cotroneo, D. (2012) · IEEE/IFIP International Conference on Dependable Systems and Networks (DSN) · doi:10.1109/dsn.2012.6263946 · Cited by 39 · cited 1×
“Log analysis is also employed in structured comparative analysis for performance problem diagnosis [9] and time coalescence assessment for failure reconstruction [47].”
MethodSystem problem detection by mining console logs
Patterson, D. A., & Xu, W. (2010) · Cited by 37
Automated Derivation of Application-Specific Error Detectors Using Dynamic Analysis
Pattabiraman, K., Saggese, G. P., Chen, D., Kalbarczyk, Z., & Iyer, R. K. (2010) · IEEE Transactions on Dependable and Secure Computing · doi:10.1109/tdsc.2010.19 · Cited by 34 · cited 1×
“Besides, Pattabiraman et al. [46] design an assertion generator based on execution logs to detect application runtime errors.”
MethodA comparison of general-purpose distributed systems for data processing
Li, J., Cheng, J. S.-C., Zhao, Y., Yang, F., Huang, Y., Chen, H., & Zhao, R. (2016) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata.2016.7840626 · Cited by 12
Self-citationSystem Problem Detection by Mining Console Logs
Xu, W. (2010) · eScholarship (California Digital Library) · Cited by 11
Research works printed but not linked: 4
- salfner, 2004, Comprehensive logfiles for autonomic systems, Proc 18th Int Parallel Distrib Process Symp
- krause, 1987, Taxicab Geometry
- gu, 2015, Proc 45th Annu IEEE/IFIP Int Conf Dependable Syst Netw, 57
- lang, 2013, Using SEC, LOGIN The USENIX Magazine, 38, 38
Cited by
Robust log-based anomaly detection on unstable log data
Zhang, X., Xu, Y., Lin, Q., Qiao, B., Zhang, H., Dang, Y., Xie, C., Yang, X., Cheng, Q., Li, Z., et al. (2019) · Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3338906.3338931 · Cited by 761
“Over the years, many automated log-based approaches have been proposed to detect system anomalies [3, 10, 18, 29, 44].”
BackgroundMethodInfluentialTools and benchmarks for automated log parsing
Zhu, J., He, S., Liu, J., He, P., Xie, Q., Zheng, Z., & Lyu, M. R. (2019) · 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1109/icse-seip.2019.00021 · Cited by 574
“[53] have studied large-scale log parsing through the parallelization on Spark.”
MethodSelf-citationSarker, I. H. (2021) · SN Computer Science · doi:10.1007/s42979-021-00765-8 · Cited by 570
A Survey on Automated Log Analysis for Reliability Engineering
He, S., He, P., Chen, Z., Yang, T., Su, Y., & Lyu, M. R. (2022) · ACM Computing Surveys · doi:10.1145/3460345 · Cited by 334
“ch [82] in logging practice and Deeplog [42] in anomaly detection. Besides machine learning, parallelization has been employed in various recent papers, such as Logzip [90] in log compression and POP [59] in log parsing. These extensive studies on automated log analysis across multiple core directions have largely boo…”
BackgroundMethodInfluentialSelf-citationUllah, F., Ullah, S., Srivastava, G., & Lin, J. C.-W. (2024) · Digital Communications and Networks · doi:10.1016/j.dcan.2023.03.008 · Cited by 266
Metadata
- Authors
- He, P., Zhu, J., He, S., Li, J., & Lyu, M. R.
- Affiliations
- Chinese University of Hong Kong, Shenzhen (CN)
- Huawei Technologies (China) (CN)
- Venue
- IEEE Transactions on Dependable and Secure Computing
- Published
- 2018-11-01
- Volume
- 15
- Issue
- 6
- Pages
- 931-944
- ISSN
- 1545-5971 (ISSN-L)
- 1941-0018
- 2160-9209
- OpenAlex
- W2762028850
- Cited by
- 230
- References
- 42
- Citations by year
- 2026: 12 · 2025: 26 · 2024: 28 · 2023: 40 · 2022: 23 · 2021: 45 · 2020: 30 · 2019: 18 · 2018: 8
- FWCI
- 10.19
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- IEEE Transactions on Dependable and Secure Computing (publishedVersion)
- Funders
- National Natural Science Foundation of China (61332010)
- Research Grants Council of the Hong Kong Special Administrative Region, China (CUHK 14205214)
- 2015 Microsoft Research Asia Collaborative Research Program (FY16-RES-THEME-005)
- Topics
- Physical Sciences › Computer Science › Computer Networks and Communications › Software System Performance and Reliability
- Physical Sciences › Computer Science › Information Systems › Advanced Database Systems and Queries
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Keywords
- log parsing, log mining, Apache Spark, dependability assurance
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.