Abstracting log lines to log event types for mining software system logs
The DOI matches its Crossref record; 8 of the 8 references it cites were found in a registry or an index.
Abstract
Log files contain valuable information about the execution of a system. This information is often used for debugging, operational profiling, finding anomalies, detecting security threats, measuring performance etc. The log files are usually too big for extracting this valuable information manually, even though manual perusal is still one of the more widely used techniques. Recently a variety of data mining and machine learning algorithms are being used to analyze the information in the log files. A major road block for the efficient use of these algorithms is the inherent variability present in every log line of a log file. Each log line is a combination of a static message type field and a variable parameter field. Even though both these fields are required, the analyses algorithm often requires that these be separated out, in order to find correlations in the repeating log event types. This disentangling of the message and parameter fields to find the event types is called abstraction of log lines. Each log line is abstracted to a unique ID or event type and the dynamic parameter value is extracted to give an insight on the current state of the system. In this paper we present a technique based on a clustering technique used in the Simple Log file Clustering Tool for log file abstraction. This solution is especially useful when we don't have access to the source code of the application or when the lines in the log file do not conform to a rigid structure. We evaluated our implementation on log files from the Virtual Computing Lab, a cloud computer management system at North Carolina State University, and abstracted it to 727 unique event types.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“The reason we choose WordPiece is that it can effectively handle the OOV words and reduce the vocabulary size.”
From Log-based Anomaly Detection Without Log Parsing (2021) · cites this paper as [41]
Citations
Cites
A data clustering algorithm for mining patterns from event logs
Vaarandi, R. (2004) · Proceedings of the 3rd IEEE Workshop on IP Operations & Management (IPOM 2003) (IEEE Cat. No.03EX764) · doi:10.1109/ipom.2003.1251233 · Cited by 453 · cited 1×
“The log lines in the log files can then be matched to these regular expressions and the ID for the best match is used for each line in the log file.”
MethodAutomated Identification of Failure Causes in System Logs
Mariani, L., & Pastore, F. (2008) · 2008 19th International Symposium on Software Reliability Engineering (ISSRE) · doi:10.1109/issre.2008.48 · Cited by 127 · cited 2×
“These include but are not limited to, fault detection by monitoring, fault isolation [5], operational profiling [6] etc. Tools like Splunk [1], and Swatch [3], are used to monitor log files.”
BackgroundMethodInfluentialAutomated System Monitoring and Notification With Swatch
Hansen, S., & Atkins, E. T. (1993) · Cited by 125
Mining console logs for large-scale system problem detection
Xu, W., Huang, L., Fox, A., Patterson, D. A., & Jordan, M. I. (2008) · Cited by 105
An automated approach for abstracting execution logs to execution events
Jiang, Z. M., Hassan, A. E., Hamann, G., & Flora, P. (2008) · Journal of Software Maintenance and Evolution: Research and Practice · doi:10.1002/smr.374 · Cited by 83
Error log processing for accurate failure prediction
Salfner, F., & Tschirpke, S. (2008) · Cited by 63
Efficiently Extracting Operational Profiles from Execution Logs Using Suffix Arrays
Nagappan, M., Wu, K., & Vouk, M. A. (2009) · 2009 20th International Symposium on Software Reliability Engineering · doi:10.1109/issre.2009.23 · Cited by 63 · cited 3×
“These include but are not limited to, fault detection by monitoring, fault isolation [5], operational profiling [6] etc. Tools like Splunk [1], and Swatch [3], are used to monitor log files.”
BackgroundMethodSelf-citationAbstracting Execution Logs to Execution Events for Enterprise Applications
Jiang, Z. M., Hassan, A. E., Flora, P. K., & Hamann, G. (2008) · Cited by 18
Research works printed but not linked: 2
- 0, Virtual Computing Lab
- 0, Splunk
Cited by
Tools and benchmarks for automated log parsing
Zhu, J., He, S., Liu, J., He, P., Xie, Q., Zheng, Z., & Lyu, M. R. (2019) · 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1109/icse-seip.2019.00021 · Cited by 574
“1) Frequent Pattern Mining: Among all the log parsing methods, three of them belong to this category, i.e., SLCT [20], LFA [42], and LogCluster [21].”
BackgroundLog-based Anomaly Detection Without Log Parsing
Le, V.-H., & Zhang, H. (2021) · 2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase51524.2021.9678773 · Cited by 361
“There are many log parsing techniques, including frequent pattern mining [17], [18], [19], clustering [20], [21], [22], language modeling [23], heuristics [24], [25], [26], etc.”
BackgroundLoghub: a large collection of system log datasets for AI-driven log analytics
Zhu, J., He, S., He, P., Liu, J., & Lyu, M. R. (2023) · 2023 IEEE 34th International Symposium on Software Reliability Engineering (ISSRE) · doi:10.1109/issre59848.2023.00071 · Cited by 302
“Conversely, LogSig, LFA, MoLFI, and LKE have the lowest average accuracy.”
BackgroundMethodInfluentialLog-based anomaly detection with deep learning: how far are we?
Le, V.-H., & Zhang, H. (2022) · Proceedings of the 44th International Conference on Software Engineering · doi:10.1145/3510003.3510155 · Cited by 257
“There are many log parsing techniques, based on frequent pattern mining [10, 38, 45], clustering [16, 42, 44], and heuristics [17, 22, 34].”
MethodHitAnomaly: Hierarchical Transformers for Anomaly Detection in System Log
Huang, S., Liu, Y., Fung, C., He, R., Zhao, Y., Yang, H., & Luan, Z. (2020) · IEEE Transactions on Network and Service Management · doi:10.1109/tnsm.2020.3034647 · Cited by 228
“To further evaluate the impact of log parser on log anomaly detection tasks, we use LFA [28] to preprocess the OpenStack dataset.”
Method
Metadata
- Authors
- Nagappan, M., & Vouk, M. A.
- Affiliations
- North Carolina State University (US)
- ORCID
- Nagappan, Meiyappan
- Venue
- 2010 7th IEEE Working Conference on Mining Software Repositories (MSR 2010)
- Publisher
- IEEE
- Published
- 2010-05
- Pages
- 114-117
- OpenAlex
- W1994625872
- Cited by
- 171
- References
- 8
- Citations by year
- 2026: 12 · 2025: 20 · 2024: 27 · 2023: 29 · 2022: 14 · 2021: 19 · 2020: 17 · 2019: 9 · 2018: 2 · 2016: 2 · 2015: 2 · 2014: 4 · 2013: 6 · 2012: 4
- Type
- conference abstract
- Language
- en
- Open access
- closed
- Locations
- International Conference on Mining Software Repositories (MSR) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Networks and Communications › Software System Performance and Reliability
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Software › Advanced Software Engineering Methodologies
- Keywords
- log mining
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Abstracting log lines to log event types for mining software system logs. https://app.cochat.ai/papers/10.1109/msr.2010.5463281
- The paper
- Nagappan, M., & Vouk, M. A. (2010). Abstracting log lines to log event types for mining software system logs. 2010 7th IEEE Working Conference on Mining Software Repositories (MSR 2010). https://doi.org/10.1109/msr.2010.5463281
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.