Fast Dimensional Analysis for Root Cause Investigation in a Large-Scale Service Environment
Open accessThe DOI matches its Crossref record; 3 of the 3 references it cites were found in a registry or an index.
Abstract
Root cause analysis in a large-scale production environment is challenging due to the complexity of the services running across global data centers. Due to the distributed nature of a large-scale system, the various hardware, software, and tooling logs are often maintained separately, making it difficult to review the logs jointly for understanding production issues. Another challenge in reviewing the logs for identifying issues is the scale - there could easily be millions of entities, each described by hundreds of features. In this paper we present a fast dimensional analysis framework that automates the root cause analysis on structured logs with improved scalability. We first explore item-sets, i.e. combinations of feature values, that could identify groups of samples with sufficient support for the target failures using the Apriori algorithm and a subsequent improvement, FP-Growth. These algorithms were designed for frequent item-set mining and association rule learning over transactional databases. After applying them on structured logs, we select the item-sets that are most unique to the target failures based on lift. We propose pre-processing steps with the use of a large-scale real-time database and post-processing techniques and parallelism to further speed up the analysis and improve interpretability, and demonstrate that such optimization is necessary for handling large-scale production datasets. We have successfully rolled out this approach for root cause investigation purposes within Facebook's infrastructure. We also present the setup and results from multiple production use cases in this paper.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Apache Hadoop YARN: yet another resource negotiator
Vavilapalli, V. K., Murthy, A. C., Douglas, C., Agarwal, S., Konar, M., Evans, R., Graves, T., Lowe, J., Shah, H., Seth, S., et al. (2013) · Proceedings of the 4th annual Symposium on Cloud Computing · doi:10.1145/2523616.2523633 · Cited by 1,848
(2011) · doi:10.5555/1972457.1972488 · Cited by 1,619
Large-scale cluster management at Google with Borg
Verma, A., Pedrosa, L., Korupolu, M., Oppenheimer, D., Tune, E., & Wilkes, J. (2015) · Proceedings of the Tenth European Conference on Computer Systems · doi:10.1145/2741948.2741964 · Cited by 1,468
Cited by
A survey of AIOps methods for failure management
Notaro, P., Cardoso, J., & Gerndt, M. (2021) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3483424 · Cited by 142
Yu, G., Chen, P., Li, Y., Chen, H., Li, X., & Zheng, Z. (2023) · Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3611643.3616249 · Cited by 140
AI for IT Operations (AIOps) on Cloud Platforms: Reviews, Opportunities and Challenges
Cheng, Q., Sahoo, D., Saha, A., Yang, W., Liu, C., Woo, G., Singh, M., Saverese, S., & Hoi, S. C. H. (2023) · arXiv · doi:10.48550/arxiv.2304.04661 · Cited by 23
(2024) · doi:10.1145/3597503.3639088 · Cited by 22
Sarda, K., Namrud, Z., Litoiu, M., Shwartz, L., & Watts, I. (2024) · Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering · doi:10.1145/3663529.3663855 · Cited by 19
Holistic Root Cause Analysis for Failures in Cloud-Native Systems Through Observability Data
Han, Y., Du, Q., Huang, Y., Li, P., Shi, X., Wu, J., Fang, P., Tian, F., & He, C. (2024) · IEEE Transactions on Services Computing · doi:10.1109/tsc.2024.3478759 · Cited by 16
“Many approaches rely on single-modal data, including approaches based on traces [7], [8], [9], [10], [11], logs [12], [13], [14], [15], [16], [17], [18], and metrics [19], [20], 1939-1374 © 2024 IEEE.”
The Effectiveness of Compact Fine-Tuned LLMs in Log Parsing
Mehrabi, M., Hamou-Lhadj, A., & Moosavi, H. (2024) · 2024 IEEE International Conference on Software Maintenance and Evolution (ICSME) · doi:10.1109/icsme58944.2024.00047 · Cited by 4
“Log analysis is instrumental in facilitating a variety of tasks including the detection of system anomalies [2], [3], root cause analysis of system faults [4], [5], and performance optimization [6].”
MethodMicroKGCL: A Knowledge Graph for Root Cause Localization of Feedback Issues in Microservices
Yao, X.-W., Lu, Q.-C., Li, Q., Liu, L.-L., & Zhu, Z.-C. (2023) · 2023 IEEE 23rd International Conference on Software Quality, Reliability, and Security (QRS) · doi:10.1109/qrs60937.2023.00067 · Cited by 3
Metadata
- Authors
- Lin, F., Muzumdar, K., Laptev, N. P., Curelea, M.-V., Lee, S., & Sankar, S.
- Venue
- Proceedings of the ACM on Measurement and Analysis of Computing Systems
- Published
- 2020-06-09
- Volume
- 4
- Issue
- 2
- Pages
- 1-23
- ISSN
- 2476-1249 (ISSN-L)
- OpenAlex
- W2987006149
- Cited by
- 33
- References
- 3
- FWCI
- 1.53
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.