Fault Analysis and Debugging of Microservice Systems: Industrial Survey, Benchmark System, and Empirical Study
The DOI matches its Crossref record; 40 of the 42 references it cites were found in a registry or an index, and 1 are not checked yet.
Abstract
The complexity and dynamism of microservice systems pose unique challenges to a variety of software engineering tasks such as fault analysis and debugging. In spite of the prevalence and importance of microservices in industry, there is limited research on the fault analysis and debugging of microservice systems. To fill this gap, we conduct an industrial survey to learn typical faults of microservice systems, current practice of debugging, and the challenges faced by developers in practice. We then develop a medium-size benchmark microservice system (being the largest and most complex open source microservice system within our knowledge) and replicate 22 industrial fault cases on it. Based on the benchmark system and the replicated fault cases, we conduct an empirical study to investigate the effectiveness of existing industrial debugging practices and whether they can be further improved by introducing the state-of-the-art tracing and visualization techniques for distributed systems. The results show that the current industrial practices of microservice debugging can be improved by employing proper tracing and visualization techniques and strategies. Our findings also suggest that there is a strong need for more intelligent trace analysis and visualization, e.g., by combining trace visualization and improved fault localization, and employing data-driven and learning-based recommendation for guided visual exploration and comparison of traces.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Based on various testimonials [46], the debugging process is divided into seven phases: Initial Understanding (IU), Environment Setup (ES), Failure Reproduction (FR), Failure Identification (FI), Fault Scoping (FS), Fault Localization (FL), and Fault Fixing (FF).”
From Anomaly detection and root-cause identification in microservices: a survey (2026) · cites this paper as [46]
In the citing paper
“Research has focused on tasks such as anomaly detection [90, 237], incident classification [36, 68], and root cause analysis [228, 247].”
From A 2030 Roadmap for Software Engineering (2025) · cites this paper as [247]
Citations
Cites
Microservices: Yesterday, Today, and Tomorrow
Dragoni, N., Giallorenzo, S., Lafuente, A. L., Mazzara, M., Montesi, F., Mustafin, R., & Safina, L. (2017) · Springer eBooks · doi:10.1007/978-3-319-67425-4_12 · Cited by 2,058
A Survey on Software Fault Localization
Wong, W. E., Gao, R., Li, Y., Abreu, R., & Wotawa, F. (2016) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2016.2521368 · Cited by 1,110 · cited 1×
“In addition, existing fault localization techniques [36] are ineffective for microservices.”
Simplifying and isolating failure-inducing input
Zeller, A., & Hildebrandt, R. (2002) · IEEE Transactions on Software Engineering · doi:10.1109/32.988498 · Cited by 1,079 · cited 2×
“, spectrum-based fault localization [38] and delta debugging [56]) for microservice debugging.”
Scalable statistical bug isolation
Liblit, B., Naik, M., Zheng, A. X., Aiken, A., & Jordan, M. I. (2005) · ACM SIGPLAN Notices · doi:10.1145/1064978.1065014 · Cited by 729
Microservices: The Journey So Far and Challenges Ahead
Jamshidi, P., Pahl, C., Mendonca, N. C., Lewis, J., & Tilkov, S. (2018) · IEEE Software · doi:10.1109/ms.2018.2141039 · Cited by 554 · cited 1×
“There have been appeals that practitioners and researchers develop and share a common microservice infrastructure that can emulate the production environments of typical microservice applications for more repeatable and industry-focused empirical studies [23], [24].”
Using Mutation Analysis for Assessing and Comparing Testing Coverage Criteria
Andrews, J. H., Briand, L. C., Labiche, Y., & Namin, A. S. (2006) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2006.83 · Cited by 481 · cited 1×
“Similarly related fault localization techniques are designed mainly for sequential programs, such as statistic-based fault localization [39], and machine-learning-based ones [40], [41].”
Scalable statistical bug isolation
Liblit, B., Naik, M., Zheng, A. X., Aiken, A., & Jordan, M. I. (2005) · Proceedings of the 2005 ACM SIGPLAN conference on Programming language design and implementation · doi:10.1145/1065010.1065014 · Cited by 472 · cited 1×
“Similarly related fault localization techniques are designed mainly for sequential programs, such as statistic-based fault localization [39], and machine-learning-based ones [40], [41].”
Spectrum-Based Multiple Fault Localization
Abreu, R., Zoeteweij, P., & van Gemund, A. J. C. (2009) · 2009 IEEE/ACM International Conference on Automated Software Engineering · doi:10.1109/ase.2009.25 · Cited by 401 · cited 4×
“There has been some research on debugging concurrent programs [38], [59], [60] and distributed systems [10], [61], [62], [63].”
InfluentialA Systematic Mapping Study in Microservice Architecture
Alshuqayran, N., Ali, N., & Evans, R. (2016) · 2016 IEEE 9th International Conference on Service-Oriented Computing and Applications (SOCA) · doi:10.1109/soca.2016.15 · Cited by 399 · cited 1×
“[57] present a systematic mapping study on microservice architecture, focusing on the architec-”
Research on Architecting Microservices: Trends, Focus, and Potential for Industrial Adoption
Francesco, P. D., Malavolta, I., & Lago, P. (2017) · 2017 IEEE International Conference on Software Architecture (ICSA) · doi:10.1109/icsa.2017.24 · Cited by 311 · cited 3×
“[2] present a systematic mapping study on the current state of the art on architect-”
Research works printed but not linked: 2
- 10.1109/ecase.2017..4 · doi:10.1109/ecase.2017..4
- 10.1109/scc.2016.113 · DOI · Paper page
Cited by
Items 501–548 of 548
Scalable and Adaptive Log-based Anomaly Detection: A Synergistic Approach
Liu, J., Huang, J., Jiang, Z., Huo, Y., Chen, Z., Gu, J., Feng, C. Q., Yan, M., & Lyu, M. R.-T. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3846182
Tong, W., Liu, X., & Zhou, J. (2026) · International Conference on Software Engineering and Computer Applications · doi:10.1145/3829641.3829680
MARCA: Multi-Agent Root Cause Analysis with Multi-Modal DataPreprint
Wang, Y. (2026) · Research Square · doi:10.21203/rs.3.rs-10852927/v1
SRE-Marathon: A Continuous, Change-Driven Benchmark for Autonomous Site Reliability AgentsPreprint
Tian, Y., Bai, Y., He, Y., Chong, Z., Gao, Y., Li, Y., & Jacobsen, H. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.33023
“We use three microservice applications spanning different architectures, topologies, and failure surfaces: the 20-service OpenTelemetry Demo Astronomy Shop (OpenTelemetry, 2023), the 11-service Online Boutique (Google Cloud Platform, 2019), and the 41-service Train Ticket (Zhou et al., 2021).”
InfluentialSafePar: Monitoring Asynchrony in MicroservicesPreprint
Grewal, K., Godfrey, P. B., Hsu, J., & Mathur, U. (2026) · arXiv (Cornell University)
Gleaner: A Semantically-Rich and Efficient Online Sampler for Microservice Diagnostics
Yang, Y., Fang, A., Zhang, S., & He, P. (2026) · Proceedings of the ACM on software engineering. · doi:10.1145/3832241
“Microservice architectures, while enabling rapid development and deployment, introduce profound operational complexity [8, 24, 50, 52].”
Muggalla, B. K. K. (2026) · Journal of Cyber-Physical Security and Robotics · doi:10.64235/xd3wma19
Yedmala, A. (2026) · Journal of Computer Science and Technology Studies · doi:10.32996/jcsts.2026.8.9.12
Kothakonda, V. V. (2026) · International Journal of Intelligent Systems and Applications in Engineering
Contrastive Failure Embeddings for Lakehouse Reliability Engineering
Sivaraman, M. K. (2023) · International Journal of Intelligent Systems and Applications in Engineering · doi:10.5281/zenodo.23381837
Markovic, D. E. V. (2024) · International Journal of Next-Generation Engineering and Technology
Graph-Based Fault Localization in Microservices Using Static Code Metrics
Can, T., & Buzluca, F. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-39236-7_11
From Alert Floods to Precedence Forests: Zero-Prior-Knowledge Incident Triage with LOGOSPreprint
Mysore, R. N. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.02297
“To verify whether log lines naturally link events across subsystems, we analyzed telemetry across three platforms: 25 enterprise virtual infrastructure bundles, 4 OpenStack cloud datasets [53], and 8 TrainTicket microservice benchmarks [52].”
Cloud systems management with efficient and robust online learning
Qiu, H. (2024) · Illinois Digital Environment for Access to Learning and Scholarship (University of Illinois at Urbana-Champaign)
Kong, F., Fang-Fang, C., & Lim, A. H. L. (2026) · Research Square · doi:10.21203/rs.3.rs-10249260/v1
Enhancing Trace-Based Root Cause Analysis for Microservice Systems via Code Change Understanding
Zhang, M., Zhang, C., Xie, S., Chen, S., Wu, L., & Peng, X. (2026) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3832783.3837424
“Distributed tracing provides end-to-end visibility into request paths and inter-service interactions, and has been widely used to diagnose anomalies in microservice systems [15, 20, 39].”
Self-citationRamina, P. K., & Enduri, M. K. (2026) · Informatics · doi:10.3390/informatics13100167
Ardawan, D., Kurniadi, D., Budayawan, K., & Sandra, R. P. (2026) · Progresif Jurnal Ilmiah Komputer · doi:10.35889/progresif.v22i3.3977
Frameworks and Algorithms for Virtual Network Function Decomposition and Upgrade
Azhdari, A. (2026) · Spectrum Research Repository (Concordia University)
Yiğit, İ., & Buzluca, F. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7607638
Abbas, S. F., Ahmed, M., Almashfi, N., Bakhshi, A., Aamir, S., Humayun, M., & Almufareh, M. F. (2026) · Research Square · doi:10.21203/rs.3.rs-10516053/v1
(2026) · Ìnformatika, kulʹtura, tehnìka. · doi:10.15276/ict.03.2026.68
Federated Observability Architectures for Zero-Downtime Data Engineering Pipelines
Adusumilli, S. P., Shrivastava, N., Marri, M., John, S., Reddi, V. P., & Vennamaneni, P. R. (2026) · Journal of Data Analysis and Critical Management · doi:10.64235/w5sdfs91
Hussain, M. S., & Farrukh, A. (2026) · Research Square · doi:10.21203/rs.3.rs-11401166/v1
Large Language Model-Assisted Root Cause Analysis for Complex Distributed Microservice Architectures
Chinnam, N. B. (2025) · International Journal of Future Innovative Science and Technology · doi:10.15662/ijfist.2025.0801003
Osman, A., Baydan, B., & Yurttakal, A. H. (2026) · Concurrency and Computation Practice and Experience · doi:10.1002/cpe.71009
Semantic Observability for Autonomous Systems DiagnosticsPreprint
Solanki, J., Todkar, A., & Min, Z. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.28611
“2) Train-Ticket Distributed Microservices Suite [3]: An open-source benchmark application composed of 41 containerized microservices communicating via REST and gRPC.”
Bi, T., Zhang, X., Li, J., Wang, P., & Ma, M. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.35767
Arquitetura de Aplicações para a Cloud
Perdigão, J. M. T. F. C. (2021) · Portuguese National Funding Agency for Science, Research and Technology (RCAAP Project by FCT)
Slant: Shaping Traffic for Resource-Efficient MicroservicesPreprint
Zhang, C., Liang, C.-J. M., Xiao, M., Zou, Y., Wu, S., Gao, X., Cheng, X., & Yu, D. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.38778
“Train Ticket , developed by Fudan University [48], simulates a comprehensive ticket booking system with 68 mi-croservices.”
Jinna, K. R., & Thilagaraj, S. V. (2024) · International Journal of Humanities and Information Technology · doi:10.21590/ijhit.06.03.13
Fault-Free Profiles and Decision-Level Training for Root Cause Analysis via Small Language ModelsPreprint
Lee, K., Park, W., Kim, T., Baik, S., & Lee, K. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2611.02416
Toniutti, J., Baldo, M., Marino Miculan, A., & Riccio, V. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2611.04769
SWEOps-Bench: Evaluating Agents for End-to-End Resolution of Software-Induced Cloud IncidentsPreprint
Chen, D., Yu, G., & Chen, P. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2611.06099
Prabhat, S. (2024) · The Eastasouth Journal of Information System and Computer Science · doi:10.58812/esiscs.v2i02.1310
Prabhat, S. (2023) · The Eastasouth Journal of Information System and Computer Science · doi:10.58812/esiscs.v1i02.1309
Balasubramanian, S. A. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2611.11040
Prabhat, S. (2025) · Sciences du Nord Nature Science and Technology · doi:10.71238/snnst.v2i02.248
Tyolani, M., Kabaso, B., & Hendricks, W. (2026) · doi:10.1109/icssa70263.2026.11765311
AI-AUGMENTED MICROSERVICES: A FRAMEWORK FOR PREDICTIVE DATA FLOW OPTIMIZATION
Chelimela, J. (2026) · Acta Innovations · doi:10.62441/actainnovations.vi.3125
Defigium: A Modular Framework for Trace-Based Workload Generation
Kettl, F. B., Fraga, B., Vaz, M. A. B., & Mendizabal, O. M. (2026) · doi:10.5753/sscad.2026.34367
GCTformer: Graph Convolutional Transformer for Log-Trace System Anomaly Detection
Zhang, Y., Jian, S., Ye, Z., & Lü, K. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-7195-5_1
CoreCapture: Anomaly-Aware System Trace Sampling for Resource-Constrained Mobile Devices
Sun, Y., Wang, X., Sui, Y., Feng, J., Zhang, Y., Zhang, S., Zheng, B., Li, Q., Liang, W., & Pei, D. (2026) · doi:10.1145/3828523.3857128
InterfaceRCA: fault root cause analysis of microservice system based on interface multimodal data
Wang, Y., Hu, Z., Huang, H., Lin, W., & Li, K. (2026) · Cluster Computing · doi:10.1007/s10586-026-06819-x
Metadata
- Authors
- Zhou, X., Peng, X., Xie, T., Sun, J., Ji, C., Li, W., & Ding, D.
- Affiliations
- Fudan University (CN)
- University of Illinois Urbana-Champaign (US)
- Singapore University of Technology and Design (SG)
- Venue
- IEEE Transactions on Software Engineering
- Published
- 2021-02-01
- Volume
- 47
- Issue
- 2
- Pages
- 243-260
- ISSN
- 0098-5589 (ISSN-L)
- 1939-3520
- 2326-3881
- OpenAlex
- W2903799441
- Cited by
- 410
- References
- 42
- Citations by year
- 2026: 70 · 2025: 64 · 2024: 82 · 2023: 68 · 2022: 58 · 2021: 40
- FWCI
- 10.53
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- Locations
- IEEE Transactions on Software Engineering (publishedVersion)
- Singapore Management University Institutional Knowledge (InK) (Singapore Management University) (submittedVersion)
- Funders
- National Key Research and Development Program of China (2018YFB1004803)
- National Science Foundation (CNS-1513939, CNS-1564274, CCF-1816615)
- Topics
- Physical Sciences › Computer Science › Computer Networks and Communications › Software System Performance and Reliability
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Keywords
- microservices, fault analysis, debugging, distributed systems, tracing, trace visualization, fault location, benchmark systems, industrial survey, developer practices, trace analysis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.