Fault Analysis and Debugging of Microservice Systems: Industrial Survey, Benchmark System, and Empirical Study
The DOI matches its Crossref record; 40 of the 42 references it cites were found in a registry or an index, and 1 are not checked yet.
Abstract
The complexity and dynamism of microservice systems pose unique challenges to a variety of software engineering tasks such as fault analysis and debugging. In spite of the prevalence and importance of microservices in industry, there is limited research on the fault analysis and debugging of microservice systems. To fill this gap, we conduct an industrial survey to learn typical faults of microservice systems, current practice of debugging, and the challenges faced by developers in practice. We then develop a medium-size benchmark microservice system (being the largest and most complex open source microservice system within our knowledge) and replicate 22 industrial fault cases on it. Based on the benchmark system and the replicated fault cases, we conduct an empirical study to investigate the effectiveness of existing industrial debugging practices and whether they can be further improved by introducing the state-of-the-art tracing and visualization techniques for distributed systems. The results show that the current industrial practices of microservice debugging can be improved by employing proper tracing and visualization techniques and strategies. Our findings also suggest that there is a strong need for more intelligent trace analysis and visualization, e.g., by combining trace visualization and improved fault localization, and employing data-driven and learning-based recommendation for guided visual exploration and comparison of traces.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Based on various testimonials [46], the debugging process is divided into seven phases: Initial Understanding (IU), Environment Setup (ES), Failure Reproduction (FR), Failure Identification (FI), Fault Scoping (FS), Fault Localization (FL), and Fault Fixing (FF).”
From Anomaly detection and root-cause identification in microservices: a survey (2026) · cites this paper as [46]
In the citing paper
“Research has focused on tasks such as anomaly detection [90, 237], incident classification [36, 68], and root cause analysis [228, 247].”
From A 2030 Roadmap for Software Engineering (2025) · cites this paper as [247]
Citations
Cites
Microservices: Yesterday, Today, and Tomorrow
Dragoni, N., Giallorenzo, S., Lafuente, A. L., Mazzara, M., Montesi, F., Mustafin, R., & Safina, L. (2017) · Springer eBooks · doi:10.1007/978-3-319-67425-4_12 · Cited by 2,058
A Survey on Software Fault Localization
Wong, W. E., Gao, R., Li, Y., Abreu, R., & Wotawa, F. (2016) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2016.2521368 · Cited by 1,110 · cited 1×
“In addition, existing fault localization techniques [36] are ineffective for microservices.”
Simplifying and isolating failure-inducing input
Zeller, A., & Hildebrandt, R. (2002) · IEEE Transactions on Software Engineering · doi:10.1109/32.988498 · Cited by 1,079 · cited 2×
“, spectrum-based fault localization [38] and delta debugging [56]) for microservice debugging.”
Scalable statistical bug isolation
Liblit, B., Naik, M., Zheng, A. X., Aiken, A., & Jordan, M. I. (2005) · ACM SIGPLAN Notices · doi:10.1145/1064978.1065014 · Cited by 729
Microservices: The Journey So Far and Challenges Ahead
Jamshidi, P., Pahl, C., Mendonca, N. C., Lewis, J., & Tilkov, S. (2018) · IEEE Software · doi:10.1109/ms.2018.2141039 · Cited by 554 · cited 1×
“There have been appeals that practitioners and researchers develop and share a common microservice infrastructure that can emulate the production environments of typical microservice applications for more repeatable and industry-focused empirical studies [23], [24].”
Using Mutation Analysis for Assessing and Comparing Testing Coverage Criteria
Andrews, J. H., Briand, L. C., Labiche, Y., & Namin, A. S. (2006) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2006.83 · Cited by 481 · cited 1×
“Similarly related fault localization techniques are designed mainly for sequential programs, such as statistic-based fault localization [39], and machine-learning-based ones [40], [41].”
Scalable statistical bug isolation
Liblit, B., Naik, M., Zheng, A. X., Aiken, A., & Jordan, M. I. (2005) · Proceedings of the 2005 ACM SIGPLAN conference on Programming language design and implementation · doi:10.1145/1065010.1065014 · Cited by 472 · cited 1×
“Similarly related fault localization techniques are designed mainly for sequential programs, such as statistic-based fault localization [39], and machine-learning-based ones [40], [41].”
Spectrum-Based Multiple Fault Localization
Abreu, R., Zoeteweij, P., & van Gemund, A. J. C. (2009) · 2009 IEEE/ACM International Conference on Automated Software Engineering · doi:10.1109/ase.2009.25 · Cited by 401 · cited 4×
“There has been some research on debugging concurrent programs [38], [59], [60] and distributed systems [10], [61], [62], [63].”
InfluentialA Systematic Mapping Study in Microservice Architecture
Alshuqayran, N., Ali, N., & Evans, R. (2016) · 2016 IEEE 9th International Conference on Service-Oriented Computing and Applications (SOCA) · doi:10.1109/soca.2016.15 · Cited by 399 · cited 1×
“[57] present a systematic mapping study on microservice architecture, focusing on the architec-”
Research on Architecting Microservices: Trends, Focus, and Potential for Industrial Adoption
Francesco, P. D., Malavolta, I., & Lago, P. (2017) · 2017 IEEE International Conference on Software Architecture (ICSA) · doi:10.1109/icsa.2017.24 · Cited by 311 · cited 3×
“[2] present a systematic mapping study on the current state of the art on architect-”
GZoltar: an eclipse plug-in for testing and debugging
Campos, J., Riboira, A., Perez, A., & Abreu, R. (2012) · Proceedings of the 27th IEEE/ACM International Conference on Automated Software Engineering · doi:10.1145/2351676.2351752 · Cited by 208 · cited 1×
“as spectrum based fault localization [38], [55] and delta debugging [56] can be combined with visualization analysis for microservice debugging.”
BP NEURAL NETWORK-BASED EFFECTIVE FAULT LOCALIZATION
WONG, W. E., & QI, Y. (2009) · International Journal of Software Engineering and Knowledge Engineering · doi:10.1142/s021819400900426x · Cited by 196 · cited 1×
“Similarly related fault localization techniques are designed mainly for sequential programs, such as statistic-based fault localization [39], and machine-learning-based ones [40], [41].”
Benchmarking microservice systems for software engineering research
Zhou, X., Peng, X., Xie, T., Sun, J., Xu, C., Ji, C., & Zhao, W. (2018) · Proceedings of the 40th International Conference on Software Engineering: Companion Proceeedings · doi:10.1145/3183440.3194991 · Cited by 168
Self-citationFalcon: fault localization in concurrent programs
Park, S., Vuduc, R. W., & Harrold, M. J. (2010) · Proceedings of the 32nd ACM/IEEE International Conference on Software Engineering - Volume 1 · doi:10.1145/1806799.1806838 · Cited by 158 · cited 1×
“There has been some research on debugging concurrent programs [38], [59], [60] and distributed systems [10], [61], [62], [63].”
BackgroundGremlin: Systematic Resilience Testing of Microservices
Heorhiadi, V., Rajagopalan, S., Jamjoom, H., Reiter, M. K., & Sekar, V. (2016) · 2016 IEEE 36th International Conference on Distributed Computing Systems (ICDCS) · doi:10.1109/icdcs.2016.11 · Cited by 134 · cited 2×
“The existing research focuses on a wide range of topics about microservices, including design [11], testing [3], [12], [13], [14], deployment [15], [16], [17], verification [18], composition [19], architecture recovery [20], legacy migration [21], and runtime adaptation [22].”
Performance Engineering for Microservices: Research Challenges and Directions
Heinrich, R., van Hoorn, A., Knoche, H., Li, F., Lwakatare, L. E., Pahl, C., Schulte, S., & Wettinger, J. (2017) · Proceedings of the 8th ACM/SPEC on International Conference on Performance Engineering Companion · doi:10.1145/3053600.3053653 · Cited by 108 · cited 1×
“The existing research focuses on a wide range of topics about microservices, including design [11], testing [3], [12], [13], [14], deployment [15], [16], [17], verification [18], composition [19], architecture recovery [20], legacy migration [21], and runtime adaptation [22].”
Microservices and Their Design Trade-Offs: A Self-Adaptive Roadmap
Hassan, S., & Bahsoon, R. (2016) · 2016 IEEE International Conference on Services Computing (SCC) · doi:10.1109/scc.2016.113 · Cited by 98 · cited 1×
“The existing research focuses on a wide range of topics about microservices, including design [11], testing [3], [12], [13], [14], deployment [15], [16], [17], verification [18], composition [19], architecture recovery [20], legacy migration [21], and runtime adaptation [22].”
BackgroundOverload Control for Scaling WeChat Microservices
Zhou, H., Chen, M., Lin, Q., Wang, Y., She, X., Liu, S., Gu, R., Ooi, B. C., & Yang, J. (2018) · Proceedings of the ACM Symposium on Cloud Computing · doi:10.1145/3267809.3267823 · Cited by 92
MicroART: A Software Architecture Recovery Tool for Maintaining Microservice-Based Systems
Granchelli, G., Cardarelli, M., Francesco, P. D., Malavolta, I., Iovino, L., & Salle, A. D. (2017) · IEEE International Conference on Software Architecture Companion (ICSA-C) · doi:10.1109/icsaw.2017.9 · Cited by 90 · cited 1×
“The existing research focuses on a wide range of topics about microservices, including design [11], testing [3], [12], [13], [14], deployment [15], [16], [17], verification [18], composition [19], architecture recovery [20], legacy migration [21], and runtime adaptation [22].”
Microservices for Scalability: Keynote Talk Abstract
Hasselbring, W. (2016) · ACM/SPEC International Conference on Performance Engineering (ICPE) · doi:10.1145/2851553.2858659 · Cited by 79 · cited 1×
“The existing research focuses on a wide range of topics about microservices, including design [11], testing [3], [12], [13], [14], deployment [15], [16], [17], verification [18], composition [19], architecture recovery [20], legacy migration [21], and runtime adaptation [22].”
Beschastnikh, I., Wang, P., Brun, Y., & Ernst, M. D. (2016) · Communications of the ACM · doi:10.1145/2909480 · Cited by 75 · cited 6×
“There has been some research on debugging concurrent programs [38], [59], [60] and distributed systems [10], [61], [62], [63].”
InfluentialModelling and managing deployment costs of microservice-based cloud applications
Leitner, P., Cito, J., & Stoeckli, E. (2016) · IEEE/ACM International Conference on Utility and Cloud Computing (UCC) · doi:10.1145/2996890.2996901 · Cited by 68
Failure Diagnosis for Distributed Systems using Targeted Fault Injection
Pham, C., Wang, L., Tak, B., Baset, S., Tang, C., Kalbarczyk, Z., & Iyer, R. (2016) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2016.2575829 · Cited by 65 · cited 1×
“In recent years, fault localization has been extended to concurrent programs [42], [43], [44] and distributed systems [45], [46], [47].”
Schermann, G., Schöni, D., Leitner, P., & Gall, H. C. (2016) · International Middleware Conference · doi:10.1145/2988336.2988348 · Cited by 65 · cited 1×
“The existing research focuses on a wide range of topics about microservices, including design [11], testing [3], [12], [13], [14], deployment [15], [16], [17], verification [18], composition [19], architecture recovery [20], legacy migration [21], and runtime adaptation [22].”
Workload-Based Clustering of Coherent Feature Sets in Microservice Architectures
Klock, S., van der Werf, J. M. E. M., Guelen, J. P., & Jansen, S. (2017) · IEEE International Conference on Software Architecture (ICSA) · doi:10.1109/icsa.2017.38 · Cited by 65 · cited 1×
“The existing research focuses on a wide range of topics about microservices, including design [11], testing [3], [12], [13], [14], deployment [15], [16], [17], verification [18], composition [19], architecture recovery [20], legacy migration [21], and runtime adaptation [22].”
An architecture to automate performance tests on microservices
de Camargo, A. S., Salvadori, I. L., Mello, R. D. S., & Siqueira, F. (2016) · International Conference on Information Integration and Web-based Applications & Services (iiWAS) · doi:10.1145/3011141.3011179 · Cited by 64 · cited 1×
“The existing research focuses on a wide range of topics about microservices, including design [11], testing [3], [12], [13], [14], deployment [15], [16], [17], verification [18], composition [19], architecture recovery [20], legacy migration [21], and runtime adaptation [22].”
Fault detection and localization in distributed systems using invariant relationships
Sharma, A. B., Chen, H., Ding, M., Yoshihira, K., & Jiang, G. (2013) · International Conference on Dependable Systems & Networks/Proceedings · doi:10.1109/dsn.2013.6575304 · Cited by 51 · cited 1×
“In recent years, fault localization has been extended to concurrent programs [42], [43], [44] and distributed systems [45], [46], [47].”
Wait Analysis of Distributed Systems Using Kernel Tracing
Giraldeau, F., & Dagenais, M. (2015) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2015.2488629 · Cited by 47 · cited 2×
“There has been some research on debugging concurrent programs [38], [59], [60] and distributed systems [10], [61], [62], [63].”
BackgroundMethodVerification in the Age of Microservices
Panda, A., Sagiv, M., & Shenker, S. J. (2017) · Workshop on Hot Topics in Operating Systems · doi:10.1145/3102980.3102986 · Cited by 42 · cited 1×
“The existing research focuses on a wide range of topics about microservices, including design [11], testing [3], [12], [13], [14], deployment [15], [16], [17], verification [18], composition [19], architecture recovery [20], legacy migration [21], and runtime adaptation [22].”
10 Years of research on debugging concurrent and multicore software: a systematic mapping study
Asadollah, S. A., Sundmark, D., Eldh, S., Hansson, H., & Afzal, W. (2016) · Software Quality Journal · doi:10.1007/s11219-015-9301-7 · Cited by 37 · cited 1×
“[64] present a systematic mapping study on debugging concurrent and multicore software in the decade between 2005 and 2014.”
Probabilistic diagnosis of performance faults in large-scale parallel applications
Laguna, I., Ahn, D. H., de Supinski, B. R., Bagchi, S., & Gamblin, T. (2012) · International Conference on Parallel Architectures and Compilation Techniques (PACT) · doi:10.1145/2370816.2370848 · Cited by 23 · cited 1×
“In recent years, fault localization has been extended to concurrent programs [42], [43], [44] and distributed systems [45], [46], [47].”
UNICORN : a unified approach for localizing non‐deadlock concurrency bugs
Park, S., Vuduc, R. W., & Harrold, M. J. (2014) · Software Testing Verification and Reliability · doi:10.1002/stvr.1523 · Cited by 22 · cited 1×
“There has been some research on debugging concurrent programs [38], [59], [60] and distributed systems [10], [61], [62], [63].”
Migrating web applications to clouds with microservice architectures
Lin, J., Lin, L. C., & Huang, S. (2016) · International Conference on Applied System Innovation · doi:10.1109/icasi.2016.7539733 · Cited by 21 · cited 1×
“The existing research focuses on a wide range of topics about microservices, including design [11], testing [3], [12], [13], [14], deployment [15], [16], [17], verification [18], composition [19], architecture recovery [20], legacy migration [21], and runtime adaptation [22].”
Publishing linked data through semantic microservices composition
Salvadori, I. L., Huf, A., Mello, R. D. S., & Siqueira, F. (2016) · International Conference on Information Integration and Web-based Applications & Services (iiWAS) · doi:10.1145/3011141.3011155 · Cited by 18
Bailis, P. D., Alvaro, P., & Gulwani, S. (2017) · Communications of the ACM · doi:10.1145/3052942 · Cited by 16
Quantitative program slicing: Separating statements by relevance
Santelices, R., Zhang, Y., Jiang, S., Cai, H., & Zhang, Y. (2013) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2013.6606695 · Cited by 16 · cited 1×
“Program-slicing-based fault localization [37] works by identifying program statements that are irrelevant to the faulty statement and allowing the developers to investigate the fault based on the remaining statements.”
Spectrum-Based Fault Localization for Diagnosing Concurrency Faults
Koca, F., Sözer, H., & Abreu, R. (2013) · Lecture notes in computer science · doi:10.1007/978-3-642-41707-8_16 · Cited by 15 · cited 1×
“In recent years, fault localization has been extended to concurrent programs [42], [43], [44] and distributed systems [45], [46], [47].”
Qi, G., Yao, L., & Uzunov, A. V. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-69179-4_3 · Cited by 15 · cited 1×
“In recent years, fault localization has been extended to concurrent programs [42], [43], [44] and distributed systems [45], [46], [47].”
Quantitative program slicing: separating statements by relevance
Santelices, R., Zhang, Y., Jiang, S., Cai, H., & Zhang, Y. (2013) · doi:10.5555/2486788.2486981 · Cited by 12
Fault Localization in Multi-threaded C Programs Using Bounded Model Checking
Alves, E. H. D. S., Cordeiro, L. C., & Filho, E. B. D. L. (2015) · Brazilian Symposium on Computing System Engineering (SBESC) · doi:10.1109/sbesc.2015.25 · Cited by 6 · cited 1×
“In recent years, fault localization has been extended to concurrent programs [42], [43], [44] and distributed systems [45], [46], [47].”
FULLY-DISTRIBUTED DEBUGGING AND VISUALIZATION OF DISTRIBUTED SYSTEMS IN ANONYMOUS NETWORKS
n.n. (2012) · Cited by 4
Research works printed but not linked: 2
- 10.1109/ecase.2017..4 · doi:10.1109/ecase.2017..4
- 10.1109/scc.2016.113 · DOI · Paper page
Cited by
FIRM: An Intelligent Fine-Grained Resource Management Framework for SLO-Oriented MicroservicesPreprint
Qiu, H., Banerjee, S. S., Jha, S., Kalbarczyk, Z., & Iyer, R. K. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.08509 · Cited by 446
“Train-Ticket Booking Service provides typical train-ticket booking functionalities, such as ticket inquiry, reservation, payment, change, and user notification.”
MethodInfluentialA Survey on Automated Log Analysis for Reliability Engineering
He, S., He, P., Chen, Z., Yang, T., Su, Y., & Lyu, M. R. (2022) · ACM Computing Surveys · doi:10.1145/3460345 · Cited by 334
“gnificantly facilitated by logs. In addition, they appealed to engineers to employ automated method to speedup the problem resolution time, which is also the focus of this survey. Likewise, Zhou etal.[163] empirically investigated the faults and debugging practices in microservice systems. The results show that proper…”
MethodZhou, X., Peng, X., Xie, T., Sun, J., Ji, C., Liu, D., Xiang, Q., & He, C. (2019) · Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3338906.3338961 · Cited by 277
“According to an existing empirical study [61], almost half of all studied microservice application faults are of these three types.”
BackgroundMethodInfluentialSelf-citationMicroRCA: Root Cause Localization of Performance Issues in Microservices
Wu, L., Tordsson, J., Elmroth, E., & Kao, O. (2020) · NOMS 2020 - 2020 IEEE/IFIP Network Operations and Management Symposium · doi:10.1109/noms47738.2020.9110353 · Cited by 265
“Similarly, trace-based approaches [8], [9], [18]–[23] gather information through complete tracing of the execution paths, then identify root causes through analyzing the deviation of latencies along the paths, e.g., based on machine learning.”
BackgroundSelf-citationSoldani, J., & Brogi, A. (2023) · ACM Computing Surveys · doi:10.1145/3501297 · Cited by 256
“[96] DT visual trace comparison E F, P A events previous and runtime traces GMTA [28] DT visual trace comparison E F, P A services previous and runtime traces CloudDiag [56] DT RPCA on traces E P A ranked services app deployment TraceAnomaly [45] DT direct KPI correlation I F, P A services previous and runtime traces…”
BackgroundMethodInfluential
Metadata
- Authors
- Zhou, X., Peng, X., Xie, T., Sun, J., Ji, C., Li, W., & Ding, D.
- Affiliations
- Fudan University (CN)
- University of Illinois Urbana-Champaign (US)
- Singapore University of Technology and Design (SG)
- Venue
- IEEE Transactions on Software Engineering
- Published
- 2021-02-01
- Volume
- 47
- Issue
- 2
- Pages
- 243-260
- ISSN
- 0098-5589 (ISSN-L)
- 1939-3520
- 2326-3881
- OpenAlex
- W2903799441
- Cited by
- 410
- References
- 42
- Citations by year
- 2026: 70 · 2025: 64 · 2024: 82 · 2023: 68 · 2022: 58 · 2021: 40
- FWCI
- 10.53
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- Locations
- IEEE Transactions on Software Engineering (publishedVersion)
- Singapore Management University Institutional Knowledge (InK) (Singapore Management University) (submittedVersion)
- Funders
- National Key Research and Development Program of China (2018YFB1004803)
- National Science Foundation (CNS-1513939, CNS-1564274, CCF-1816615)
- Topics
- Physical Sciences › Computer Science › Computer Networks and Communications › Software System Performance and Reliability
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Keywords
- microservices, fault analysis, debugging, distributed systems, tracing, trace visualization, fault location, benchmark systems, industrial survey, developer practices, trace analysis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.