Fault Analysis and Debugging of Microservice Systems: Industrial Survey, Benchmark System, and Empirical Study
The DOI matches its Crossref record; 40 of the 42 references it cites were found in a registry or an index, and 1 are not checked yet.
Abstract
The complexity and dynamism of microservice systems pose unique challenges to a variety of software engineering tasks such as fault analysis and debugging. In spite of the prevalence and importance of microservices in industry, there is limited research on the fault analysis and debugging of microservice systems. To fill this gap, we conduct an industrial survey to learn typical faults of microservice systems, current practice of debugging, and the challenges faced by developers in practice. We then develop a medium-size benchmark microservice system (being the largest and most complex open source microservice system within our knowledge) and replicate 22 industrial fault cases on it. Based on the benchmark system and the replicated fault cases, we conduct an empirical study to investigate the effectiveness of existing industrial debugging practices and whether they can be further improved by introducing the state-of-the-art tracing and visualization techniques for distributed systems. The results show that the current industrial practices of microservice debugging can be improved by employing proper tracing and visualization techniques and strategies. Our findings also suggest that there is a strong need for more intelligent trace analysis and visualization, e.g., by combining trace visualization and improved fault localization, and employing data-driven and learning-based recommendation for guided visual exploration and comparison of traces.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Based on various testimonials [46], the debugging process is divided into seven phases: Initial Understanding (IU), Environment Setup (ES), Failure Reproduction (FR), Failure Identification (FI), Fault Scoping (FS), Fault Localization (FL), and Fault Fixing (FF).”
From Anomaly detection and root-cause identification in microservices: a survey (2026) · cites this paper as [46]
In the citing paper
“Research has focused on tasks such as anomaly detection [90, 237], incident classification [36, 68], and root cause analysis [228, 247].”
From A 2030 Roadmap for Software Engineering (2025) · cites this paper as [247]
Citations
Cites
Microservices: Yesterday, Today, and Tomorrow
Dragoni, N., Giallorenzo, S., Lafuente, A. L., Mazzara, M., Montesi, F., Mustafin, R., & Safina, L. (2017) · Springer eBooks · doi:10.1007/978-3-319-67425-4_12 · Cited by 2,058
A Survey on Software Fault Localization
Wong, W. E., Gao, R., Li, Y., Abreu, R., & Wotawa, F. (2016) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2016.2521368 · Cited by 1,110 · cited 1×
“In addition, existing fault localization techniques [36] are ineffective for microservices.”
Simplifying and isolating failure-inducing input
Zeller, A., & Hildebrandt, R. (2002) · IEEE Transactions on Software Engineering · doi:10.1109/32.988498 · Cited by 1,079 · cited 2×
“, spectrum-based fault localization [38] and delta debugging [56]) for microservice debugging.”
Scalable statistical bug isolation
Liblit, B., Naik, M., Zheng, A. X., Aiken, A., & Jordan, M. I. (2005) · ACM SIGPLAN Notices · doi:10.1145/1064978.1065014 · Cited by 729
Microservices: The Journey So Far and Challenges Ahead
Jamshidi, P., Pahl, C., Mendonca, N. C., Lewis, J., & Tilkov, S. (2018) · IEEE Software · doi:10.1109/ms.2018.2141039 · Cited by 554 · cited 1×
“There have been appeals that practitioners and researchers develop and share a common microservice infrastructure that can emulate the production environments of typical microservice applications for more repeatable and industry-focused empirical studies [23], [24].”
Using Mutation Analysis for Assessing and Comparing Testing Coverage Criteria
Andrews, J. H., Briand, L. C., Labiche, Y., & Namin, A. S. (2006) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2006.83 · Cited by 481 · cited 1×
“Similarly related fault localization techniques are designed mainly for sequential programs, such as statistic-based fault localization [39], and machine-learning-based ones [40], [41].”
Scalable statistical bug isolation
Liblit, B., Naik, M., Zheng, A. X., Aiken, A., & Jordan, M. I. (2005) · Proceedings of the 2005 ACM SIGPLAN conference on Programming language design and implementation · doi:10.1145/1065010.1065014 · Cited by 472 · cited 1×
“Similarly related fault localization techniques are designed mainly for sequential programs, such as statistic-based fault localization [39], and machine-learning-based ones [40], [41].”
Spectrum-Based Multiple Fault Localization
Abreu, R., Zoeteweij, P., & van Gemund, A. J. C. (2009) · 2009 IEEE/ACM International Conference on Automated Software Engineering · doi:10.1109/ase.2009.25 · Cited by 401 · cited 4×
“There has been some research on debugging concurrent programs [38], [59], [60] and distributed systems [10], [61], [62], [63].”
InfluentialA Systematic Mapping Study in Microservice Architecture
Alshuqayran, N., Ali, N., & Evans, R. (2016) · 2016 IEEE 9th International Conference on Service-Oriented Computing and Applications (SOCA) · doi:10.1109/soca.2016.15 · Cited by 399 · cited 1×
“[57] present a systematic mapping study on microservice architecture, focusing on the architec-”
Research on Architecting Microservices: Trends, Focus, and Potential for Industrial Adoption
Francesco, P. D., Malavolta, I., & Lago, P. (2017) · 2017 IEEE International Conference on Software Architecture (ICSA) · doi:10.1109/icsa.2017.24 · Cited by 311 · cited 3×
“[2] present a systematic mapping study on the current state of the art on architect-”
Research works printed but not linked: 2
- 10.1109/ecase.2017..4 · doi:10.1109/ecase.2017..4
- 10.1109/scc.2016.113 · DOI · Paper page
Cited by
FIRM: An Intelligent Fine-Grained Resource Management Framework for SLO-Oriented MicroservicesPreprint
Qiu, H., Banerjee, S. S., Jha, S., Kalbarczyk, Z., & Iyer, R. K. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.08509 · Cited by 446
“Train-Ticket Booking Service provides typical train-ticket booking functionalities, such as ticket inquiry, reservation, payment, change, and user notification.”
MethodInfluentialA Survey on Automated Log Analysis for Reliability Engineering
He, S., He, P., Chen, Z., Yang, T., Su, Y., & Lyu, M. R. (2022) · ACM Computing Surveys · doi:10.1145/3460345 · Cited by 334
“gnificantly facilitated by logs. In addition, they appealed to engineers to employ automated method to speedup the problem resolution time, which is also the focus of this survey. Likewise, Zhou etal.[163] empirically investigated the faults and debugging practices in microservice systems. The results show that proper…”
MethodZhou, X., Peng, X., Xie, T., Sun, J., Ji, C., Liu, D., Xiang, Q., & He, C. (2019) · Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3338906.3338961 · Cited by 277
“According to an existing empirical study [61], almost half of all studied microservice application faults are of these three types.”
BackgroundMethodInfluentialSelf-citationMicroRCA: Root Cause Localization of Performance Issues in Microservices
Wu, L., Tordsson, J., Elmroth, E., & Kao, O. (2020) · NOMS 2020 - 2020 IEEE/IFIP Network Operations and Management Symposium · doi:10.1109/noms47738.2020.9110353 · Cited by 265
“Similarly, trace-based approaches [8], [9], [18]–[23] gather information through complete tracing of the execution paths, then identify root causes through analyzing the deviation of latencies along the paths, e.g., based on machine learning.”
BackgroundSelf-citationSoldani, J., & Brogi, A. (2023) · ACM Computing Surveys · doi:10.1145/3501297 · Cited by 256
“[96] DT visual trace comparison E F, P A events previous and runtime traces GMTA [28] DT visual trace comparison E F, P A services previous and runtime traces CloudDiag [56] DT RPCA on traces E P A ranked services app deployment TraceAnomaly [45] DT direct KPI correlation I F, P A services previous and runtime traces…”
BackgroundMethodInfluentialMAGIS: LLM-Based Multi-Agent Framework for GitHub Issue Resolution
Tao, W., Zhou, Y., Wang, Y., Zhang, W., Zhang, H., & Cheng, Y. (2024) · neural information processing systems · doi:10.52202/079017-1647 · Cited by 258
Unsupervised Detection of Microservice Trace Anomalies through Service-Level Deep Bayesian Networks
Liu, P., Xu, H., Ouyang, Q., Jiao, R., Chen, Z., Zhang, S., Yang, J., Mo, L., Zeng, J., Xue, W., et al. (2020) · 2020 IEEE 31st International Symposium on Software Reliability Engineering (ISSRE) · doi:10.1109/issre5003.2020.00014 · Cited by 252
“1) Testbed Evaluation: We choose a benchmarking microservice system, TrainTicket [32], as our testbed.”
BackgroundMethodInfluentialDeepTraLog: trace-log combined microservice anomaly detection through graph-based deep learning
Zhang, C., Peng, X., Sha, C., Zhang, K., Fu, Z., Wu, X., Lin, Q., & Zhang, D. (2022) · Proceedings of the 44th International Conference on Software Engineering · doi:10.1145/3510003.3510180 · Cited by 207
“Traditional approaches [10, 42] rely on the visualization of traces to support manual trace analysis.”
BackgroundMethodInfluentialSelf-citationMicroHECL: High-Efficient Root Cause Localization in Large-Scale Microservice Systems
Liu, D., He, C., Peng, X., Lin, F., Zhang, C., Gong, S., Li, Z., Ou, J., & Wu, Z. (2021) · 2021 IEEE/ACM 43rd International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1109/icse-seip52600.2021.00043 · Cited by 192
“A service can have several to thousands of instances running on different containers and dynamically created or destroyed according to the scaling requirements at runtime; the execution of a microservice system may involve a huge number of microservice interactions and most of them are asynchronous and involve complex…”
BackgroundMethodSelf-citationData Quality Matters: A Case Study on Data Label Correctness for Security Bug Report Prediction
Wu, X., Zheng, W., Xia, X., & Lo, D. (2022) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2021.3063727 · Cited by 132
“Data annotation is an extremely time-consuming work [24], [40], [41].”
BackgroundA Spatiotemporal Deep Learning Approach for Unsupervised Anomaly Detection in Cloud Systems
He, Z., Chen, P., Li, X., Wang, Y., Yu, G., Chen, C., Li, X., & Zheng, Z. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2020.3027736 · Cited by 148
“For example, Zhou et al. [29] investigate application faults in microservice systems with different root causes and the debugging practice for them.”
BackgroundAdaptive Resource Efficient Microservice Deployment in Cloud-Edge Continuum
Fu, K., Zhang, W., Chen, Q., Zeng, D., & Guo, M. (2022) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2021.3128037 · Cited by 170
“We use all the 3 open-sourced benchmarks SocialNetwork (SN), MediaService (MS), and HotelReservation (HR) in the DeathStarBench [1] microservice benchmark suite, and a microservice-based train ticket booking system TrainTicket (TT)[34] to perform the investigation.”
BackgroundMethodPractical Root Cause Localization for Microservice Systems via Trace Analysis
Li, Z., Chen, J., Jiao, R., Zhao, N., Wang, Z., Zhang, S., Wu, Y., Jiang, L., Yan, L., Wang, Z., et al. (2021) · 2021 IEEE/ACM 29th International Symposium on Quality of Service (IWQOS) · doi:10.1109/iwqos52092.2021.9521340 · Cited by 159
“For example, Train-Ticket [22] is organized as many microservices, such as UI , seat , train , station , order , and price .”
InfluentialDesign, monitoring, and testing of microservices systems: The practitioners’ perspective
Waseem, M., Liang, P., Shahin, M., Di Salle, A., & Márquez, G. (2021) · Journal of Systems and Software · doi:10.1016/j.jss.2021.111061 · Cited by 158
Microservices: architecture, container, and challenges
Liu, G., Huang, B., Liang, Z., Qin, M., Zhou, H., & Li, Z. (2020) · IEEE International Conference on Software Quality, Reliability and Security Companion (QRS-C) · doi:10.1109/qrs-c51114.2020.00107 · Cited by 152
“These asynchronous interactions involve complex call chains, which can easily lead to improper collaboration, which in turn can lead to runtime failures [36].”
BackgroundMethodInfluentialPromises and challenges of microservices: an exploratory study
Wang, Y., Kadiyala, H., & Rubin, J. (2021) · Empirical Software Engineering · doi:10.1007/s10664-020-09910-y · Cited by 153
“Zhou et al. (2018) focused specifically on fault analysis and debugging of microservices.”
BackgroundEadro: An End-to-End Troubleshooting Framework for Microservices on Multi-source Data
Lee, C., Yang, T., Chen, Z., Su, Y., & Lyu, M. R. (2023) · 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) · doi:10.1109/icse48619.2023.00150 · Cited by 149
“An anomaly in one microservice could propagate to others and magnify its impact, resulting in considerable revenue and reputation loss for companies [2].”
Yu, G., Chen, P., Li, Y., Chen, H., Li, X., & Zheng, Z. (2023) · Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3611643.3616249 · Cited by 140
“The complexity of fault scenarios is alleviated by injecting resource issues and code defects from real faults in industrial systems [33, 76] into microservices.”
BackgroundResultPlacement of Microservices-based IoT Applications in Fog Computing: A Taxonomy and Future Directions
Pallewatta, S., Kostakos, V., & Buyya, R. (2023) · ACM Computing Surveys · doi:10.1145/3592598 · Cited by 133
“[36] also uses benchmark applications from DeathStartBench along with the benchmark application provided in [115].”
MethodAutoMAP: Diagnose Your Microservice-based Web Applications Automatically
Ma, M., Xu, J., Wang, Y., Chen, P., Zhang, Z., & Wang, P. (2020) · Proceedings of The Web Conference 2020 · doi:10.1145/3366423.3380111 · Cited by 129
“Various research efforts have been devoted to similar issues such as network traffic analysis [9], web application anomaly detection [10] and debugging [11], service failure detection [28] and prediction [29].”
BackgroundIdentifying bad software changes via multimodal anomaly detection for online service systems
Zhao, N., Chen, J., Yu, Z., Wang, H., Li, J., Qiu, B., Xu, H., Zhang, W., Sui, K., & Pei, D. (2021) · Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3468264.3468543 · Cited by 89
“The first system is Train-Ticket [65], an open-source microservice system, which has been widely applied in the literature [31, 42, 68, 82].”
Enjoy your observability: an industrial survey of microservice tracing and analysis
Li, B., Peng, X., Xiang, Q., Wang, H., Xie, T., Sun, J., & Liu, X. (2022) · Empirical Software Engineering · doi:10.1007/s10664-021-10063-9 · Cited by 121
Self-citationGraph-based trace analysis for microservice architecture understanding and problem diagnosis
Guo, X., Peng, X., Wang, H., Li, W., Jiang, H., Ding, D., Xie, T., & Su, L. (2020) · Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3368089.3417066 · Cited by 118
Self-citationRobust Failure Diagnosis of Microservice System Through Multimodal Data
Zhang, S., Jin, P., Lin, Z., Sun, Y., Zhang, B., Xia, S., Li, Z., Zhong, Z., Ma, M., Jin, W., et al. (2023) · IEEE Transactions on Services Computing · doi:10.1109/tsc.2023.3290018 · Cited by 118
On Microservice Analysis and Architecture Evolution: A Systematic Mapping Study
Bushong, V., Abdelfattah, A. S., Maruf, A. A., Das, D., Lehman, A., Jaroszewski, E., Coffey, M., Cerny, T., Frajták, K., Tisnovsky, P., et al. (2021) · Applied Sciences · doi:10.3390/app11177856 · Cited by 118
“Finally, one of the greatest deficiencies in related research is the lack of benchmarks [45].”
BackgroundMethodInfluentialActionable and interpretable fault localization for recurring failures in online service systems
Li, Z., Zhao, N., Li, M., Lu, X., Wang, L., Chang, D., Nie, X., Cao, L., Zhang, W., Sui, K., et al. (2022) · Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3540250.3549092 · Cited by 70
“For D, we deploy Train-Ticket [59] on a 4-node Kubernetes cluster.”
BackgroundMethodInfluentialRobust Multimodal Failure Detection for Microservice Systems
Zhao, C., Ma, M., Zhong, Z., Zhang, S., Tan, Z., Xiong, X., Yu, L., Feng, J., Sun, Y., Zhang, Y., et al. (2023) · Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining · doi:10.1145/3580305.3599902 · Cited by 69
“As an increasing number of Internet applications migrate to the cloud, the microservice architecture, which allows each microservice to be independently developed, deployed, upgraded, and scaled, has attractedwidespread attention recently [47].”
BackgroundPham, L., Ha, H., & Zhang, H. (2024) · Proceedings of the ACM on Software Engineering · doi:10.1145/3660805 · Cited by 48
“4.8.2 Hyperparameter Sensitivity.”
InfluentialSHOWAR: Right-Sizing And Efficient Scheduling of Microservices
Baarzi, A. F., & Kesidis, G. (2021) · ACM Symposium on Cloud Computing (SoCC) · doi:10.1145/3472883.3486999 · Cited by 97
“…application consisting 36 microservices in which users can follow others, compose posts, and read and interact with others’ posts; b) Train-Ticket [64], an application consisting 41 microservices which allows its users to reserve online tickets and make payments; and c) Google Cloud Platform’s…”
BackgroundA Survey of Software Log Instrumentation
Chen, B., & Jiang, Z. M. (2021) · ACM Computing Surveys · doi:10.1145/3448976 · Cited by 94
Jiao, Y., Yang, K., Song, D., & Tao, D. (2022) · IEEE Transactions on Network Science and Engineering · doi:10.1109/tnse.2022.3148276 · Cited by 96
“Datasets and Baselines: TimeAutoAD is utilized for detecting collective anomalies in this paper, which have been studied in many scenarios as ECG time series anomaly detection [25], IoT time series anomaly detection [63], Trace time series anomaly detection [57].”
BackgroundMethodAn empirical study of log analysis at Microsoft
He, S., Zhang, X., He, P., Xu, Y., Li, L., Kang, Y., Ma, M., Wei, Y., Dang, Y., Rajmohan, S., et al. (2022) · Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3540250.3558963 · Cited by 51
“One is reconstructing a failed execution or separating different execution flows based on log data [14, 27, 48].”
MethodRoot-Cause Metric Location for Microservice Systems via Log Anomaly Detection
Wang, L., Zhao, N., Chen, J., Li, P., Zhang, W., & Sui, K. (2020) · 2020 IEEE International Conference on Web Services (ICWS) · doi:10.1109/icws49710.2020.00026 · Cited by 84
“To evaluate the effectiveness of our proposed approach, we conduct experiments based on a widely-used microservice benchmark system named TrainTicket, which contains more than 30 microservices [1].”
BackgroundMethodInfluentialQoS-Aware and Resource Efficient Microservice Deployment in Cloud-Edge Continuum
Fu, K., Zhang, W., Chen, Q., Zeng, D., Peng, X., Zheng, W., & Guo, M. (2021) · IEEE International Parallel and Distributed Processing Symposium (IPDPS) · doi:10.1109/ipdps49936.2021.00102 · Cited by 80
Self-citationHao, J., Chen, P., Chen, J., & Li, X. (2024) · Future Generation Computer Systems · doi:10.1016/j.future.2024.05.006 · Cited by 76
A Survey of AIOps for Failure Management in the Era of Large Language ModelsPreprint
Zhang, L., Jia, T., Jia, M., Wu, Y., Liu, A., Yang, Y., Wu, Z., Hu, X., Yu, P. S., & Li, Y. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2406.11213 · Cited by 77
“This type of data is the most commonly used in traditional ML-based and DL-based AIOps for failure management [69, 77, 117, 138, 140, 141, 143, 155, 157].”
MethodDetecting anomalies in microservices with execution trace comparison
Meng, L., Ji, F., Sun, Y., & Wang, T. (2021) · Future Generation Computer Systems · doi:10.1016/j.future.2020.10.040 · Cited by 70
Testing Microservices Architecture-Based Applications: A Systematic Mapping Study
Waseem, M., Liang, P., Márquez, G., & Salle, A. D. (2020) · Asia-Pacific Software Engineering Conference (APSEC) · doi:10.1109/apsec51365.2020.00020 · Cited by 69
“We found only one study (i.e., [39]) that present the fault analysis and debugging of MSA-based application by considering the medium-size benchmark MSA-based application.”
BackgroundReal-time incident prediction for online service systems
Zhao, N., Chen, J., Wang, Z., Peng, X., Wang, G., Wu, Y., Zhou, F., Feng, Z., Nie, X., Zhang, W., et al. (2020) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3368089.3409672 · Cited by 67
“, metrics [46], logs [19, 31], and traces [55, 56]) from various service components, and engineers manually define many rules to”
BackgroundSelf-citationDeep Learning or Classical Machine Learning? An Empirical Study on Log-Based Anomaly Detection
Yu, B., Yao, J., Fu, Q., Zhong, Z., Xie, H., Wu, Y., Ma, Y., & He, P. (2024) · Proceedings of the IEEE/ACM 46th International Conference on Software Engineering · doi:10.1145/3597503.3623308 · Cited by 65
“There are various topics within log mining, including failure prediction [68], failure diagnosis [71, 87, 88], anomaly detection [80, 81, 85], log-based slicing [59], log visualization [55], and root cause analysis [44].”
BackgroundPractical Efficient Microservice Autoscaling with QoS Assurance
Hossen, M. R., Islam, M. A., & Ahmed, K. (2022) · ACM International Symposium on High-Performance Parallel and Distributed Computing (HPDC) · doi:10.1145/3502181.3531460 · Cited by 62
“13(a), we show the resource allocation of PEMA for TrainTicket as our workload varies between 200 and 300 requests per second.”
MethodInfluentialWang, L., Zhang, C., Ding, R., Xu, Y., Chen, Q., Zou, W., Chen, Q., Zhang, M., Gao, X., Fan, H., et al. (2023) · Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining · doi:10.1145/3580305.3599934 · Cited by 33
“Unexpected latency at any of the component may lead to a huge number of slow email deliveries and impact customer experience, which can further result in substantial financial loss [3].”
BackgroundAn empirical study of the systemic and technical migration towards microservices
Ayas, H. M., Leitner, P., & Hebig, R. (2023) · Empirical Software Engineering · doi:10.1007/s10664-023-10308-9 · Cited by 60
“There are some formal models on how to track and split technical artifacts of the system (Waseem et al., 2021; Zhou et al., 2021).”
BackgroundDing, R., Zhang, C., Wang, L., Xu, Y., Ma, M., Wu, X., Zhang, M., Chen, Q., Gao, X., Gao, X., et al. (2023) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3611643.3613864 · Cited by 60
“To ensure the reliability of cloud services, on-call engineers rely on a range of system data sources such as key performance indicators [28], logs [23], and traces [56] to develop automated and intelligent incident detection and root cause analysis (RCA) solutions [27, 50, 56, 61, 62].”
BackgroundFailure Diagnosis in Microservice Systems: A Comprehensive Survey and Analysis
Zhang, S., Xia, S., Fan, W., Shi, B., Xiong, X., Zhong, Z., Ma, M., Sun, Y., & Pei, D. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3715005 · Cited by 57
Service-Level Fault Injection Testing
Meiklejohn, C. S., Estrada, A., Song, Y., Miller, H., & Padhye, R. (2021) · ACM Symposium on Cloud Computing (SoCC) · doi:10.1145/3472883.3487005 · Cited by 56
“Other work [50] has attempted to build a large microservice application seeded with bugs from industrial surveys, but only a small subset of the bugs are actually specific to microservice architectures.”
BackgroundEvolution of Microservices Identification in Monolith Decomposition: A Systematic Review
Oumoussa, I., & Saidi, R. (2024) · IEEE Access · doi:10.1109/access.2024.3365079 · Cited by 56
“This encompasses testing, monitoring, debugging, versioning, and evolution [85], [86].”
BackgroundTraceGra: A trace-based anomaly detection for microservice using graph deep learning
Chen, J., Liu, F., Jiang, J., Zhong, G., Xu, D., Tan, Z., & Shi, S. (2023) · Computer Communications · doi:10.1016/j.comcom.2023.03.028 · Cited by 55
Monitoring tools for DevOps and microservices: A systematic grey literature review
Giamattei, L., Guerriero, A., Pietrantuono, R., Russo, S., Malavolta, I., Islam, T., Dînga, M., Koziolek, A., Singh, S., Armbruster, M., et al. (2024) · Journal of Systems and Software · doi:10.1016/j.jss.2023.111906 · Cited by 55
Workflow-Aware Automatic Fault Diagnosis for Microservice-Based Applications With Statistics
Wang, T., Zhang, W., Xu, J., & Gu, Z. (2020) · IEEE Transactions on Network and Service Management · doi:10.1109/tnsm.2020.3022028 · Cited by 54
“[31] conduct an empirical study to investigate the effectiveness of existing industrial debugging practices and whether they can be fur-”
BackgroundKGroot: A knowledge graph-enhanced method for root cause analysis
Wang, T., Qi, G., & Wu, T. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.124679 · Cited by 56
Han, Y., Du, Q., Huang, Y., Wu, J., Tian, F., & He, C. (2024) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3691620.3695475 · Cited by 55
“However, in such systems, failures are inevitable due to the large scale and complexity, which may result in significant costs in terms of customer impact [39, 56].”
STRATUS: A Multi-agent System for Autonomous Reliability Engineering of Modern Clouds
Chen, Y., Pan, J., Clark, J., Su, Y., Zheutlin, N., Bhavya, B., Arora, R., Deng, Y., Jha, S., & Xu, T. (2025) · neural information processing systems · doi:10.52202/085713-1671 · Cited by 54
Artificial Intelligence for Software Engineering: The Journey So Far and the Road Ahead
Ahmed, I., Aleti, A., Cai, H., Chatzigeorgiou, A., He, P., Hu, X., Pezzè, M., Poshyvanyk, D., & Xia, X. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3719006 · Cited by 54
Jin, M., Lv, A., Zhu, Y., Wen, Z., Zhong, Y., Zhao, Z., Wu, J., Li, H., He, H., & Chen, F. (2020) · IEEE Access · doi:10.1109/access.2020.3044610 · Cited by 50
White-Box Fuzzing RPC-Based APIs with EvoMaster: An Industrial Case Study
Zhang, M., Arcuri, A., Li, Y., Liu, Y., & Xue, K. (2023) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3585009 · Cited by 51
“It is a common practice in industry to develop large enterprise systems with microservice architectures [59, 68, 79].”
BackgroundPanahandeh, M., Hamou-Lhadj, A., Hamdaqa, M., & Miller, J. (2024) · Journal of Systems and Software · doi:10.1016/j.jss.2023.111917 · Cited by 49
“We evaluate the effectiveness of our approach using two different benchmark systems, Teastore [37] and TrainTicket [38].”
BackgroundMethodA Survey of AIOps in the Era of Large Language Models
Zhang, L., Jia, T., Jia, M., Wu, Y., Liu, A., Yang, Y., Wu, Z., Hu, X., Yu, P., & Li, Y. (2026) · ACM Computing Surveys · doi:10.1145/3746635 · Cited by 49
“To begin, we review the data sources traditionally employed in ML-based and DL-based AIOps [68, 73, 122, 147, 149, 150, 152, 154, 157, 176, 178].”
Multivariate Log-based Anomaly Detection for Distributed Database
Zhang, L., Jia, T., Jia, M., Li, Y., Yang, Y., & Wu, Z. (2024) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3637528.3671725 · Cited by 47
“…an analysis of the definitions and classifications of anomalies in system and database work and discover that database anomalies can be classified into two categories[3, 6, 20, 31, 34, 35, 39, 51, 56, 62, 63]: (1) anomalies caused by the system and (2) anomalies generated by the database itself.”
Gu, S., Rong, G., Ren, T., Zhang, H., Shen, H., Yu, Y., Li, X., Ouyang, J., & Chen, C. (2023) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2023.3241299 · Cited by 46
“Second, according to a recent industrial survey [7], most anomalies in a microservice system are caused by functional faults.”
BackgroundServiceRank: Root Cause Identification of Anomaly in Large-Scale Microservice Architectures
Ma, M., Lin, W., Pan, D., & Wang, P. (2022) · IEEE Transactions on Dependable and Secure Computing · doi:10.1109/tdsc.2021.3083671 · Cited by 46
“Significant research efforts have been devoted to related topics in previous literature, such as network traffic analysis [5][6], web application anomaly detection [7], debugging [8], service failure detection [9], and prediction [10].”
BackgroundOn the Nature of Issues in Five Open Source Microservices Systems: An Empirical Study
Waseem, M., Liang, P., Shahin, M., Ahmad, A., & Nassab, A. R. (2021) · International Conference on Evaluation and Assessment in Software Engineering (EASE) · doi:10.1145/3463274.3463337 · Cited by 47
“To conclude, we categorize the most relevant existing research as (a) repository mining approaches such as [6, 17, 19] and (b) surveybased studies including [3, 24, 34] to investigateMSA-based systems and their underlying issues.”
InfluentialDeep Attentive Anomaly Detection for Microservice Systems with Multimodal Time-Series Data
Chen, Y., Yan, M. L., Yang, D., Zhang, X., & Wang, Z. (2022) · IEEE International Conference on Web Services (ICWS) · doi:10.1109/icws55610.2022.00062 · Cited by 46
“We conduct our fault injection experiments following the work in [19].”
MethodTraceCRL: contrastive representation learning for microservice trace analysis
Zhang, C., Peng, X., Zhou, T., Sha, C., Yan, Z., Chen, Y., & Yang, H. (2022) · Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3540250.3549146 · Cited by 36
“[39] propose an improved trace visualization approach to support fault analysis and debugging.”
BackgroundMethodInfluentialSelf-citationSelf-Healing Dilemmas in Distributed Systems: Fault Correction vs. Fault Tolerance
Nikolić, J., Jubatyrov, N., & Pournaras, E. (2021) · IEEE Transactions on Network and Service Management · doi:10.1109/tnsm.2021.3092939 · Cited by 43
When Failure is (Not) an Option: Reliability Models for Microservices Architectures
Jagadeesan, L. J., & Mendiratta, V. (2020) · IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW) · doi:10.1109/issrew51248.2020.00031 · Cited by 43
“The work by Zhou et al. [13] presents the results of an industrial survey and empirical study on the fault analysis and debugging of microservice systems.”
BackgroundFaster, deeper, easier: crowdsourcing diagnosis of microservice kernel failure from user space
Pan, Y., Ma, M., Jiang, X., & Wang, P. (2021) · Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis · doi:10.1145/3460319.3464805 · Cited by 42
“These failures have caused extensive impacts on online businesses and severe economic losses [35].”
BackgroundMicroservices integrated performance and reliability testing
Camilli, M., Guerriero, A., Janes, A., Russo, B., & Russo, S. (2022) · doi:10.1145/3524481.3527233 · Cited by 42
“To answer these questions, MIPaRT is evaluated through controlled experiments with the Train Ticket microservices system benchmark [13], reproducing two scenarios of evolutionary and operational changes typically occurring in DevOps.”
MethodGan, Y., Liu, G., Zhang, X., Zhou, Q., Wu, J., & Jiang, J. (2023) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3623278.3624758 · Cited by 39
Delta Debugging Microservice Systems with Parallel Optimization
Zhou, X., Peng, X., Xie, T., Sun, J., Ji, C., Li, W., & Ding, D. (2022) · IEEE Transactions on Services Computing · doi:10.1109/tsc.2019.2919823 · Cited by 33
“Our recent industrial survey [10] reveals that they often need to manually examine a large number of logs, and the debugging depends heavily on the developers’ experience on the system (e.g., overall architecture and error-prone microservices) and similar fault cases, as well as the technology…”
BackgroundMethodInfluentialSelf-citationRoot Cause Analysis for Microservice System based on Causal Inference: How Far Are We?
Pham, L., Ha, H., & Zhang, H. (2024) · Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering · doi:10.1145/3691620.3695065 · Cited by 38
“However, we believe that the datasets and systems we use are representative since they have been used in many previous studies [27, 28, 32, 35, 58– 60, 62, 63, 67] and could help us to derive various important insights for causal inference-based RCA methods for microservice systems.”
Huang, L., & Zhu, T. (2021) · Proceedings of the ACM Symposium on Cloud Computing · doi:10.1145/3472883.3486994 · Cited by 37
Xie, Z., Zhang, S., Geng, Y., Zhang, Y., Ma, M., Nie, X., Yao, Z., Xu, L., Sun, Y., Li, W., et al. (2024) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3637528.3671530 · Cited by 37
Self-citationChen, H., Chen, P., & Yu, G. (2020) · IEEE Access · doi:10.1109/access.2020.2977464 · Cited by 36
“In previous work [38] and [39], researchers collected 22 representative microservice faults and listed the detail description of these faults.”
BackgroundUrsa: Lightweight Resource Management for Cloud-Native Microservices
Zhang, Y., Zhou, Z., Elnikety, S., & Delimitrou, C. (2024) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca57654.2024.00077 · Cited by 36
“Representative benchmarks include DeathstarBench [28] and ticket reservation [62], which implement several end-to-end user-facing applications.”
BackgroundMethodRouf, R., Rasolroveicy, M., Litoiu, M., Nagar, S., Mohapatra, P., Gupta, P., & Watts, I. (2024) · ACM/SPEC International Conference on Performance Engineering (ICPE) · doi:10.1145/3629526.3645047 · Cited by 36
DevOpRET: Continuous reliability testing in DevOps
Bertolino, A., Angelis, G. D., Guerriero, A., Miranda, B., Pietrantuono, R., & Russo, S. (2020) · Journal of Software Evolution and Process · doi:10.1002/smr.2298 · Cited by 35
Causal reasoning in Software Quality Assurance: A systematic review
Giamattei, L., Guerriero, A., Pietrantuono, R., & Russo, S. (2024) · Information and Software Technology · doi:10.1016/j.infsof.2024.107599 · Cited by 35
“Many papers refer to open-source datasets and software (P3, P11, P15, P21, P74, P75, P81) for their experiments, such as SockShop 17 (P31, P38) and TrainTicket by Zhou et al. [162] (P39), even both (P35), for microservices, and CARLA simulator by Dosovitskiy et al. [163] (P34) for automotive.”
MethodPUTraceAD: Trace Anomaly Detection with Partial Labels based on GNN and PU Learning
Zhang, K., Zhang, C., Peng, X., & Sha, C. (2022) · 2022 IEEE 33rd International Symposium on Software Reliability Engineering (ISSRE) · doi:10.1109/issre55969.2022.00032 · Cited by 33
“1) Dataset: Our experiments are conducted on TrainTicket [25], [26], a microservice benchmark system widely used in researches on microservice architecture, anomaly detection, and root cause analysis [5], [7], [27].”
BackgroundMethodSelf-citationMicroservice-based projects in agile world: A structured interview
Ünlü, H., Kennouche, D. E., Soylu, G. K., & Demirörs, O. (2023) · Information and Software Technology · doi:10.1016/j.infsof.2023.107334 · Cited by 33
Kub-Sec, an automatic Kubernetes cluster AppArmor profile generation engine
Zhu, H., & Gehrmann, C. (2022) · International Conference on Communication Systems & Networks (COMSNETS) · doi:10.1109/comsnets53615.2022.9668504 · Cited by 32
AID: Efficient Prediction of Aggregated Intensity of Dependency in Large-scale Cloud Systems
Yang, T., Shen, J., Su, Y., Ling, X., Yang, Y., & Lyu, M. R.-T. (2021) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase51524.2021.9678534 · Cited by 32
“Figure 2 illustrates a span generated by the train-ticket benchmark [18].”
BackgroundMethodA Study on the Aging and Fault Tolerance of Microservices in Kubernetes
Flora, J., Gonçalves, P., Teixeira, M., & Antunes, N. (2022) · IEEE Access · doi:10.1109/access.2022.3231191 · Cited by 32
“The faults presented in this work are based on an industrial survey that condenses the typical faults observed in microser-vice systems and the difficulties developers face [7].”
BackgroundMethodInfluentialA Choreography Analysis Approach for Microservice Composition in Cyber-Physical-Social Systems
Dai, F., Mo, Q., Qiang, Z., Huang, B., Kou, W., & Yang, H. (2020) · IEEE Access · doi:10.1109/access.2020.2980891 · Cited by 33
“2) In the context of microservice composition, the asynchronous communication model can either peer-to-peer communication or mailbox communication [24].”
BackgroundFuzzing microservices: A series of user studies in industry on industrial systems with EvoMaster
Zhang, M., Arcuri, A., Li, Y., Liu, Y., Xue, K., Wang, Z., Huo, J., & Huang, W. (2025) · Science of Computer Programming · doi:10.1016/j.scico.2025.103322 · Cited by 31
“Other testing techniques, such as mutation testing [23, 59], fault debugging [82] and flaky tests [47] have also been evaluated in industry.”
Proceedings of the XI International Scientific and Practical Conference
Валерійович, С. О., Оружак, І. В., Сергійович, К. Д., В., Р. А., & Юрійович, С. Є. (2022) · doi:10.46299/isg.2022.2.11 · Cited by 31
Graph based Incident Extraction and Diagnosis in Large-Scale Online Systems
He, Z., Chen, P., Luo, Y. B., Yan, Q., Chen, H., Yu, G., & Li, F. (2022) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3551349.3556904 · Cited by 30
AutoMan: Resource-efficient provisioning with tail latency guarantees for microservices
Cai, B., Wang, B., Yang, M., & Guo, Q. (2023) · Future Generation Computer Systems · doi:10.1016/j.future.2023.01.014 · Cited by 30
Performance Modeling and Analysis of Design Patterns for Microservice Systems
Pinciroli, R., Aleti, A., & Trubiani, C. (2023) · IEEE International Conference on Software Architecture (ICSA) · doi:10.1109/icsa56044.2023.00012 · Cited by 30
“As part of our future work, we plan to measure the performance characteristics of microservice-based systems (e.g., TeaStore [39], TrainTicket [40], or Sock Shop [41]).”
BackgroundMicroFI: Non-Intrusive and Prioritized Request-Level Fault Injection for Microservice Applications
Chen, H., Chen, P., Yu, G., Li, X., & He, Z. (2024) · IEEE Transactions on Dependable and Secure Computing · doi:10.1109/tdsc.2024.3363902 · Cited by 29
“Therefore, these application are representative and widely used as the experiment benchmarks [4], [38], [58], [61], [62], [63], [64].”
MethodNo More Data Silos: Unified Microservice Failure Diagnosis With Temporal Knowledge Graph
Zhang, S., Zhao, Y., Xia, S., Wei, S., Sun, Y., Zhao, C., Ma, S., Kuang, J., Zhu, B., Pan, L., et al. (2024) · IEEE Transactions on Services Computing · doi:10.1109/tsc.2024.3489444 · Cited by 32
“However, the complexity and dynamism of microservice systems pose unique challenges for maintenance [2].”
InfluentialMicroIRC: Instance-level Root Cause Localization for Microservice Systems
Zhu, Y., Wang, J., Li, B., Zhao, Y., Zhang, Z., Xiong, Y., & Chen, S. (2024) · Journal of Systems and Software · doi:10.1016/j.jss.2024.112145 · Cited by 29
Wang, P., Zhang, X., Chen, Y., & Cao, Z. (2024) · Information Processing & Management · doi:10.1016/j.ipm.2024.104013 · Cited by 28
Chen, Y., Xu, D., Chen, N., & Wu, X. (2022) · Information and Software Technology · doi:10.1016/j.infsof.2022.107083 · Cited by 28
An Empirical Study of Web Services Topics in Web Developer Discussions on Stack Overflow
Mahmood, K., Rasool, G., Sabir, F., & Athar, A. (2023) · IEEE Access · doi:10.1109/access.2023.3238813 · Cited by 28
“Zhou et al. [57] investigated defects in industrial micro-service systems and debugging procedures using 22 typical situations.”
MethodCromlech: Semi-Automated Monolith Decomposition Into Microservices
Quattrocchi, G., Cocco, D., Staffa, S., Margara, A., & Cugola, G. (2024) · IEEE Transactions on Services Computing · doi:10.1109/tsc.2024.3354457 · Cited by 27
“Tutored 7 isareal-worldmonolithicapplication,while TrainTicket has been proposed as a benchmark for microservices systems by the software engineering research community [24].”
MethodAsynchronous Real-Time Federated Learning for Anomaly Detection in Microservice Cloud Applications
Raeiszadeh, M., Ebrahimzadeh, A., Glitho, R., Eker, J., & Mini, R. A. F. (2025) · IEEE Transactions on Machine Learning in Communications and Networking · doi:10.1109/tmlcn.2025.3527919 · Cited by 27
Fitness-guided Resilience Testing of Microservice-based Applications
Long, Z., Wu, G., Chen, X., Cui, C., Chen, W., & Wei, J. (2020) · IEEE International Conference on Web Services (ICWS) · doi:10.1109/icws49710.2020.00027 · Cited by 27
“In the work [22], they further leverages ShiViz [10], a debugging visualization tool for distributed system to identify faulty microservice invocation.”
MethodA systematic literature review on benchmarks for evaluating debugging approaches
Hirsch, T., & Hofer, B. (2022) · Journal of Systems and Software · doi:10.1016/j.jss.2022.111423 · Cited by 27
Automated functional and robustness testing of microservice architectures
Giamattei, L., Guerriero, A., Pietrantuono, R., & Russo, S. (2023) · Journal of Systems and Software · doi:10.1016/j.jss.2023.111857 · Cited by 27
Metadata
- Authors
- Zhou, X., Peng, X., Xie, T., Sun, J., Ji, C., Li, W., & Ding, D.
- Affiliations
- Fudan University (CN)
- University of Illinois Urbana-Champaign (US)
- Singapore University of Technology and Design (SG)
- Venue
- IEEE Transactions on Software Engineering
- Published
- 2021-02-01
- Volume
- 47
- Issue
- 2
- Pages
- 243-260
- ISSN
- 0098-5589 (ISSN-L)
- 1939-3520
- 2326-3881
- OpenAlex
- W2903799441
- Cited by
- 410
- References
- 42
- Citations by year
- 2026: 70 · 2025: 64 · 2024: 82 · 2023: 68 · 2022: 58 · 2021: 40
- FWCI
- 10.53
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- Locations
- IEEE Transactions on Software Engineering (publishedVersion)
- Singapore Management University Institutional Knowledge (InK) (Singapore Management University) (submittedVersion)
- Funders
- National Key Research and Development Program of China (2018YFB1004803)
- National Science Foundation (CNS-1513939, CNS-1564274, CCF-1816615)
- Topics
- Physical Sciences › Computer Science › Computer Networks and Communications › Software System Performance and Reliability
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Keywords
- microservices, fault analysis, debugging, distributed systems, tracing, trace visualization, fault location, benchmark systems, industrial survey, developer practices, trace analysis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.