Cited by 34Software System Performance and ReliabilitySoftware Engineering ResearchSoftware Testing and Debugging Techniques

Flow-of-action: SOP-enhanced LLM-based multi-agent system for root cause analysis

Open access
Pei, Changhua, Wang, Zexin, Liu, Fengrui, Li, Zeyan, Liu, Yang, He, Xiao, Kang, Rong, Zhang, Tieying +4 more·2025·Companion Proceedings of the ACM on Web Conference 2025

Abstract

In the realm of microservices architecture, the occurrence of frequent incidents necessitates the employment of Root Cause Analysis (RCA) for swift issue resolution. It is common that a serious incident can take several domain experts hours to identify the root cause. Consequently, a contemporary trend involves harnessing Large Language Models (LLMs) as automated agents for RCA. Though the recent ReAct framework aligns well with the Site Reliability Engineers (SREs) for its thought-action-observation paradigm, its hallucinations often lead to irrelevant actions and directly affect subsequent results. Additionally, the complex and variable clues of the incident can overwhelm the model one step further. To confront these challenges, we propose Flow-of-Action, a pioneering Standard Operation Procedure (SOP) enhanced LLM-based multi-agent system. By explicitly summarizing the diagnosis steps of SREs, SOP imposes constraints on LLMs at crucial junctures, guiding the RCA process towards the correct trajectory. To facilitate the rational and effective utilization of SOPs, we design an SOP-centric framework called SOP flow. SOP flow contains a series of tools, including one for finding relevant SOPs for incidents, another for automatically generating SOPs for incidents without relevant ones, and a tool for converting SOPs into code. This significantly alleviates the hallucination issues of ReAct in RCA tasks. We also design multiple auxiliary agents to assist the main agent by removing useless noise, narrowing the search space, and informing the main agent whether the RCA procedure can stop. Compared to the ReAct method's 35.50% accuracy, our Flow-of-Action method achieves 64.01%, meeting the accuracy requirements for RCA in real-world systems.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3701716.3715225

Cites

22 of 22 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Budgets prevent open-ended tool churn during incidents and provide a mechanical point at which an agent must stop, escalate, or roll back.

    “Budgets are not only a cost-control device.They prevent openended tool churn during incidents and provide a mechanical point at which an agent must stop, escalate, or roll back [43], [40], [79].”

    From Large language models for agentic NetOps and AIOps: Architectures, evaluation, and safety (2027) · cites this paper as [43]

    Section 3.4 (page unavailable) says the authors imposed a 20-step limit to prevent unbounded loops in root-cause analysis, supporting the claim that a budget prevents open-ended activity. The paper does not say that reaching the limit triggers escalation or rollback, or specify what the agent must do at that point. (LLM, conf=0.87)

    Experimental — accuracy not yet measured
  • An agent may identify the right cause and still fail operationally if it cannot show evidence, pass checks, prepare rollback, or leave an audit trail.

    “An agent may identify the right cause and still fail operationally if it cannot show evidence, pass checks, prepare rollback, or leave an audit trail [85], [43], [74].”

    From Large language models for agentic NetOps and AIOps: Architectures, evaluation, and safety (2027) · cites this paper as [43]

    The Introduction (page unavailable) says that identifying a root cause without showing the diagnostic process leads to poor explainability and SRE distrust. That supports the point about showing how a cause was identified, but our copy does not establish the claimed requirements to pass checks, prepare rollback, or leave an audit trail. (LLM, conf=0.86)

    Experimental — accuracy not yet measured
  • The reference architecture exposes observables that can be measured across tasks, traces, and deployment stages.

    “It exposes observables that can be measured across tasks, traces, and deployment stages [43], [79], [107].”

    From Large language models for agentic NetOps and AIOps: Architectures, evaluation, and safety (2027) · cites this paper as [43]

    Section 3.1.1 and Appendix A (page numbers unavailable) describe collecting metrics, logs, and traces from a deployed microservices system, supporting the claim about measurable observables and traces. The paper does not describe those observables as spanning tasks or deployment stages, or present its system as the cited document’s reference architecture. (LLM, conf=0.84)

    Experimental — accuracy not yet measured
  • Autonomy changes which failures dominate.

    “Autonomy changes which failures dominate [85], [43], [31].”

    From Large language models for agentic NetOps and AIOps: Architectures, evaluation, and safety (2027) · cites this paper as [43]

    Section 3.2 (page unavailable) reports that smaller action sets restrict randomness but leave the model unable to handle complex scenarios, while larger sets introduce randomness and loss of control. This supports a shift in failure modes as an RCA agent’s action choices change, but the paper does not establish that autonomy changes which failures dominate generally. (LLM, conf=0.78)

    Experimental — accuracy not yet measured
  • Safe autonomy can only be measured when tools, gates, and action envelopes are part of the benchmark.

    “These needs go back to Section III: safe autonomy can only be measured when tools, gates, and action envelopes are part of the benchmark [31], [43], [79].”

    From Large language models for agentic NetOps and AIOps: Architectures, evaluation, and safety (2027) · cites this paper as [43]

    Sections 2.2 and 2.4 describe tools and an action set that constrains agent choices; Section 3.4 evaluates the effects of removing the action set and SOP flow. That supports including these components in this paper’s RCA evaluation, but the paper does not say that safe autonomy can only be measured by a benchmark containing tools, gates, and action envelopes. (LLM, conf=0.75)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

2 of the 24 references couldn’t be matched to a known paper.

Research works printed but not linked: 15

  1. Sirui Hong, Xiawu Zheng, Jonathan Chen, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, et al. 2023. Metagpt: Meta programming for multi-agent collaborative framework. arXiv preprint arXiv:2308.00352 (2023). · arXiv
  2. Soyeong Jeong, Jinheon Baek, Sukmin Cho, Sung Ju Hwang, and Jong C Park. 2024. Adaptive-rag: Learning to adapt retrieval-augmented large language models through question complexity. arXiv preprint arXiv:2403.14403 (2024). · arXiv
  3. Chengshu Li, Jacky Liang, Andy Zeng, Xinyun Chen, Karol Hausman, Dorsa Sadigh, Sergey Levine, Li Fei-Fei, Fei Xia, and Brian Ichter. 2023. Chain of code: Reasoning with a language model-augmented code emulator. arXiv preprint arXiv:2312.04474 (2023). · arXiv
  4. Panagiotis Misiakos, Chris Wendler, and Markus Püschel. 2024. Learning DAGs from data with few root causes. Advances in Neural Information Processing Systems 36 (2024).
  5. Liangming Pan, Alon Albalak, Xinyi Wang, and William Yang Wang. 2023. Logic- lm: Empowering large language models with symbolic solvers for faithful logical reasoning. arXiv preprint arXiv:2305.12295 (2023). · arXiv
  6. Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al. 2023. Toolllm: Facilitating large language models to master 16000+ real-world apis. arXiv preprint arXiv:2307.16789 (2023). · arXiv
  7. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog 1, 8 (2019), 9.
  8. Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2024. Toolformer: Language models can teach themselves to use tools. Advances in Neural Information Processing Systems 36 (2024).
  9. Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2024. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems 36 (2024).
  10. Shreshth Tuli, Giuliano Casale, and Nicholas R Jennings. 2022. Tranad: Deep transformer networks for anomaly detection in multivariate time series data. arXiv preprint arXiv:2201.07284 (2022). · arXiv
  11. Lu Wang, Chaoyun Zhang, Ruomeng Ding, Yong Xu, Qihang Chen, Wentao Zou, Qingjun Chen, Meng Zhang, Xuedong Gao, Hao Fan, et al. 2023. Root cause analysis for microservice systems via hierarchical reinforcement learning from human feedback. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 5116-5125.
  12. Zexin Wang, Jianhui Li, Minghua Ma, Ze Li, Yu Kang, Chaoyun Zhang, Chetan Bansal, Murali Chintalapati, Saravan Rajmohan, Qingwei Lin, et al. 2024. Large Language Models Can Provide Accurate and Interpretable Incident Triage. In 2024 IEEE 35th International Symposium on Software Reliability Engineering (ISSRE). IEEE, 523-534.
  13. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems 35 (2022), 24824-24837.
  14. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2022. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629 (2022). · arXiv
  15. Zhenhe Yao, Haowei Ye, Changhua Pei, Guang Cheng, Guangpei Wang, Zhiwei Liu, Hongwei Chen, Hang Cui, Zeyan Li, Jianhui Li, et al. 2024. SparseRCA: Unsupervised Root Cause Analysis in Sparse Microservice Testing Traces. In 2024 IEEE 35st International Symposium on Software Reliability Engineering (ISSRE).

Cited by

Sorted by citations

Authors
Pei, C., Wang, Z., Liu, F., Li, Z., Liu, Y., He, X., Kang, R., Zhang, T., Chen, J., Li, J., Xie, G., & Pei, D.
Affiliations
  • Chinese Academy of Sciences (CN)
  • ByteDance (China) (CN)
  • ByteDance, San Jose, USA
  • Tsinghua University (CN)
Venue
Companion Proceedings of the ACM on Web Conference 2025
Publisher
ACM
Published
2025-05-08
Pages
422-431
OpenAlex
W4410636386
Cited by
33
References
24
Citations by year
2026: 29 · 2025: 4
FWCI
41.13
Type
conference paper
Language
en
Open access
Locations
  • International World Wide Web Conference (WWW) (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • Chinese Academy of Sciences (241711KYSB20200023)
  • National Natural Science Foundation of China (62202445)
  • National Natural Science Foundation of China-Research Grants Council (RGC) Joint Research Scheme (62321166652)
Topics
  • Physical Sciences › Computer Science › Computer Networks and Communications › Software System Performance and Reliability
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
Keywords
root cause analysis, large language models, multi-agent systems, microservices, incident diagnosis, hallucination mitigation, ReAct framework, site reliability engineering

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review