CoChat
Sign Up
PreprintCited by 3Software System Performance and ReliabilityCloud Computing and Resource ManagementSoftware Engineering Research

The DOI matches its DataCite record; the references it cites are not checked yet.

AIOpsLab: A Holistic Framework to Evaluate AI Agents for Enabling Autonomous Clouds

Open access
Chen, Yinfang, Shetty, Manish, Somashekar, Gagan, Ma, Minghua, Simmhan, Yogesh, Mace, Jonathan, Bansal, Chetan, Wang, Rujia, Rajmohan, Saravan·2025·arXiv
DOI 10.48550/arxiv.2501.06706arXiv 2501.06706
Open this paper in CoChat — free accountOpen PDF

Abstract

AI for IT Operations (AIOps) aims to automate complex operational tasks, such as fault localization and root cause analysis, to reduce human workload and minimize customer impact. While traditional DevOps tools and AIOps algorithms often focus on addressing isolated operational tasks, recent advances in Large Language Models (LLMs) and AI agents are revolutionizing AIOps by enabling end-to-end and multitask automation. This paper envisions a future where AI agents autonomously manage operational tasks throughout the entire incident lifecycle, leading to self-healing cloud systems, a paradigm we term AgentOps. Realizing this vision requires a comprehensive framework to guide the design, development, and evaluation of these agents. To this end, we present AIOPSLAB, a framework that not only deploys microservice cloud environments, injects faults, generates workloads, and exports telemetry data but also orchestrates these components and provides interfaces for interacting with and evaluating agents. We discuss the key requirements for such a holistic framework and demonstrate how AIOPSLAB can facilitate the evaluation of next-generation AIOps agents. Through evaluations of state-of-the-art LLM agents within the benchmark created by AIOPSLAB, we provide insights into their capabilities and limitations in handling complex operational tasks in cloud environments.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2501.06706

Cites

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • A change may be correct in its final state yet unsafe while it is being applied.

    “A change may be correct in its final state yet unsafe while it is being applied, so the agent should stage the change, run a canary, validate the result, and only then widen the rollout [11], [73], [41].”

    From Large language models for agentic NetOps and AIOps: Architectures, evaluation, and safety (2027) · cites this paper as [41]

    Section 2.4.3 discusses faults caused by incorrect deployments or operations, and Section 3.3 discusses how injected faults can propagate through services. Neither addresses whether a change that is correct in its final state can be unsafe while being applied. Searches for intermediate-state safety and staged or canary rollouts found no passage establishing that distinction. (LLM, conf=0.72)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

The citation lists haven’t been fetched yet

Cited by 3

Research works printed but not linked: 59

  1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ah- mad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023). · arXiv
  2. Toufique Ahmed, Supriyo Ghosh, Chetan Bansal, Thomas Zimmermann, Xuchao Zhang, and Saravan Rajmohan. 2023. Recommending Root-Cause and Mitigation Steps for Cloud Incidents using Large Language Models. In Proceedings of the 45th International Conference on Soft- ware Engineering (ICSE'23).
  3. Ramnatthan Alagappan, Aishwarya Ganesan, Yuvraj Pa- tel, Thanumalayan Sankaranarayana Pillai, Andrea C. Arpaci-Dusseau, and Remzi H. Arpaci-Dusseau. 2016. Correlated Crash Vulnerabilities. In Proceedings of the 12th USENIX Conference on Operating Systems Design and Implementation (OSDI'16).
  4. Ahmed Alquraan, Hatem Takruri, Mohammed Alfatafta, and Samer Al-Kiswany. 2018. An Analysis of Network- Partitioning Failures in Cloud Systems. In Proceedings of the 13th USENIX Conference on Operating Systems Design and Implementation (OSDI'18).
  5. Radu Banabic and George Candea. 2012. Fast Black-Box Testing of System Recovery Code. In Proceedings of the 7th European Conference on Computer Systems (Eu- roSys'12).
  6. BIG-bench authors. 2023. Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models. Transactions on Machine Learning Research (2023).
  7. Marco Canini, Daniele Venzano, Peter Perešíni, Dejan Kostić, and Jennifer Rexford. 2012. A NICE Way to Test OpenFlow Applications. In Proceedings of the 9th USENIX Conference on Networked Systems Design and Implementation (NSDI'12).
  8. Uzay Çetin and Mursel Tasgin. 2020. Anomaly detection with multivariate K-sigma score using Monte Carlo. In 2020 5th International Conference on Computer Science and Engineering.
  9. Haicheng Chen, Wensheng Dou, Dong Wang, and Feng Qin. 2020. CoFI: Consistency-Guided Fault Injection for Cloud Systems. In Proceedings of the 35th ACM/IEEE International Conference on Automated Software Engi- neering (ASE'20).
  10. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Hen- rique Ponde De Oliveira Pinto, Jared Kaplan, Harri Ed- wards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 (2021). · arXiv
  11. Yinfang Chen, Huaibing Xie, Minghua Ma, Yu Kang, Xin Gao, Liu Shi, Yunjie Cao, Xuedong Gao, Hao Fan, Ming Wen, Jun Zeng, Supriyo Ghosh, Xuchao Zhang, Chaoyun Zhang, Qingwei Lin, Saravan Rajmohan, Dong- mei Zhang, and Tianyin Xu. 2024. Automatic Root Cause Analysis via Large Language Models for Cloud Incidents. In Proceedings of the Nineteenth European Conference on Computer Systems (EuroSys'24).
  12. Maria Christakis, Patrick Emmisberger, Patrice Godefroid, and Peter Müller. 2017. A General Framework for Dy- namic Stub Injection. In Proceedings of the 39th Interna- tional Conference on Software Engineering (ICSE'17).
  13. Yuanshun Dai, Yanping Xiang, and Gewei Zhang. 2009. Self-healing and Hybrid Diagnosis in Cloud Computing. In Cloud Computing, Martin Gilje Jaatun, Gansen Zhao, and Chunming Rong (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg.
  14. Yu Gan, Yanqi Zhang, Dailun Cheng, Ankitha Shetty, Priyal Rathi, Nayan Katarki, Ariana Bruno, Justin Hu, Brian Ritchken, Brendon Jackson, et al. 2019. An open-source benchmark suite for microservices and their hardware- software implications for cloud & edge systems. In Pro- ceedings of the 24th International Conference on Archi- tectural Support for Programming Languages and Oper- ating Systems (ASPLOS'19).
  15. Vaibhav Ganatra, Anjaly Parayil, Supriyo Ghosh, Yu Kang, Minghua Ma, Chetan Bansal, Suman Nath, and Jonathan Mace. 2023. Detection Is Better Than Cure: A Cloud Incidents Perspective. In Proceedings of the 31st Joint European Software Engineering Conference and Sympo- sium on the Foundations of Software Engineering (ES- EC/FSE'23).
  16. Haryadi S. Gunawi, Thanh Do, Pallavi Joshi, Peter Al- varo, Joseph M. Hellerstein, Andrea C. Arpaci-Dusseau, Remzi H. Arpaci-Dusseau, Koushik Sen, and Dhruba Borthakur. 2011. Fate and Destini: A Framework for Cloud Recovery Testing. In Proceedings of the 8th USENIX Symposium on Networked Systems Design and Implementation (NSDI'11).
  17. Songqiao Han, Xiyang Hu, Hailiang Huang, Minqi Jiang, and Yue Zhao. 2022. ADBench: Anomaly Detection Benchmark. In Thirty-sixth Conference on Neural Infor- mation Processing Systems Datasets and Benchmarks Track.
  18. Shilin He, Xu Zhang, Pinjia He, Yong Xu, Liqun Li, Yu Kang, Minghua Ma, Yining Wei, Yingnong Dang, Sara- vanakumar Rajmohan, et al. 2022. An empirical study of log analysis at Microsoft. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineer- ing (ESEC/FSE).
  19. Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt. 2021a. Aligning AI With Shared Human Values. Proceedings of the International Conference on Learning Representa- tions (ICLR'21) (2021).
  20. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021b. Measuring Massive Multitask Language Under- standing. Proceedings of the International Conference on Learning Representations (ICLR'21) (2021).
  21. Victor Heorhiadi, Shriram Rajagopalan, Hani Jamjoom, Michael K Reiter, and Vyas Sekar. 2016. Gremlin: Sys- tematic Resilience Testing of Microservices. In Proceed- ings of the IEEE 36th International Conference on Dis- tributed Computing Systems (ICDCS'16).
  22. Chuanjia Hou, Tong Jia, Yifan Wu, Ying Li, and Jing Han. 2021. Diagnosing performance issues in microservices with heterogeneous data source. In 2021 IEEE Intl Conf on Parallel & Distributed Processing with Applications, Big Data & Cloud Computing, Sustainable Computing & Communications, Social Computing & Networking (ISPA/BDCloud/SocialCom/SustainCom).
  23. Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Jiayi Lei, Yao Fu, Maosong Sun, and Junxian He. 2023. C-Eval: A Multi-Level Multi- Discipline Chinese Evaluation Suite for Foundation Mod- els. In Advances in Neural Information Processing Sys- tems (NeurIPS'23).
  24. Vincent Jacob, Fei Song, Arnaud Stiegler, Yanlei Diao, and Nesime Tatbul. 2020. AnomalyBench: An Open Bench- mark for Explainable Anomaly Detection. CoRR (2020).
  25. Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. 2024a. LiveCodeBench: Holistic and Contamination Free Evaluation of Large Lan- guage Models for Code. arXiv preprint arXiv:2403.07974 (2024). · arXiv
  26. Naman Jain, Manish Shetty, Tianjun Zhang, King Han, Koushik Sen, and Ion Stoica. 2024b. R2E: Turning any Github Repository into a Programming Agent Environ- ment. In Forty-first International Conference on Machine Learning (ICML'24).
  27. Yuxuan Jiang, Chaoyun Zhang, Shilin He, Zhihao Yang, Minghua Ma, Si Qin, Yu Kang, Yingnong Dang, Sara- van Rajmohan, Qingwei Lin, and Dongmei Zhang. 2024. Xpert: Empowering Incident Management with Query Recommendations via Large Language Models. In Pro- ceedings of the 46th IEEE/ACM International Conference on Software Engineering (ICSE'24).
  28. Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. 2024. SWE-bench: Can Language Models Resolve Real- world Github Issues?. In The Twelfth International Con- ference on Learning Representations (ICLR'24).
  29. Xiaoen Ju, Livio Soares, Kang G. Shin, Kyung Dong Ryu, and Dilma Da Silva. 2013. On Fault Resilience of Open- Stack. In Proceedings of the 12th ACM Symposium on Cloud Computing (SOCC'13).
  30. Tony Lee, Michihiro Yasunaga, Chenlin Meng, Yifan Mai, Joon Sung Park, Agrim Gupta, Yunzhi Zhang, Deepak Narayanan, Hannah Teufel, Marco Bellagente, et al. 2024. Holistic evaluation of text-to-image models. Advances in Neural Information Processing Systems (NeurIPS'24) (2024).
  31. Tanakorn Leesatapornwongsa, Mingzhe Hao, Pallavi Joshi, Jeffrey F. Lukman, and Haryadi S. Gunawi. 2014. SAMC: Semantic-Aware Model Checking for Fast Discovery of Deep Bugs in Cloud Systems. In Proceedings of the 11th USENIX Conference on Operating Systems Design and Implementation (OSDI'14).
  32. Wenrui Li, Pengcheng Zhang, and Zhongxue Yang. 2012. A Framework for Self-Healing Service Compositions in Cloud Computing Environments. In 2012 IEEE 19th International Conference on Web Services (ICWS'12).
  33. Yuhe Liu, Changhua Pei, Longlong Xu, Bohan Chen, Mingze Sun, Zhirui Zhang, Yongqian Sun, Shenglin Zhang, Kun Wang, Haiming Zhang, et al. 2023. Op- sEval: A Comprehensive Task-Oriented AIOps Bench- mark for Large Language Models. arXiv preprint arXiv:2310.07637 (2023). · arXiv
  34. Jie Lu, Chen Liu, Lian Li, Xiaobing Feng, Feng Tan, Jun Yang, and Liang You. 2019. CrashTuner: Detecting Crash-Recovery Bugs in Cloud Systems via Meta-Info Analysis. In Proceedings of the 26th ACM Symposium on Operating System Principles (SOSP'19).
  35. Minghua Ma, Shenglin Zhang, Dan Pei, Xin Huang, and Hongwei Dai. 2018. Robust and rapid adaption for con- cept drift in software system anomaly detection. In 2018 IEEE 29th International Symposium on Software Relia- bility Engineering (ISSRE). IEEE, 13-24.
  36. Rupak Majumdar and Filip Niksic. 2018. Why is Ran- dom Testing Effective for Partition Tolerance Bugs?. In Proceedings of the 45th ACM SIGPLAN Symposium on Principles of Programming Languages (POPL'18).
  37. Paul D Marinescu and George Candea. 2009. LFI: A Practi- cal and General Library-Level Fault Injector. In Proceed- ings of the 39th IEEE/IFIP International Conference on Dependable Systems and Networks (DSN'09).
  38. Grégoire Mialon, Roberto Dessì, Maria Lomeli, Christo- foros Nalmpantis, Ram Pasunuru, Roberta Raileanu, Bap- tiste Rozière, Timo Schick, Jane Dwivedi-Yu, Asli Ce- likyilmaz, et al. 2023. Augmented language models: a survey. arXiv preprint arXiv:2302.07842 (2023). · arXiv
  39. Jayashree Mohan, Ashlie Martinez, Soujanya Ponnapalli, Pandian Raju, and Vijay Chidambaram. 2018. Find- ing Crash-Consistency Bugs with Bounded Black-Box Crash Testing. In Proceedings of the 13th USENIX Con- ference on Operating Systems Design and Implementation (OSDI'18). Netflix. 2011. ChaosMonkey. https://github.com/ Netflix/chaosmonkey. Accessed: 2024-07-08.
  40. Thanumalayan Sankaranarayana Pillai, Vijay Chidambaram, Samer Al Kiswany, Andrea C. Arpaci-Dusseau, and Remzi H. Arpaci-Dusseau. 2014. All File Systems Are Not Created Equal: On the Complexity of Craft- ing Crash-Consistent Applications. In Proceedings of the 11th USENIX Conference on Operating Systems Design and Implementation (OSDI'14). Prometheus Authors. 2024. Prometheus. https:// prometheus.io.
  41. Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2024. Tool- former: Language models can teach themselves to use tools. Advances in Neural Information Processing Sys- tems (NeurIPS'24) (2024).
  42. Gagan Somashekar, Anurag Dutt, Mainak Adak, Tania Lorido Botran, and Anshul Gandhi. 2024. GAMMA: Graph Neural Network-Based Multi-Bottleneck Localiza- tion for Microservices Applications. In Proceedings of the ACM Web Conference 2024.
  43. Akshitha Sriraman and Thomas F Wenisch. 2018. µ suite: a benchmark suite for microservices. In 2018 IEEE Inter- national Symposium on Workload Characterization.
  44. Xudong Sun, Wenqing Luo, Jiawei Tyler Gu, Aishwarya Ganesan, Ramnatthan Alagappan, Michael Gasch, Lalith Suresh, and Tianyin Xu. 2022. Automatic Reliability Test- ing for Cluster Management Controllers. In Proceedings of the 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI'22).
  45. Yongqian Sun, Binpeng Shi, Mingyu Mao, Minghua Ma, Sibo Xia, Shenglin Zhang, and Dan Pei. 2024. ART: A Unified Unsupervised Framework for Incident Man- agement in Microservice Systems. In Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering.
  46. Lingzhi Wang, Nengwen Zhao, Junjie Chen, Pinnong Li, Wenchi Zhang, and Kaixin Sui. 2020. Root-cause metric location for microservice systems via log anomaly de- tection. In 2020 IEEE international conference on web services (ICWS'20).
  47. Zefan Wang, Zichuan Liu, Yingying Zhang, Aoxiao Zhong, Lunting Fan, Lingfei Wu, and Qingsong Wen. 2023. Rcagent: Cloud root cause analysis by autonomous agents with tool-augmented large language models. arXiv preprint arXiv:2310.16340 (2023). · arXiv
  48. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. 2022a. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903 (2022). · arXiv
  49. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022b. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems (NeurIPS'22) (2022).
  50. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023. ReAct: Syn- ergizing Reasoning and Acting in Language Models. In International Conference on Learning Representations (ICLR'23).
  51. Zhaoyang Yu, Minghua Ma, Chaoyun Zhang, Si Qin, Yu Kang, Chetan Bansal, Saravan Rajmohan, Yingnong Dang, Changhua Pei, Dan Pei, Qingwei Lin, and Dong- mei Zhang. 2024a. MonitorAssistant: Simplifying Cloud Service Monitoring via Large Language Models. In Com- panion Proceedings of the 32nd ACM International Con- ference on the Foundations of Software Engineering (FSE'24).
  52. Zhaoyang Yu, Changhua Pei, Xin Wang, Minghua Ma, Chetan Bansal, Saravan Rajmohan, Qingwei Lin, Dong- mei Zhang, Xidao Wen, Jianhui Li, et al. 2024b. Pre- trained KPI Anomaly Detection Model Through Disen- tangled Transformer. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining.
  53. Pingyu Zhang and Sebastian Elbaum. 2012. Amplifying Tests to Validate Exception Handling Code. In Proceed- ings of the 34th International Conference on Software Engineering (ICSE'12).
  54. Qiao Zhang, Guo Yu, Chuanxiong Guo, Yingnong Dang, Nick Swanson, Xinsheng Yang, Randolph Yao, , Murali Chintalapati, Arvind Krishnamurthy, and Thomas Ander- son. 2018. Deepview: Virtual Disk Failure Diagnosis and Pattern Detection for Azure. In Proceedings of the 15th USENIX Symposium on Networked Systems Design and Implementation (NSDI'18).
  55. Xuchao Zhang, Supriyo Ghosh, Chetan Bansal, Rujia Wang, Minghua Ma, Yu Kang, and Saravan Rajmohan. 2024a. Automated Root Causing of Cloud Incidents using In- Context Learning with GPT-4. In Companion Proceed- ings of the 32nd ACM International Conference on the Foundations of Software Engineering.
  56. Chenyu Zhao, Minghua Ma, Zhenyu Zhong, Shenglin Zhang, Zhiyuan Tan, Xiao Xiong, LuLu Yu, Jiayi Feng, Yongqian Sun, Yuzhi Zhang, et al. 2023. Robust Mul- timodal Failure Detection for Microservice Systems. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining.
  57. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuo- han Li, Dacheng Li, Eric Xing, et al. 2024. Judging llm- as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems (NeurIPS'24) (2024).
  58. Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Yonatan Bisk, Daniel Fried, Uri Alon, et al. 2023. WebArena: A Realistic Web Environment for Building Autonomous Agents. arXiv preprint arXiv:2307.13854 (2023). · arXiv
  59. Xiang Zhou, Xin Peng, Tao Xie, Jun Sun, Chao Ji, Wenhai Li, and Dan Ding. 2021. Fault Analysis and Debugging of Microservice Systems: Industrial Survey, Benchmark System, and Empirical Study. IEEE Transactions on Software Engineering (2021).

Metadata

Authors
Chen, Y., Shetty, M., Somashekar, G., Ma, M., Simmhan, Y., Mace, J., Bansal, C., Wang, R., & Rajmohan, S.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2025
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2501.06706
arXiv
2501.06706
OpenAlex
W6891996985
Cited by
3
References
0
Citations by year
2026: 3
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by-nc-nd)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Computer Networks and Communications › Software System Performance and Reliability
  • Physical Sciences › Computer Science › Information Systems › Cloud Computing and Resource Management
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
Keywords
AIOps, AI agents, large language models, self-healing systems, fault location, root cause analysis, telemetry data, DevOps automation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review