CoChat
Sign Up
Cited by 1Software System Performance and ReliabilityMulti-Agent Systems and NegotiationAdvanced Software Engineering Methodologies

LumiMAS: A Comprehensive Framework for Real-Time Monitoring and Enhanced Observability in Multi-Agent Systems

Open access
Solomon, Ron, Yerushalmi Levi, Yarin, Waknin, Lior, Aizikovich, Eran, Baras, Amit, Ohana, Etai, Giloni, Amit, Bose, Shamik +3 more·2026·Proceedings of the 25th International Conference on Autonomous Agents and Multiagent Systems
DOI 10.65109/pnwl9707
Open this paper in CoChat — free accountOpen PDF

Abstract

The incorporation of large language models in multi-agent systems (MASs) has the potential to significantly improve our ability to autonomously solve complex problems. However, such systems introduce unique challenges in monitoring, interpreting, and detecting system failures. Most existing MAS observability frameworks focus on analyzing each individual agent separately, overlooking failures associated with the entire MAS. To bridge this gap, we propose LumiMAS, a novel MAS observability framework that incorporates advanced analytics and monitoring techniques. The proposed framework consists of three key components: a monitoring and logging layer, anomaly detection layer, and anomaly explanation layer. LumiMAS's first layer monitors MAS executions, creating detailed logs of the agents' activity. These logs serve as input to the anomaly detection layer, which detects anomalies across the MAS workflow in real time. Then, the anomaly explanation layer performs classification and root cause analysis (RCA) of the detected anomalies. LumiMAS was evaluated on seven different MAS applications, implemented using two popular MAS platforms, and a diverse set of possible failures. The applications include two novel failure-tailored applications that illustrate the effects of a hallucination or bias on the MAS. The evaluation results demonstrate LumiMAS's effectiveness in failure detection, classification, and RCA.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.65109/pnwl9707

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Effective monitoring must extend beyond surface-level interactions to include visibility into trigger activations, internal state transitions, and execution chains.

    “As a result, effective monitoring must extend beyond surface-level interactions to include visibility into trigger activations, internal state transitions, and execution chains (Chan et al., 2024b;Solomon et al., 2026;Nanda et al., 2022).”

    From Open Challenges in Multi-Agent Security: Towards Secure Systems of Interacting AI Agents (2025) · cites this paper as Solomon et al., 2026

    Section 2.3 (p. 2) calls for observability beyond system- and agent-level data, and Section 4.1 (p. 3) describes chronological logs of agent execution, LLM calls, and tool use. These support visibility beyond surface-level interactions and into execution chains. The paper does not describe monitoring trigger activations or internal state transitions as such. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • LLM-based approaches may assist threat attribution by summarizing large-scale interaction logs, extracting structured event representations, or generating hypotheses about plausible attack paths.

    “LLM-based approaches may assist by summarizing large-scale interaction logs, extracting structured event representations, or generating hypotheses about plausible attack paths; recent work on tool-augmented language models and automated log analysis suggests their usefulness in triage and forensic workflows (Schick et al., 2023;Solomon et al., 2026;Yao et al., 2022).”

    From Open Challenges in Multi-Agent Security: Towards Secure Systems of Interacting AI Agents (2025) · cites this paper as Solomon et al., 2026

    The Anomaly Explanation Layer (p. 4) says LLM-based agents analyze suspected logs, classify vulnerabilities, and explain the causal chain leading to a failure, supporting the claim about assisting investigation of plausible attack paths. The System Logging Layer (p. 3) records structured events, but does not say an LLM extracts them; the cited passages also do not establish LLM summarization of large-scale logs or use in forensic workflows. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 34

  1. [1][1] Roy Betser, Shamik Bose, Amit Giloni, Chiara Picardi, Sindhu Padakandla, and Roman Vainshtein. 2026. AgenTRIM: Tool Risk Mitigation for Agentic AI. arXiv preprint arXiv:2601.12449 (2026). - · arXiv
  2. [2][2] Mert Cemri, Melissa Z Pan, Shuyi Yang, Lakshya A Agrawal, Bhavya Chopra, Rishabh Tiwari, Kurt Keutzer, Aditya Parameswaran, Dan Klein, Kannan Ramchandran, et al. 2025. Why do multi-agent llm systems fail? arXiv preprint arXiv:2503.13657 (2025). - · arXiv
  3. [3][3] Alan Chan, Carson Ezell, Max Kaufmann, Kevin Wei, Lewis Hammond, Herbie Bradley, Emma Bluemke, Nitarshan Rajkumar, David Krueger, Noam Kolt, et al. 2024. Visibility into AI agents. In Proceedings of the 2024 ACM conference on fairness, accountability, and transparency . 958-973. -
  4. [4][4] Chi-Min Chan, Jianxuan Yu, Weize Chen, Chunyang Jiang, Xinyu Liu, Weijie Shi, Zhiyuan Liu, Wei Xue, and Yike Guo. 2024. Agentmonitor: A plug-andplay framework for predictive and secure multi-agent systems. arXiv preprint arXiv:2408.14972 (2024). - · arXiv
  5. [7][7] Zehang Deng, Yongjian Guo, Changzhou Han, Wanlun Ma, Junwu Xiong, Sheng Wen, and Yang Xiang. 2025. Ai agents under threat: A survey of key security challenges and future pathways. Comput. Surveys 57, 7 (2025), 1-36. -
  6. [8][8] Liming Dong, Qinghua Lu, and Liming Zhu. 2024. AgentOps: Enabling Observability of LLM Agents. arXiv preprint arXiv:2411.05285 (2024). - · arXiv
  7. [9][9] Finale Doshi-Velez and Been Kim. 2017. Towards a rigorous science of interpretable machine learning. arXiv preprint arXiv:1702.08608 (2017). - · arXiv
  8. [10][10] Haishuo Fang, Xiaodan Zhu, and Iryna Gurevych. [n.d.]. Inferact: Inferring safe actions for llm-based agents through preemptive evaluation and human feedback, 2024. URL https://arxiv. org/abs/2407.11843 ([n. d.]). -
  9. [11][11] Kai Greshake, Sahar Abdelnabi, Shailesh Mishra, Christoph Endres, Thorsten Holz, and Mario Fritz. 2023. Not what you've signed up for: Compromising realworld llm-integrated applications with indirect prompt injection. In Proceedings of the 16th ACM workshop on artificial intelligence and security . 79-90. -
  10. [12][12] Haixuan Guo, Shuhan Yuan, and Xintao Wu. 2021. Logbert: Log anomaly detection via bert. In 2021 international joint conference on neural networks (IJCNN) . IEEE, 1-8. -
  11. [13][13] Taicheng Guo, Xiuying Chen, Yaqi Wang, Ruidi Chang, Shichao Pei, Nitesh V Chawla, Olaf Wiest, and Xiangliang Zhang. 2024. Large language model based multi-agents: A survey of progress and challenges. arXiv preprint arXiv:2402.01680 (2024). - · arXiv
  12. [16][16] Sirui Hong, Mingchen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, et al. 2023. MetaGPT: Meta programming for a multi-agent collaborative framework. In The twelfth international conference on learning representations . -
  13. [17][17] Wenyue Hua, Xianjun Yang, Mingyu Jin, Zelong Li, Wei Cheng, Ruixiang Tang, and Yongfeng Zhang. 2024. Trustagent: Towards safe and trustworthy llm-based agents. In Findings of the Association for Computational Linguistics: EMNLP 2024 . 10000-10016. -
  14. [18][18] Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276 (2024). - · arXiv
  15. [19][19] Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of hallucination in natural language generation. ACM computing surveys 55, 12 (2023), 1-38. -
  16. [21][21] Younjeong Lee, Chanho Park, Namji Kim, Jisu Ahn, and Jongpil Jeong. 2024. LSTM-autoencoder based anomaly detection using vibration data of wind turbines. Sensors 24, 9 (2024), 2833. -
  17. [22][22] Potsawee Manakul, Adian Liusie, and Mark Gales. 2023. Selfcheckgpt: Zeroresource black-box hallucination detection for generative large language models. In Proceedings of the 2023 conference on empirical methods in natural language processing . 9004-9017. -
  18. [23][23] Silen Naihin, David Atkinson, Marc Green, Merwane Hamadi, Craig Swift, Douglas Schonholtz, Adam Tauman Kalai, and David Bau. 2023. Testing language 24. model agents safely in the wild. arXiv preprint arXiv:2311.10538 (2023). - · arXiv
  19. [24][24] Roberto Navigli, Simone Conia, and Björn Ross. 2023. Biases in large language models: origins, inventory, and discussion. ACM Journal of Data and Information Quality 15, 2 (2023), 1-21. -
  20. [27][27] Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thompson, Phu Mon Htut, and Samuel Bowman. 2022. BBQ: A handbuilt bias benchmark for question answering. In Findings of the Association for Computational Linguistics: ACL 2022 . 2086-2105. -
  21. [28][28] Pierre Peigné, Mikolaj Kniejski, Filip Sondej, Matthieu David, Jason HoelscherObermaier, Christian Schroeder de Witt, and Esben Kran. 2025. Multi-agent security tax: Trading off security and collaboration capabilities in multi-agent systems. In Proceedings of the AAAI Conference on Artificial Intelligence , Vol. 39. 27573-27581. -
  22. [29][29] Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250 (2016). - · arXiv
  23. [30][30] Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP) . 3982-3992. -
  24. [31][31] Yangjun Ruan, Honghua Dong, Andrew Wang, Silviu Pitis, Yongchao Zhou, Jimmy Ba, Yann Dubois, Chris J Maddison, and Tatsunori Hashimoto. 2023. Identifying the risks of lm agents with an lm-emulated sandbox. arXiv preprint arXiv:2309.15817 (2023). - · arXiv
  25. [33][33] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems 30 (2017). -
  26. [34][34] Yuanyuan Wei, Julian Jang-Jaccard, Wen Xu, Fariza Sabrina, Seyit Camtepe, and Mikael Boulic. 2023. LSTM-autoencoder-based anomaly detection for indoor air quality time-series data. IEEE Sensors Journal 23, 4 (2023), 3787-3800. -
  27. [35][35] Zhenjie Xu, Wenqing Chen, Yi Tang, Xuanying Li, Cheng Hu, Zhixuan Chu, Kui Ren, Zibin Zheng, and Zhichao Lu. 2025. Mitigating social bias in large language models: A multi-objective approach within a multi-agent framework. In Proceedings of the AAAI Conference on Artificial Intelligence , Vol. 39. 25579-25587. -
  28. [36][36] Tongxin Yuan, Zhiwei He, Lingzhong Dong, Yiming Wang, Ruijie Zhao, Tian Xia, Lizhen Xu, Binglin Zhou, Fangqi Li, Zhuosheng Zhang, et al. 2024. R-judge: Benchmarking safety risk awareness for llm agents. In Findings of the Association for Computational Linguistics: EMNLP 2024 . 1467-1490. -
  29. [37][37] Cong Zhang, Xin Deik Goh, Dexun Li, Hao Zhang, and Yong Liu. 2025. Planning with multi-constraints via collaborative language agents. In Proceedings of the 31st International Conference on Computational Linguistics . 10054-10082. -
  30. [38][38] Hanrong Zhang, Jingyuan Huang, Kai Mei, Yifei Yao, Zhenting Wang, Chenlu Zhan, Hongwei Wang, and Yongfeng Zhang. 2024. Agent security bench (asb): Formalizing and benchmarking attacks and defenses in llm-based agents. arXiv preprint arXiv:2410.02644 (2024). - · arXiv
  31. [39][39] Wei Zhang, Hongcheng Guo, Jian Yang, Zhoujin Tian, Yi Zhang, Yan Chaoran, Zhoujun Li, Tongliang Li, Xu Shi, Liangfan Zheng, et al. 2024. mABC: multiAgent Blockchain-Inspired Collaboration for root cause analysis in micro-services architecture. In Findings of the Association for Computational Linguistics: EMNLP 2024 . 4017-4033. -
  32. [40][40] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in neural information processing systems 36 (2023), 46595-46623. -
  33. [41][41] Xuanhe Zhou, Guoliang Li, Zhaoyan Sun, Zhiyuan Liu, Weize Chen, Jianming Wu, Jiesi Liu, Ruohang Feng, and Guoyang Zeng. 2023. D-bot: Database diagnosis system using large language models. arXiv preprint arXiv:2312.01454 (2023). - · arXiv
  34. [42][42] Mingchen Zhuge, Changsheng Zhao, Dylan Ashley, Wenyi Wang, Dmitrii Khizbullin, Yunyang Xiong, Zechun Liu, Ernie Chang, Raghuraman Krishnamoorthi, Yuandong Tian, et al. 2024. Agent-as-a-judge: Evaluate agents with agents. arXiv preprint arXiv:2410.10934 (2024). · arXiv

Cited by

Sorted by citations

Authors
Solomon, R., Yerushalmi Levi, Y., Waknin, L., Aizikovich, E., Baras, A., Ohana, E., Giloni, A., Bose, S., Picardi, C., Elovici, Y., & Shabtai, A.
Affiliations
Ben-Gurion University of the Negev (IL)
ORCID
Venue
Proceedings of the 25th International Conference on Autonomous Agents and Multiagent Systems
Publisher
International Foundation for Autonomous Agents and Multiagent Systems
Published
2026-05-24
ISSN
1548-8403 (ISSN-L)
DOI
10.65109/pnwl9707
OpenAlex
W4414487281
Cited by
1
References
0
Citations by year
2026: 1
FWCI
10.50
Type
conference paper
Language
en
Open access
Locations
  • Adaptive Agents and Multi-Agents Systems (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Computer Networks and Communications › Software System Performance and Reliability
  • Physical Sciences › Computer Science › Artificial Intelligence › Multi-Agent Systems and Negotiation
  • Physical Sciences › Computer Science › Artificial Intelligence › Advanced Software Engineering Methodologies
Keywords
multi-agent systems, large language models, observability, real-time monitoring, anomaly detection, anomaly explanation, root cause analysis, fault detection, hallucinations

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review