AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents
Open accessAbstract
Evaluating Large Language Models (LLMs) as general-purpose agents is essential for understanding their capabilities and facilitating their integration into practical applications. However, the evaluation process presents substantial challenges. A primary obstacle is the benchmarking of agent performance across diverse scenarios within a unified framework, especially in maintaining partially-observable environments and ensuring multi-round interactions. Moreover, current evaluation frameworks mostly focus on the final success rate, revealing few insights during the process and failing to provide a deep understanding of the model abilities. To address these challenges, we introduce AgentBoard, a pioneering comprehensive benchmark and accompanied open-source evaluation framework tailored to analytical evaluation of LLM agents. AgentBoard offers a fine-grained progress rate metric that captures incremental advancements as well as a comprehensive evaluation toolkit that features easy assessment of agents for multi-faceted analysis. This not only sheds light on the capabilities and limitations of LLM agents but also propels the interpretability of their performance to the forefront. Ultimately, AgentBoard serves as a step towards demystifying agent behaviors and accelerating the development of stronger LLM agents.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Task completion remains a predominant and essential measure of overall agent performance.
“Although sometimes noted for providing limited fine-grained insight into failures, especially when most models achieve low success rates [64], task completion remains a predominant and essential measure of overall agent performance [64].”
From Evaluation and Benchmarking of LLM Agents: A Survey (2025) · cites this paper as [64]
Section 2.2 describes success rate—the proportion of tasks completed—as the predominant metric for agent evaluation and uses it alongside progress rate. It does not establish that task completion is essential; Section 4.2 instead says progress rate is a more effective measure for distinguishing overall model performance. (LLM, conf=0.82)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 19
- Chevalier-Boisvert, M., Bahdanau, D., Lahlou, S., Willems, L., Saharia, C., Nguyen, T. H., and Bengio, Y. (2019). Babyai: A platform to study the sample efficiency of grounded language learning. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net.
- Hausknecht, M. J., Ammanabrolu, P., Côté, M., and Yuan, X. (2020). Interactive fiction games: A colossal adventure. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 7903-7910. AAAI Press.
- Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J., Zhang, H., and Stoica, I. (2023). Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, pages 611-626.
- LeCun, Y. (2022). A path towards autonomous machine intelligence version 0.9. 2, 2022-06-27. Open Review, 62.
- OpenAI (2022). Introducing chatgpt. OpenAI (2023). Gpt-4 technical report. arXiv, pages 2303-08774.
- Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022). Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744.
- Pourchot, A. and Sigaud, O. (2019). CEM-RL: combining evolutionary and gradient-based methods for policy search. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net.
- Russell, S. and Norvig, P. (2005). Ai a modern approach. Learning, 2(3):4.
- Shi, T., Karpathy, A., Fan, L., Hernandez, J., and Liang, P. (2017a). World of bits: An open-domain platform for web-based agents. In Precup, D. and Teh, Y. W., editors, Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pages 3135-3144.
- PMLR.
- Shi, T., Karpathy, A., Fan, L., Hernandez, J., and Liang, P. (2017b). World of bits: An open-domain platform for web-based agents. In Precup, D. and Teh, Y. W., editors, Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pages 3135-3144. PMLR.
- Shinn, N., Cassano, F., Gopinath, A., Narasimhan, K. R., and Yao, S. (2023). Reflexion: Language agents with verbal reinforcement learning. In Thirty-seventh Conference on Neural Information Processing Systems.
- Shridhar, M., Yuan, X., Côté, M., Bisk, Y., Trischler, A., and Hausknecht, M. J. (2021). Alfworld: Aligning text and embodied environments for interactive learning. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.
- Song, C. H., Wu, J., Washington, C., Sadler, B. M., Chao, W.-L., and Su, Y. (2023). Llm-planner: Few-shot grounded planning for embodied agents with large language models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2998-3009.
- Vallati, M., Chrpa, L., Grześ, M., McCluskey, T. L., Roberts, M., Sanner, S., et al. (2015). The 2014 international planning competition: Progress and trends. Ai Magazine, 36(3):90-98.
- Wang, R., Jansen, P., Côté, M.-A., and Ammanabrolu, P. (2022). ScienceWorld: Is your agent smarter than a 5th grader? In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11279-11298, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
- Yao, S., Chen, H., Yang, J., and Narasimhan, K. (2022). Webshop: Towards scalable real-world web interaction with grounded language agents. Advances in Neural Information Processing Systems, 35:20744-20757.
- Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K. R., and Cao, Y. (2023). ReAct: Synergizing reason- ing and acting in language models. In The Eleventh International Conference on Learning Representations.
- press_enter_after = |1]': Use this to type the content into the field with id. By default, the "Enter" key is pressed after typing unless press_enter_after is set to 0. 'hover [id]': Hover over an element with id. 'press [key_comb]': Simulates the pressing of a key combination on the keyboard (e.g., Ctrl+v). 'scroll [directio n= down |up]': Scroll the page up or down. Tab Management Actions: 'new_tab': Open a new, empty browser tab. 'tab_focus [tab_index]': Switch the browser's focus to a specific tab using its index. 'close_tab': Close the currently active tab. URL Navigation Actions: 'goto [url]': Navigate to a specific URL. 'go_back': Navigate to the previously viewed page. 'go_forward': Navigate to the next page (if a previous 'go_back' action was performed). Completion Action: 'stop [answer]': Apply this action when you believe the task is complete. If it is a operation-type task, use 'stop [Done]' when finished. If the objective is to give a text-based answer, provide the answer in the bracket. To be successful, it is very important to follow the following rules:
Cited by
Sorted by citations
MedAgentBench: a virtual EHR environment to benchmark medical LLM agents
Jiang, Y., Black, K. C., Geng, G., Park, D., Zou, J., Ng, A. Y., & Chen, J. H. (2025) · NEJM AI · doi:10.1056/aidbp2500144 · Cited by 66 · see citing works
Toward large reasoning models: A survey of reinforced reasoning with large language models
Xu, F., Hao, Q., Shao, C., Zong, Z., Li, Y., Wang, J., Zhang, Y., Wang, J., Lan, X., Gong, J., et al. (2025) · Patterns · doi:10.1016/j.patter.2025.101370 · Cited by 186 · see citing works
Evaluation and Benchmarking of LLM Agents: A Survey
Mohammadi, M., Li, Y., Lo, J., & Yip, W. (2025) · Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2 · doi:10.1145/3711896.3736570 · Cited by 74 · see citing works
Understanding Software Engineering Agents: A Study of Thought-Action-Result Trajectories
Bouzenia, I., & Pradel, M. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00234 · Cited by 19 · see citing works
The bewitching AI: The Illusion of Communication with Large Language Models
Bottazzi Grifoni, E., & Ferrario, R. (2025) · Philosophy & Technology · doi:10.1007/s13347-025-00893-6 · Cited by 15 · see citing works
The Evolution of Research on AI and Education Across Four Decades: Insights from the AIxEd Framework
Rismanchian, S., & Doroudi, S. (2025) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-025-00483-9 · Cited by 17 · see citing works
MedAgentBench: A Realistic Virtual EHR Environment to Benchmark Medical LLM AgentsPreprint
Jiang, Y., Black, K. C., Park, D., Zou, J., Ng, A. Y., & Chen, J. H. (2025) · Qeios · doi:10.32388/vn3yh7 · Cited by 10 · see citing works
Kapoor, S., Stroebl, B., Siegel, Z. S., Nadgir, N., & Narayanan, A. (2024) · arXiv · doi:10.48550/arxiv.2407.01502 · Cited by 13 · see citing works
“To select agent benchmarks, we relied on the benchmarks included in AgentBench [30], AgentBoard [31], OpenDevin [49], and the ICLR 2024 workshop on LLM agents, in addition to benchmarks we had previously come across.”
AgentBuilder: Automating agent creation via large language model-driven systems
Tang, W., Zhang, H. W., Huang, J., Wang, S., Yu, F., Yang, H., & Wang, Y. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.130476 · Cited by 7 · see citing works
An Evaluation Mechanism of LLM-based Agents on Manipulating APIs
Liu, B., Jianxiang, Z., Meng, D., & Lu, H. (2024) · Findings of the Association for Computational Linguistics: EMNLP 2024 · doi:10.18653/v1/2024.findings-emnlp.267 · Cited by 4 · see citing works
AgentSquare: Automatic LLM Agent Search in Modular Design Space
Shang, Y., Li, Y., Zhao, K., Ma, L., Liu, J., Xu, F., & Li, Y. (2024) · arXiv · doi:10.48550/arxiv.2410.06153 · Cited by 3 · see citing works
“OPRO (Yang et al., 2024) and Prompt-breeder (Fernando et al., 2024) can be viewed as leveraging the reasoning power of LLMs to improve the prompt of LLM agents.”
InfluentialThe enhanced context for AI-generated learning advisors with Advanced RAG
Dieu, A. N. T., Nguyen, H. T., & Cong, C. T. D. (2024) · International Conference on Advanced Computing and Analytics (ACOMPA) · doi:10.1109/acompa64883.2024.00021 · Cited by 3 · see citing works
FlowBench: Revisiting and Benchmarking Workflow-Guided Planning for LLM-based Agents
Xiao, R., Ma, W., Wang, K., Wu, Y., Zhao, J., Wang, H., Huang, F., & Li, Y. (2024) · arXiv · doi:10.48550/arxiv.2406.14884 · Cited by 15 · see citing works
Observational Scaling Laws and the Predictability of Language Model Performance
Ruan, Y., Maddison, C. J., & Hashimoto, T. (2024) · arXiv · doi:10.48550/arxiv.2405.10938 · Cited by 3 · see citing works
Lu, J., Holleis, T., Zhang, Y., Aumayer, B., Nan, F., Bai, F., Ma, S., Ma, S., Li, M., Yin, G., et al. (2024) · arXiv · doi:10.48550/arxiv.2408.04682 · Cited by 2 · see citing works
EvoAgentX: An Automated Framework for Evolving Agentic Workflows
Wang, Y., Liu, S., Fang, J., & Meng, Z. (2025) · Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/2025.emnlp-demos.47 · Cited by 1 · see citing works
Beyond Task Success: Measuring Workflow Fidelity in LLM-Based Agentic Payment Systems
Huang, D., Chua, J. K., & Wang, Z. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-2014-4_29 · Cited by 1 · see citing works
Sun, X., Wang, Q., Li, H., & Chen, L. (2026) · arXiv · doi:10.48550/arxiv.2608.03137
Yu, Y., Yao, L., Xie, Y., Tan, Q., Feng, J., Li, Y., & Wu, L. (2026) · arXiv · doi:10.48550/arxiv.2601.01885
“In long-horizon agentic tasks involving multi-step reasoning and complex workflows (Chang et al., 2024), the effectiveness of large language model (LLM) agents is fundamentally constrained by the information they can attend to at any given time, which we collectively refer to as the agent’s memory…”
Heterogeneous Swarms: Jointly Optimizing Model Roles and Weights for Multi-LLM Systems
Feng, S., Wang, Z., Goyal, P., Wang, Y., Shi, W., Xia, H., Palangi, H., Zettlemoyer, L., Tsvetkov, Y., Lee, C.-Y., et al. (2025) · arXiv · doi:10.48550/arxiv.2502.04510
“…for each other’s responses to collectively self-refine [25, 7]; (multi-)agent systems could divide and conquer complex problems [98, 33, 91, 60, 118, 71, 47, 65, 55, 99, 43, 56, 92, 86, 53, 75, 68, 9, 17, 119]; multiple LLMs could debate and compete with each other to find better answers [57, 21].”
Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation
Kapoor, S., Stroebl, B., Kirgis, P., Nadgir, N., Siegel, Z. S., Wei, B., Xue, T., Chen, Z., Chen, F., Utpala, S., et al. (2025) · arXiv · doi:10.48550/arxiv.2510.11977
Evaluating and Guarding Citation Faithfulness in Agentic Scientific Synthesis
Goo, T., Kim, J., Han, K., Jo, G., Kim, J.-S., & Kim, T.-H. (2026) · arXiv · doi:10.48550/arxiv.2607.20527
A dual-channel converter valve fault prediction model based on attention-LSTM and LLM agent
Wang, Z., Xu, L., Zheng, R., Yang, P., Feng, A., & Zhang, Y. (2025) · Journal of Computational Methods in Sciences and Engineering · doi:10.1177/14727978251374350
NoRMA: A Multi-agent Communication-Centric Dataset for Enhanced Customs Nomenclature Classification
Bouchtib, H., Karboub, K., Tabaa, M., & Hamlich, M. (2026) · Communications in computer and information science · doi:10.1007/978-3-032-15632-7_18
H 2 R: Hierarchical Hindsight Reflection for Multi-Task LLM Agents
Ye, S., Yu, C., Ke, K., Xu, C., & Wei, Y. (2025) · IEEE International Conference on Agents (ICA) · doi:10.1109/ica67499.2025.00030 · Cited by 4 · see citing works
"Here, Let Me Help": An Empirical Study of User Interventions in Human–Web Agent Collaboration
Kim, J., Cho, S., Nguyen, D. M., Jeon, J., Shin, M., & Ko, S. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791536
AdaInertia: Dynamic Tool Usage Inertia Control via Deep Q-Learning for Efficient LLM Agent Reasoning
Hu, Y., Zhao, Y., Huang, Y., Dong, G., Zhang, B., Di, M., & Wang, C. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-3381-6_37
Agentic AI Unleashed: Power, Promise, and Ethical Frontiers
Wang, M., Wang, Y., & Ma, Q. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-3761-6_3
When Prompts Fail to Differentiate: Understanding Discriminative Power in LLM Agent Evaluation
Chen, Z., Wong, W. E., Hsu, C.-W., & Dai, G. (2026) · International Symposium on System Security, Safety, and Reliability (ISSSR) · doi:10.1109/isssr69973.2026.00023
Chung, 鍾. H. I., Lee, J., Kwak, S., Lee, M., & Lee, S. (2026) · Journal of Industrial Information Integration · doi:10.1016/j.jii.2026.101193
Auditable Closed-Loop AI Agents for Urban Bridge Digital Twins
Shinohara, T. (2026) · Street Art & Urban Creativity · doi:10.62161/sauc.v12.6387
Robsky, A., Lavitas, L., & Wang, Y. (2026) · Springer eBooks · doi:10.1007/978-3-032-26749-8_4
The Illusion of Success: Procedure-Aware Evaluation is Critical for LLM Agents
Cao, H., Driouich, I., & Thomas, E. M. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37682-4_29 · Cited by 5 · see citing works
Ranking Is Not Authority: Executable Context-Routing Contracts for Coding Agents
Mahesh, N. (2026) · Proceedings of the 1st International Workshop on PromptOps and Vibe Coding · doi:10.1145/3843779.3844639
Agentic AI for Computational Fluid Dynamics: A Review
Lee, K., & Kim, I. (2026) · Applied Mechanics · doi:10.3390/applmech7040081
Liu, S., Li, H., & Qian, J. (2026) · Electronics · doi:10.3390/electronics15204688
AUDITABLE CLOSED-LOOP AI AGENTS FOR URBAN BRIDGE DIGITAL TWINS
Shinohara, T. (2026) · Street Art & Urban Creativity · doi:10.66838/sauc.6368
In-Context Teacher–Student Guidance for Open-Weight Browser Agents
Cardoso, W. S., & Marcacini, R. M. (2026) · Anais do Symposium on Knowledge Discovery, Mining and Learning (KDMiLe) · doi:10.5753/kdmile.2026.32021
More Capable, Less Cooperative? When LLMs Fail At Zero-Cost Collaboration
Yadav, A., Black, S., & Sourbut, O. (2026) · arXiv · doi:10.48550/arxiv.2604.07821
“We add a sender-side bonus of $1 , 000 per piece shared with another agent (equal to 10% of the base task value r = $10 , 000 ).”
- Authors
- Ma, C., Zhang, J., Zhu, Z., Yang, C., Yang, Y., Jin, Y., Lan, Z., Kong, L., & He, J.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2024
- Pages
- 74325-74362
- ISSN
- 1049-5258 (ISSN-L)
- arXiv
- 2401.13178
- OpenAlex
- W4415797149
- Cited by
- 26
- References
- 0
- Citations by year
- 2026: 16 · 2025: 9 · 2024: 1
- FWCI
- 7.51
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- neural information processing systems (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- Rare & Special e-Zone (The Hong Kong University of Science and Technology) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Ministry of Education, India
- Shanghai Jiao Tong University
- Fundamental Research Funds for the Central Universities (2021SHZDZX0102)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Physical Sciences › Computer Science › Artificial Intelligence › Multi-Agent Systems and Negotiation
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- agent evaluation, evaluation board, large language models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
