Establishing Best Practices in Building Rigorous Agentic Benchmarks
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Prior work has shown that failures can be systematic and can be clustered into several overlapping categories.
“Prior work has shown that failures can be systematic and can be clustered into several overlapping categories (Nathani et al., 2025; Pan et al., 2025; Zhu et al., 2025; Yang et al., 2025c).”
From Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces (2026) · cites this paper as Zhu et al., 2025
The paper applied the Agentic Benchmark Checklist (ABC) to further troubleshoot and demonstrate the evaluation rigor.
“We applied the Agentic Benchmark Checklist (ABC) (Zhu et al., 2025) to further troubleshoot and demonstrate the evaluation rigor.”
From Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces (2026) · cites this paper as Zhu et al., 2025
Cites
Still being checked.
Cited by
Sorted by citations
Failure as a Process: An Anatomy of CLI Coding Agent Trajectories
Zhao, X., Li, H., Li, S., Zhao, T., Barr, E. T., Sarro, F., & Ye, H. (2026) · arXiv · doi:10.48550/arxiv.2607.09510 · Cited by 1 · see citing works
Towards a Science of Scaling Agent Systems
Kim, Y., Gu, K., Park, C., Park, C., Schmidgall, S., Heydari, A. A., Yan, Y., Zhang, Z., Zhuang, Y., Liu, Y., et al. (2025) · arXiv · doi:10.48550/arxiv.2512.08296 · Cited by 1 · see citing works
Self-citationHolistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation
Kapoor, S., Stroebl, B., Kirgis, P., Nadgir, N., Siegel, Z. S., Wei, B., Xue, T., Chen, Z., Chen, F., Utpala, S., et al. (2025) · arXiv · doi:10.48550/arxiv.2510.11977
Self-citationLife After Benchmark Saturation: A Case Study of CORE-Bench
Nadgir, N., Kapoor, S., Liu, K., Kirgis, P., Orona, M., Rabanser, S., Bayer, T., Shetty, A., Ling, Y., Chan-Sew, D., et al. (2026) · arXiv · doi:10.48550/arxiv.2606.26158
Self-citationTerminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces
Merrill, M. A., Shaw, A. G., Carlini, N., Li, B., Raj, H., Bercovich, I., Shi, L., Shin, J. Y., Walshe, T., Buchanan, E. K., et al. (2026) · arXiv · doi:10.48550/arxiv.2601.11868
Self-citation
- Authors
- Zhu, Y., Jin, T., Pruksachatkun, Y., Zhang, A., Liu, S., Cui, S., Kapoor, S., Longpre, S., Meng, K., Weiss, R., Barez, F., Gupta, R., Dhamala, J., Merizian, J., Giulianelli, M., Coppock, H., Ududec, C., Kellermann, A., Sekhon, J., Steinhardt, J., Schwettmann, S., Narayanan, A., Zaharia, M. A., Stoica, I., Liang, P., & Kang, D.
- Affiliations
- University of Illinois Urbana-Champaign (US)
- Salesforce (United States) (US)
- Stanford University (US)
- University of California, Berkeley (US)
- Yale University (US)
- Princeton University (US)
- Massachusetts Institute of Technology (US)
- MLCommons Association
- University of Oxford (GB)
- Amazon (United States) (US)
- Virginia Tech (US)
- University of London (GB)
- Imperial College London (GB)
- UK AI Security Institute
- Transluce
- ORCID
- Venue
- Advances in Neural Information Processing Systems 38
- Published
- 2025
- Pages
- 184435-184475
- ISSN
- 1049-5258 (ISSN-L)
- OpenAlex
- W7197047848
- Cited by
- 0
- References
- 0
- FWCI
- 0.00
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- neural information processing systems (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
- Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
- Social Sciences › Decision Sciences › Information Systems and Management › Scientific Computing and Data Management
- Keywords
- agentic AI, benchmarking, large language models, AI agents, evaluation metrics
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
