Agents' Last Exam
Open accessAbstract
Recent AI systems have achieved strong results on a wide range of benchmarks, yet these gains have not translated into economically meaningful deployment across many professional domains. We argue that this gap is largely an evaluation problem: widely used benchmarks lack sustained performance measurement on real and economically valuable workflows. This paper introduces Agents' Last Exam (ALE), a benchmark designed to evaluate AI agents on long horizon, economically valuable, real world tasks with verifiable outcomes. Developed in collaboration with 250+ industry experts, ALE covers non-physical industries defined with reference to O*NET / SOC 2018 (the U.S. federal occupational taxonomy). It is organized around a task taxonomy with 55 sub fields grouped into 13 industry clusters covering 1K+ tasks. Current results show that the hardest tier remains far from saturated: across mainstream harness and backbone configurations, the average full pass rate is below 1%. ALE is designed as a living benchmark: its task pool grows continuously as new workflows and industries are onboarded. More broadly, ALE is intended not merely as another leaderboard, but as an instrument for closing the gap between benchmark success and GDP relevant impact.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
QB is related to BIP, which solves the same assignment with inequality constraints; the induced non-negativity constraints add a max(0, •) clipping to both updates, which can only suppress over-selected experts without promoting under-selected ones, and markedly slows equilibration in the authors' experiments.
“QB is likewise related to BIP [116], which solves the same assignment with inequality constraints j x i,j ≤ k and i x i,j ≤ mk/n; the induced non-negativity constraints on α and β add a max(0, •) clipping to both updates, which can only suppress over-selected experts without promoting under-selected ones, and markedly slows equilibration in our experiments.”
From Kimi K3: Open Frontier Intelligence (2026) · cites this paper as [116]
Cites
The publisher hasn’t deposited this paper’s reference list.
Cited by
Sorted by citations
Kimi K3: Open Frontier Intelligence
Kimi Team, Bai, T., Bai, Y., Bao, Y., C., M., Cai, J., Cai, X., Cao, P., Cao, Y., Chai, Z., et al. (2026) · arXiv · doi:10.48550/arxiv.2607.24653
“…[70], MCPMark-Verified [135], MCP-Atlas [11], AutomationBench [109], JobBench [71], GDPval-AA v2 [91], AA-Briefcase [8, 2], Agents’ Last Exam (ALE) [4, 117], APEX-Agents [129], OfficeQA Pro [88], SpreadsheetBench 2 [152], OSWorld-Verified [138] and OSWorld 2.0 [145], SaaS-Bench [110], τ 3 -Banking…”
InfluentialFinding the Right Fit: Model-Harness Interactions across Agent Tasks
Li, Y., Zhou, Y., Teng, Y. L., Yang, F., Deng, Y., Lyu, Z., Dong, X., Chen, F., & An, B. (2026) · arXiv
Self-citation
- Authors
- Sun, Y., Han, X., Zhang, W., Pang, Y., Wang, T., Cao, Y., Huang, Y., Liu, X., Di, Z., Duroiu, C., Zhang, H., Lin, J., Zhang, W., Zeng, T., Yan, Y., Liu, B., Wen, H., Xu, M., Liu, X., Chen, Z., Zhang, Z., Shi, W., Dsouza, A., Chen, V. S., Xie, J., Lei, D., Zhang, H., Wang, Y., Zhang, X., Zhou, C., Huang, S., Bryant, P., Boettiger, C., Rangan, Y., Rothenberg, B., Steinfeld, K., Rao, A., Schneider, T., Yannakakis, G., Zanna, L., Ozbay, K., Sim, I., Zohdi, T., Karniadakis, G. E., Gallant, J., Head-Gordon, T., Liu, C. Y., Hu, R., Cheng, Y., Bacho, A., Qin, Z., Yuille, A., Chen, Y., Liu, H., Zeng, L., Chen, B., Li, Z., Gong, L., Bharadwaj, S. M., Jansen, T., Tang, H., Tu, J., Liu, J., Song, M., Wang, R., Cao, S., Zhang, Y., Yang, Y., Zhang, Z., Bao, H., Kim, H., Xu, P., Lu, S., Zhang, Z., Wang, Z., Prakash, A., Lyu, B., Ma, C., Chen, H., Fan, H., Tao, M., Sitelew, R., Poldrack, R. A., Liu, Y., Liu, Y., Wu, Y., Lin, Y., Li, Z., Wei, Z., Chen, Z., Bendahi, A., Schubert, A., Broderick, C., Liu, C., Sun, H., Li, H., Hua, H., Ding, J., Huang, J., Gu, S., Vinh, T., Sun, X., Jiang, Y., Hu, Y., Xu, Z., Yang, Z., Wang, C., Spanos, C., Yang, D., Wang, H., Cheng, J., Wang, J., Peng, J., Zhang, K., Kuang, K., Sun, L., Yin, M., Xu, O., Siu, V., Liao, W., Shi, W., Deng, W., Zhang, X., Zhu, Y., Shen, Y., Jiang, Y., Li, Y., Pan, Y., Li, Y., Zhu, Y., Zhang, Y., Yang, Z., Rao, Z., Rafiei, A., Glover, A., Ren, B., Yuan, B., Cai, C., Lei, D., Xing, H., Jin, H., Gomez, H., Chandok, I. S., Hu, J., Sun, J., Purnomo, J., Sun, K., Ding, L., Liao, L., Kim, L., Liu, M., Wuwu, Q., Jin, R., Zhang, R., Sun, T., Huang, W., Zhao, X., Chen, P., Jin, C., Bao, Q., Shen, X., Zhang, Y., Li, Y., Wang, Y., Wang, Y., Yang, Y., Dong, Z., Yan, Z., Xu, Z., Zhao, Z., Jiang, Z., Emami, A., Liu, A. Z., Wu, A., Caliskan, A., Huang, B., Huang, B., Yang, B., Zhang, B., Zhao, B., Bosio, C., Guo, C., Liu, C., Du, C., Dai, C., Zhang, C., Li, C., Zhu, C., Zhang, C., Park, C. F., Li, D., Wu, D., Wong, D., Kasson, E., Wu, F., Xu, F., Shen, F., Hu, H., Li, H., Zhou, H., Szlyk, H., Zhang, H., Zhang, H., Fan, H., Ye, H., Shang, H., Bai, H., Fu, H., Li, H., Wang, H., Li, J., Wang, J., Liu, J., Shen, J., Zuo, J., Fan, J., Li, J., Yang, J., So, J., Overwiening, J., Cavanagh, J., Lee, J., Zhang, J., Liu, K., Dhole, K., Lin, K. Q., Zhang, K., Montgomery, K., Paraboschi, L., Liana, Liu, L., Zhou, L., Lu, L., Wu, M. "., Tolonen, M., Yan, M., Shi, M., Niu, M., Lu, P., Cavazos-Rehg, P., Zhao, Q., Mang, Q., Jaiswal, R., Xu, R., Fan, R.-Z., Ning, R., Antu, S. B., Rankin, S., Yang, S., Gu, S., Qiu, S., Zhu, S., Xu, S., Zhu, S., Zhang, S., Sun, S., De Castro, S., Rajwal, S., Shi, T., Chen, T., Zhou, T., Wang, T., Shaw, V., Susoy, V., Xiao, W., Wang, W., Lv, W., Zhang, W., Zhang, W., Liang, W., Yu, X., Li, X., Yang, X., Lan, X., Wang, X., Liu, X., Chen, X., Pan, X., Zhu, Y., Lyu, Y., Gu, Y., Zhao, Y., Zhang, Y., Ren, Y., Chen, Y., Rong, Y., Shao, Y., Zhu, Y., Zhang, Y., Xiong, Y., Li, Y., Wu, Y., Zheng, Y., Wu, Y., Deng, Y., Li, Y., Wu, Y., Xu, Y., Nie, Y., Wang, Z., Du, Z., Yu, Z., Jiang, Z., Li, Z., Li, Z., Yi, Z., Sun, T., Wang, Z., Xu, J., Bo, S., Casademunt, H., Chen, W., Cheng, C., Deng, N., Fu, R., Fu, T., Han, Y., He, Z., Jin, Q., Li, L., Li, Y., Liu, S., Mukherjee, S., Ouyang, Y., Ren, H., Shi, D., Wu, H., Wu, Z., Yao, H., Yu, J., Zhan, R., Zhou, L., Zhu, B., Zhu, J., Das, S., Gao, Y., Guo, W., Schneider, E. J., Liu, B., Lu, Z., Ma, Y., Mueller, M., Poovendran, R., Sojoudi, S., Yao, H., Zhu, Y., Zsolnai-Fehér, K., Denisov-Blanch, Y., Hu, C., Koc, V., Yin, H., Xia, N., Bai, X., Dilbagi, A. S., Gu, J., Liu, Y., Shi, L., Zhang, Q., Hao, Y., Li, X., Li, X., You, B., Ke, W., Zhang, D., Nemet, A., Zhang, T., Lyu, X., Chawla, P., Winata, G. I., Tang, M., Chiniya, P., Chen, Q., Dillmann, S., Xu, W., Yang, W., Wu, X., Luo, X., Huang, Y., & Song, D.
- ORCID
- Zhang, Weichen
- Di, Zonglin
- Dsouza, Amanda
- Xie, Jingxu
- Steinfeld, Kyle
- Sim, Ida
- Bacho, Aras
- Qin, Zengyi
- Jansen, Thorben
- Tu, Jianhong
- Wang, Ruheng
- Bao, Honglin
- Lyu, Bohan
- Bendahi, Abderrahim
- Schubert, Alexander
- Gu, Shangding
- Sun, Xiatao
- Liao, Wanzhe
- Rafiei, Alireza
- Ren, Bixuan
- Chandok, Ishaan Singh
- Wuwu, Qingpo
- Zhao, Brian
- Overwiening, Jorin
- Paraboschi, Layna
- Cavazos-Rehg, Patricia
- Zhao, Qijian
- Mang, Qiuyang
- Jaiswal, Rimika
- Rajwal, Swati
- Susoy, Vladislav
- Lyu, Yicheng
- Zheng, Yuhaohua
- Xu, Yuzheng
- Casademunt, Helena
- Yao, Hannah
- Zhou, Luqing
- Schneider, Erika J
- Ma, Yian
- Sojoudi, Somayeh
- Zsolnai-Fehér, Károly
- Denisov-Blanch, Yegor
- Koc, Vincent
- Zhang, Daoan
- Chawla, Pranit
- Chiniya, Purva
- Dillmann, Steven
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2026
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2606.05405
- OpenAlex
- W7163679042
- Cited by
- 0
- References
- 0
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
- Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- AI agents, benchmark evaluation, long-horizon tasks, task taxonomy, industrial clusters
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
