TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks
Open accessAbstract
We interact with computers on an everyday basis, be it in everyday life or work, and many aspects of work can be done entirely with access to a computer and the Internet. At the same time, thanks to improvements in large language models (LLMs), there has also been a rapid development in AI agents that interact with and affect change in their surrounding environments. But how performant are AI agents at accelerating or even autonomously performing work-related tasks? The answer to this question has important implications both for industry looking to adopt AI into their workflows and for economic policy to understand the effects that adoption of AI may have on the labor market. To measure the progress of these LLM agents' performance on performing real-world professional tasks, in this paper we introduce TheAgentCompany, an extensible benchmark for evaluating AI agents that interact with the world in similar ways to those of a digital worker: by browsing the Web, writing code, running programs, and communicating with other coworkers. We build a self-contained environment with internal web sites and data that mimics a small software company environment, and create a variety of tasks that may be performed by workers in such a company. We test baseline agents powered by both closed API-based and open-weights language models (LMs), and find that the most competitive agent can complete 30% of tasks autonomously. This paints a nuanced picture on task automation with LM agents--in a setting simulating a real workplace, a good portion of simpler tasks could be solved autonomously, but more difficult long-horizon tasks are still beyond the reach of current systems. We release code, data, environment, and experiments on https://the-agent-company.com.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
TheAgentCompany evaluates enterprise AI agents under structured correctness constraints, requiring them to follow predefined organizational policies when completing tasks.
“For example, TheAgentCompany [97] evaluates enterprise AI agents under structured correctness constraints, requiring them to follow predefined organizational policies when completing tasks.”
From Evaluation and Benchmarking of LLM Agents: A Survey (2025) · cites this paper as [97]
Section 4 (Task Structure; page unavailable) describes task checkpoints and evaluators that assess actions, data accuracy, and collaboration, supporting the claim that agents are evaluated against structured correctness criteria. The copy also gives an example involving a departmental budget, but it does not establish that agents are generally required to follow predefined organizational policies when completing tasks. (LLM, conf=0.85)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 26
- Bonatti, R., Zhao, D., Bonacci, F., Dupont, D., Abdali, S., Li, Y., Lu, Y., Wagle, J., Koishida, K., Bucker, A., et al. Windows agent arena: Evaluating multi-modal os agents at scale. arXiv preprint arXiv:2409.08264, 2024. · arXiv
- Chezelles, D., Le Sellier, T., Gasse, M., Lacoste, A., Drouin, A., Caccia, M., Boisvert, L., Thakkar, M., Marty, T., Assouel, R., et al. The browsergym ecosystem for web agent research. arXiv preprint arXiv:2412.05467, 2024. · arXiv
- Deng, X., Gu, Y., Zheng, B., Chen, S., Stevens, S., Wang, B., Sun, H., and Su, Y. Mind2web: Towards a generalist agent for the web. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023. URL https://openreview.net/forum? id=kiYqbO3wqw.
- Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. · arXiv
- He, H., Yao, W., Ma, K., Yu, W., Dai, Y., Zhang, H., Lan, Z., and Yu, D. Webvoyager: Building an end-to-end web agent with large multimodal models. arXiv preprint arXiv:2401.13919, 2024. · arXiv
- Huang, K.-H., Prabhakar, A., Dhawan, S., Mao, Y., Wang, H., Savarese, S., Xiong, C., Laban, P., and Wu, C.-S. Crmarena: Understanding the capacity of llm agents to perform professional crm tasks in realistic environments. arXiv preprint arXiv:2411.02305, 2024. · arXiv
- Jang, L., Li, Y., Ding, C., Lin, J., Liang, P. P., Zhao, D., Bonatti, R., and Koishida, K. Videowebarena: Evaluating long context multimodal agents with video understanding web tasks, 2024. URL https://arxiv.org/abs/2410.19100. · arXiv
- Jimenez, C. E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., and Narasimhan, K. R. SWE-bench: Can Language Models Resolve Real-world Github Issues? In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=VTF8yNQM66.
- Kambhampati, S., Valmeekam, K., Guan, L., Verma, M., Stechly, K., Bhambri, S., Saldyt, L., and Murthy, A. Llms can't plan, but can help planning in llm-modulo frameworks. arXiv preprint arXiv:2402.01817, 2024. · arXiv
- Koh, J. Y., Lo, R., Jang, L., Duvvur, V., Lim, M., Huang, P.-Y., Neubig, G., Zhou, S., Salakhutdinov, R., and Fried, D. VisualWebArena: Evaluating multimodal agents on realistic visual web tasks. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, 2024. URL https://aclanthology.org/ 2024.acl-long.
- Li, B., Wu, W., Tang, Z., Shi, L., Yang, J., Li, J., Yao, S., Qian, C., Hui, B., Zhang, Q., et al. Devbench: A comprehensive benchmark for software development. arXiv preprint arXiv:2403.08604, 2024. · arXiv
- Liu, E. Z., Guu, K., Pasupat, P., Shi, T., and Liang, P. Reinforcement learning on web interfaces using workflow-guided exploration. In International Conference on Learning Representations (ICLR), 2018. URL https://arxiv.org/abs/1802.08802. · arXiv
- Lù, X. H., Kasner, Z., and Reddy, S. Weblinx: Real-world website navigation with multi-turn dialogue, 2024. URL https://arxiv.org/abs/2402.05930. · arXiv
- Patil, S. G., Zhang, T., Wang, X., and Gonzalez, J. E. Gorilla: Large language model connected with massive apis. arXiv preprint arXiv:2305.15334, 2023. · arXiv
- Song, Y., Xu, F., Zhou, S., and Neubig, G. Beyond browsing: Api-based web agents. arXiv preprint arXiv:2410.16464, 2024. · arXiv
- Team, G., Anil, R., Borgeaud, S., Alayrac, J.-B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A. M., Hauth, A., Millican, K., et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023. · arXiv
- Wang, X., Chen, Y., Yuan, L., Zhang, Y., Li, Y., Peng, H., and Ji, H. Executable Code Actions Elicit Better LLM Agents. In ICML, 2024a.
- Wang, X., Li, B., Song, Y., Xu, F. F., Tang, X., Zhuge, M., Pan, J., Song, Y., Li, B., Singh, J., Tran, H. H., Li, F., Ma, R., Zheng, M., Qian, B., Shao, Y., Muennighoff, N., Zhang, Y., Hui, B., Lin, J., Brennan, R., Peng, H., Ji, H., and Neubig, G. Openhands: An open platform for ai software developers as generalist agents, 2024b. URL https://arxiv.org/abs/2407.16741. · arXiv
- Xie, T., Zhang, D., Chen, J., Li, X., Zhao, S., Cao, R., Hua, T. J., Cheng, Z., Shin, D., Lei, F., Liu, Y., Xu, Y., Zhou, S., Savarese, S., Xiong, C., Zhong, V., and Yu, T. OSWorld: Benchmarking multimodal agents for open-ended tasks in real computer environments. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024. URL https://openreview.net/forum?id=tN61DTr4Ed.
- Yang, A., Yang, B., Hui, B., Zheng, B., Yu, B., Zhou, C., Li, C., Li, C., Liu, D., Huang, F., et al. Qwen2 technical report. arXiv preprint arXiv:2407.10671, 2024. · arXiv
- Yang, J., Zhang, H., Li, F., Zou, X., Li, C., and Gao, J. Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v. arXiv preprint arXiv:2310.11441, 2023. · arXiv
- Yao, S., Shinn, N., Razavi, P., and Narasimhan, K. τ -bench: A benchmark for tool-agent-user interaction in real-world domains. arXiv preprint arXiv:2406.12045, 2024. · arXiv
- Yoran, O., Amouyal, S. J., Malaviya, C., Bogin, B., Press, O., and Berant, J. Assistantbench: Can web agents solve realistic and time-consuming tasks?, 2024. URL https://arxiv.org/abs/2407.15711. · arXiv
- Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E., et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595-46623, 2023.
- Zhou, S., Xu, F. F., Zhu, H., Zhou, X., Lo, R., Sridhar, A., Cheng, X., Ou, T., Bisk, Y., Fried, D., et al. Webarena: A realistic web environment for building autonomous agents. In The Twelfth International Conference on Learning Representations, 2023.
- Zhou, X., Zhu, H., Mathur, L., Zhang, R., Yu, H., Qi, Z., Morency, L.-P., Bisk, Y., Fried, D., Neubig, G., and Sap, M. SOTOPIA: Interactive evaluation for social intelligence in language agents. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=mM7VurbA4r.
Cited by
Sorted by citations
Evaluation and Benchmarking of LLM Agents: A Survey
Mohammadi, M., Li, Y., Lo, J., & Yip, W. (2025) · Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2 · doi:10.1145/3711896.3736570 · Cited by 74 · see citing works
How malicious AI swarms can threaten democracy
Schroeder, D. T., Cha, M., Baronchelli, A., Bostrom, N., Christakis, N. A., Garcia, D., Goldenberg, A., Kyrychenko, Y., Leyton-Brown, K., Lutz, N., et al. (2026) · Science · doi:10.1126/science.adz1697 · Cited by 25 · see citing works
Measuring AI Ability to Complete Long Software Tasks
Kwa, T., West, B., Becker, J., Deng, A., Garcia, K., Hasin, M., Jawhar, S., Kinniment, M., Rush, N., Von Arx, S., et al. (2025) · arXiv · doi:10.48550/arxiv.2503.14499 · Cited by 14 · see citing works
Are Autonomous Web Agents Good Testers?
Chevrot, A., Vernotte, A., Falleri, J., Blanc, X., Legeard, B., & Cretin, A. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3728879 · Cited by 11 · see citing works
Cao, H., Ma, R., Zhai, Y., & Shen, J. (2024) · Applied Computing and Intelligence · doi:10.3934/aci.2024019 · Cited by 7 · see citing works
Humanist in the Loop: Teaching Critical AI Literacies, Episode 1
Goodlad, L. M. E. (2025) · Critical AI · doi:10.1215/2834703x-11908331 · Cited by 5 · see citing works
Gao, H.-A., Geng, J., Hua, W., Hu, M., Juan, X., Liu, H., Liu, S., Qiu, J., Qi, X., Wu, Y., et al. (2025) · arXiv · doi:10.48550/arxiv.2507.21046 · Cited by 3 · see citing works
Self-citationMeasuring AI Ability to Complete Long Software Tasks
Kwa, T., West, B., Becker, J., Deng, A., Garcia, K., Hasin, M., Jawhar, S., Kinniment, M., Rush, N., Von Arx, S., et al. (2025) · Advances in Neural Information Processing Systems 38 · doi:10.52202/085713-3086 · Cited by 1 · see citing works
“Agentic AI capability benchmarks Many recent benchmarks aim to measure the capability of AI systems to act as agents in general, including AgentBench [9], ToolBench [10], GAIA [11], TheAgentCompany [12], the Berkeley Function Calling Leaderboard [13].”
SWE-Marathon: Can Agents Autonomously Complete Ultra-Long-Horizon Software Work?
Desai, R., Hu, J., Cabezas, J., Harsola, N., Shukla, P., Chaim, R. B., Assadi, A. E., Kamath, O. M., Faldu, F., Hebbar, P., et al. (2026) · arXiv · doi:10.48550/arxiv.2606.07682
HCAST: Human-Calibrated Autonomy Software Tasks
Rein, D., Becker, J., Deng, A., Nix, S., Canal, C., O'Connel, D., Arnott, P., Bloom, R., Broadley, T., Garcia, K., et al. (2025) · arXiv · doi:10.48550/arxiv.2503.17354
“TheAgentCom-pany (Xu et al., 2024) creates tasks that simulate work done in a small software company, with Claude-3.”
Better Harnesses, Smaller Models: Building 90% Cheaper Agents via Automated Harness Adaptation
Yang, C., Zhao, X., Wu, T., & Kästner, C. (2026) · arXiv · doi:10.48550/arxiv.2607.08938
OpenAgentSafety: A Comprehensive Framework for Evaluating Real-World AI Agent Safety
Vijayvargiya, S., Soni, A. B., Zhou, X., Wang, Z. Z., Dziri, N., Neubig, G., & Sap, M. (2025) · arXiv · doi:10.48550/arxiv.2507.06134
“We use locally hosted instances of OwnCloud (file sharing), GitLab (version control), and Plane (issue tracking), adapted from The Agent Company [36].”
Self-citationMS-Eval: Benchmarking Sub-Agent Delegation and Orchestration in MCP-Intensive Working Scenarios
Ding, B., Wang, Q., Ruan, L., Lu, W., Ge, Q., Fang, R., & Lin, T. (2026) · arXiv · doi:10.48550/arxiv.2610.22869
“Driven by the scaling of parameters and data (Kaplan et al., 2020; Ding et al., 2026), LLMs have evolved into tool-empowered autonomous agents (GLM-5-Team et al., 2026; DeepSeek-AI, 2026) that are increasingly integrated into daily works to augment human productivity (Xu et al., 2026).”
Seemann, M. (2026) · Springer eBooks · doi:10.1007/978-3-658-50335-2_11
"Here, Let Me Help": An Empirical Study of User Interventions in Human–Web Agent Collaboration
Kim, J., Cho, S., Nguyen, D. M., Jeon, J., Shin, M., & Ko, S. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791536
ClaimPilot: An Open Multi-Agent Pipeline for Veterinary Insurance Claims Adjudication
Hoang, M. L., & Shunmugam, N. (2026) · Future Internet · doi:10.3390/fi18090465
Xie, H., Sun, Q.-R., & Zhang, D.-H. (2026) · 2026 International Conference on Large Models and Intelligent Agent Technologies (LMIAT) · doi:10.1109/lmiat70705.2026.11682665
iBRAHIMOĞLU, N., Demi̇rezen, M. U., & Yıldız, F. (2026) · Cluster Computing · doi:10.1007/s10586-026-06537-4
A Survey of Evaluation Methods for the Security and Trustworthiness of Large Language Models
Zhao, L., Xu, Z., Zhao, R., Li, C., Yang, B., & Feng, D. (2026) · International Conference on Machine Learning Technologies (ICMLT) · doi:10.1109/icmlt69916.2026.11689272
Lv, Z., Xiang, L., Wang, D., Wu, H., Wang, Z., & He, Z. (2026) · International Conference on Artificial Intelligence, Internet of Things and Cloud Computing Technology (AIoTC) · doi:10.1109/aiotc71250.2026.11688714
Self-citationIDV: Customer-Context-Aware Multi-Engine Architecture for AI-Agent Intent Drift Detection in Finance
Zhou, M., & Xue, Y. (2026) · IEEE Conference on Computational Intelligence for Financial Engineering & Economics · doi:10.1109/cifer67845.2026.11692374
Liu, J., & Zhang, S. (2026) · International Conference on Parallel Processing Workshops (ICPPW) · doi:10.1145/3816891.3834855
From Chatbot Hype to Enterprise Reality
Hossain, M. M., & Ponnusamy, A. A. N. (2026) · Apress eBooks · doi:10.1007/979-8-8688-3118-8_2
A survey on AI agent security: Reasoning, Acting, and Self-Evolving
Liu, D., Luo, T., Huang, S., Yang, J., Liang, S., Jia, X., Lau, C. P., Dong, J., Liu, Y., Bai, Y., et al. (2026) · AI Plus · doi:10.59717/ipj.aiplus.2026.100004
Self-citationGodakanda, S., Vidanage, K., Bhattacharyya, S., & Nirkhi, D. S. (2026) · International Conference on Computing Communication Control and automation · doi:10.1109/iccubea69821.2026.11702006
Conversational Inventory Management: A Proof-of-Concept for LLM-Mediated ERP Inventory Operations
Assidiqi, M. H., Alghazzawi, D., & Alarifi, S. (2026) · International Electronics Symposium (IES) · doi:10.1109/ies71033.2026.11702636
LLM-based agents for generalized web automation and schema-guided data extraction: a survey
Khayoub, I., Chadi, M.-A., Mousannif, H., & Mohamed, F. A. (2026) · International Journal of Data Science and Analytics · doi:10.1007/s41060-026-01335-y
- Authors
- Xu, F. F., Song, Y., Li, B., Tang, Y., Jain, K., Bao, M., Wang, Z. Z., Zhou, X., Guo, Z., Cao, M., Yang, M., Lu, H. Y., Martin, A., Su, Z., Maben, L., Mehta, R., Chi, W., Jang, L., Xie, Y., Zhou, S., & Neubig, G.
- Affiliations
- Microsoft (United States) (US)
- Carnegie Mellon University (US)
- Oracle (United States) (US)
- University of Hong Kong (HK)
- Alibaba Group (China) (CN)
- Meta (United Kingdom) (GB)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2024
- Pages
- 10291-10321
- ISSN
- 1049-5258 (ISSN-L)
- arXiv
- 2412.14161
- OpenAlex
- W4405627213
- Cited by
- 17
- References
- 0
- Citations by year
- 2026: 14 · 2025: 2 · 2024: 1
- FWCI
- 15.19
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- neural information processing systems (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- Open Philanthropy Project
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
- Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- AI agents, large language models, benchmarking, task automation, digital workers, workplace simulation, web browsing, writing software, long-horizon tasks, open-weight language models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
