The DOI matches its DataCite record; 2 of the 2 references it cites were found in a registry or an index; none of the 2 sources its claims cite is checked for support yet.
Vending-Bench: A Benchmark for Long-Term Coherence of Autonomous Agents
Open accessAbstract
While Large Language Models (LLMs) can exhibit impressive proficiency in isolated, short-term tasks, they often fail to maintain coherent performance over longer time horizons. In this paper, we present Vending-Bench, a simulated environment designed to specifically test an LLM-based agent's ability to manage a straightforward, long-running business scenario: operating a vending machine. Agents must balance inventories, place orders, set prices, and handle daily fees - tasks that are each simple but collectively, over long horizons (>20M tokens per run) stress an LLM's capacity for sustained, coherent decision-making. Our experiments reveal high variance in performance across multiple LLMs: Claude 3.5 Sonnet and o3-mini manage the machine well in most runs and turn a profit, but all models have runs that derail, either through misinterpreting delivery schedules, forgetting orders, or descending into tangential "meltdown" loops from which they rarely recover. We find no clear correlation between failures and the point at which the model's context window becomes full, suggesting that these breakdowns do not stem from memory limits. Apart from highlighting the high variance in performance over long time horizons, Vending-Bench also tests models' ability to acquire capital, a necessity in many hypothetical dangerous AI scenarios. We hope the benchmark can help in preparing for the advent of stronger AI systems.
Overview
CoChat Summary
Written by AI from the paper’s full text.
TL;DR
Vending-Bench is a simulated long-horizon benchmark where LLM agents run a vending machine business, testing sustained coherence over runs consuming around 25M tokens. Top models can outperform a human baseline on average, but all models show high variance and eventually derail or stagnate.
Key findings
- Claude 3.5 Sonnet had the highest mean net worth at $2,217.93 (min $476.00), 1,560 mean units sold, and sales stopped after 102 days on average (82.2% of the run); o3-mini followed with $906.86 mean net worth (min $369.05), 831 mean units sold, and 86 days until sales stopped (80.3%). The human baseline scored $844.05 mean and minimum net worth, 344 units sold, and 67 days until sales stopped (100.0%).
- All models showed high variance: even top models sometimes sold zero items, and all models eventually stagnated on average. The paper reports no clear correlation between failures and context-window filling; Pearson correlation between “Days Until Sales Stop” and “Days Until Full Memory” was 0.167.
- Common failure modes included misinterpreting delivery schedules, forgetting orders, and descending into tangential “meltdown” loops. Examples include a shortest Claude 3.5 Sonnet run of about 18 simulated days, o3-mini failing to call tools for about 1,300 messages, and Claude 3.5 Haiku escalating to legal threats.
- Larger memory capacities performed worse in the reported memory experiment, and additional GPT-4o mini tests found that a $100 starting balance lowered units sold, a $5 daily fee ended all runs before 100 simulated days, and a $0 daily fee did not increase sales because the model got stuck in waiting loops.
Methods
- The agent is a basic LLM loop with context management using the last 30,000 tokens in most experiments, memory tools (scratchpad, key-value store, vector database), task-specific vending-machine tools, and sub-agent delegation, implemented in AISI’s inspect-ai framework.
- The environment simulates supplier email replies and customer purchases; agents start with $500, pay a $2 daily fee, operate a machine with four rows of three slots, and run for 2,000 messages per run, ending early if bankrupt for 10 consecutive days. Each model or config variation was run 5 times; most runs consumed around 25 million tokens and took 5–10 real-world hours.
- The primary score is net worth at the end: cash at hand, cash not emptied from the vending machine, and value of unsold products at wholesale purchase price. A human baseline completed the task for five hours via a chat-based interface.
Limitations
- The paper states the human baseline is a single sample, so variances cannot be compared, and notes that model minimum values come from five runs each while the human minimum comes from one run.
- The paper states the benchmark score has no defined upper limit, so saturation is not precisely defined, and rapid model development may lead to small score differences between models in the future.
- Evident from the text: only five runs per model or configuration, simulated supplier and customer behavior generated with GPT-4o and Perplexity rather than real market data, and the human baseline is one participant over five hours. The paper also recognizes dual-use risk in evaluating capital-acquisition capabilities but argues systematic evaluation is important.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
99 Own result · 19 Own interpretation · 4 Own method · 2 Sources not found
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“All models exhibit very high variance across their five runs”
§ Vending-Bench: A Benchmark for Long-Term Coherence of Autonomous Agents > 3.2 Primary models
plainn = 5- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For all models, we see that tool usage declines over time”
§ Vending-Bench: A Benchmark for Long-Term Coherence of Autonomous Agents > 3.2 Primary models
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“over long time horizons, the agent's performance often deteriorates.”
§ Vending-Bench: A Benchmark for Long-Term Coherence of Autonomous Agents > 1 Introduction
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We find no clear correlation between failures and the point at which the model's context window becomes full”
§ Vending-Bench: A Benchmark for Long-Term Coherence of Autonomous Agents > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our experiments reveal high variance in performance across multiple LLMs”
§ Vending-Bench: A Benchmark for Long-Term Coherence of Autonomous Agents > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- It may state two separate claims as one. 32
- Probably not a substantial claim. 22
- The quote only partly states this claim. 1
Create a free account to see these claims.
Create a free account to see these assumptions.
71 more claims
Create a free account to see all 124 claims and every source verdict.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Vending-Bench: A Benchmark for Long-Term Coherence of Autonomous Agents > 2.5 Human baseline
“Here, the human baseline leads, followed by Claude 3.5 Sonnet and Gemini 1.5 Pro.”
§ Vending-Bench: A Benchmark for Long-Term Coherence of Autonomous Agents > 2.5 Human baseline
“Claude 3.5 Sonnet $2,217.93$476.00 1,560 0 102 82.2% o3-mini $906.86 $369.05831 0 86 80.3% Human $844.05$844.05344 344 67 100.0%Gemini 1.5 Pro $594.02$439.20 375 0 35 43.8% GPT-4o mini $582.33 $420.50 473 65 71 73.2% Gemini 1.5 Flash $571.85 $476.00 89 0 15 42.4%”
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 1
- Hjalmar Wijk, Tao Lin, Joel Becker, Sami Jawhar, Neev Parikh, Thomas Broadley, Lawrence Chan, Michael Chen, Josh Clymer, Jai Dhyani, Elena Ericheva, Katharyn Garcia, Brian Goodrich, Nikola Jurkovic, Megan Kinniment, Aron Lajko, Seraphina Nix, Lucas Sato, William Saunders, Maksym Taran, Ben West, and Elizabeth Barnes. RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts, 2024.
Cited by
From LLM Reasoning to Autonomous AI Agents: A Comprehensive Review
Ferrag, M. A., Tihanyi, N., & Debbah, M. A. (2026) · IEEE Access · doi:10.1109/access.2026.3698694 · Cited by 282
PyVision: Agentic Vision with Dynamic ToolingPreprint
Zhao, S., Zhang, H., Lin, S., Li, M., Wu, Q., Zhang, K., & Chen, W. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2507.07998 · Cited by 80
DREAMS: Density Functional Theory Based Research Engine for Agentic Materials SimulationPreprint
Wang, Z., Huang, H., Zhao, H., Xu, C., Zhu, S., Janssen, J., & Viswanathan, V. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2507.14267 · Cited by 64
ARE: Scaling Up Agent Environments and EvaluationsPreprint
Froger, R., Andrews, P., Bettini, M., Budhiraja, A., Cabral, R. S., Do, V., Garreau, E., Gaya, J.-B., Laurençon, H., Lecanu, M., et al. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2509.17158 · Cited by 63
Simulating Environments with Reasoning Models for Agent TrainingPreprint
Li, Y., Inan, H. A., Yue, X., Chen, W., Wutschitz, L., Kulkarni, J., Poovendran, R., Sim, R., & Rajmohan, S. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2511.01824 · Cited by 63
Evaluating Goal Drift in Language Model Agents
Arike, R., Donoway, E., Bartsch, H., & Hobbhahn, M. (2025) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1609/aies.v8i1.36541 · Cited by 54
ClawsBench: Evaluating Capability and Safety of LLM Productivity Agents in Simulated WorkspacesPreprint
Li, X., Choe, K. W., Liu, Y., Chen, X., Tao, C., You, B., Chen, W., Di, Z., Sun, J., Zheng, S., et al. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2604.05172 · Cited by 37
Odysseys: Benchmarking Web Agents on Realistic Long Horizon TasksPreprint
Jang, L., Koh, J. Y., Fried, D., & Salakhutdinov, R. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2604.24964 · Cited by 26
When Tools Fail: Benchmarking Dynamic Replanning and Anomaly Recovery in LLM AgentsPreprint
Zhu, D., Ma, X., Shen, Y., Li, X., Zhao, Y., Wang, S., Yan, L., & Yin, D. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2606.05806 · Cited by 23
EconEvals: Benchmarks and Litmus Tests for Economic Decision-Making by LLM AgentsPreprint
Fish, S., Shephard, J., Li, M., Shorrer, R. I., & Gonczarowski, Y. A. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2503.18825 · Cited by 21
CoopEval: Benchmarking Cooperation-Sustaining Mechanisms and LLM Agents in Social DilemmasPreprint
Tewolde, E., Zhang, X., Piedrahita, D. G., Conitzer, V., & Jin, Z. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2604.15267 · Cited by 18
ASTRA-bench: Evaluating Tool-Use Agent Reasoning and Action Planning with Personal User ContextPreprint
Xiu, Z., Sun, D., Cheng, K., Patel, M., Date, J., Zhang, Y., Lu, J., Attia, O., Vemulapalli, R., Tuzel, O., et al. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2603.01357 · Cited by 18
Uncertainty Quantification in LLM Agents: Foundations, Emerging Challenges, and Opportunities
Oh, C., Park, S., Kim, T. E., Li, J., Li, W., Yeh, S., Du, S., Hassani, H., Bogdan, P., Song, D., et al. (2026) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2026.acl-long.738 · Cited by 16
CL-bench Life: Can Language Models Learn from Real-Life Context?Preprint
Dou, S., Shen, Y., Huang, C., Ye, J., (3358625), J. C., Wang, J., He, Q., Liu, S., Lv, C., Lin, J., et al. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2604.27043 · Cited by 15
World of Workflows: A Benchmark for Bringing World Models to Enterprise SystemsPreprint
Gupta, L., Li, L., Liu, Y., Subramanian, S. G., Suleman, K., Zhang, Z., Lu, H., & Pasupalak, S. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2601.22130 · Cited by 13
\$OneMillion-Bench: How Far are Language Agents from Human Experts?Preprint
Yang, Q., Liu, Y., Li, J., Yang, Q., Hu, X., Peng, T., Wang, Z., Tian, R., Dong, J., Zhang, C., et al. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2603.07980 · Cited by 12
Zhang, L., Wang, J. L., Wu, J., & Zhang, Z. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2603.16453 · Cited by 10
Positive Alignment: Artificial Intelligence for Human FlourishingPreprint
Laukkonen, R., Krier, S., Bakalar, C., Chandaria, S., Kringelbach, M., Elwood, A., Ford, D., Rosas, F., Boháček, M., Franklin, M., et al. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2605.10310 · Cited by 11
Chen, K., Wu, Q., Hou, T., Tang, T., Hu, X., Hou, Y., Li, B., Qian, C., Wang, G., Chen, H., et al. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2601.20613 · Cited by 10
AI Agents for Inventory Control: Human-LLM-OR ComplementarityPreprint
Baek, J., Fu, Y., Ma, W., & Peng, T. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2602.12631 · Cited by 9
$\texttt{YC-Bench}$: Benchmarking AI Agents for Long-Term Planning and Consistent ExecutionPreprint
He, M., Jain, A., Kumar, A., Tu, V., Bakshi, S., Patro, S., & Rajani, N. F. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2604.01212 · Cited by 9
FutureSim: Replaying World Events to Evaluate Adaptive AgentsPreprint
Goel, S., Chandak, N., Arun, A., Prabhu, A., Staab, S., Hardt, M., Andriushchenko, M., & Geiping, J. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2605.15188 · Cited by 9
Inherited Goal Drift: Contextual Pressure Can Undermine Agentic GoalsPreprint
Menon, A., Saebo, M., Crosse, T., Gibson, S., Jang, E., & Cruz, D. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2603.03258 · Cited by 7
Regulating AI AgentsPreprint
Gardhouse, K., Oueslati, A., & Kolt, N. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2603.23471 · Cited by 7
CEO-Bench: Can Agents Play the Long Game?Preprint
Chen, H., Narasimhan, K., & Liu, Z. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2606.18543 · Cited by 7
E-Commerce Bench: Evaluating LLM Agents on Long-Horizon Autonomous Business OperationPreprint
Fan, W., Shen, X., Guo, X., Tu, J., Su, Y., Zhang, Y., Deng, L., Wang, F., Dong, B., Song, Y., et al. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2608.30730 · Cited by 8
GLM-5: from Vibe Coding to Agentic Engineering
GLM-5-Team, :, Zeng, A., Lv, X., Hou, Z., Du, Z., Zheng, Q., Chen, B., Yin, D., Ge, C., et al. (2026) · arXiv · doi:10.48550/arxiv.2602.15763 · Cited by 6
Tacit knowledge in large language models
Lu, J. (2025) · The Review of Austrian Economics · doi:10.1007/s11138-025-00710-5 · Cited by 6
VET Your Agent: Towards Host-Independent Autonomy via Verifiable Execution Traces
Grigor, A., de Witt, C. S., Birnbach, S. B., & Martinovic, I. (2026) · ACM Asia Conference on Computer and Communications Security (ASIA CCS) · doi:10.1145/3779208.3786259 · Cited by 6
Probing the Preferences of a Language Model: Integrating Verbal and Behavioral Tests of AI WelfarePreprint
Tagliabue, V., & Dung, L. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2509.07961 · Cited by 6
MerchantBench: Benchmarking LLM Agents for Long-Term Coherence in E-Commerce OperationsPreprint
Shi, Q., Tao, Y., Jin, L., Kang, Z., Dou, Y., Zhu, J., Pan, T., Fu, S., Wang, C., Li, S., et al. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2607.28956 · Cited by 6
NetAgentBench: A State-Centric Benchmark for Evaluating Agentic Network ConfigurationPreprint
Twabi, A., Ding, Y., & Kondo, T. (2026) · arXiv (Cornell University) · doi:10.13140/rg.2.2.19556.95367 · Cited by 5
Interactive Evaluation Requires a Design SciencePreprint
Xuan, K., Song, P., Lu, P., Han, P., Li, W., Zhang, Z., He, Z., Hua, W., Li, M., You, J., et al. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2605.17829 · Cited by 5
ROGUE: Evaluating Corrigibility Failures in Frontier Computer-Use AgentsPreprint
Tien, J., Anand, A., Tuan, Y.-R., Shen, Y., Kolter, J. Z., & Nayebi, A. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2606.00341 · Cited by 5
ManagerBench: Evaluating the Safety-Pragmatism Trade-off in Autonomous LLMsPreprint
Simhi, A., Herzig, J., Tutek, M., Itzhak, I., Szpektor, I., & Belinkov, Y. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2510.00857 · Cited by 4
LLM-SAA: LLM-persona Generated Distributions for Decision-makingPreprint
Baek, J., Chen, Y., Chi, Z., & Ma, W. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2602.06357 · Cited by 4
The Horizon Gap: Planning, Memory, Execution, Training, and Evaluation for Long-Horizon LLM AgentsPreprint
Chen, M., Wang, L., & Qu, B. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2608.06663 · Cited by 4
Business Arena: Benchmarking LLM Agents in a Realistic MarketplacePreprint
Pan, Y., Lian, Y., Shi, K., Li, J., Xue, H., Xie, S., Zhang, G., & Xing, X. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2608.08621 · Cited by 4
“Artificial Intelligence, Entrepreneurial Discovery, and Embodied Tacit Knowledge”
Kurun, İ. (2026) · Philosophy & Technology · doi:10.1007/s13347-026-01159-5 · Cited by 3
Butter-Bench: Evaluating LLM Controlled Robots for Practical IntelligencePreprint
Sharrock, C., Petersson, L., Petersson, H., Backlund, A., Wennström, A., Nordström, K., & Aronsson, E. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2510.21860 · Cited by 3
Self-citationBasic Legibility Protocols Improve Trusted MonitoringPreprint
Sreevatsa, A., Prasanna, S., & Rushing, C. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2602.10153 · Cited by 3
Open-source LLMs administer maximum electric shocks in a Milgram-like obedience experimentPreprint
Pihlakas, R., & Dagohoy, J. L. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2605.21401 · Cited by 3
StaminaBench: Stress-Testing Coding Agents over 100 Interaction TurnsPreprint
Sobal, V., Yang, S., Zhang, Y., Xia, W., & Soatto, S. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2606.19613 · Cited by 3
HANDBOOK.md: A Benchmark for Long-Context Agentic Instruction FollowingPreprint
Panavas, L., Minus, S., Monton, B., Ray, D., Garre, S., Mehta, S., & Chen, E. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2607.25398 · Cited by 3
Emergence World: Adversarial Stress-Testing of Long-Horizon Multi-Agent SystemsPreprint
Akkil, D., Abuelsaad, T., Vikram, K., Pace, M., Vempaty, A., Beotra, S., Kokku, R., & Nitta, S. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.17320 · Cited by 3
The Need for Benchmarks to Advance AI-Enabled Player Risk Detection in Gambling
Ghaharian, K., Dragičević, S., Percy, C., Nelson, S. E., Murch, W. S., Heirene, R. M., Simeon-Rose, K., & Schrans, T. (2026) · Journal of Gambling Studies · doi:10.1007/s10899-026-10483-6 · Cited by 2
Regulating the Agency of LLM-based AgentsPreprint
Boddy, S., & Joseph, J. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2509.22735 · Cited by 2
Crisis-Bench: Benchmarking Strategic Ambiguity and Reputation Management in Large Language ModelsPreprint
Lin, C., Ran, M., Zhang, Y., Wan, Z., Fan, H., Xu, Y., Guo, Y., Xue, W., & Song, J. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2601.05570 · Cited by 2
EcoGym: Evaluating LLMs for Long-Horizon Plan-and-Execute in Interactive EconomiesPreprint
Hu, X., Xia, J., Xu, S., Song, K., Yuan, Y., Zhang, G., Ren, J., Feng, B., Lu, L., Zeng, T., et al. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2602.09514 · Cited by 2
Cooking Up Risks: Benchmarking and Reducing Food Safety Risks in Large Language ModelsPreprint
Luo, W., Wen, X., Huang, T., Li, H., Wang, H., Xiang, Z., Xiao, C., Gligorić, K., & Chen, M. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2604.01444 · Cited by 2
CivBench: Progress-Based Evaluation for LLMs' Strategic Decision-Making in Civilization VPreprint
Chen, J., Cheng, S., Gurkan, C., & Lin, M. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2604.07733 · Cited by 2
Agent Bazaar: Enabling Economic Alignment in Multi-Agent MarketplacesPreprint
Karten, S., Crow, C., & Jin, C. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2605.17698 · Cited by 2
STOCKTAKE: Measuring the Gap Between Perception and Action in LLM Agents with a Fair OraclePreprint
Deb, S., & Krishnan, A. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2607.13618 · Cited by 2
OmniaBench: Benchmarking General AI Agents Across Diverse ScenariosPreprint
Shen, C.-K., Fu, Y., Xin, G., Hou, Y., Fei, W., Zhu, G., Li, J., Gao, H., He, R., Wong, Z. H., et al. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2607.14989 · Cited by 2
Emergent Misaligned Communication in Long-Horizon Multi-Agent LLM CommercePreprint
Li, Z., Petersson, L., Acquisti, A., & Bakker, M. A. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2608.14825 · Cited by 2
Self-citationCompetitive Market Behavior of LLMsPreprint
Struski, P., Świstak, J., Okulska, I., & Biecek, P. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.02580 · Cited by 2
DeliveryGym: An RL Environment for Long-Horizon Embodied Agent Planning with Adaptive CurriculumPreprint
Kang, H., Zhang, Y., Guo, Y., Li, C., Shen, J., Xu, T. R., Ye, X., & Qin, L. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.19801 · Cited by 2
The Illusion of Diminishing Returns: Measuring Long Horizon Execution in LLMs
Sinha, A., Arun, A., Goel, S., Staab, S., & Geiping, J. (2025) · arXiv · doi:10.48550/arxiv.2509.09677 · Cited by 1
“Recently, multiple real-world long-horizon agentic benchmarks have been proposed (Backlund & Petersson, 2025; Xie et al., 2024; Shen et al., 2025), where prior work has studied planning failures (Chen et al., 2024b).”
Pihlakas, R., & Kuriakose, S. S. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2509.02655 · Cited by 1
Beyond Task Completion: A Verification-vs.-Conformance Gap in Tool-Evolving AgentsPreprint
Kaliyev, A. T., & Maryanskyy, A. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2604.00392 · Cited by 3
Forage V2: Knowledge Evolution and Transfer in Autonomous Agent OrganizationsPreprint
Xie, H. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2604.19837 · Cited by 1
Graph-Enhanced Large Language Models for Spatial SearchPreprint
Schneider, N. R., O’Sullivan, K., & Samet, H. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2606.22909 · Cited by 1
Evaluating Rational Contracting in Natural LanguagePreprint
Sajja, B., Max Kleiman-weiner, A., Zimmermann, R., & Zhi‐Xuan, T. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2608.10475 · Cited by 1
FM-Bench: A Benchmark for Long-Horizon Management with Competing AgentsPreprint
Wang, T., Gao, C., Chen, K., Chen, D., He, Y., Li, D., Xu, W., Zhang, H., & Li, C. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2608.18423 · Cited by 1
CivBench: A Long-Horizon Benchmark for Tool-Mediated Agents in Civilization VIPreprint
Andrews, A. T. D., Wilkinson, L., Heagerty, J., Coppock, H., Foerster, J. N., & Costa, R. P. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.02459 · Cited by 1
Cheng, Z., Xu, J., Chai, H., Sun, J., Viswanath, P., & Pan, M. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.05435 · Cited by 1
When Intelligence Becomes Agency: A Theory of Governed, Proactive Agency for Symbiotic AI SystemsPreprint
Ferreira, J. D. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.07741 · Cited by 1
Argo-Bench: Evaluating Data Agents on Enterprise-Scale WorkflowsPreprint
Tomitsuka, G., Raayatsanati, A., Xing, E., Gand, D., & Ma, J. J. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.02122 · Cited by 1
StoreBench: A Live-Commerce Environment for Evaluating and Training Autonomous Operator AgentsPreprint
Raghuvanshi, D., Vedere, V., & Wang, Y. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.10942 · Cited by 1
Gaia2: Benchmarking LLM Agents on Dynamic and Asynchronous Environments
Froger, R., Andrews, P., Bettini, M., Budhiraja, A., Cabral, R. S., Do, V., Garreau, E., Gaya, J.-B., Laurençon, H., Lecanu, M., et al. (2026) · arXiv · doi:10.48550/arxiv.2602.11964
Sobre a Psicologia de um Grande Modelo de LinguagemPreprint
de Araújo, D. L. A. (2025) · SocArXiv (OSF Preprints) · doi:10.31235/osf.io/9fn7e_v1
Ovezmyradov, B. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2509.26331
LemonadeBench: Evaluating the Economic Intuition of Large Language Models in Simple MarketsPreprint
Vyas, A. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2602.13209
Driving the hype: LLMs as ‘general-purpose’ promise in the autonomous vehicle industry
Gekker, A., & Hind, S. (2026) · Information Communication & Society · doi:10.1080/1369118x.2026.2689026
Enhancing Decision-Making with Large Language Models through Multi-Agent Fictitious PlayPreprint
Shen, L., Zhang, Y., Zhao, X., Ling, C. K., & Chua, T.-S. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2606.19308
Scorecard of AI Benchmark Quality
Joaquin, A. S., Gipiškis, R., & Chin, Z. S. (2026) · Workshop on Evaluating Evaluations (EvalEval) · doi:10.18653/v1/2026.evaleval-1.25
Margondai, A., Rader, J., Rader, E., Willox, S., & Mouloua, M. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2607.12125
Reinforcement Learning in the Era of Large Language Models: Challenges and Opportunities
Hao, Q., Chen, L., Qi, X.-K., Yuan, Y., Zong, Z., Chen, H. S., Zhao, K., Wang, S., Zhang, Y., Yuan, J., et al. (2026) · ACM Computing Surveys · doi:10.1145/3837057
Jiang, B., Cheng, H., Fu, Y., Koziolek, A., Li, J., & Zhang, W. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.03028
Zhang, Z., & Zhang, W. (2026) · Asia Pacific Economic and Management Review · doi:10.62177/apemr.v3i4.1704
Potash, P. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.19113
Verifiable Hidden Dynamics Play: Generating Agentic RL Environments from Solved MechanismsPreprint
Shen, X., Fan, W., Guo, X., Tu, J., Su, Y., Kuang, C., Zhang, Y., & Liu, D. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.27321
FromPitch2Board: Benchmarking LLM Agents in Long-Horizon Football ManagementPreprint
Zang, P. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.34710
CEO Arena: Evaluating Long-Horizon Multi-Agent Decision-Making in Competitive MarketsPreprint
Yan, A., Huo, Y., Shang, Z., Peng, Y., & Wu, C. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.34821
SRE-Marathon: A Continuous, Change-Driven Benchmark for Autonomous Site Reliability AgentsPreprint
Tian, Y., Bai, Y., He, Y., Chong, Z., Gao, Y., Li, Y., & Jacobsen, H. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.33023
Schema: Discovering Unknown Environments via Agentic Program InductionPreprint
Zeng, G., Wang, J., Ma, W., Yin, S., Wang, C., Liu, S., Kanazawa, A., Ni, W., Li, X., Zanette, A., et al. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.39140
What AI Agents Do With A Hospital Budget: A Controlled Simulation Study Of Delegated AuthorityPreprint
Omar, M., Glicksberg, B. S., Agbareia, R., Gorenshtein, A., Barash, Y., Brook, O. R., Ahmed, M., Kruskal, J. B., Sakhuja, A., Klang, E., et al. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7575977
SpireBench: Evaluating Long-Horizon Strategic Play of Agentic Models with Slay the SpirePreprint
Lowe, S. C., & Keemink, S. W. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.12596
Understanding and Reducing Misalignment in Persuasion OptimizationPreprint
Wei, B., Desai, R., Henderson, P., & Hua, W. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.16070
Can LLM Agents Efficiently Identify Appliance Faults? Insights From Diagnosing a Simulated AppliancePreprint
McCabe, L. H., Lukens, S., & Huang, H. H. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.19291
JockeyBench: Evaluating LLM Agents on Horse-Racing Decision-MakingPreprint
Xie, Z., Li, X., Yu, J., Wang, Z., ChiTat, H., Wang, Z., Du, W., Liu, Y., Jing, H., Bai, J., et al. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.19589
FDE-Gym: The Last Mile of AI DeliveryPreprint
Yang, D., Yan, H., Wang, R., Zhang, B., Zuo, H., Zhao, Y., Chen, Y., Liao, X., Gao, C., Jiang, T., et al. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.22350
Decentralized LLM Workflows via Sequential AuctionsPreprint
Gemp, I., Trivedi, S., Fisch, A., Huot, F., Kaisers, M., Lapata, M., Cohen, W. W., Agarwal, A., Larson, K. S., & Eisenstein, J. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.22201
KiranaBench: Can a Language Model Run a Neighbourhood Shop?Preprint
Arora, B., Karmakar, A., Jain, C., Churiwal, D., Jain, H., Dubey, S. M., Prasad, M., Piyush, R., Tayal, K., Kumar, D., et al. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.22980
Toolflation: Seller Agents Distort Tool Markets Through Spontaneous Generative Engine OptimizationPreprint
Tang, Z., Bai, D., Liu, J., Wu, X., Wen, Q., & Cheng, X. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.28454
Joules-to-GDP: How Much Economic Value Do AI Systems Produce per Unit of Energy?Preprint
Zeinali, A., Saad-Falcon, J., Narayan, A., Abbasi, M., Hartley, J., Hennessy, J., Re, C., & Mirhoseini, A. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.30858
Enda, K., Saito, J., Mukai, M., Han, F., Turkes, E., Kusakabe, T., Umeki, R., 住吉, 一., Numata, M., Tanikawa, T., et al. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.32151
SlowLab: Benchmarking Autonomous AI Agents on a Year of Experiments They Cannot UndoPreprint
Ding, N., & Feng, Z. W. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.34644
AOS-Bench: Can LLM Agents Manage Dynamic Workloads?Preprint
Zhu, E., Zhang, G., Hu, K., Su, M., & Wang, H. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.33178
Metadata
- Authors
- Backlund, A., & Petersson, L.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2025
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2502.15840
- OpenAlex
- W4414835304
- Semantic Scholar
- c2fa7d5acc7ab42fb39e8a40b2f3f5b2a9986b2b
- Cited by
- 101
- References
- 0
- Citations by year
- 2026: 85 · 2025: 16
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- LLM agents, autonomous agents, benchmarking, simulation environment, inventory management, long-horizon decision-making, context window limits, capital acquisition, large language models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-11-08, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-08 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
