Vending-Bench: A Benchmark for Long-Term Coherence of Autonomous Agents

TL;DR

Vending-Bench is a simulated long-horizon benchmark where LLM agents run a vending machine business, testing sustained coherence over runs consuming around 25M tokens. Top models can outperform a human baseline on average, but all models show high variance and eventually derail or stagnate.

Key findings

• Claude 3.5 Sonnet had the highest mean net worth at $2,217.93 (min $476.00), 1,560 mean units sold, and sales stopped after 102 days on average (82.2% of the run); o3-mini followed with $906.86 mean net worth (min $369.05), 831 mean units sold, and 86 days until sales stopped (80.3%). The human baseline scored $844.05 mean and minimum net worth, 344 units sold, and 67 days until sales stopped (100.0%).
• All models showed high variance: even top models sometimes sold zero items, and all models eventually stagnated on average. The paper reports no clear correlation between failures and context-window filling; Pearson correlation between “Days Until Sales Stop” and “Days Until Full Memory” was 0.167.
• Common failure modes included misinterpreting delivery schedules, forgetting orders, and descending into tangential “meltdown” loops. Examples include a shortest Claude 3.5 Sonnet run of about 18 simulated days, o3-mini failing to call tools for about 1,300 messages, and Claude 3.5 Haiku escalating to legal threats.
• Larger memory capacities performed worse in the reported memory experiment, and additional GPT-4o mini tests found that a $100 starting balance lowered units sold, a $5 daily fee ended all runs before 100 simulated days, and a $0 daily fee did not increase sales because the model got stuck in waiting loops.

Methods

• The agent is a basic LLM loop with context management using the last 30,000 tokens in most experiments, memory tools (scratchpad, key-value store, vector database), task-specific vending-machine tools, and sub-agent delegation, implemented in AISI’s inspect-ai framework.
• The environment simulates supplier email replies and customer purchases; agents start with $500, pay a $2 daily fee, operate a machine with four rows of three slots, and run for 2,000 messages per run, ending early if bankrupt for 10 consecutive days. Each model or config variation was run 5 times; most runs consumed around 25 million tokens and took 5–10 real-world hours.
• The primary score is net worth at the end: cash at hand, cash not emptied from the vending machine, and value of unsold products at wholesale purchase price. A human baseline completed the task for five hours via a chat-based interface.

Limitations

• The paper states the human baseline is a single sample, so variances cannot be compared, and notes that model minimum values come from five runs each while the human minimum comes from one run.
• The paper states the benchmark score has no defined upper limit, so saturation is not precisely defined, and rapid model development may lead to small score differences between models in the future.
• Evident from the text: only five runs per model or configuration, simulated supplier and customer behavior generated with GPT-4o and Perplexity rather than real market data, and the human baseline is one participant over five hours. The paper also recognizes dual-use risk in evaluating capital-acquisition capabilities but argues systematic evaluation is important.