Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat
Sign Up
PreprintCited by 1Multi-Agent Systems and NegotiationArtificial Intelligence in GamesArtificial Intelligence Applications

The DOI matches its DataCite record; 31 of the 31 references it cites were found in a registry or an index.

SidConArena: An Environment Evaluating Agents in Open-Ended,Positive-Sum Bargaining Game

Open access
Feng, Yeqi, Chen, Yuxin, He, Tianxing·2026·arXiv
DOI 10.48550/arxiv.2606.27397arXiv 2606.27397
Open this paper in CoChat — free accountOpen PDF

Abstract

Evaluating LLM agents requires dynamic environments that go beyond static reasoning and zero-sum games. Real-world economic interaction is often open-ended and mixed-motive: agents must negotiate, create positive-sum surplus, compete for scarce assets, and plan under delayed returns. We introduce SidConArena, a new benchmark framework for evaluating LLM agents in open-ended, positive-sum bargaining. SidConArena formalizes a multi-player economy as a finite-horizon partially observable stochastic game with three coupled phases: natural-language negotiation with binding trades, deterministic converter-based production, and sealed-bid auctions for long-term assets. The framework combines structured observations, phase-aware agent dispatching, a neural-symbolic action interface, and asynchronous execution, enabling free-form interaction while preserving rule-grounded evaluation. Across homogeneous and heterogeneous tournaments, stronger frontier models achieve higher economic outcomes, yet agents still misvalue resources, bargain passively, and remain limited in long-horizon investment planning.

Overview

CoChat Summary

TL;DR

SidConArena is a finite-horizon partially observable stochastic game benchmark for evaluating LLM agents in open-ended, positive-sum bargaining across negotiation, production, and sealed-bid auction phases. Experiments show frontier models achieve higher terminal economic outcomes, but agents still misvalue resources, bargain passively, and fail at long-horizon investment planning.

Key findings

  • In homogeneous self-play, stronger frontier models generally obtain higher terminal scores; GPT-5 and Gemini-3-Flash-Preview achieve the strongest overall performance, while GPT-4o-mini obtains much lower scores and GLM-4.7 is weak.
  • Heterogeneous Elo tournaments (20 games, baseline 1500, zero-sum) show frontier models remain competitive in mixed-agent economies; absolute rollout performance and competitive strength are related but not identical.
  • Agents can execute valid structured actions but misground local valuation: Table 1 examples include trading Industry×3 for Ship×1 when listed values are 3 and 1, Ship×1 for Culture×2 when listed values are 1 and 2, and Ship×1 for Culture×4+Biotech×1 when listed values are 1 vs 5.5.
  • Trajectory analysis finds passive/overly polite bargaining (accepting first reasonable offer without counteroffering or leveraging competing demand) and short-horizon planning failures that miss early investment, compounding production, and late-game score conversion.

Methods

  • Formalizes SidConArena as a finite-horizon POSG with asymmetric agents and composite action space \(A_i = A_{neg} \cup A_{bid} \cup A_{prod}\); phases are natural-language binding negotiation, deterministic converter production, and simultaneous sealed-bid Right-of-Choice auction for colonies/technologies.
  • Evaluates LLM agents via homogeneous self-play (all seats same model) and heterogeneous Elo tournaments (different model backbones per seat; 20 games total), using shared observation format, action schema, prompting interface, and environment rules.
  • Models include GPT-4o-mini, o3-mini, Qwen-Plus, DeepSeek-V3, Gemini-3-Flash-Preview, GPT-5, Claude-Opus-4, GLM-4.7, and others; primary outcome is terminal economic performance, with Elo ratings and trajectory-log diagnostics; temperature=0, default sampling otherwise; each agent consumes ~0.6 million tokens per complete game round.

Limitations

  • Paper-stated: quantitative evaluation focuses on LLM-agent rollouts, not systematic mixed human-LLM games; negotiation protocol uses binding structured proposals plus natural-language messages and does not model non-binding promises, conditional agreements, deferred compensation, reputation, or strategic betrayal.
  • Paper-stated: no fine-tuning or environment-specific training, so results characterize zero-shot/prompt-based agency; finite rollout scale and stochastic auction outcomes/asymmetric initial conditions/interaction paths limit statistical robustness; benchmark is grounded in one positive-sum board-game economy.
  • Evident from text: Table 1 valuation examples are qualitative illustrations, not a full-dataset estimate; Elo tournament is only 20 games, and the paper calls for larger-scale tournaments, repeated-seed evaluation, and more human reference data.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2606.27397

Cites

31 of 31 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Figure 1 Overview of the SidConArena framework. SidConArena decomposes each agent's turn into four stages. First, the environment assembles a structured observation from private state, public board information, market context, and interaction history. Second, a phase-aware agent brain routes the task to specialized callers for turn planning, trading, production, bidding, and item selection. Third, the routed context is passed to an LLM agent through a neural-symbolic action interface, where free-form reasoning is converted into validated function-call actions. Finally, the execution engine applies the submitted actions through the negotiation ledger, production engine, auction mechanism, and state update module, advancing the game to the next turn.
Figure 1 Overview of the SidConArena framework. SidConArena decomposes each agent's turn into four stages. First, the environment assembles a structured observation from private state, public board information, market context, and interaction history. Second, a phase-aware agent brain routes the task to specialized callers for turn planning, trading, production, bidding, and item selection. Third, the routed context is passed to an LLM agent through a neural-symbolic action interface, where free-form reasoning is converted into validated function-call actions. Finally, the execution engine applies the submitted actions through the negotiation ledger, production engine, auction mechanism, and state update module, advancing the game to the next turn.CC-BY · From the PDF
Figure 2 Self-play multi-turn rollout protocol in SidConArena . Each evaluation game instantiates LLM agents under the shared agent-environment pipeline in Figure 1 and repeats an observe-plan-act-execute/update cycle until terminal scoring. Agents receive asynchronous updates from other players, emit phase-conditioned structured actions for negotiation, production, bidding, and asset selection, and are evaluated by terminal economic outcomes.
Figure 2 Self-play multi-turn rollout protocol in SidConArena . Each evaluation game instantiates LLM agents under the shared agent-environment pipeline in Figure 1 and repeats an observe-plan-act-execute/update cycle until terminal scoring. Agents receive asynchronous updates from other players, emit phase-conditioned structured actions for negotiation, production, bidding, and asset selection, and are evaluated by terminal economic outcomes.CC-BY · From the PDF
Figure 3 Distribution of self-play performance scores across different LLM backbones. Stronger frontier models tend to achieve higher scores, whereas lightweight or earlier-generation models obtain substantially lower economic returns.
Figure 3 Distribution of self-play performance scores across different LLM backbones. Stronger frontier models tend to achieve higher scores, whereas lightweight or earlier-generation models obtain substantially lower economic returns.CC-BY · From the PDF
Figure 4 Elo of models. We apply 1500 as baseline elo, and the tournament is zero-sum.
Figure 4 Elo of models. We apply 1500 as baseline elo, and the tournament is zero-sum.CC-BY · From the PDF
Figure 5 Representative qualitative patterns observed in SidConArena . (a) Passive bargaining under scarcity: even when holding a scarce resource demanded by multiple partners, the agent accepts the first reasonable offer without counteroffering or leveraging competing demand. (b) Short-horizon planning failure: the agent's actions appear locally reasonable across turns, but the trajectory does not follow the preferred path from early investment to compound production and late-game scoring.
Figure 5 Representative qualitative patterns observed in SidConArena . (a) Passive bargaining under scarcity: even when holding a scarce resource demanded by multiple partners, the agent accepts the first reasonable offer without counteroffering or leveraging competing demand. (b) Short-horizon planning failure: the agent's actions appear locally reasonable across turns, but the trajectory does not follow the preferred path from early investment to compound production and late-game scoring.CC-BY · From the PDF
Figure 6 Web-based human interface for participating in SidConArena .
Figure 6 Web-based human interface for participating in SidConArena .CC-BY · From the PDF

Verification

Claims the paper makes

0 Valid · 2 Not valid · 0 Not checked · 74 Own results and interpretations

29 more claims

Create a free account to see all 76 claims and every source verdict.

See all claims — free account
  • It may state two separate claims as one. 26
  • Probably not a substantial claim. 23

Create a free account to see these claims.

Create a free account to see these assumptions.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

1 of 1

Citations

Cites

Show all 18 references

Research works printed but not linked: 3

  1. Chang Ma, Junlei Zhang, Zhihao Zhu, Cheng Yang, Yujiu Yang, Yaohui Jin, Zhenzhong Lan, Lingpeng Kong, and Junxian He. 2024a. Agentboard: An an- alytical evaluation board of multi-turn llm agents. Advances in neural information processing systems, 37:74325-74362.
  2. Weiyu Ma, Qirui Mi, Yongcheng Zeng, Xue Yan, Yuqiao Wu, Runji Lin, Haifeng Zhang, and Jun Wang. 2024b. Large language models play starcraft ii: Benchmarks and a chain of summarization ap- proach. Advances in Neural Information Processing Systems, 37:133386-133442.
  3. Yue Wu, Xuan Tang, Tom Mitchell, and Yuanzhi Li. 2024. Smartplay: A benchmark for llms as intelligent agents. In International Conference on Learning Representations, volume 2024, pages 1543-1561.

Cited by

See all 1

Related

See all

Metadata

Authors
Feng, Y., Chen, Y., & He, T.
Affiliations
Tsinghua University (CN)
ORCID
He, Tianxing
Venue
arXiv
Publisher
arXiv
Published
2026
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2606.27397
arXiv
2606.27397
OpenAlex
W7166511072
Semantic Scholar
a16df857bc621bb522d32ee2e313acfa12f746f5
Cited by
1
References
18
Citations by year
2026: 1
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Multi-Agent Systems and Negotiation
  • Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence in Games
  • Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
Keywords
LLM agent evaluation, partially observable stochastic games, sealed-bid auctions, economic surplus, long-term investment planning, benchmarking framework

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-11-14, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-11-14 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: SidConArena: An Environment Evaluating Agents in Open-Ended,Positive-Sum Bargaining Game. https://app.cochat.ai/papers/10.48550/arxiv.2606.27397
The paper
Feng, Y., Chen, Y., & He, T. (2026). SidConArena: An Environment Evaluating Agents in Open-Ended,Positive-Sum Bargaining Game. arXiv. https://doi.org/10.48550/arxiv.2606.27397

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.