Magentic Marketplace: An Open-Source Environment for Studying Agentic Markets

Current research on agentic systems has focused mainly on individual agent performance and structured interactions between agents for isolated economic tasks.

“Current research on agentic systems has focused mainly on individual agent performance and structured interactions between agents for isolated economic tasks (Wang et al., 2024; Buscemi et al., 2025; Mao et al., 2025; Abdelnabi et al., 2024).”

• cites Wang et al., 2024 sourcePartly supported

  The survey discusses individual GUI-agent performance, including task-completion success rates (2 GUI Agent Data Resources, page unavailable), and describes a multi-agent architecture (Learning-based, page unavailable). It does not establish that agentic-systems research has focused mainly on structured interactions between agents for isolated economic tasks; its examples also span social forums, software development, and mobile applications. (LLM, conf=0.78)

  Experimental — accuracy not yet measured
• cites Buscemi et al., 2025 sourcePartly supported

  The Introduction (p. 2) describes structured interactions among three LLM agents in a game-theoretic regulatory setting, so the paper supports that part of the sentence. It does not establish that current agentic-systems research focuses mainly on individual agent performance or isolated economic tasks; its own task models a regulatory ecosystem. (LLM, conf=0.82)

  Experimental — accuracy not yet measured
• cites Mao et al., 2025Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

All models exhibit severe first-proposal bias.

“all models exhibit severe first-proposal bias”

First-proposal bias creates 10-30x advantages for response speed over quality.

“creating 10-30x advantages for response speed over quality”

Performance degrades sharply with scale.

“Performance degrades sharply with scale”

Platform extensibility is supported through the protocol discovery mechanism.

“The second objective, platform extensibility , is supported through the protocol discovery mechanism”

Manual analysis of seven trials revealed more fundamental issues beyond prompt-model misalignment.

“While some degradation may stem from prompt-model misalignment, manual analysis of seven trials revealed more fundamental issues.”

Small changes in protocols lead to meaningful outcome differences.

“The Magentic Marketplace environment enables experimental exploration of these trade-offs, revealing that small changes in protocols lead to meaningful outcome differences.”

This represents a 10-30 fold advantage for businesses that respond first, dwarfing any other competitive factor we measured.

“This represents a 10-30 fold advantage for businesses that respond first, dwarfing any other competitive factor we measured.”

Near-zero selection rates for second and third proposals (often 0-7%) were observed.

“The near-zero selection rates for second and third proposals (often 0-7%) suggest agents are not genuinely comparing options but rather satisficing with the first acceptable offer.”

Manual evaluation of Qwen3-14b revealed significant performance limitations.

“Our manual evaluation of Qwen3-14b revealed significant performance limitations.”

Giving agents more results to consider does not necessarily mean they will contact more businesses.

“Giving agents more results to consider does not necessarily mean they will contact more businesses (Figure 6a, Figure 6b).”

Agent heterogeneity is supported because the protocol defines only the communication interface, not implementation requirements.

“Agent heterogeneity is supported because the protocol defines only the communication interface, not implementation requirements.”

For GPT-4o, consumer welfare declines by 4.3% when providing one hundred search results versus three search results in the initial consideration set (Mexican 100-300).

“For GPT-4o, which shows the least decline across search results, consumer welfare declines by 4.3% when providing one hundred search results versus three search results in the initial consideration set (Mexican 100-300).”

Experiments with consideration set size revealed a paradox of choice effect where increased options from search results reduced welfare.

“Experiments with consideration set size revealed a paradox of choice effect where surprisingly increased options (from search results) reduced welfare.”

Open-source models show an overall notable drop under lexical search.

“However, there is an overall notable drop under lexical search , suggesting difficulty in identifying optimal transactions when the consideration set is noisy.”

For other models, consumer welfare declines more drastically as consideration set size increases (Sonnet-4 - 65.4%, GPT-5 - 44% on Mexican 100-300).

“For other models, consumer welfare declines more drastically as the consideration set size increases (Sonnet-4 - 65.4%, GPT-5 - 44% on Mexican 100-300), though we emphasize that in these experiments we use GPT-5 with minimal reasoning.”

Qwen3-14b exhibited role confusion where it critiqued its own wrong actions while simultaneously executing them.

“The model exhibited three primary failure modes including premature termination without completing payment, role confusion where it critiqued its own wrong actions while simultaneously executing them, and excessive purchasing without selection criteria, all pointing to fundamental challenges beyond prompt optimization of Qwen3-14b.”

Qwen3-14b performs poorly across both conditions.

“Qwen3-14b performs poorly across both conditions, mainly stemming from limited reasoning ability inherently.”

Only Gemini-2.5-Flash's outreach increases proportionally with the size of the consideration set.

“Only Gemini-2.5-Flash's outreach increases proportionally with the size of the consideration set, suggesting true exploratory capacity.”

Frontier models (GPT-4.1, Sonnet-4.5, Gemini-2.5-Flash) showed near-uniform selection rates across all three positions in search results.

“Results in Figure 8 show that frontier models (GPT-4.1, Sonnet-4.5, Gemini-2.5-Flash) showed near-uniform selection rates across all three positions, suggesting they can effectively process search results in parallel rather than sequentially.”

There is a negative relationship between the consideration set size available to consumer agents and welfare outcomes.

“Our experiments revealed a negative relationship between the consideration set size available to consumer agents and welfare outcomes (Figure 5a, Figure 5b).”

Our results reveal systematic behavioral biases and vulnerabilities across models.

“Our results reveal systematic behavioral biases and vulnerabilities across models”

Every model tested exhibited severe anchoring on the first proposal received, though with varying degrees.

“Every model tested exhibited severe anchoring on the first proposal received, though with varying degrees.”

GPT-4o, GPT-OSS-20B, and Qwen3-4B-2507 showed increased payments to malicious actors under traditional psychological manipulation conditions.

“These models not only fell victim to prompt injection attacks which often redirect all payments to manipulative agents, but also showed increased payments to malicious actors under traditional psychological manipulation conditions.”

Qwen3-4b-2507 performed close to GPT-4o in the Contractors dataset.

“Notice that GPT-OSS-20b outperforms GPT-4o in both lexical search and perfect search in Mexican dataset; and Qwen3-4b-2507 performed close to GPT-4o as well in Contractors dataset.”

Qwen3-14b exhibited premature termination without completing payment.

“The model exhibited three primary failure modes including premature termination without completing payment, role confusion where it critiqued its own wrong actions while simultaneously executing them, and excessive purchasing without selection criteria, all pointing to fundamental challenges beyond prompt optimization of Qwen3-14b.”

Under Agentic: Lexical search, proprietary models such as Sonnet-4, Sonnet4.5, GPT-5, GPT-4.1, GPT-4o, and Gemini-2.5-Flash outperform two of the three baseline conditions: random selection among businesses with matching menu items, and cheapest selection based on price without amenity information.

“Even under these imperfect search conditions, proprietary models such as Sonnet-4, Sonnet4.5, GPT-5, GPT-4.1, GPT-4o, and Gemini-2.5-Flash outperform two of the three baseline conditions: random selection among businesses with matching menu items, and cheapest selection based on price without amenity information.”

The majority of models exhibit limited exploration, contacting only a small fraction of available businesses regardless of how many options were presented.

“The majority of models exhibit limited exploration, contacting only a small fraction of available businesses regardless of how many options were presented.”

Consideration set size experiments revealed that the majority of models contact only a small subset of businesses, even when provided with more options.

“Consideration set size experiments (Section 5.2) also revealed that the majority of models contact only a small subset of businesses, even when provided with more options.”

Qwen3-14b exhibited excessive purchasing without selection criteria.

“The model exhibited three primary failure modes including premature termination without completing payment, role confusion where it critiqued its own wrong actions while simultaneously executing them, and excessive purchasing without selection criteria, all pointing to fundamental challenges beyond prompt optimization of Qwen3-14b.”

Gemini-2.5-Flash consistently sent messages to all businesses provided for consideration.

“Gemini-2.5-Flash consistently sent messages to all businesses provided for consideration.”

Magentic Marketplace is a simulated multiagentic marketplace environment for controlled experimentation in agentic markets.

“we introduce Magentic Marketplace , a simulated m ultiagentic marketplace environment for controlled experimentation in agentic markets.”

Even the 'best performing' model in terms of proposal diversity (GPT-4.1 in the contractor scenario) still selected first proposals at 60% compared to 13.3% for third proposals, a 4.5x advantage.

“Even the 'best performing' model in terms of proposal diversity (GPT-4.1 in the contractor scenario) still selected first proposals at 60% compared to 13.3% for third proposals, a 4.5x advantage.”

GPT-4o, GPT-OSS-20B, and Qwen3-4B-2507 demonstrated significant vulnerability across multiple attack vectors.

“In stark contrast, GPT-4o, GPT-OSS-20B, and Qwen3-4B-2507 demonstrated significant vulnerability across multiple attack vectors.”

Frontier models demonstrate robust manipulation resistance.

“Frontier models demonstrate robust manipulation resistance.”

Sonnet-4.5 exhibited the strongest resistance, showing virtually no susceptibility to any manipulation strategy tested.

“Sonnet-4.5 exhibited the strongest resistance, showing virtually no susceptibility to any manipulation strategy tested.”

Results in Figure 7 reveal a clear divide in manipulation resistance across model sizes and capabilities.

“Results in Figure 7 reveal a clear divide in manipulation resistance across model sizes and capabilities.”

Figure 4 reveals a clear hierarchy in performance under different conditions and models.

“Figure 4 reveals a clear hierarchy in performance under different conditions and models.”

Increased exploration does not correlate with improved performance.

“However, increased exploration does not correlate with improved performance either.”

Even when individual components appear to work well in isolation, emergent outcomes and dynamic interactions can lead to unintended consequences, vulnerabilities, and inefficiencies that are apparent only at scale.

“Even when individual components appear to work well in isolation, emergent outcomes and dynamic interactions can lead to unintended consequences, vulnerabilities, and inefficiencies that are apparent only at scale.”

Two-sided agentic markets can achieve reasonable welfare outcomes by reducing information asymmetries through agent-mediated communication.

“These findings collectively demonstrate that two-sided agentic markets can achieve reasonable welfare outcomes by reducing information asymmetries through agent-mediated communication.”

• AssumptionRests on an assumption: The synthetic restaurant and contractor marketplace scenarios adequately represent real-world two-sided markets for drawing conclusions about agentic markets.
• AssumptionRests on an assumption: The LLM agents' behavior in the simulated marketplace is representative of autonomous agents in real two-sided markets.

These failure modes point to fundamental challenges beyond prompt optimization of Qwen3-14b.

“The model exhibited three primary failure modes including premature termination without completing payment, role confusion where it critiqued its own wrong actions while simultaneously executing them, and excessive purchasing without selection criteria, all pointing to fundamental challenges beyond prompt optimization of Qwen3-14b.”

Gemini-2.5-Flash's proportional outreach suggests true exploratory capacity.

“Only Gemini-2.5-Flash's outreach increases proportionally with the size of the consideration set, suggesting true exploratory capacity.”

Proposal bias emerged as a dominant behavioral pattern that fundamentally distorts marketplace dynamics.

“Unlike the relatively modest position effects observed in search results, proposal bias emerged as a dominant behavioral pattern that fundamentally distorts marketplace dynamics.”

In a marketplace where agents exhibit such extreme proposal bias, competitive dynamics can shift entirely from product quality or pricing to response latency.

“In a marketplace where agents exhibit such extreme proposal bias, competitive dynamics can shift entirely from product quality or pricing to response latency.”

Agents are not genuinely comparing options but rather satisficing with the first acceptable offer.

“The near-zero selection rates for second and third proposals (often 0-7%) suggest agents are not genuinely comparing options but rather satisficing with the first acceptable offer.”

The authors use fully synthetic data to ensure experimental control, reproducibility, and safe exploration of agent behaviors.

“In this work, we use fully synthetic data to ensure experimental control, reproducibility, and safe exploration of agent behaviors.”

These results illustrate a strikingly consistent paradox of choice whereby presenting more options to an assistant agent ultimately leads to lower-quality selections.

“These results illustrate a strikingly consistent paradox of choice whereby presenting more options to an assistant agent ultimately leads to lower-quality selections.”

• AssumptionRests on an assumption: The synthetic restaurant and contractor marketplace scenarios adequately represent real-world two-sided markets for drawing conclusions about agentic markets.
• AssumptionRests on an assumption: The LLM agents' behavior in the simulated marketplace is representative of autonomous agents in real two-sided markets.

Our experimental study focused on two-sided agentic markets populated entirely by AI agents.

“Our experimental study focused on two-sided agentic markets populated entirely by AI agents.”

The drop under lexical search suggests difficulty in identifying optimal transactions when the consideration set is noisy.

“However, there is an overall notable drop under lexical search , suggesting difficulty in identifying optimal transactions when the consideration set is noisy.”

This bias can create market distortions that undermine quality and price competition.

“This bias can create market distortions that undermine quality and price competition.”

Frontier models can effectively process search results in parallel rather than sequentially.

“Results in Figure 8 show that frontier models (GPT-4.1, Sonnet-4.5, Gemini-2.5-Flash) showed near-uniform selection rates across all three positions, suggesting they can effectively process search results in parallel rather than sequentially.”

When autonomous agents are equipped with capabilities for discovery, communication, and transaction execution empowered by sufficiently advanced language models, they can effectively mediate between consumers and service providers.

“Our results, with baseline ReACT-style agents, indicate that when autonomous agents are equipped with capabilities for discovery, communication, and transaction execution empowered by sufficiently advanced language models, they can effectively mediate between consumers and service providers.”

• AssumptionRests on an assumption: The synthetic restaurant and contractor marketplace scenarios adequately represent real-world two-sided markets for drawing conclusions about agentic markets.
• AssumptionRests on an assumption: The LLM agents' behavior in the simulated marketplace is representative of autonomous agents in real two-sided markets.

Coordinated agentic interactions can approximate optimal welfare outcomes when discovery is accurate and communication is effective.

“These results demonstrate the potential of coordinated agentic interactions to approximate optimal welfare outcomes when discovery is accurate and communication is effective.”

• AssumptionRests on an assumption: The LLM agents' behavior in the simulated marketplace is representative of autonomous agents in real two-sided markets.

Agents can effectively communicate and reason to navigate noise in the discovery layer and still make high-quality decisions.

“These results suggest that agents can effectively communicate and reason to navigate noise in the discovery layer and still make high-quality decisions.”

• AssumptionRests on an assumption: The LLM agents' behavior in the simulated marketplace is representative of autonomous agents in real two-sided markets.

Qwen3-14b's poor performance mainly stems from limited reasoning ability inherently.

“Qwen3-14b performs poorly across both conditions, mainly stemming from limited reasoning ability inherently.”