CoChat
Sign Up
PreprintCited by 33Multi-Agent Systems and NegotiationAI-based Problem Solving and PlanningNatural Language Processing Techniques

MAS-ZERO: Designing Multi-Agent Systems with Zero Supervision

Open access
Ke, Zixuan, Xu, Austin, Ming, Yifei, Nguyen, Xuan-Phi, Chin, Ryan, Xiong, Caiming, Joty, Shafiq·2025·arXiv
DOI 10.48550/arxiv.2505.14996arXiv 2505.14996
Open this paper in CoChat — free accountOpen PDF

Abstract

Multi-agent systems (MAS) leveraging the impressive capabilities of Large Language Models (LLMs) hold significant potential for tackling complex tasks. However, most current MAS depend on manually designed agent roles and communication protocols. These manual designs often fail to align with the underlying LLMs' strengths and struggle to adapt to novel tasks. Recent automatic MAS approaches attempt to mitigate these limitations but typically necessitate a validation set for tuning and yield static MAS designs lacking adaptability during inference, while also removing the flexibility to reduce to simpler systems. We introduce MAS-ZERO, the first self-evolved, inference-time framework for automatic MAS design. MAS-ZERO employs meta-level design to iteratively design, critique, and refine MAS configurations tailored to each problem instance, without requiring a validation set. Critically, it enables dynamic problem decomposition and agent composition through meta-feedback on solvability and completeness, and reduction to simpler systems when appropriate. Experiments across reasoning (math and graduate-level QA), coding, and agentic (search-based) benchmarks, using both closed-source and open-source LLM backbones of varying sizes, demonstrate that MAS-ZERO outperforms strong manual and automatic MAS baselines. It achieves substantial average accuracy improvements of up to 16.69% on reasoning, 16.66% on coding, and 5.45% on agentic tasks, while maintaining cost efficiency.

AI snapshot

Written from the paper’s full text.

TL;DR

MAS-ZERO is an inference-time, zero-supervision framework that uses a meta-agent to iteratively design, critique, and verify multi-agent system configurations per problem instance. It outperforms manual and automatic MAS baselines across reasoning, coding, and agentic benchmarks while remaining cost-efficient.

Key findings

  • MAS-ZERO achieves average accuracy improvements of up to 16.69% on reasoning, 16.66% on coding, and 5.45% on agentic tasks over baselines.
  • On reasoning, it exceeds the strongest baseline AFlow by 13.03% on average with GPT-4o; ADAS fails completely on AIME24 (0% accuracy), and MAS-ZERO only falls behind AFlow on AIME24 with Qwen by one sample out of 24.
  • On coding, MAS-ZERO scores 25.83 vs. AFlow’s 16.25 with GPT-4o and 16.74 vs. 6.67 with Llama, corresponding to 58% and 149% relative gains over AFlow.
  • Ablations show all three steps contribute: removing MAS-Init, MAS-Evolve, meta-design, meta-feedback, or MAS-Verify degrades performance, with using the last iteration alone causing the largest drop; an oracle verifier pushes GPQA close to 95%.

Methods

  • MAS-ZERO runs entirely at inference time with a meta-agent and three steps: MAS-Init executes building blocks (CoT, CoT-SC, Debate, Self-Refine); MAS-Evolve iteratively performs meta-design and meta-feedback with an experience library; MAS-Verify selects the final answer from candidate outputs.
  • Evaluated on AIME24, GPQA-diamond, SWE-Bench-Lite-Oracle, BrowseComp, and Frames, using GPT-4o, Llama3.3-70B-inst, Qwen2.5-32B-inst, and o3-mini; it uses 5 MAS-Evolve iterations plus 4 building blocks, yielding 9 candidate answers, with the same LLM for meta-agent and individual agents unless testing heterogeneous settings.
  • Baselines include CoT, CoT-SC, manual MAS (Debate, Self-Refine, ReConcile, MAD, SPP, DyLAN), automatic MAS (MaAS, ADAS, AFlow), and training-based MAS-GPT; cost is computed using official OpenAI API pricing.

Limitations

  • Stated: meta-design and meta-feedback signals may be imperfect and depend on the underlying LLM; weaker LLMs (GPT-OSS-20B, Qwen2.5-7B, Qwen2.5-Coder-3B, GPT-4.1-nano) cannot reliably follow instructions and often produce syntactically incorrect Python code, suggesting the meta-agent role requires sufficiently strong capabilities.
  • Stated: on Frames, MAS-ZERO underperforms Debate and Self-Refine; the authors speculate that search-agent errors grounded in retrieved content are harder to detect during MAS-Verify, highlighting the need to strengthen the verifier.
  • Stated: the inference-time mechanism incurs higher token usage during testing, and MAS-Verify does not apply to SWE; the paper also reports high variance in MAS results due to multi-agent interactions and non-zero agent temperature.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2505.14996

Cites

64 of 64 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 14 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    MAS-ZERO achieves average accuracy improvements of up to 16.66% on coding tasks.

    “It achieves substantial average accuracy improvements of up to 16.69% on reasoning, 16.66% on coding, and 5.45% on agentic tasks, while maintaining cost efficiency.”
    Quote in the paperSubstantial claim
  • Findingown data

    MAS-ZERO achieves average accuracy improvements of up to 16.69% on reasoning tasks.

    “It achieves substantial average accuracy improvements of up to 16.69% on reasoning, 16.66% on coding, and 5.45% on agentic tasks, while maintaining cost efficiency.”
    Quote in the paperSubstantial claim
  • Findingown data

    MAS-ZERO outperforms strong manual and automatic MAS baselines across reasoning, coding, and agentic benchmarks.

    “Experiments across reasoning (math and graduate-level QA), coding, and agentic (search-based) benchmarks, using both closed-source and open-source LLM backbones of varying sizes, demonstrate that MAS-ZERO outperforms strong manual and automatic MAS baselines.”
    • Also stated in: § 1 Introduction · boosted
      “Evaluations across three challenging domains-reasoning (math and graduate-level QA), coding, and agentic (search-based), using both closed-source and open-source LLM backbones of varying sizes (including GPT-4o, 32B, and 70B models) demonstrate that MAS-ZERO consistently outperforms strong manual and automatic MAS baselines.”
    Quote in the paperSubstantial claim
  • Findingown data

    MAS-ZERO maintains cost efficiency.

    “while maintaining cost efficiency.”
    • Also stated in: § 1 Introduction · plain
      “It also consistently lies on the Pareto frontier of accuracy and cost (Fig. 1).”
    Quote in the paperSubstantial claim
  • Findingown data

    MAS-ZERO consistently outperforms strong manual and automatic MAS baselines across diverse domains and LLMs.

    “Evaluations across three challenging domains-reasoning (math and graduate-level QA), coding, and agentic (search-based), using both closed-source and open-source LLM backbones of varying sizes (including GPT-4o, 32B, and 70B models) demonstrate that MAS-ZERO consistently outperforms strong manual and automatic MAS baselines.”
    Quote in the paperSubstantial claim
  • May combine two claims. 27
  • The quote only partly states this claim. 4
  • Classified as probably not a substantial claim. 11

  • May combine two claims.

    Findingcitation

    The inference-time mechanism incurs higher token usage during testing but avoids expensive validation-time optimization and shifts design effort to testing phase.

    “While the inference-time mechanism incurs higher token usage during testing, it avoids expensive validation-time optimization and shifts the design effort to the testing phase, where it can flexibly handle new tasks, and often be more effective [1].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [1]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Findingown data

    On challenging tasks like AIME24, many baselines show little to no improvement over simple CoT, meaning the integrated system does not even outperform a single component.

    “On more challenging tasks (e.g., AIME24), however, the inability to revert or decompose becomes critical: as shown in Fig. 1, many baselines show little to no improvement over simple CoT, meaning the integrated system does not even outperform a single component.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingnone given

    MAS-ZERO achieves substantial performance gains over both manually designed and strong automatic baselines, while remaining cost-efficient and Pareto-optimal across diverse LLMs and domains.

    “We present a new SoTA automatic MAS system that achieves substantial performance gains over both manually designed and strong automatic baselines, while remaining cost-efficient and Pareto-optimal across diverse LLMs and domains.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    MAS-ZERO consistently outperforms strong single-agent and manual MAS, with only two exceptions: GPQA with MAD using GPT-4o, and GPQA with Self-Refine using Llama.

    “(2) MAS-ZERO also consistently outperforms strong single-agent and manual MAS, with only two exceptions: GPQA with MAD using GPT-4o, and GPQA with Self-Refine using Llama.”
    Quote in the paperSubstantial claim

  • The quote only partly states this claim.

    Findingown data

    MAS-ZERO achieves an average accuracy of 41.97% with GPT-4o on the reasoning tasks.

    “MAS-ZERO 33.33 | 50.60 41.97 | 37.50 | 52.41 44.96 | 29.17 | 51.81 40.49”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    When GPT-4o is the individual agent and the meta-agent is replaced with o3-mini, performance improves notably but still falls short of directly using o3-mini for both roles.

    “As shown in Table 5, when GPT-4o is the individual agent and the meta-agent is replaced with o3-mini, performance improves notably but still falls short of directly using o3-mini for both roles.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Ablation using last iteration as output without additional judgment causes sharp performance decline, the largest drop among all ablations.

    “The last row in Table 6 shows a sharp performance decline (the largest drop among all ablations).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingnone given

    Automatic MAS baselines cannot integrate external verifiers, yielding zero improvement.

    “Automatic MAS baselines cannot integrate external verifiers, yielding zero improvement.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    MAS-ZERO comes with 58% (GPT-4o) and 149% (Llama) relative gains over strongest baseline AFlow on coding tasks.

    “Notably, it comes with 58% (GPT-4o) and 149% (Llama) relative gains over the strongest baseline AFlow.”
    Quote in the paperSubstantial claim

  • The quote only partly states this claim.

    Findingown data

    MAS-ZERO achieves an average accuracy of 40.49% with Qwen2.5-32B on the reasoning tasks.

    “MAS-ZERO 33.33 | 50.60 41.97 | 37.50 | 52.41 44.96 | 29.17 | 51.81 40.49”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingnone given

    CoT and CoT-SC consistently outperform MAS-GPT, ADAS, and MaAS.

    “For example, CoT and CoT-SC consistently outperform MAS-GPT, ADAS, and MaAS.”
    Quote in the paperSubstantial claim

  • The quote only partly states this claim.

    Findingown data

    MAS-ZERO achieves an average accuracy of 44.96% with Llama3.3-70B on the reasoning tasks.

    “MAS-ZERO 33.33 | 50.60 41.97 | 37.50 | 52.41 44.96 | 29.17 | 51.81 40.49”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Ensemble meta-feedback (generating multiple feedback candidates and selecting best) fails to improve performance and even reduces it.

    “Surprisingly, the ensemble approach not only fails to improve performance but even reduces it.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    MAS-ZERO delivers high performance at lower cost than comparable automatic MAS methods, establishing a new frontier for accuracy vs. cost trade-off.

    “MAS-ZERO delivers high performance at lower cost than comparable automatic MAS methods, establishing a new frontier for accuracy vs. cost trade-off.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Of automatic MAS frameworks, MAS-ZERO delivers the highest performance at relatively low cost.

    “Of automatic MAS frameworks, MAS-ZERO delivers the highest performance at relatively low cost.”
    Quote in the paperSubstantial claim

  • May combine two claims. The quote only partly states this claim.

    Findingown data

    MAS-ZERO consistently outperforms all automatic MAS methods across all LLM backbones and benchmarks.

    “From Table 1, we observe that (1) MAS-ZERO consistently outperforms all automatic MAS methods.”
    • Also stated in: § 4.1 Overall Results · plain
      “Across all LLM backbones and benchmarks, it surpasses SoTA baselines”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Weaker LLMs (GPT-OSS-20B, Qwen2.5-7B, Qwen2.5-Coder-3B, GPT-4.1-nano) are unable to reliably follow instructions and often produce syntactically incorrect Python code.

    “We find that these models are unable to reliably follow instructions and often produce syntactically incorrect Python code, suggesting that the meta-agent role requires sufficiently strong capabilities to handle its multiple responsibilities.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    When o3-mini is the individual agent and the meta-agent is replaced with GPT-4o, performance decreases, though it remains better than the setting where GPT-4o is the agent and o3-mini is the meta-agent.

    “Conversely, when o3-mini is the individual agent and the meta-agent is replaced with GPT-4o, performance decreases, though it remains better than the setting where GPT-4o is the agent and o3-mini is the meta-agent.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    MAS-ZERO is significantly stronger than other baselines, with only one exception in AIME24 and two in GPQA.

    “Nevertheless, MAS-ZERO is significantly stronger than other baselines, with only one exception in AIME24 and two in GPQA, as mentioned in Sec.??.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    MAS-ZERO is significantly more cost-efficient than AFlow, MaAS, and ADAS, with lone exception ADAS on GPQA.

    “It is significantly more cost-efficient than AFlow, MaAS, and ADAS, with the lone exception of ADAS on GPQA, where the cost increase comes with a 12% accuracy improvement.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Curated instruction design produces strong feedback, outperforming a simple ensemble alternative.

    “and that our curated instruction design produces strong feedback-outperforming a simple ensemble alternative (Sec.4.2).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    MAS-ZERO works in a fully self-evolved way by learning from the behavior of the underlying LLM agents at inference-time, enabling per-instance adaptivity with zero supervision.

    “It works in a fully self-evolved way by learning from the behavior of the underlying LLM agents at inference-time, enabling per-instance adaptivity with zero supervision.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The meta-agent can learn the capabilities of the underlying LLM agents at inference time without internal details.

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given

    MAS-ZERO is the only system that can dynamically revert to simpler yet effective strategies.

    “Crucially, MAS-ZERO is the only system that can dynamically revert to these simpler yet effective strategies, ensuring that such strengths are not overlooked.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    Existing automatic MAS methods cannot reduce to a simpler MAS or single-agent system when such strategies would be stronger, nor flexibly decompose a problem into smaller sub-tasks.

    “Even worse, these methods eliminate critical dynamics: they cannot reduce to a simpler MAS or a single-agent system when such strategies would be stronger [19], nor can they flexibly decompose a problem into smaller, more manageable sub-tasks.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [19]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Interpretationreasoning

    MAS-ZERO operates entirely at test time, allowing for unique MAS designs per-problem.

    “This process operates entirely at test time, allowing for unique MAS designs per-problem.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The meta-agent can learn the capabilities of the underlying LLM agents at inference time without internal details.

  • May combine two claims.

    Interpretationreasoning

    MAS-ZERO enables dynamic problem decomposition and agent composition through meta-feedback on solvability and completeness, and reduction to simpler systems when appropriate.

    “Critically, it enables dynamic problem decomposition and agent composition through meta-feedback on solvability and completeness, and reduction to simpler systems when appropriate.”
    • Also stated in: § Automatic MAS design. · plain
      “MAS-ZERO can dynamically revert to simpler building blocks when they are sufficient.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Solvability and completeness are sufficient criteria for evaluating and refining MAS designs.

  • Probably not a substantial claim: may not pass S4 (based).

    Premisenone given

    Existing automatic MAS methods largely restrict their evaluations to relatively simple reasoning tasks.

    “Existing automatic MAS methods largely restrict their evaluations to relatively simple reasoning tasks.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    Ranking candidates by final answer frequency acts as a prior favoring majority responses, a strategy shown effective in prior work.

    “This acts as a prior favoring majority responses, a strategy shown to be effective in prior work [48].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [48]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Classified as probably not a substantial claim.

    Study design and conductown data

    For methods that do not use validation sets (including MAS-ZERO), the authors evaluate on the same 80% split.

    “For methods do not use validation sets (including MAS-ZERO), we evaluate on the same 80% split.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationreasoning

    MAS-Verify does not apply to SWE, as correctness in SWE is determined directly by the compiler.

    “Note that MAS-Verify does not apply to SWE, as correctness in SWE is determined directly by the compiler.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisenone given

    MAS-ZERO can seamlessly incorporate external verifiers.

    “Unlike existing automatic MAS frameworks, MAS-ZERO can seamlessly incorporate external verifiers, making it naturally positioned to benefit from advances in verification techniques.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    To fairly compare with validation-based baselines, the authors split each benchmark's original test set into 20% for validation and 80% for testing.

    “To fairly compare with validation-based baselines, we split each benchmark's original test set into 20% for validation and 80% for testing.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Study design and conductnone given

    For baselines, we strictly use parameters found in original papers and provided code.

    “For baselines, we strictly use parameters found in original papers and provided code.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductown data

    The authors provide a template with utility functions and apply sanity checks (syntax validation and field consistency) to support code generation.

    “To support code generation, we provide a template with utility functions and apply sanity checks (syntax validation and field consistency; see App.I).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    MAS-ZERO's margins on coding exceed those observed in reasoning tasks, highlighting effectiveness in challenging tasks.

    “These margins exceed those observed in reasoning tasks, highlighting the effectiveness of MAS-ZERO in challenging tasks.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    MAS can exhibit high variance due to the inherent nature of multi-agent systems; the variance may be amplified by the interactions among multiple agents.

    “We can see that the MAS can exhibit high variance due to the inherent nature of multi-agent systems: the variance may be amplified by the interactions among multiple agents [3], and the generated temperature of the agents is typically non-zero.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [3]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Agents in a multi-agent system can interact with external environmental tools (e.g., search tools) or collaborate with other agents to address tasks requiring diverse capabilities or multiple steps.

    “Agents in a MAS can interact with external environmental tools e.g., search tools[21], or collaborate with other agents to address tasks requiring diverse capabilities or multiple steps[28,5].”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites [21]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [28]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [5]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    MAS-ZERO uses the same LLM for both the meta-agent and individual agents to fairly evaluate relative performance.

    “To fairly evaluate how well MAS-ZERO performs relative to the underlying LLM used to construct the MAS, we always use the same LLM for both the meta-agent and individual agents (heterogeneous settings in Sec.4.2).”
    Quote in the paperSubstantial claim
38 of 38

  • The paper relies on this without support.

    Validation sets are often unavailable in real-world scenarios.

    “which are often unavailable in real-world scenarios and may not generalize.”

  • The paper relies on this without support.

    The meta-agent can learn the capabilities of the underlying LLM agents at inference time without internal details.

    “Initially, it has no knowledge of the underlying LLM agents' internal capabilities and may produce suboptimal designs.”
    “Importantly, the whole process functions without prior knowledge or internal details of the underlying LLM agents.”

  • The paper relies on this without support.

    Solvability and completeness are sufficient criteria for evaluating and refining MAS designs.

    “MAS-ZERO evaluates solvability and completeness.”
    “Solvability requires that each sub-task be independently and completely solvable by its sub-MAS”

  • The paper relies on this without support.

    MAS-ZERO is the first inference-time-only automatic MAS design framework with zero supervision.

    “We presented MAS-ZERO, the first inference-time-only automatic MAS design framework with zero supervision.”

  • The paper relies on this without support.

    The generated temperature of the agents is typically non-zero.

    “the generated temperature of the agents is typically non-zero.”
5 of 5

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Another automatic baseline exceeds MAS-ZERO on Llama GPQA

    Major

    The text says the Qwen AIME24 result is the only case where MAS-ZERO falls behind an automatic method. Table 1 also shows ADAS at 53.60 on Llama GPQA, above MAS-ZERO's 52.41.

  • GPQA split is not 20% validation and 80% test

    Minor

    The stated GPQA split has 32 validation and 166 test questions. That makes validation 32 of 198 questions, or about 16.16%, rather than the stated 20%.

2 of 2

Cites

Flagged first, then by citations

Cited by

Sorted by citations

Authors
Ke, Z., Xu, A., Ming, Y., Nguyen, X.-P., Chin, R., Xiong, C., & Joty, S.
Affiliations
Salesforce (United States) (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2025
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2505.14996
arXiv
2505.14996
OpenAlex
W4414580636
Cited by
0
References
57
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by-sa)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Multi-Agent Systems and Negotiation
  • Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Keywords
multi-agent systems, large language models, self-evolution, test-time adaptation, problem decomposition, agent composition, reasoning, coding, agentic search

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-24.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-24.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review