CoChat
Sign Up
PreprintCited by 7Software Engineering ResearchAI-based Problem Solving and PlanningMulti-Agent Systems and Negotiation

FlowReasoner: Reinforcing Query-Level Meta-Agents

Open access
Gao, Hongcheng, Liu, Yue, He, Yufei, Dou, Longxu, Du, Chao, Deng, Zhijie, Hooi, Bryan, Lin, Min +1 more·2025·arXiv
DOI 10.48550/arxiv.2504.15257arXiv 2504.15257
Open this paper in CoChat — free accountOpen PDF

Abstract

This paper proposes a query-level meta-agent named FlowReasoner to automate the design of query-level multi-agent systems, i.e., one system per user query. Our core idea is to incentivize a reasoning-based meta-agent via external execution feedback. Concretely, by distilling DeepSeek R1, we first endow the basic reasoning ability regarding the generation of multi-agent systems to FlowReasoner. Then, we further enhance it via reinforcement learning (RL) with external execution feedback. A multi-purpose reward is designed to guide the RL training from aspects of performance, complexity, and efficiency. In this manner, FlowReasoner is enabled to generate a personalized multi-agent system for each user query via deliberative reasoning. Experiments on both engineering and competition code benchmarks demonstrate the superiority of FlowReasoner. Remarkably, it surpasses o1-mini by 10.52% accuracy across three benchmarks. The code is available at https://github.com/sail-sg/FlowReasoner.

AI snapshot

Written from the paper’s full text.

TL;DR

FLOWREASONER is a query-level meta-agent that generates a personalized multi-agent system for each user query by distilling DeepSeek R1 and then reinforcing reasoning with external execution feedback. It outperforms baselines on three code-generation benchmarks, with FLOWREASONER-14B reaching 81.89 overall accuracy and surpassing o1-mini by 10.52%.

Key findings

  • FLOWREASONER-14B outperforms all competing approaches across BigCodeBench, HumanEval, and MBPP, achieving 63.53, 97.26, and 92.15 accuracy respectively, for 81.89 overall.
  • It improves over the strongest baseline, MaAS, by 5 percentage points overall (81.89 vs. 76.81) and exceeds its underlying worker o1-mini by 10.52% overall (81.89 vs. 71.37).
  • Ablations show 14B variants outperform 7B variants, and SFT+RL improves over SFT alone: 7B SFT 78.89 → SFT+RL 80.53; 14B SFT 81.50 → SFT+RL 81.89.
  • The trained meta-agent generalizes across workers: with o1-mini, Qwen2.5Coder-32b, Claude3.5, and GPT-4o-mini, FLOWREASONER-14B maintains performance, though o1-mini yields the best overall results.

Methods

  • The pipeline first synthesizes warm-up SFT data using DeepSeek R1-671B as a meta-agent, then fine-tunes DeepSeek-R1-Distill-Qwen-7B/14B on approximately 1,400 items from three datasets.
  • It then applies GRPO reinforcement learning with external execution feedback and a multi-purpose reward covering performance (pass rate), complexity (AST complexity score), and diversity (distinctness ratio).
  • Workflows are represented as code using six operators (Code Generator, Format Generator, Ensemble, Review, Revise, Code Test); evaluation uses BigCodeBench, HumanEval, and MBPP with pass@1, 10 workflow iterations, and each workflow evaluated 3 times.

Limitations

  • The paper does not include an explicit limitations section; no limitations are stated by the authors.
  • Evident from the text: the study is restricted to code-generation tasks because they provide test cases as external execution feedback, so applicability to non-code domains is not demonstrated.
  • Evident from the text: the main evaluation uses three code benchmarks, and the appendix includes failure cases without a systematic analysis of failure modes.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2504.15257

Cites

0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 18 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    FLOWREASONER surpasses o1-mini by 10.52% accuracy across three benchmarks.

    “Remarkably, it surpasses o1-mini by 10.52% accuracy across three benchmarks.”
    • Also stated in: § 6 Conclusion · boosted
      “notably improving o1-mini's performance by overall 10.52% on three benchmarks”
    Quote in the paperSubstantial claim
  • Findingown data

    The 14B variant consistently outperformed the 7B counterpart across all benchmarks.

    “We observed that the 14B variant consistently outperformed the 7B counterpart across all benchmarks”
    Quote in the paperSubstantial claim
  • Findingown data

    Experiments on engineering and competition code benchmarks demonstrate the superiority of FLOWREASONER.

    “Experiments on both engineering and competition code benchmarks demonstrate the superiority of FLOWREASONER.”
    • Also stated in: § 1 Introduction · boosted
      “We demonstrate the superiority of FLOWREASONER via extensive experiments and open-source it.”
    Quote in the paperSubstantial claim
  • Findingown data

    FLOWREASONER-14B consistently outperforms all competing approaches across the three benchmark datasets.

    “FLOWREASONER-14B consistently outperforms all competing approaches across the three benchmark datasets.”
    Quote in the paperSubstantial claim
  • Findingown data

    Within each model size, versions trained with both SFT and RL exhibited notable improvements over those trained with SFT alone.

    “within each model size, versions trained with both SFT and RL exhibited notable improvements over those trained with SFT alone”
    Quote in the paperSubstantial claim
  • May combine two claims. 18
  • The quote only partly states this claim. 3
  • Classified as probably not a substantial claim. 19

  • May combine two claims.

    Findingown data

    FLOWREASONER-14B (SFT+RL) with Qwen2.5Coder-32b achieves 52.67 on BigCodeBench, 93.69 on HumanEval, and 78.90 on MBPP.

    “| Qwen2.5Coder-32b | 50.17 | 92.89 | 80.40 | 52.67 | 93.69 | 78.90 |”
    Quote in the paperSubstantial claim

  • The quote only partly states this claim.

    Findingown data

    For 7B models, adding RL to SFT improves overall accuracy from 78.89 to 80.53.

    “| SFT | 7B | 61.79 | 96.38 | 87.22 | 78.89 |”
    • Also stated in: § # Extract key problem requirements and constraints · plain
      “| SFT+RL 7B | 62.78 | 96.95 | 89.86 | 80.53 |”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    FLOWREASONER generated successful cases for BigCodeBench, HumanEval, and MBPP.

    “Fig. 6, Fig. 8 and Fig. 7 demonstrate three successful cases generated by FLOWREASONER for BigCodeBench, HumanEval and MBPP.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    FLOWREASONER-7B (SFT+RL) with Claude3.5 achieves 60.67 on BigCodeBench, 96.07 on HumanEval, and 87.63 on MBPP.

    “| Claude3.5 | 60.67 | 96.07 | 87.63 | 61.12 | 96.52 | 89.82 |”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    FLOWREASONER generated failure cases for BigCodeBench, HumanEval, and MBPP.

    “Fig. 10, Fig. 11 and Fig. 9 demonstrate three failure cases generated by FLOWREASONER for BigCodeBench, HumanEval and MBPP.”
    Quote in the paperNot a substantial claim

  • The quote only partly states this claim.

    Findingown data

    For 14B models, adding RL to SFT improves overall accuracy from 81.50 to 81.89.

    “| SFT | 14B | 62.83 | 97.18 | 91.91 | 81.50 |”
    • Also stated in: § # Extract key problem requirements and constraints · plain
      “| SFT+RL 14B | 63.53 | 97.26 | 92.15 | 81.89 |”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingreasoning

    The HumanEval workflow is substantially more concise than the BigCodeBench workflow.

    “In contrast, the HumanEval workflow is substantially more concise”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    FLOWREASONER-14B (SFT+RL) with Claude3.5 achieves 61.12 on BigCodeBench, 96.52 on HumanEval, and 89.82 on MBPP.

    “| Claude3.5 | 60.67 | 96.07 | 87.63 | 61.12 | 96.52 | 89.82 |”
    Quote in the paperSubstantial claim

  • The quote only partly states this claim.

    Findingown data

    FLOWREASONER-14B achieves higher overall accuracy (81.89) than the best baseline MaAS (76.81).

    “| MaAS [57] | 60.33 | 95.42 | 84.16 | 76.81 |”
    • Also stated in: § # Extract key problem requirements and constraints · plain
      “| Ours | FLOWREASONER-14B | 63.53 | 97.26 | 92.15 | 81.89 |”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [57]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingreasoning

    The workflow corresponding to the BigCodeBench task exhibits greater complexity than the HumanEval workflow.

    “The workflow corresponding to the BigCodeBench task exhibits greater complexity”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    FLOWREASONER-7B (SFT+RL) with GPT-4o-mini achieves 59.18 on BigCodeBench, 94.24 on HumanEval, and 82.19 on MBPP.

    “| GPT-4o-mini | 59.18 | 94.24 | 82.19 | 59.75 | 94.52 | 82.27 |”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    FLOWREASONER-7B (SFT+RL) with Qwen2.5Coder-32b achieves 50.17 on BigCodeBench, 92.89 on HumanEval, and 80.40 on MBPP.

    “| Qwen2.5Coder-32b | 50.17 | 92.89 | 80.40 | 52.67 | 93.69 | 78.90 |”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    FLOWREASONER-14B (SFT+RL) with GPT-4o-mini achieves 59.75 on BigCodeBench, 94.52 on HumanEval, and 82.27 on MBPP.

    “| GPT-4o-mini | 59.18 | 94.24 | 82.19 | 59.75 | 94.52 | 82.27 |”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    FLOWREASONER-7B (SFT+RL) with o1-mini achieves 62.77 on BigCodeBench, 96.95 on HumanEval, and 89.86 on MBPP.

    “| o1-mini | 62.77 | 96.95 | 89.86 | 63.53 | 97.26 | 92.15 |”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Premisenone given

    Previous methods focus on task-level meta-agents, generating merely a single task-specific multi-agent system that applies to one kind of task.

    “These previous methods focus on task-level meta-agents, generating merely a single task-specific multi-agent system that applies to one kind of task, e.g., code generation task, as in Figure 1 (a).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    Graph-based methods formulate workflows as graphs or networks and automate workflow designs, but structural complexity of graphs limits their scalability.

    “However, the structural complexity of graphs limits their scalability [18].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [18]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

    Interpretationreasoning

    The greater complexity of the BigCodeBench workflow reflects the more challenging and engineering-oriented nature of the task.

    “reflecting the more challenging and engineering-oriented nature of the task.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Premisereasoning

    Search-based approaches prompt an LLM to act as a meta-agent, generating multiple candidate multi-agent system designs, and then employ complex search algorithms on a carefully designed search set to identify the optimal system.

    “Subsequently, complex search algorithms are employed on the carefully designed search set to identify the optimal system for completing the task.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    Current open-source models struggle to produce reliable workflows solely based on instruction prompts and are heavily reliant on predefined, manually crafted workflows.

    “This highlights a limitation of current open-source models, which struggle to produce reliable workflows solely based on instruction prompts and are heavily reliant on predefined, manually crafted workflows.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    The selection of BigCodeBench, HumanEval, and MBPP enables comprehensive evaluation of workflow discovery across a diverse spectrum of code generation challenges.

    “This selection enables us to comprehensively evaluate workflow discovery across a diverse spectrum of code generation challenges.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    FLOWREASONER reduces human resource costs while enhancing scalability by enabling more adaptive and efficient multi-agent systems that dynamically optimize their structure based on specific user queries rather than relying on fixed workflows for entire task categories.

    “Our approach reduces human resource costs while enhancing scalability by enabling more adaptive and efficient multi-agent systems that dynamically optimize their structure based on specific user queries rather than relying on fixed workflows for entire task categories.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    The comparison against three categories of baselines enables a comprehensive assessment of FLOWREASONER's effectiveness relative to both static and adaptive baselines.

    “This comparison enables a comprehensive assessment of FLOWREASONER's effectiveness relative to both static and adaptive baselines.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given

    Through SFT, the model's reasoning ability regarding workflow generation is unlocked.

    “Through SFT, we unlock the model's reasoning ability regarding workflow generation.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    The conciseness of the HumanEval workflow aligns with the relative simplicity and algorithmic focus of the task.

    “aligning with the relative simplicity and algorithmic focus of the task.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Premisenone given

    Manually designed multi-agent systems lead to extensive human costs.

    “This kind of manually designed multi-agent system leads to extensive human costs.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Most multi-agent systems are manually designed.

  • Probably not a substantial claim: may not pass S4 (based).

    Premisenone given

    Code generation tasks can provide test cases as external execution feedback for the workflow construction process.

    “we restrict our scope to code generation tasks, as they can provide test cases as external execution feedback for the workflow construction process.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Premisereasoning

    For individual user queries, these one-size-fits-all systems lack the capability for automatic adaptation.

    “However, for individual user queries, these one-size-fits-all systems lack the capability for automatic adaptation.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    State-of-the-art methods represent multi-agent systems as programming codes and prompt a performant LLM as a meta-agent to optimize workflows via complex search algorithms on carefully designed search sets.

    “state-of-the-art methods represent the multi-agent systems as programming codes [18] and prompt a performant LLM, e.g., GPT-4o, as a meta-agent to optimize workflows via complex search algorithms on carefully designed search sets [58,42,57].”
    Quote in the paperSubstantial claim0 of 4 sources checked
    • cites [18]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [58]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [42]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [57]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Classified as probably not a substantial claim.

    Premisecitation

    Search-based automatic multi-agent systems are proposed to reduce human effort.

    “search-based automatic multi-agent systems [58,18,5] are proposed to reduce human effort.”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites [58]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [18]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [5]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Interpretationown data

    The planner is not tightly bound to a specific worker and can adapt its strategies effectively across diverse execution agents.

    “These results suggest that the planner is not tightly bound to a specific worker and can adapt its strategies effectively across diverse execution agents.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationreasoning

    The stronger performance of API-based models is likely attributable to their superior instruction-following capabilities.

    “likely attributable to their superior instruction-following capabilities.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    Early automatic methods optimize prompts or hyper-parameters but still rely on the fixed workflow of the multi-agent system, which requires human effort to manually design workflows for each new scenario.

    “But they still rely on the fixed workflow of the multi-agent system, which requires human effort to manually design workflows for each new scenario.”
    Quote in the paperSubstantial claim0 of 5 sources checked
    • cites [55]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [22]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [61]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [53]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [41]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Premisenone given

    Search algorithms are time-consuming and rely on the search set, which is absent in one user query.

    “Besides, the search algorithm is time-consuming and relies on the search set, which is absent in one user query.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Search sets are unavailable in the setting of query-specific multi-agent systems.

  • May combine two claims.

    Premisecitation

    In traditional multi-agent systems, agents and workflows are designed manually according to one kind of task, and the task-level system is fixed for all queries in one task.

    “in traditional multi-agent systems [17,51,26], the agents and the workflows are designed manually according to one kind of task t = P (q), as formulated S task = H(t), where H denotes human experts, and S task denotes the task-level multi-agent system which is fixed for all queries in one task.”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites [17]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [51]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [26]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductcitation

    The standard pass@1 metric for code accuracy was used to assess performance.

    “To assess performance, we used the standard pass@1 metric for code accuracy, consistent with prior work [6].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [6]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S4 (based).

    Premisereasoning

    The success of task-level meta-agents largely depends on carefully designed search sets, as they rely on complex search algorithms.

    “We first identify that the success of task-level meta-agents largely depends on carefully designed search sets, as they rely on complex search algorithms.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Search sets are unavailable in the setting of query-specific multi-agent systems.

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

    Interpretationreasoning

    The proposed method is more practical in real-world scenarios, as it can design an optimal multi-agent system for each specific query.

    “Our proposed method is more practical in real-world scenarios, as it can design an optimal multi-agent system for each specific query.”
    Quote in the paperNot a substantial claim
37 of 37

  • The paper relies on this without support.

    Most multi-agent systems are manually designed.

    “While effective, most of them are manually designed, increasing human resource costs and limiting scalability.”

  • The paper relies on this without support.

    Search sets are unavailable in the setting of query-specific multi-agent systems.

    “However, such search sets are unavailable in the setting of query-specific multi-agent systems.”

  • The paper relies on this without support.

    External execution feedback (pass rate) is a reliable signal for optimizing multi-agent systems.

    “The pass rate of the proposed solution for the given query, as the key performance indicator of the system, serves as the external execution feedback.”
3 of 3

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • DeepSeek R1 citation points to a Claude reference

    Minor

    Reference [44] is an Anthropic Claude document. The DeepSeek R1 paper appears instead as reference [45].

  • Reward's third component is described as both efficiency and diversity

    Minor

    The paper says the reward targets efficiency, but its detailed definition identifies diversity-not efficiency-as the third component.

  • Ablation discussion cites the wrong figure

    Minor

    The meta-agent and worker performance comparison is captioned Figure 4, not Figure 1.

  • 7B BigCodeBench accuracy differs between ablation tables

    Minor

    For FLOWREASONER-7B with SFT+RL and o1-mini as worker, Table 2 reports 62.78% BigCodeBench accuracy, while Table 3 reports 62.77%.

4 of 4

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 4

  1. Yilun Du, Shuang Li, Antonio Torralba, Joshua B Tenenbaum, and Igor Mordatch. Improving factuality and reasoning in language models through multiagent debate, 2023. URL https://arxiv. org/abs/2305.14325, 3, 2023.
  2. Yichong Huang, Xiaocheng Feng, Baohang Li, Yang Xiang, Hui Wang, Ting Liu, and Bing Qin. Ensemble learning for heterogeneous large language models with deep parallel collaboration. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors, Advances in Neural Information Processing Systems, volume 37, pages 119838-119860.
  3. Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Saiful Haq, Ashutosh Sharma, Thomas T Joshi, Hanna Moazam, Heather Miller, et al. Dspy: Compiling declarative language model calls into state-of-the-art pipelines. In The Twelfth International Conference on Learning Representations, 2024.
  4. Guohao Li, Hasan Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. Camel: Communicative agents for" mind" exploration of large language model society. Advances in Neural Information Processing Systems, 36:51991-52008, 2023.

Cited by

Sorted by citations

Authors
Gao, H., Liu, Y., He, Y., Dou, L., Du, C., Deng, Z., Hooi, B., Lin, M., & Pang, T.
Affiliations
  • University of Chinese Academy of Sciences (CN)
  • National University of Singapore (SG)
  • Shanghai Jiao Tong University (CN)
ORCID
Pang, Tianyu
Venue
arXiv
Publisher
arXiv
Published
2025
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2504.15257
arXiv
2504.15257
OpenAlex
W6910639071
Cited by
0
References
0
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
  • Physical Sciences › Computer Science › Artificial Intelligence › Multi-Agent Systems and Negotiation
Keywords
multi-agent system design, reinforcement learning, reward function design, code generation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-24.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-24.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review