CoChat
Sign Up
PreprintCited by 22Topic ModelingNatural Language Processing TechniquesAdvanced Neural Network Applications

More Agents Is All You Need

Open access
Li, Junyou, Zhang, Qin, Yu, Yangbin, Fu, Qiang, Ye, Deheng·2024·arXiv
DOI 10.48550/arxiv.2402.05120arXiv 2402.05120
Open this paper in CoChat — free accountOpen PDF

Abstract

We find that, simply via a sampling-and-voting method, the performance of large language models (LLMs) scales with the number of agents instantiated. Also, this method, termed as Agent Forest, is orthogonal to existing complicated methods to further enhance LLMs, while the degree of enhancement is correlated to the task difficulty. We conduct comprehensive experiments on a wide range of LLM benchmarks to verify the presence of our finding, and to study the properties that can facilitate its occurrence. Our code is publicly available at: https://github.com/MoreAgentsIsAllYouNeed/AgentForest

AI snapshot

Written from the paper’s full text.

TL;DR

Agent Forest, a simple sampling-and-voting method, shows LLM performance scales with the number of instantiated agents across reasoning and code-generation benchmarks. It is orthogonal to existing prompting and multi-agent methods, with larger gains on harder tasks and weaker models.

Key findings

  • Agent Forest improves accuracy across tasks and LLMs: e.g., GSM8K rises from 0.35 to 0.59 for Llama2-13B, 0.54 to 0.74 for Llama2-70B, and 0.73 to 0.85 for GPT-3.5-Turbo; MATH rises from 0.03 to 0.09, 0.05 to 0.11, and 0.29 to 0.39, respectively.
  • Scaling smaller models can beat larger ones: enhanced Llama2-13B reaches 59% on GSM8K, outperforming Llama2-70B at 54%.
  • The method is compatible with other approaches: on Llama2-13B GSM8K, CoT improves from 0.39 to 0.56 (+0.17), ZS-COT from 0.40 to 0.61 (+0.21), SPP from 0.19 to 0.42 (+0.23), and Reflection from 0.36 to 0.59 (+0.23); Debate integration fails on HumanEval for Llama2-13B and Llama2-70B.
  • Relative gains are larger for harder tasks and weaker models: GSM8K gains are 69%, 37%, and 16% for Llama2-13B, Llama2-70B, and GPT-3.5-Turbo, versus MATH gains of 200%, 120%, and 34%; gains increase then decrease with inherent difficulty, increase with number of steps, and performance increases with prior probability.

Methods

  • Agent Forest has two phases: sample N outputs from a single LLM or a method f_M(x), then majority-vote by cumulative similarity; BLEU is used for open-ended code generation and occurrence frequency for close-ended tasks.
  • Evaluated on GSM8K, MATH, MMLU, Chess, and HumanEval using Llama2-Chat 13B/70B, GPT-3.5-Turbo, and GPT-4; ensemble size scales to 40, except Debate is limited to 10 due to computational overhead, with results averaged over 10 independent runs.
  • Integrated with prompt engineering (CoT, Zero-Shot CoT, SPP) and multi-agent collaboration (Debate, Reflection); one-way ANOVA across ensemble sizes (1, 10, 20, 30, 40) reports all p-values < 0.05.

Limitations

  • Stated in the paper: token usage increases proportionally with the number of agents or when combined with other methods; escalating costs commonly exist in works requiring multiple LLM calls, and the paper leaves optimizing sampling cost as future work.
  • Stated in the paper: Debate integration is limited to ensemble size 10 due to significant computational overhead, and Debate combined with code generation fails for Llama2-13B and Llama2-70B because referencing other agents’ answers disrupts code logic.
  • Evident from the text: GPT-4 is used only for comparison and presents only “Single” results, so Agent Forest is not evaluated on GPT-4.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2402.05120

Cites

32 of 32 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

4 of 11 cited sources checked · 1 supported · 0 flagged · accuracy not yet measured

  • Premisereasoning
    § 2 Related Work · p. 3

    Prior multiple LLM agents collaboration methods primarily focus on interaction structures between agents rather than the relationship between the number of agents and performance.

    “However, these methods primarily focus on the interaction structures between LLM agents, rather than the relationship between the number of agents and performance.”
    Quote in the paperSubstantial claim2 of 3 sources checked
    • cites Du et al. (2023)

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Liang et al. (2023) source

      The paper presents a multi-agent debate framework built around interactions among debaters and a judge (Section 2). But it also explicitly examines the relationship between agent number and performance: Section 4.3 reports that translation performance declined as the number of debaters increased (Table 7). Thus the contrast with studying agent number overstates what this source supports. (LLM, conf=0.86)

      Experimental — accuracy not yet measured
    • cites Xiong et al. (2023) source

      The Introduction (page not provided) says that multiple LLMs have been used to collaborate, with each targeting a different subtask or aspect. It does not characterize prior methods as primarily focused on interaction structures rather than the relationship between agent count and performance. Searches for that comparison found no passage making it. (LLM, conf=0.72)

      Experimental — accuracy not yet measured
  • Findingnone given
    § Abstract · p. 1

    Agent Forest is orthogonal to existing complicated methods to further enhance LLMs.

    “this method, termed as Agent Forest, is orthogonal to existing complicated methods to further enhance LLMs”
    Quote in the paperSubstantial claim
  • Findingown data
    § Accuracy Curves In GSM8K · p. 2

    Combining Agent Forest with other existing methods further improves performance.

    “Moreover, by combining our method with other existing methods, we find the performance can be further improved.”
    • Also stated in: § 5.2 Compatibility · p. 7 · plain
      “Table 3 shows that by integrating our method with other methods, the performance can be further improved across different LLMs and tasks, despite these methods have different implementations.”
    Quote in the paperNot a substantial claim
  • Findingown data
    § 6.3 Number of Steps · p. 10

    Performance gains from Agent Forest increase with the number of steps.

    “Property 2.1 : Gains increase with the number of steps.”
    • Also stated in: § 6.3 Number of Steps · p. 10 · hedged
      “Figure 6 (middle) shows that as the number of steps increases, there is a corresponding increase in performance gain.”
    • Also stated in: § 7 Conclusions and Future Work · p. 12 · plain
      “2) performance gains increase with the number of steps”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Task difficulty can be decomposed into three orthogonal dimensions: inherent difficulty, number of steps, and prior probability of a correct answer.
    • Rests on an assumption: The designed mathematical task can isolate each of the three difficulty dimensions.
  • Findingown data
    § Accuracy Curves In GSM8K · p. 2

    In GSM8K, accuracy increases with ensemble size across Llama2-13B, Llama2-70B and GPT-3.5-Turbo.

    “The accuracy increases with ensemble size across Llama2-13B, Llama2-70B and GPT-3.5-Turbo in GSM8K.”
    Quote in the paperSubstantial claim
  • May combine two claims. 22
  • Classified as probably not a substantial claim. 11

  • May combine two claims.

    Premisecitation
    § 2 Related Work · p. 3

    Prior supervised LLM ensemble approaches require task-specific annotated data and exhibit limited generalizability.

    “These approaches primarily employ supervised learning, necessitating task-specific annotated data, and exhibit limited generalizability.”
    Quote in the paperSubstantial claim1 of 3 sources checked
    • cites Wan et al. (2024) source

      The Additional information section says that LLM-Blender may generalize inadequately to test tasks, supporting a narrower version of the generalizability point. The Model Fusion section describes ensemble methods, but our copy does not say that these approaches primarily use supervised learning or require task-specific annotated data. No page numbers are available in the indexed passages. (LLM, conf=0.79)

      Experimental — accuracy not yet measured
    • cites Jiang et al. (2023)

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Chen et al. (2023b)

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation
    § 7 Conclusions and Future Work · p. 12

    Current benchmarks focus narrowly on accuracy, leading to costly agents.

    “This aligns with recent findings as discussed in Kapoor et al. (2024) that current benchmarks focus narrowly on accuracy, leading to costly agents.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Kapoor et al. (2024) source

  • May combine two claims.

    Findingown data
    § 6.2 Inherent Difficulty · p. 10

    The most notable performance gains are seen at I = 100 and I = 200, consistent across all S and K settings.

    “The most notable gains are seen at I = 100 and I = 200, consistent across all S and K settings.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Task difficulty can be decomposed into three orthogonal dimensions: inherent difficulty, number of steps, and prior probability of a correct answer.
    • Rests on an assumption: The designed mathematical task can isolate each of the three difficulty dimensions.

  • May combine two claims.

    Findingown data
    § 6 Understanding the Performance Gains · p. 8

    The more challenging task MATH yields relative performance gains of 34%-200%, compared to 16%-69% on the easier task GSM8K.

    “Moreover, the more challenging task MATH yields gains of 34%-200%, in contrast to only 16%-69% on the easier task GSM8K.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 5.1 Generalizability · p. 6

    In general reasoning tasks, Agent Forest accuracy gains range from 1% to 4% on Chess and from 5% to 11% on MMLU.

    “In general reasoning tasks, the accuracy gains range from 1% to 4% on the Chess and from 5% to 11% on the MMLU.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 5.2 Compatibility · p. 7

    When integrating Agent Forest with other methods, arithmetic reasoning gains range from 10% to 21% on GSM8K and from 1% to 15% on MATH.

    “To be specific, in arithmetic reasoning tasks, our method enhances these methods to further improvement, yielding increases between 10% and 21% on the GSM8K dataset, and between 1% and 15% on the MATH dataset.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 5.1 Generalizability · p. 6

    The enhanced Llama2-13B model achieves 59% accuracy on GSM8K, outperforming Llama2-70B, which scores 54%.

    “For instance, the enhanced Llama2-13B model achieves 59% accuracy on the GSM8K dataset, outperforming the Llama2-70B model, which scores 54%.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingnone given
    § 2 Related Work · p. 3

    Agent Forest validates effectiveness in reasoning tasks and in generation tasks.

    “In contrast, our method not only validates effectiveness in reasoning tasks but also in generation tasks.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 5.1 Generalizability · p. 7

    Agent Forest outperforms other methods used standalone in most cases and always enhances other methods across various tasks and LLMs.

    “Our method outperforms other methods used standalone in most cases and always enhances other methods across various tasks and LLMs.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 5.2 Compatibility · p. 7

    When integrating Agent Forest with other methods, general reasoning gains range from 1% to 13% on Chess and from 1% to 11% on MMLU.

    “In general reasoning tasks, integration with other methods generally achieves performance gains ranging from 1% to 13% in the Chess task and from 1% to 11% in the MMLU task.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 6.3 Number of Steps · p. 10

    When I and K are increased, performance gains are more significant, e.g., 4%-18% gains over {I=10,K=2} compared to 16%-48% over {I=100,K=4}.

    “Additionally, we find that when I and K are increased (which indicates a higher difficulty), the performance gains are more significant, e.g., 4%-18% gains over { I = 10 , K = 2 } compared to 16%-48% over { I = 100 , K = 4 } .”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Task difficulty can be decomposed into three orthogonal dimensions: inherent difficulty, number of steps, and prior probability of a correct answer.
    • Rests on an assumption: The designed mathematical task can isolate each of the three difficulty dimensions.

  • May combine two claims.

    Findingown data
    § 5.3 Effectiveness · p. 7

    Agent Forest outperforms other methods in standalone cases, except on the Chess task using Llama2-13B and Llama2-70B.

    “From Table 3, we find that our method outperforms other methods in standalone cases, except on the Chess task using Llama2-13B and Llama2-70B.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 6.3 Number of Steps · p. 10

    Accuracy decreases as the number of steps increases, although each step has equal inherent difficulty.

    “Figure 7 (left) shows that although each step has equal inherent difficulty, the accumulation of errors from previous steps lead to a decrease in accuracy as the number of steps increases.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Task difficulty can be decomposed into three orthogonal dimensions: inherent difficulty, number of steps, and prior probability of a correct answer.
    • Rests on an assumption: The designed mathematical task can isolate each of the three difficulty dimensions.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingnone given
    § 2 Related Work · p. 3

    Agent Forest is compatible with a broader range of methods, such as prompt engineering (including CoT) and multiple LLM agents collaboration.

    “Moreover, our method is compatible with a broader range of methods, such as prompt engineering (including CoT) and multiple LLM agents collaboration.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingreasoning
    § 5.5 Token usage · p. 8

    Token usage increases proportionally with the number of agents or when combined with other methods.

    “Given that our method scales up by increasing the ensemble size, the token usage increases proportionally with the number of agents or when combined with other methods.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 6.3 Number of Steps · p. 10

    Step-wise Agent Forest yields greater improvements than Agent Forest, with 15%-42% gains that increase with inherent difficulty.

    “Figure 7 (middle) shows that compared to Agent Forest, Step-wise Agent Forest yields greater improvements. e.g., we see 15%-42% gains, which increase with inherent difficulty.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Task difficulty can be decomposed into three orthogonal dimensions: inherent difficulty, number of steps, and prior probability of a correct answer.
    • Rests on an assumption: The designed mathematical task can isolate each of the three difficulty dimensions.

  • May combine two claims.

    Findingown data
    § 5.1 Generalizability · p. 6

    In arithmetic reasoning tasks, Agent Forest accuracy gains range from 12% to 24% on GSM8K and from 6% to 10% on MATH.

    “Specifically, in arithmetic reasoning tasks, the accuracy gains range from 12% to 24% on the GSM8K and from 6% to 10% on the MATH.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 6 Understanding the Performance Gains · p. 8

    Within the same task, Llama2-13B gains 28%-200% relative performance, while GPT-3.5-Turbo gains only 8%-16%.

    “Specifically, we observe that within the same task, the smaller model, Llama2-13B, gains ranging from 28%-200%, but only 8%-16% over GPT-3.5-Turbo.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § B.2 Statistical Test Results · p. 17

    All p-values from one-way ANOVA across ensemble sizes (1, 10, 20, 30, 40) are less than 0.05, demonstrating significant differences in accuracy between ensemble sizes.

    “The detailed p-values are provided in Table 7, where all p-values are less than 0.05, demonstrating that the differences in accuracy between ensemble sizes are significant.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning
    § 6.4 Prior Probability · p. 11

    Subtasks with varying prior probabilities can be addressed using different models.

    “Moreover, subtasks with varying prior probabilities can be addressed using different models.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Task difficulty can be decomposed into three orthogonal dimensions: inherent difficulty, number of steps, and prior probability of a correct answer.
    • Rests on an assumption: The designed mathematical task can isolate each of the three difficulty dimensions.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning
    § 6.4 Prior Probability · p. 11

    Cost savings can be achieved by using simpler, less expensive models for the easier, higher-probability subtasks.

    “Additionally, cost savings can be achieved by using simpler, less expensive models for the easier, higher-probability subtasks.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Task difficulty can be decomposed into three orthogonal dimensions: inherent difficulty, number of steps, and prior probability of a correct answer.
    • Rests on an assumption: The designed mathematical task can isolate each of the three difficulty dimensions.

  • Classified as probably not a substantial claim.

    Premisereasoning
    § Accuracy Curves In GSM8K · p. 2

    There lacks a dedicated in-depth study on the scaling property of raw agents instantiated by LLMs.

    “So far, there lacks a dedicated in-depth study on such a phenomenon.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data
    § 4 Experimental Setup · p. 6

    The effectiveness of Agent Forest was evaluated by averaging results across 10 independent runs.

    “Specifically, the effectiveness of our method is evaluated by averaging the results across 10 independent runs.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationnone given
    § 2 Related Work · p. 3

    Agent Forest is unsupervised and does not require additional training data.

    “In contrast, our method is unsupervised, without the need for additional training data.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning
    § 5.5 Token usage · p. 8

    One can trade a higher token budget for improved performance on specific tasks.

    “When addressing specific tasks, one can trade a higher token budget for improved performance.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given
    § 4 Experimental Setup · p. 5

    The study encompasses various LLMs, multiple tasks, and integration with multiple methods.

    “Our comprehensive study encompasses various LLMs, multiple tasks, and the integration with multiple methods.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning
    § Accuracy Curves In GSM8K · p. 2

    Comparable performance can be achieved without additional handcraft prompt design or complex collaboration frameworks.

    “This implies that comparable performance can be achieved without the need for additional handcraft prompt design or complex collaboration frameworks.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning
    § 4 Experimental Setup · p. 6

    When integrating Agent Forest with Debate, the ensemble size was limited to 10 due to computational overhead.

    “However, when integrating our method with the Debate Du et al. (2023), the ensemble size is limited to 10 due to the significant computational overhead introduced by the communication architecture.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Du et al. (2023)

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 7 Conclusions and Future Work · p. 12

    Escalating costs commonly exist in works requiring multiple LLM calls.

    “Nonetheless, such a challenge of escalating costs commonly exists in works requiring multiple LLM calls Wang et al. (2023b); Du et al. (2023).”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites Wang et al. (2023b)

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Du et al. (2023)

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Interpretationreasoning
    § 7 Conclusions and Future Work · p. 12

    Agent Forest is orthogonal to different existing methods, and combining it with them can lead to further improvements.

    “Importantly, this method is orthogonal to different existing methods, which can lead to further improvements when combined with them.”
    • Also stated in: § B.1 Additional Accuracy Curves · p. 16 · plain
      “Compatibility. Our method can generally enhance other methods by increasing the ensemble size.”
    Quote in the paperSubstantial claim
30 of 30

  • The paper relies on this without support.

    The repeated LLM samples are sufficiently diverse and independent for majority voting to improve accuracy.

    “Subsequently, majority voting is used to determine the final result.”

  • The paper relies on this without support.

    The benchmark tasks used (GSM8K, MATH, Chess, MMLU, HumanEval) are representative enough to support general conclusions about LLM performance scaling.

    “We conduct comprehensive experiments on a wide range of LLM benchmarks to verify the presence of our finding, and to study the properties that can facilitate its occurrence.”

  • The paper relies on this without support.

    The three dimensions used to classify problem difficulty (inherent difficulty, length of reasoning steps, prior probability of the correct answer) adequately capture task difficulty.

    “We classify difficulty into three dimensions: the inherent difficulty, the length of reasoning steps, and the prior probability of the correct answer.”

  • The paper relies on this without support.

    The performance gains from Agent Forest are attributable to the number of agents rather than solely to increased token usage or compute.

    “Given that our method scales up by increasing the ensemble size, the token usage increases proportionally with the number of agents or when combined with other methods.”

  • The paper relies on this without support.

    Task difficulty can be decomposed into three orthogonal dimensions: inherent difficulty, number of steps, and prior probability of a correct answer.

    “we categorize the difficulty of a given task into three orthogonal dimensions: 1) the inherent difficulty of the task; 2) the number of steps required to solve the task; 3) the prior probability of the correct answer.”

  • The paper relies on this without support.

    The designed mathematical task can isolate each of the three difficulty dimensions.

    “To explicitly explore the impact of these dimensions, we conduct a mathematical task designed to isolate each one.”
6 of 6

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Simply calling the underlying model multiple times can increase accuracy.

    “This means simply calling the underlying model multiple times can increase accuracy [27, 6, 26].”

    From AI Agents That Matter (2024) · cites this paper as [26]

  • Calling language models repeatedly and taking a majority vote can lead to non-trivial increases in accuracy across benchmarks like GSM-8K, MATH, Chess, and MMLU.

    “Calling language models repeatedly and taking a majority vote can lead to non-trivial increases in accuracy across benchmarks like GSM-8K, MATH, Chess, and MMLU [26, 6, 48].”

    From AI Agents That Matter (2024) · cites this paper as [26]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Llama2-13B HumanEval result differs between tables

    Major

    Table 2 reports 0.18 for Agent Forest with Llama2-13B on HumanEval, whereas Table 3 reports 0.25 for the same model and method.

  • GSM8K integration gains fall outside the stated range

    Minor

    The stated range is 10%-21%, but Table 3 includes a reported gain of 27 percentage points for SPP with Llama2-70B.

  • MMLU integration gain exceeds the stated maximum

    Minor

    The stated maximum gain is 11%, but Table 3 reports a 21-percentage-point MMLU gain for SPP with Llama2-13B.

  • HumanEval integration gain exceeds the stated maximum

    Minor

    The text gives a 2%-7% gain range, whereas Table 3 reports a 15-percentage-point gain for Reflection with GPT-3.5-Turbo.

  • Claim that integration always enhances conflicts with unchanged results

    Minor

    The Table 3 caption says integration always enhances other methods, but the Debate HumanEval result remains zero after integration.

  • Prior-probability experiment says K is both fixed and varied

    Minor

    The experiment says it maintains K constant, then says it varies K from 4 to 32.

  • Relative-gain ranges disagree with Table 6

    Minor

    The paragraph gives ranges of 28%-200% for Llama2-13B and 8%-16% for GPT-3.5-Turbo. Table 6 instead lists 69% and 200% for Llama2-13B, and 16% and 34% for GPT-3.5-Turbo.

7 of 7

Cites

Flagged first, then by citations

Research works printed but not linked: 6

  1. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. CoRR , abs/2110.14168, 2021b. · arXiv
  2. Yao Fu, Hao Peng, Ashish Sabharwal, Peter Clark, and Tushar Khot. Complexity-based prompting for multi-step reasoning. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023 . OpenReview.net, 2023.
  3. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021 . OpenReview.net, 2021a.
  4. Papers), ACL 2023, Toronto, Canada, July 9-14, 2023 , pp. 14165-14178. Association for Computational Linguistics, 2023.
  5. Yifei Li, Zeqi Lin, Shizhuo Zhang, Qiang Fu, Bei Chen, Jian-Guang Lou, and Weizhu Chen. Making language models better reasoners with step-aware verifier. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pp. 5315-5333, Toronto, Canada, July 2023b. Association for Computational Linguistics.
  6. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog , 1(8):9, 2019.

Cited by

Sorted by citations

Related

Authors
Li, J., Zhang, Q., Yu, Y., Fu, Q., & Ye, D.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2024
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2402.05120
arXiv
2402.05120
OpenAlex
W4391709247
Cited by
22
References
32
Citations by year
2026: 4 · 2025: 9 · 2024: 9
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
Keywords
large language models, multi-agent systems, task difficulty

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-10.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-10.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review