CoChat
Sign Up
PreprintCited by 0Topic ModelingNatural Language Processing TechniquesAI-based Problem Solving and Planning

Are Retrials All You Need? Enhancing Large Language Model Reasoning Without Verbalized Feedback

Open access
Potamitis, Nearchos, Arora, Akhil·2025·arXiv
DOI 10.48550/arxiv.2504.12951arXiv 2504.12951
Open this paper in CoChat — free accountOpen PDF

Abstract

Recent advancements in large language models (LLMs) have catalyzed the development of general-purpose autonomous agents, demonstrating remarkable performance in complex reasoning tasks across various domains. This surge has spurred the evolution of a plethora of prompt-based reasoning frameworks. A recent focus has been on iterative reasoning strategies that refine outputs through self-evaluation and verbalized feedback. However, these strategies require additional computational complexity to enable models to recognize and correct their mistakes, leading to a significant increase in their cost. In this work, we introduce the concept of ``retrials without feedback'', an embarrassingly simple yet powerful mechanism for enhancing reasoning frameworks by allowing LLMs to retry problem-solving attempts upon identifying incorrect answers. Unlike conventional iterative refinement methods, our method does not require explicit self-reflection or verbalized feedback, simplifying the refinement process. Our findings indicate that simpler retrial-based approaches often outperform more sophisticated reasoning frameworks, suggesting that the benefits of complex methods may not always justify their computational costs. By challenging the prevailing assumption that more intricate reasoning strategies inherently lead to better performance, our work offers new insights into how simpler, more efficient approaches can achieve optimal results. So, are retrials all you need?

AI snapshot

Written from the paper’s full text.

TL;DR

The paper introduces “retrials without feedback,” a simple mechanism that lets LLMs retry problem-solving attempts after incorrect answers without self-reflection or verbalized feedback. Across several benchmarks and two base models, simpler retrial-based methods such as CoT often outperform more complex reasoning frameworks while being more cost-efficient.

Key findings

  • Retrials without feedback improve reasoning frameworks by simply re-attempting unsolved problems until a correct answer is found or a computational budget is exhausted, without explicit self-evaluation or verbalized feedback.
  • CoT prompting consistently outperformed alternative methods across benchmarks and models; on Game of 24, CoT achieved a 94% success rate, a result the authors say methods such as RAFA or ToT would need multiple hundreds of dollars (≃ 600$) to achieve on GPT-4.
  • Cost efficiency depended on both task and base model: with GPT-4o-mini, IO and CoT showed comparable performance, while with LLaMA-3.3-70B, CoT substantially outperformed IO on Game of 24 and HotpotQA.
  • Higher temperature improved success rates for GPT-4o-mini; for CoT at temperature 1.0, the experiment reached 100% accuracy at half the allocated budget, while ToT also predominantly improved but not as clearly.

Methods

  • Benchmarks: Game of 24 (1362 puzzles; test set puzzles 901–1000; success rate and cost), HumanEval (161 Python programs; pass@1 and cost), and HotpotQA (100 randomly selected samples; exact match and cost).
  • Baselines: standard IO prompting, CoT, ToT, and Reflexion, evaluated with GPT-4o-mini and LLaMA-3.3-70B as base models.
  • Analysis: iterative retrials under a constrained budget, where all samples are attempted in the first trial and unsolved samples are retried until the budget is exhausted or all are solved; additional temperature analysis was run for Game of 24 across CoT and ToT.

Limitations

  • The paper states that the main limitation is that the method cannot be applied to tasks where the answer cannot be directly verified, such as HotpotQA, where the correct answer is hidden and only used for evaluation, making direct verification impossible during solving.
  • The paper states that temperature analysis was limited to Game of 24 across CoT and ToT due to budget and time restrictions, and that LLaMA-3.3-70B temperature results had not yet plateaued, limiting conclusiveness.
  • The text makes evident that HumanEval results may change with a larger budget: the authors note that Reflexion and ToT appear likely to surpass other methods if allowed more budget, though they say this does not contradict their cost-efficiency claims.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2504.12951

Cites

0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

4 of 10 cited sources checked · 3 supported · 0 flagged · accuracy not yet measured

  • Findingown data
    § 4 Main results · p. 3

    Under the retrial mechanism, simpler methods such as IO and CoT prompting are significantly more cost-efficient than complex reasoning approaches such as ToT and Reflexion across benchmarks and models.

    “Our results demonstrate that simple methods such as IO and CoT prompting are significantly more efficient than complex reasoning approaches such as ToT (Yao et al., 2024) or Reflexion (Shinn et al., 2023).”
    • Also stated in: § 5 Discussion and Concluding Insights · p. 4 · boosted
      “Our results in Figs. 1 and 4 have shown that methods such as Chain of Thought are more cost-efficient than complex reasoning strategies on a variety of models and tasks.”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites Yao et al., 2024

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Shinn et al., 2023 source
    • Rests on an assumption: The benchmarks used (Game of 24, HumanEval, HotpotQA) are representative of the broader class of reasoning tasks to which the paper's conclusions about cost-efficiency generalize.
  • Findingown data
    § 1 Introduction · p. 2

    Under the retrial mechanism, simpler methods such as CoT often outperform more sophisticated reasoning frameworks such as ToT or Reflexion.

    “Our results show that under the retrial mechanism, simpler methods such as chain-ofthoughts (Wei et al., 2022) often outperform more sophisticated reasoning frameworks such as treeof-thoughts (Yao et al., 2024) or Reflexion (Shinn et al., 2023).”
    • Also stated in: § Abstract · p. 1 · hedged
      “Our findings indicate that simpler retrial-based approaches often outperform more sophisticated reasoning frameworks, suggesting that the benefits of complex methods may not always justify their computational costs.”
    Quote in the paperSubstantial claim1 of 3 sources checked
    • cites Wei et al., 2022 source
    • cites Yao et al., 2024

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Shinn et al., 2023 source
    • Rests on an assumption: The benchmarks used (Game of 24, HumanEval, HotpotQA) are representative of the broader class of reasoning tasks to which the paper's conclusions about cost-efficiency generalize.
  • Findingown data
    § 4 Main results · p. 3

    For Game of 24, CoT achieved a 94% success rate.

    “In fact, for Game of 24, CoT achieved a 94% success rate: a result that methods such as (Liu et al., 2024; Yao et al., 2024) would need multiple hundreds of dollars ( ≃ 600$) to achieve on GPT-4.”
    Quote in the paperSubstantial claim
  • Findingown data
    § 4 Main results · p. 4

    There is a quality gap between GPT-4o-mini and Llama-3.3-70B in the Game of 24 and HotpotQA benchmarks.

    “This is particularly evident in the quality gap between GPT-4o-mini and Llama-3.3-70B in the Game of 24 and HotpotQA benchmarks.”
    Quote in the paperSubstantial claim
  • Findingown data
    § 4 Main results · p. 4

    For GPT-4o-mini, at a temperature value of 1.0, the experiment achieves 100% accuracy at half of the allocated budget.

    “Furthermore, we can even see that for a temperature value of 1 . 0 the experiment already achieves 100% accuracy at half of the allocated budget.”
    Quote in the paperNot a substantial claim
  • May combine two claims. 5
  • Classified as probably not a substantial claim. 3

  • May combine two claims.

    Premisereasoning
    § 1 Introduction · p. 1

    Iterative refinement methods that use verbalized feedback require additional computational complexity and increase cost with each iteration.

    “While effective, these approaches add an additional layer of complexity, requiring the model to both recognize its own errors and articulate useful self-correction strategies.”
    • Also stated in: § 1 Introduction · p. 1 · plain
      “Moreover, owing to an incremental increase in the context window size, verbalized feedback monotonically increases the cost for every subsequent iteration of the overall reasoning pipeline, thereby rendering iterative refinementbased methods exorbitantly expensive.”
    Quote in the paperSubstantial claim2 of 2 sources checked
    • cites Madaan et al., 2023 source
    • cites Shinn et al., 2023 source

      The Introduction (p. 1) says useful reflective feedback requires identifying mistakes and generating actionable insights, supporting the citing sentence’s point about added self-correction demands. Section 3 (pp. 3–4) describes evaluation and self-reflection on successive trials, but our copy does not state that computational complexity or cost increases with each iteration. (LLM, conf=0.86)

      Experimental — accuracy not yet measured

  • May combine two claims.

    Findingreasoning
    § 4 Main results · p. 4

    With GPT-4o-mini, CoT and IO exhibit comparable performance; with Llama-3.3-70B, CoT substantially outperforms IO.

    “While the two methods exhibit comparable performance with the former model, CoT substantially outperforms IO in the latter.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 4 Main results · p. 4

    In HumanEval and HotpotQA, both CoT and IO methods reach a performance plateau earlier, whereas in Game of 24 the efficiency peak is more gradual.

    “For instance, in the HumanEval and HotpotQA tasks, both CoT and IO methods reach a performance plateau earlier, whereas, in the Game of 24 task, the efficiency peak is more gradual.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § B Additional results · p. 6

    For HotpotQA with LLaMA 3.3 70B, the ToT approach slightly outperforms CoT.

    “However, for HotpotQA, the ToT approach slightly outperforms CoT.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingreasoning
    § B Additional results · p. 7

    For LLaMA 3.3 70B, the performance trends across temperature have not yet plateaued.

    “Unlike the results observed with GPT-4omini (Fig.2), the performance trends here have not yet plateaued.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § B Additional results · p. 6

    For LLaMA 3.3 70B, both IO and CoT prompting strategies demonstrate significantly higher cost-efficiency than complex methods.

    “Similar to the results observed with GPT-4o-mini, both IO and CoT prompting strategies demonstrate significantly higher cost-efficiency.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingnone given
    § 5 Discussion and Concluding Insights · p. 4

    The performance of the retrial concept can be improved even further by properly tuning the model's temperature.

    “Finally, we have shown that the performance of the retrial concept can be improved even further by properly tuning the model's temperature”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The temperature tuning results obtained for Game of 24 generalize to other tasks and models.

  • May combine two claims.

    Findingown data
    § B Additional results · p. 7

    For Game of 24 and HotpotQA, ToT and Reflexion exhibit greater sample efficiency, achieving strong performance with relatively few re-trials.

    “For Game of 24 and HotpotQA, ToT and Reflexion exhibit greater sample efficiency, achieving strong performance with relatively few re-trials.”
    Quote in the paperNot a substantial claim
8 of 8

  • The paper relies on this without support.

    The benchmarks used (Game of 24, HumanEval, HotpotQA) are representative of the broader class of reasoning tasks to which the paper's conclusions about cost-efficiency generalize.

    “Our results in Figs. 1 and 4 have shown that methods such as Chain of Thought are more cost-efficient than complex reasoning strategies on a variety of models and tasks.”

  • The paper relies on this without support.

    The deterministic verifiers used for Game of 24 and HumanEval are correct and trivial.

    “Finally, our method uses trivial deterministic verifiers to decide whether an answer has been found and in this case, no retry is needed.”

  • The paper relies on this without support.

    The temperature tuning results obtained for Game of 24 generalize to other tasks and models.

    “Due to budget and time restrictions, we only conduct this analysis for Game of 24 across the CoT and ToT methods.”
    “Finally, we have shown that the performance of the retrial concept can be improved even further by properly tuning the model's temperature”
3 of 3

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • CoT is not the top-performing method on every benchmark and model

    Major

    The main results say CoT consistently outperforms alternatives across all benchmarks and models, but the appendix says ToT slightly outperforms CoT on HotpotQA with LLaMA 3.3 70B.

1 of 1

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 7

  1. Askell, et al. 2020. Language models are few-shot learners. In NeurIPS , volume 33, pages 1877-1901.
  2. Zhihan Liu, Hao Hu, Shenao Zhang, Hongyi Guo, Shuqi Ke, Boyi Liu, and Zhaoran Wang. 2024. Reason for future, act for now: A principled architecture for autonomous LLM agents. In ICML .
  3. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. Self-consistency improves chain of thought reasoning in language models. In ICLR .
  4. Robert West, Joelle Pineau, and Doina Precup. 2009. Wikispeedia: An online game for inferring semantic distances between concepts. In IJCAI , page 1598-1603.
  5. Ling Yang, Zhaochen Yu, Tianjun Zhang, Shiyi Cao, Minkai Xu, Wentao Zhang, Joseph E. Gonzalez, and Bin Cui. 2024. Buffer of thoughts: Thoughtaugmented reasoning with large language models. In NeurIPS .
  6. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. 2023. ReAct: Synergizing Reasoning and Acting in Language Models. In ICLR .
  7. Yang Zhilin, Qi Peng, Zhang Saizheng, Bengio Yoshua, Cohen William, Salakhutdinov Ruslan, and Christopher D. Manning. 2018. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing .

Authors
Potamitis, N., & Arora, A.
Affiliations
Aarhus University (DK)
Venue
arXiv
Publisher
arXiv
Published
2025
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2504.12951
arXiv
2504.12951
OpenAlex
W4417090983
Cited by
0
References
0
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
Keywords
large language models, iterative reasoning, self-assessment, autonomous agents, computational cost

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-07.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-07.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review