CoChat
Sign Up
Cited by 12Topic ModelingNatural Language Processing TechniquesMultimodal Machine Learning Applications

Large Language Models Can Self-Correct with Key Condition Verification

Open access
Wu, Zhenyu, Zeng, Qingkai, Zhang, Zhihan, Tan, Zhaoxuan, Shen, Chao, Jiang, Meng·2024·Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing
DOI 10.18653/v1/2024.emnlp-main.714
Open this paper in CoChat — free accountOpen PDF

Abstract

Intrinsic self-correct was a method that instructed large language models (LLMs) to verify and correct their responses without external feedback.Unfortunately, the study concluded that the LLMs could not self-correct reasoning yet.We find that a simple yet effective prompting method enhances LLM performance in identifying and correcting inaccurate answers without external feedback.That is to mask a key condition in the question, add the current response to construct a verification question, and predict the condition to verify the response.The condition can be an entity in an open-domain question or a numerical value in an arithmetic question, which requires minimal effort (via prompting) to identify.We propose an iterative verify-then-correct framework to progressively identify and correct (probably) false responses, named PROCO.We conduct experiments on three reasoning tasks.On average, PROCO, with GPT-3.5-Turbo-1106 as the backend LLM, yields +6.8 exact match on four open-domain question answering datasets, +14.1 accuracy on three arithmetic reasoning datasets, and +9.6 accuracy on a commonsense reasoning dataset, compared to Self-Correct.Our implementation is made publicly avail

AI snapshot

Written from the paper’s full text.

TL;DR

PROCO is a zero-shot iterative verify-then-correct prompting method that masks a key condition in a question, uses the prior answer as a new condition to form a verification question, and corrects answers without external feedback. With GPT-3.5-Turbo-1106 it improves over Self-Correct by +6.8 EM on four open-domain QA datasets, +14.1 accuracy on three arithmetic datasets, and +9.6 accuracy on CSQA.

Key findings

  • On GPT-3.5-Turbo-1106, PROCO vs Self-Correct: NQ +7.9 EM (48.0 vs 40.1), TriviaQA +7.4 (78.7 vs 71.3), WebQ +7.8 (47.0 vs 39.2), HotpotQA +4.0 (33.0 vs 29.0), and CSQA +9.6 accuracy (75.5 vs 65.9). On arithmetic, PROCO reaches GSM8K 87.1 vs 75.1, AQuA 65.2 vs 48.7 (+16.5), and MATH 41.5 vs 27.6.
  • With GPT-4-0125-Preview, PROCO scores GSM8K 97.6, CSQA 86.7, and HotpotQA 61.0 EM, compared with CoT (95.5, 82.0, 49.0) and Self-Correct (91.5, 79.5, 49.0).
  • GenRead + PROCO boosts EM by +6.1 on NQ, +7.6 on TriviaQA, +5.4 on WebQ, +9.0 on HotpotQA, and accuracy by +9.1 on CSQA. PROCO also outperforms GenRead/RAG in EM across three open-domain QA benchmarks using one external document vs five, with an average 12.8% higher EM than GenRead and 9.2% higher than RAG while using half the tokens.
  • On WebQ, PROCO overall EM is 47.2 vs Self-Correct 38.0, F1 57.5 vs 44.1, average tokens 1368.3 vs 3446.0, and average time 15.32 vs 33.68. On GSM8K, PROCO incorrectly alters correct answers 2.5% of the time and corrects incorrect answers 8.2%, vs Self-Correct at 9.1% and 7.6%.

Methods

  • PROCO generates an initial answer via CoT (or RAG/GenRead), identifies a key condition (numerical values via SimCSE + regex; entities/concepts via zero-shot LLM prompting), masks it as “X,” appends the previous answer and a verification question, solves it, and verifies via match-based checking for arithmetic or proposition-based checking for open-domain/commonsense. Incorrect answers are added to a set of potentially incorrect answers and used as feedback with the hint “the answer is likely not in {set}”; the loop stops when an answer is verified correct, the corrected answer matches the previous one, or max iterations T=3 is reached.
  • Evaluated on three reasoning tasks and eight datasets: arithmetic (GSM8K, AQuA, MATH), open-domain QA (NQ, TriviaQA, WebQ, HotpotQA), and commonsense (CSQA). Backends: GPT-3.5-Turbo-1106, GPT-4-0125-Preview, and Mixtral-8x7B. Baselines: CoT, Self-Correct, CoVe, GenRead, and RAG. Metrics: EM/F1 for open-domain QA and accuracy for other tasks. Temperature 0.7; external-document baselines use M=5, and PROCO combinations use M=1.
  • Analyses include iteration-count effects, token/time efficiency, error-change rates, comparison with self-consistency, and case studies.

Limitations

  • Stated: the study focuses exclusively on English; non-English tasks were excluded and the method may not perform well for them. The problems solved are generally short, averaging 52.3 words, with answers typically numerical values or entities; longer problems or answers not numerical/entity-based are left for future research.
  • Evident from the text: key condition identification for open-domain and commonsense questions relies on zero-shot LLM prompting, so errors in that step could propagate, but the paper does not quantify this failure mode.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.18653/v1/2024.emnlp-main.714

Cites

33 of 50 references verified0 citation errors · 17 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

2 of 7 cited sources checked · 0 supported · 0 flagged · accuracy not yet measured

  • Findingown data

    The average iteration number for PROCO across all benchmarks is less than 3.

    “Additionally, the average iteration number for PROCO across all benchmarks is less than 3.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: If the verified answer and the key condition are equivalent, then the generated answer is likely correct.
    • Rests on an assumption: If the verified answer a(v)_t equals the key condition c_k, then the previous answer a_{t-1} is most likely correct.
    • Rests on an assumption: If the proposition 'If the answer to question Q(v)_t is c_k, then X could also be a(v)_t' is verified as correct, then the previously generated answer a_{t-1} is likely correct.
    • Rests on an assumption: The key condition c_k is crucial for problem-solving.
    • Rests on an assumption: Key conditions are found in context sentences with high semantic relevance to the query sentence q.
    • Rests on an assumption: The set of potentially incorrect answers as feedback will guide LLMs to avoid repeating previous mistakes.
  • Findingown data

    PROCO with GPT-3.5-Turbo-1106 yields an average +6.8 exact match on four open-domain question answering datasets compared to Self-Correct.

    “On average, PROCO, with GPT-3.5-Turbo-1106 as the backend LLM, yields +6.8 exact match on four open-domain question answering datasets, +14.1 accuracy on three arithmetic reasoning datasets, and +9.6 accuracy on a commonsense reasoning dataset, compared to Self-Correct.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: If the verified answer and the key condition are equivalent, then the generated answer is likely correct.
    • Rests on an assumption: If the verified answer a(v)_t equals the key condition c_k, then the previous answer a_{t-1} is most likely correct.
    • Rests on an assumption: If the proposition 'If the answer to question Q(v)_t is c_k, then X could also be a(v)_t' is verified as correct, then the previously generated answer a_{t-1} is likely correct.
    • Rests on an assumption: The key condition c_k is crucial for problem-solving.
    • Rests on an assumption: Key conditions are found in context sentences with high semantic relevance to the query sentence q.
    • Rests on an assumption: The set of potentially incorrect answers as feedback will guide LLMs to avoid repeating previous mistakes.
  • Findingown data

    PROCO significantly enhances problem-solving performance across five benchmarks when combined with baseline methods using external documents.

    “Table 2 demonstrates that PROCO significantly enhances problem-solving performance across five benchmarks when combined with baseline methods using external documents.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: If the verified answer and the key condition are equivalent, then the generated answer is likely correct.
    • Rests on an assumption: If the verified answer a(v)_t equals the key condition c_k, then the previous answer a_{t-1} is most likely correct.
    • Rests on an assumption: If the proposition 'If the answer to question Q(v)_t is c_k, then X could also be a(v)_t' is verified as correct, then the previously generated answer a_{t-1} is likely correct.
    • Rests on an assumption: The key condition c_k is crucial for problem-solving.
    • Rests on an assumption: Key conditions are found in context sentences with high semantic relevance to the query sentence q.
    • Rests on an assumption: The set of potentially incorrect answers as feedback will guide LLMs to avoid repeating previous mistakes.
  • Findingown data

    PROCO with GPT-3.5-Turbo-1106 yields an average +14.1 accuracy on three arithmetic reasoning datasets compared to Self-Correct.

    “On average, PROCO, with GPT-3.5-Turbo-1106 as the backend LLM, yields +6.8 exact match on four open-domain question answering datasets, +14.1 accuracy on three arithmetic reasoning datasets, and +9.6 accuracy on a commonsense reasoning dataset, compared to Self-Correct.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: If the verified answer and the key condition are equivalent, then the generated answer is likely correct.
    • Rests on an assumption: If the verified answer a(v)_t equals the key condition c_k, then the previous answer a_{t-1} is most likely correct.
    • Rests on an assumption: If the proposition 'If the answer to question Q(v)_t is c_k, then X could also be a(v)_t' is verified as correct, then the previously generated answer a_{t-1} is likely correct.
    • Rests on an assumption: The key condition c_k is crucial for problem-solving.
    • Rests on an assumption: Key conditions are found in context sentences with high semantic relevance to the query sentence q.
    • Rests on an assumption: The set of potentially incorrect answers as feedback will guide LLMs to avoid repeating previous mistakes.
  • Findingown data

    The EM score of PROCO improves with more iterations in both WebQ and TriviaQA.

    “Figure 4 (a) shows that the EM score of PROCO improves with more iterations in both WebQ and TriviaQA.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: If the verified answer and the key condition are equivalent, then the generated answer is likely correct.
    • Rests on an assumption: If the verified answer a(v)_t equals the key condition c_k, then the previous answer a_{t-1} is most likely correct.
    • Rests on an assumption: If the proposition 'If the answer to question Q(v)_t is c_k, then X could also be a(v)_t' is verified as correct, then the previously generated answer a_{t-1} is likely correct.
    • Rests on an assumption: The key condition c_k is crucial for problem-solving.
    • Rests on an assumption: Key conditions are found in context sentences with high semantic relevance to the query sentence q.
    • Rests on an assumption: The set of potentially incorrect answers as feedback will guide LLMs to avoid repeating previous mistakes.
  • Classified as probably not a substantial claim. 8
  • May combine two claims. 23
  • The quote only partly states this claim. 1

  • Classified as probably not a substantial claim.

    Premisecitation

    Numerical values are crucial in arithmetic reasoning tasks.

    “Numerical values are crucial in arithmetic reasoning tasks, so we select those relevant to the problem solving process as key conditions (Wu et al., 2024b;Jiang et al., 2019Jiang et al., , 2021;;Zeng et al., 2020).”
    Quote in the paperSubstantial claim2 of 3 sources checked
    • cites Wu et al., 2024b

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Jiang et al., 2019 source

      The Introduction describes numerical values such as temperature and pH as possible condition-tuple objects in scientific statements. Section 4.2.3 evaluates sequence tagging and tuple extraction, not arithmetic reasoning. Searches for “numerical values” and “arithmetic reasoning” found no passage addressing whether numerical values are crucial to arithmetic reasoning tasks. (LLM, conf=0.76)

      Experimental — accuracy not yet measured
    • cites Zeng et al., 2020 source

      The available abstract (%X, no page given) describes pre-fine-tuning for scientific named-entity recognition, not arithmetic reasoning. Searches for numerical values, arithmetic reasoning, and problem-solving conditions returned no passage addressing the claim. (LLM, conf=0.72)

      Experimental — accuracy not yet measured

  • May combine two claims.

    Findingown data

    For GSM8K, PROCO incorrectly alters correct answers 2.5% of the time and corrects incorrect answers 8.2% of the time, compared to Self-Correct, which makes correct answers incorrect 9.1% of the time and fixes incorrect answers 7.6% of the time.

    “For GSM8K, PROCO incorrectly alters correct answers 2.5% of the time and corrects incorrect answers 8.2% of the time, compared to Self-Correct, which makes correct answers incorrect 9.1% of the time and fixes incorrect answers 7.6% of the time.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: If the verified answer and the key condition are equivalent, then the generated answer is likely correct.
    • Rests on an assumption: If the verified answer a(v)_t equals the key condition c_k, then the previous answer a_{t-1} is most likely correct.
    • Rests on an assumption: If the proposition 'If the answer to question Q(v)_t is c_k, then X could also be a(v)_t' is verified as correct, then the previously generated answer a_{t-1} is likely correct.
    • Rests on an assumption: The key condition c_k is crucial for problem-solving.
    • Rests on an assumption: Key conditions are found in context sentences with high semantic relevance to the query sentence q.
    • Rests on an assumption: The set of potentially incorrect answers as feedback will guide LLMs to avoid repeating previous mistakes.

  • May combine two claims.

    Findingown data

    PROCO achieves an average 12.8% higher EM score than GenRead and 9.2% higher than RAG, while using half the tokens.

    “PROCO achieves an average 12.8% higher EM score than GenRead and 9.2% higher than RAG, while using half the tokens.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: If the verified answer and the key condition are equivalent, then the generated answer is likely correct.
    • Rests on an assumption: If the verified answer a(v)_t equals the key condition c_k, then the previous answer a_{t-1} is most likely correct.
    • Rests on an assumption: If the proposition 'If the answer to question Q(v)_t is c_k, then X could also be a(v)_t' is verified as correct, then the previously generated answer a_{t-1} is likely correct.
    • Rests on an assumption: The key condition c_k is crucial for problem-solving.
    • Rests on an assumption: Key conditions are found in context sentences with high semantic relevance to the query sentence q.
    • Rests on an assumption: The set of potentially incorrect answers as feedback will guide LLMs to avoid repeating previous mistakes.

  • May combine two claims.

    Findingown data

    Self-Correct shows minimal improvement in WebQ and even a 1.0 EM score decrease in TriviaQA with more iterations.

    “In contrast, Self-Correct shows minimal improvement in WebQ and even a 1.0 EM score decrease in TriviaQA with more iterations.”
    Quote in the paperSubstantial claim

  • May combine two claims. The quote only partly states this claim.

    Findingown data

    PROCO consistently outperforms baselines on all arithmetic reasoning benchmarks using significantly fewer tokens.

    “PROCO consistently outperforms baselines on all benchmarks using significantly fewer tokens.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: If the verified answer and the key condition are equivalent, then the generated answer is likely correct.
    • Rests on an assumption: If the verified answer a(v)_t equals the key condition c_k, then the previous answer a_{t-1} is most likely correct.
    • Rests on an assumption: If the proposition 'If the answer to question Q(v)_t is c_k, then X could also be a(v)_t' is verified as correct, then the previously generated answer a_{t-1} is likely correct.
    • Rests on an assumption: The key condition c_k is crucial for problem-solving.
    • Rests on an assumption: Key conditions are found in context sentences with high semantic relevance to the query sentence q.
    • Rests on an assumption: The set of potentially incorrect answers as feedback will guide LLMs to avoid repeating previous mistakes.

  • May combine two claims.

    Findingown data

    PROCO outperforms GenRead/RAG in EM scores across three open-domain question-answering benchmarks, using just one external document compared to five.

    “Table 5 shows that PROCO outperforms GenRead/RAG in EM scores across three opendomain question-answering benchmarks, using just one external document compared to five.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: If the verified answer and the key condition are equivalent, then the generated answer is likely correct.
    • Rests on an assumption: If the verified answer a(v)_t equals the key condition c_k, then the previous answer a_{t-1} is most likely correct.
    • Rests on an assumption: If the proposition 'If the answer to question Q(v)_t is c_k, then X could also be a(v)_t' is verified as correct, then the previously generated answer a_{t-1} is likely correct.
    • Rests on an assumption: The key condition c_k is crucial for problem-solving.
    • Rests on an assumption: Key conditions are found in context sentences with high semantic relevance to the query sentence q.
    • Rests on an assumption: The set of potentially incorrect answers as feedback will guide LLMs to avoid repeating previous mistakes.

  • May combine two claims.

    Findingreasoning

    PROCO uses fewer tokens and achieves better accuracy on both GSM8K and CSQA compared to CoT + SC.

    “We find that PROCO uses fewer tokens and achieves better accuracy on both tasks.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: If the verified answer and the key condition are equivalent, then the generated answer is likely correct.
    • Rests on an assumption: If the verified answer a(v)_t equals the key condition c_k, then the previous answer a_{t-1} is most likely correct.
    • Rests on an assumption: If the proposition 'If the answer to question Q(v)_t is c_k, then X could also be a(v)_t' is verified as correct, then the previously generated answer a_{t-1} is likely correct.
    • Rests on an assumption: The key condition c_k is crucial for problem-solving.
    • Rests on an assumption: Key conditions are found in context sentences with high semantic relevance to the query sentence q.
    • Rests on an assumption: The set of potentially incorrect answers as feedback will guide LLMs to avoid repeating previous mistakes.

  • May combine two claims.

    Findingown data

    Extensive experiments on eight complex reasoning datasets demonstrate the effectiveness and efficiency of PROCO.

    “Extensive experiments on eight complex reasoning datasets demonstrate the effectiveness and efficiency of our proposed method.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingnone given

    The problems solved in this paper are generally short, averaging 52.3 words, with answers typically being numerical values or entities.

    “The problems solved in this paper are generally short, averaging 52.3 words, with answers typically being numerical values or entities.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    PROCO accurately judges and corrects wrong answers.

    “PROCO accurately judges and corrects wrong answers.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Compared to CoT + SC, PROCO exhibits higher accuracy and consumes fewer tokens.

    “Compared to CoT + SC, PROCO not only exhibits higher accuracy but also consumes fewer tokens.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Findingnone given

    PROCO reduces errors generated by methods that use external documents, ensuring correct source citation.

    “Additionally, PROCO reduces errors generated by methods that use external documents, ensuring correct source citation.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    RAG + PROCO and PROCO accurately identify the first appearance of the phrase 'Patience is a virtue' in the 5th century.

    “In contrast, RAG + PROCO and PROCO accurately identify the first appearance of the phrase”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The phrase 'Patience is a virtue' first appeared in the 5th century.

  • May combine two claims.

    Findingown data

    PROCO achieves significant improvements in both black-box and open-source LLMs.

    “We conduct extensive experiments on three complex reasoning tasks and demonstrate that PROCO achieves significant improvements in both black-box and open-source LLMs.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: If the verified answer and the key condition are equivalent, then the generated answer is likely correct.
    • Rests on an assumption: If the verified answer a(v)_t equals the key condition c_k, then the previous answer a_{t-1} is most likely correct.
    • Rests on an assumption: If the proposition 'If the answer to question Q(v)_t is c_k, then X could also be a(v)_t' is verified as correct, then the previously generated answer a_{t-1} is likely correct.
    • Rests on an assumption: The key condition c_k is crucial for problem-solving.
    • Rests on an assumption: Key conditions are found in context sentences with high semantic relevance to the query sentence q.
    • Rests on an assumption: The set of potentially incorrect answers as feedback will guide LLMs to avoid repeating previous mistakes.

  • May combine two claims.

    Findingown data

    PROCO is more effective at correcting errors without introducing new ones, enhancing LLM performance in complex reasoning tasks.

    “Furthermore, PROCO is more effective at correcting errors without introducing new ones, enhancing LLM performance in complex reasoning tasks.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: If the verified answer and the key condition are equivalent, then the generated answer is likely correct.
    • Rests on an assumption: If the verified answer a(v)_t equals the key condition c_k, then the previous answer a_{t-1} is most likely correct.
    • Rests on an assumption: If the proposition 'If the answer to question Q(v)_t is c_k, then X could also be a(v)_t' is verified as correct, then the previously generated answer a_{t-1} is likely correct.
    • Rests on an assumption: The key condition c_k is crucial for problem-solving.
    • Rests on an assumption: Key conditions are found in context sentences with high semantic relevance to the query sentence q.
    • Rests on an assumption: The set of potentially incorrect answers as feedback will guide LLMs to avoid repeating previous mistakes.

  • May combine two claims.

    Findingown data

    CoT generates an incorrect answer and is unable to determine the origin of the phrase 'Patience is a virtue'.

    “CoT generates an incorrect answer, unable to determine the origin of the phrase”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingreasoning

    PROCO demonstrates superior self-correction compared to Self-Correct and CoVe, achieving gains of +9.2 and +4.1 on WebQ over Self-Correct and CoVe, respectively.

    “In performance comparisons of different prompting methods, PROCO demonstrates superior self-correction compared to Self-Correct and CoVe, achieving gains of +9.2 and +4.1 on WebQ over Self-Correct and CoVe, respectively.”
    • Also stated in: § A.6 Sample Predictions for Complex Reasoning Datasets · plain
      “PROCO shows superior self-correction compared to baseline methods that include self-correction processes.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    CoT guides LLMs to generate intermediate reasoning paths instead of generating the final answer directly, which helps the LLMs simulate the human-like reasoning process.

    “CoT guides LLMs to generate intermediate reasoning paths instead of generating the final answer directly, which helps the LLMs simulate the human-like reasoning process.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Wei et al., 2022 source

  • May combine two claims.

    Interpretationreasoning

    CoT + SC merely solves the problem multiple times, and this repeated independent process can lead to the same mistakes, making the frequent answer still incorrect.

    “This is because, unlike PROCO's verification and correctness processes, CoT + SC merely solves the problem multiple times, this repeated independent process can lead to the same mistakes, making the frequent answer still incorrect.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    CoT prompting yields promising results on several reasoning tasks.

    “Wei et al. (2022) proposed chain-of-thought (CoT) prompting and yielded promising results on several reasoning tasks, such as arithmetic reasoning (Kojima et al., 2022;Zhou et al., 2023), commonsense reasoning (Wei et al., 2022;Zhang et al., 2023 et al., 2023b), and open-domain question answering (Wang et al., 2023a), using only a few or no reasoning exemplars.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Wei et al., 2022 source

  • Classified as probably not a substantial claim. May combine two claims.

    Premisenone given

    CoT enables LLMs to handle complex reasoning tasks, but they are sensitive to mistakes in the reasoning path, as any mistake can lead to an incorrect answer.

    “Although CoT enables LLMs to handle complex reasoning tasks, they are sensitive to mistakes in the reasoning path, as any mistake can lead to an incorrect answer.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationreasoning

    PROCO progressively identifies incorrect answers, preventing LLMs from repeating previous mistakes and achieving continuous improvement.

    “This is because PROCO progressively identifies incorrect answers, preventing LLMs from repeating previous mistakes and achieving continuous improvement.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    This study focused exclusively on addressing complex reasoning tasks in English, with non-English tasks excluded from training and test data.

    “This study focused exclusively on addressing complex reasoning tasks in English, with non-English tasks excluded from our training and test data.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    By providing natural language feedback based on verification results, we can steer LLMs away from incorrect answers, thus enhancing their performance in various reasoning tasks.

    “By providing natural language feedback based on verification results, we can steer LLMs away from incorrect answers, thus enhancing their performance in various reasoning tasks.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: If the verified answer and the key condition are equivalent, then the generated answer is likely correct.
    • Rests on an assumption: If the verified answer a(v)_t equals the key condition c_k, then the previous answer a_{t-1} is most likely correct.
    • Rests on an assumption: If the proposition 'If the answer to question Q(v)_t is c_k, then X could also be a(v)_t' is verified as correct, then the previously generated answer a_{t-1} is likely correct.
    • Rests on an assumption: The key condition c_k is crucial for problem-solving.
    • Rests on an assumption: Key conditions are found in context sentences with high semantic relevance to the query sentence q.
    • Rests on an assumption: The set of potentially incorrect answers as feedback will guide LLMs to avoid repeating previous mistakes.

  • May combine two claims.

    Interpretationreasoning

    If the answer is verified to be correct, it is taken as the final answer, exiting the loop, thus saving significant time and tokens.

    “If the answer is verified to be correct, it is taken as the final answer, exiting the loop, thus saving significant time and tokens.”
    Quote in the paperSubstantial claim
25 of 25

  • The paper relies on this without support.

    If the verified answer and the key condition are equivalent, then the generated answer is likely correct.

    “If the verified answer and the key condition are equivalent, it indicates that the generated answer is likely to be correct.”

  • The paper relies on this without support.

    If the verified answer a(v)_t equals the key condition c_k, then the previous answer a_{t-1} is most likely correct.

    “For arithmetic questions, if a (v) t is equal to c k , it indicates that the previous answer a t-1 is most likely correct.”

  • The paper relies on this without support.

    If the proposition 'If the answer to question Q(v)_t is c_k, then X could also be a(v)_t' is verified as correct, then the previously generated answer a_{t-1} is likely correct.

    “If the proposition is verified as correct, it indicates that the previously generated answer a t-1 is likely correct, and we select a t-1 as the final answer â and exit the loop.”

  • The paper relies on this without support.

    The key condition c_k is crucial for problem-solving.

    “Among these conditions, the key condition c k is crucial for problem-solving and is used in the substitute verification process”

  • The paper relies on this without support.

    Key conditions are found in context sentences with high semantic relevance to the query sentence q.

    “Key conditions are found in context sentences {s j } m j=1 with high semantic relevance to the query sentence q.”

  • The paper relies on this without support.

    The set of potentially incorrect answers as feedback will guide LLMs to avoid repeating previous mistakes.

    “This instructs the large language model to re-answer the question while avoiding repeating previous mistakes.”

  • The paper relies on this without support.

    The condition can be an entity in an open-domain question or a numerical value in an arithmetic question, and it requires minimal effort to identify via prompting.

    “The condition can be an entity in an open-domain question or a numerical value in an arithmetic question, which requires minimal effort (via prompting) to identify.”

  • The paper relies on this without support.

    The phrase 'Patience is a virtue' first appeared in the 5th century.

    “In contrast, RAG + PROCO and PROCO accurately identify the first appearance of the phrase”
8 of 8

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Open-domain procedure cites the numerical-answer prompt for reasoning generation

    Minor

    The open-domain procedure says Prompt A.2 generates a reasoning path, but Prompt A.2 is labeled and written as a numerical answer-extraction prompt that already takes a reasoning path as input.

  • Key conditions are defined as being in context sentences, but an example places one in the query

    Minor

    The formal definition assigns every condition to a context sentence. In the open-domain example, "Skylar" is identified as a key condition even though it appears in the question's only sentence, which is the query.

2 of 2

Cites

Flagged first, then by citations

Research works printed but not linked: 1

  1. Hung Le, Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese, and Steven Hoi. 2022. CodeRL: Master- ing code generation through pretrained models and deep reinforcement learning. In Advances in Neural Information Processing Systems.

Cited by

Sorted by citations

Related

Authors
Wu, Z., Zeng, Q., Zhang, Z., Tan, Z., Shen, C., & Jiang, M.
Affiliations
  • University of Notre Dame (US)
  • Xi'an Jiaotong University (CN)
ORCID
Venue
Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing
Publisher
Association for Computational Linguistics
Published
2024
Pages
12846-12867
DOI
10.18653/v1/2024.emnlp-main.714
OpenAlex
W4404782643
Semantic Scholar
dc03957b23354055ec9d517902b294b941575c20
Cited by
10
References
40
Citations by year
2026: 6 · 2025: 4
FWCI
3.20
Type
conference paper
Language
en
Open access
Licence
cc-by
Locations
Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
Keywords
large language models, self-correction, prompting, open-domain question answering, arithmetic reasoning, commonsense reasoning

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-07.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-07.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review