LLMs cannot find reasoning errors, but can correct them given the error location

Huang et al. (2023) demonstrate that improvements from Reflexion and RCI are only from using oracle correctness ans information.

“also builds upon results from Huang et al. (2023), who demonstrate that improvements from Reflexion (Shinn et al., 2023) and RCI (Kim et al., 2023) are only from using oracle correctness ans information.”

• cites Huang et al., 2023 sourceSupported
• cites Shinn et al., 2023 sourceSupported
• cites Kim et al., 2023 sourceDisputed

  The Introduction reports that RCI outperformed other approaches on computer tasks without a task-specific reward function, and B.3 attributes improved reasoning to self-critiquing with implicit feedback. C.2 does say the reasoning-task experiments used correct labels to decide when to stop the RCI loop, but it does not say that improvements from RCI and Reflexion came only from oracle correctness information. The source is Kim et al.’s RCI paper, not a demonstration by Huang et al. (LLM, conf=0.85)

  Checked by a reasoning model — may be wrong.

  Experimental — accuracy not yet measured

BIG-Bench Mistake contains 2186 sets of CoT-style traces generated with PaLM 2 Unicorn and annotated with the location of the first logical error.

“BIG-Bench Mistake contains 2186 sets of CoTstyle traces.Each trace is generated with PaLM 2 Unicorn (Anil et al., 2023), and annotated with the location of the first logical error.”

• cites Anil et al., 2023 sourcePartly supported

  The Introduction identifies PaLM 2 as a model family, and the Model Card (Additional information; page unavailable) describes its small, medium, and large variants. Our copy does not identify a variant called “Unicorn,” so it supports the PaLM 2 identification but not the specific model name. The citation is attached to the model name, not to the citing paper’s dataset count or annotations. (LLM, conf=0.82)

  Experimental — accuracy not yet measured

Prompting-only setups require external feedback for reasoning tasks.

“It has been demonstrated that prompting-only setups can work well for stylistic or qualitative improvements (e.g.Madaan et al., 2023;Chen et al., 2023b), but would require external feedback for reasoning tasks (Huang et al., 2023;Shinn et al., 2023;Kim et al., 2023;Madaan et al., 2023).”

• cites Huang et al., 2023 sourcePartly supported

  The Conclusion and Discussion (p. 8) says current LLMs struggle to self-correct reasoning without external feedback and can benefit from valid external feedback. This supports the point about prompting-only self-correction, but the claim is broader: the paper does not say that external feedback is required for reasoning tasks generally. (LLM, conf=0.84)

  Experimental — accuracy not yet measured
• cites Shinn et al., 2023 sourcePartly supported

  In §4.2, the paper uses an external binary success signal from the environment to improve performance on HotPotQA reasoning questions. Its analysis of Figure 4 reports that self-reflection improves on a chain-of-thought baseline. This supports the value of feedback in that setup, but not the broader claim that prompting-only setups require external feedback for reasoning tasks generally. (LLM, conf=0.85)

  Experimental — accuracy not yet measured
• cites Kim et al., 2023 sourcePartly supported

  Section 3.1 (page not provided) reports that RCI prompting without external feedback matched zero-shot performance on half the reasoning benchmarks but underperformed on the others. External feedback helped this particular method; the paper does not establish that prompting-only approaches generally require it for reasoning tasks. (LLM, conf=0.87)

  Experimental — accuracy not yet measured

Prompting-only setups can work well for stylistic or qualitative improvements.

“It has been demonstrated that prompting-only setups can work well for stylistic or qualitative improvements (e.g.Madaan et al., 2023;Chen et al., 2023b), but would require external feedback for reasoning tasks (Huang et al., 2023;Shinn et al., 2023;Kim et al., 2023;Madaan et al., 2023).”

• cites Madaan et al., 2023 sourceSupported
• cites Chen et al., 2023bNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

PaLM 2 performs poorly on the selected tasks.

“These tasks are selected because 1)Anil et al. (2023) demonstrate that PaLM 2 performs poorly on these tasks, so it is likely to generate mistakes in CoT traces;”

• cites Anil et al., 2023 sourceCould not verify

  The citing sentence does not identify the selected tasks, so they cannot be matched to the report’s evaluations. The report’s §4.3 Reasoning and BIG-Bench Hard sections describe improvements over PaLM, but the passages found do not establish that PaLM 2 performs poorly on the citing paper’s selected tasks. I searched for poor performance and errors in reasoning and chain-of-thought tasks. (LLM, conf=0.72)

  Experimental — accuracy not yet measured

• AssumptionRests on an assumption: Mistakes in the selected tasks are likely to be unambiguous, minimising subjectivity during annotation.
• AssumptionRests on an assumption: Identifying mistakes for these tasks does not require expertise knowledge.

A small classifier trained with out-of-domain data exhibits stronger mistake-finding performance than prompting a large model.

“We train a small classifier with out-of-domain data, which exhibits stronger mistake-finding performance than prompting a large model.”

• Also stated in: § 1 Introduction · plain

  “As a proof-of-concept, we train a small mistakefinding classifier using out-of-domain data, which performs better than prompting a large model.”

It is possible to obtain mistake location information without ground truth labels or in-domain training data.

“Finally, we show that it is possible to obtain mistake location information without ground truth labels or in-domain training data.”

• Also stated in: § 1 Introduction · boosted

  “We demonstrate that LLMs' low accuracy at few-shot mistake-finding can be improved upon without using in-domain training data.”

The naive baseline of predicting all traces as incorrect achieves a weighted F1 average of 78.

“Note that the naive baseline of predicting all traces as incorrect achieves a weighted F1 average of 78.”

Feeding oracle mistake location information to LLMs through a backtracking setup boosts downstream task performance across the paper's 5 reasoning tasks.

“Secondly, we test the correction abilities of LLMs -separately from mistake finding -using a backtracking setup that feeds ground truth mistake location information to the model.We show that this boosts downstream task performance across our 5 reasoning tasks, indicating that LLMs' correction abilities are robust.”

• Also stated in: § 4.2 Discussion · plain

  “Our results show that, with mistake location information available, LLMs can correct their own outputs and improve overall downstream performance.”

State-of-the-art LLMs generally struggle with mistake-finding in CoT traces, even in highly objective, unambiguous cases.

“Firstly, we benchmark several state-of-the-art LLMs on their mistake-finding ability and demonstrate that they generally struggle with the task, even in highly objective, unambiguous cases.”

• Also stated in: § 3.1 Discussion · hedged

  “All five models appear to struggle with our mistake finding dataset.”

• Also stated in: § 3.1 Discussion · boosted

  “We show that state-of-the-art LLMs clearly struggle with mistake finding, even in the most simple and unambiguous cases.”

The random baseline also obtained improvements, but they are considerably smaller than if the true mistake location was used.

“Additionally, while the random baseline also obtained improvements, they are considerably smaller than if the true mistake location was used.”

The accuracy on traces with no mistakes decreases considerably from direct, trace-level prompting to CoT, step-level prompting.

“As we compare results across the three methods, we find that the accuracy on traces with no mistakes goes downfoot_6 considerably from direct, trace-level prompting to CoT, step-level prompting.”

LLMs generally struggle to find mistakes.

“We find that LLMs generally struggle to find mistakes, but, when given mistake location information, are able to correct outputs to boost performance.”

BIG-Bench Mistake covers a wider range of tasks than PRM800K to explore the reasoning capabilities of LLMs more thoroughly.

“Our dataset BIG-Bench Mistake covers a wider range of tasks to explore the reasoning capabilities of LLMs more thoroughly.”

The trained classifier does not reach the threshold required for effective backtracking.

“Despite this, the performance of our classifiers do not meet the threshold required for effective backtracking, as demonstrated in subsection 5.1.”

• Also stated in: § 7 Conclusion · plain

  “While the classifier does not reach the threshold required for effective backtracking, our results show that it is possible to improve on mistake-finding accuracy using standard machine learning techniques.”

The traces in BIG-Bench Mistake are sampled to contain more incorrectans traces than correctans traces.

“Note that the traces in BIG-Bench Mistake are sampled to contain more incorrectans traces than correctans traces (and therefore more incorrectmis traces than correctmis traces), so the overall mistake location accuracy appears higher for per-step prompting in Table4, despite the poor accuracy for correctmis traces.”

When given mistake location information, LLMs are able to correct outputs to boost performance.

“We find that LLMs generally struggle to find mistakes, but, when given mistake location information, are able to correct outputs to boost performance.”

Using PaLM 2 itself to do mistake finding will produce different and likely biased results.

“Therefore, we expect that using PaLM 2 itself to do mistake finding will produce different and likely biased results.”

Poor self-correction performance on reasoning tasks stems from LLMs' inability to find logical mistakes rather than their ability to correct a known mistake.

“In this paper, we show that poor self-correction performance stems from LLMs' inability to find logical mistakes, rather than their ability to correct a known mistake.”

• Also stated in: § 4.2 Discussion · hedged

  “This suggests that the main bottleneck in self-correction methods is the identification of mistakes, rather than the correction process.”

With Chain-of-Thought prompting, any task can be turned into a mistake-finding task.

“With Chain-of-Thought prompting, any task can be turned into a mistake-finding task.”

Changes in overall accuracy from backtracking depend on the original accuracy achieved on the task.

“While our numbers do show that our gains are higher than our losses, it should be noted that changes in the overall accuracy depends on the original accuracy achieved on the task.”

Tasks involving fewer steps are more likely to improve performance in the random baseline, as the true mistake location is more likely to be identified.

“Note that tasks involving fewer steps are more likely to improve performance in the random baseline, as the true mistake location is more likely to be identified.”

Prompting for mistakes is likely a poor strategy for determining the correctness of the final answer.

“The weighted F1 scores show that prompting for mistakes is likely a poor strategy for determining the correctness of the final answer.”

It may be possible to train classifiers to assist in backtracking.

“This suggests that it may be possible to train classifiers to assist in backtracking, and that these classifiers do not have to be large.”

It is possible to improve on mistake-finding accuracy using standard machine learning techniques.

“While the classifier does not reach the threshold required for effective backtracking, our results show that it is possible to improve on mistake-finding accuracy using standard machine learning techniques.”

Such a classifier can work on out-of-distribution mistakes.

“Further, such a classifier can work on out-of-distribution mistakes.”

PaLM 2 is likely to generate mistakes in CoT traces on these tasks.

“These tasks are selected because 1)Anil et al. (2023) demonstrate that PaLM 2 performs poorly on these tasks, so it is likely to generate mistakes in CoT traces;”

• AssumptionRests on an assumption: Mistakes in the selected tasks are likely to be unambiguous, minimising subjectivity during annotation.
• AssumptionRests on an assumption: Identifying mistakes for these tasks does not require expertise knowledge.

These classifiers do not have to be large.

“This suggests that it may be possible to train classifiers to assist in backtracking, and that these classifiers do not have to be large.”