Understanding the Dark Side of LLMs' Intrinsic Self-Correction

Probably not a substantial claim: may not pass S3 (load-bearing).

PACT gives each token or sequence's contribution to LLMs final answer.

“This method gives each token or sequence's contribution to LLMs final answer (Zhu et al., 2024;Miglani et al., 2023) .”

• cites Zhu et al., 2024Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Miglani et al., 2023 sourcePartly supported

  Section 3, “Language Model Attribution,” says Captum can quantify the impact of input tokens or segments on a specified output sequence or likely generation. Section 3.1.1 also describes attribution for tokens, phrases, and sentences. The paper supports the described function, but it discusses Captum, not PACT, so it does not establish that PACT provides it. (LLM, conf=0.83)

  Experimental — accuracy not yet measured

May combine two claims.

On both closed-sourced ChatGPT and open-sourced Llama, question repeating increases ACC by 3.2-4.9% and decreases ✓→✗(%) by 5.3-10.9%.

“On both closesourced ChatGPT and open-sourced Llama, ACC is increased by 3.2-4.9%and ✓ → ✗(%) is decreased by 5.3-10.9%.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The cost of SFT is only 0.004 dollars and 3 minutes due to the usage of very few training samples.

“Besides, the cost of SFT is only 0.004 $ and 3 minutes due to the usage of very few training samples.”

May combine two claims.

With SFT, ACC is surprisingly increased and almost all ✓→✗ cases are fixed.

“ACC is even surprisingly increased and almost all ✓ → ✗ cases are fixed.”

May combine two claims.

For SOTA LLMs, self-correction failures are reduced but not solved, and are even worse in certain tasks.

“Observation 1: Self-correction can fail in diverse tasks.For SOTA LLMs, self-correction failures are reduced but not solved.They are even worse in certain tasks.”

May combine two claims.

During Initial response generation, the confidence score of the correct answer increases with deeper layers; after Feedback and Refinement, the internal answer wavers and results in a wrong final answer.

“Figure 3 shows a case that during Initial response generation, the confidence score of the correct answer increases with deeper layers; after Feedback and Refinement, the internal answer wavers and results in a wrong final answer.”

May combine two claims.

In decision making, the LLM successfully completes the task by picking up two pillows one after the other during Initial response generation, but wants to improve efficiency by picking up two pillows at the same time, leading to failures because the environment restricts it.

“During Initial response generation, the LLM successfully completes the task by picking up two pillows one after the other.However, it wants to improve efficiency by picking up two pillows at the same time.This behavior leads to failures because the environment restricts it from doing so.”

May combine two claims.

Llama-3.1-8B suffers the greatest performance loss, with a 20.4% drop in ACC and 58.8% correct answers overturned.

“For instance, Llama-3.1-8Bsuffers the greatest performance loss, with a 20.4% drop in ACC and 58.8% correct answers overturned.”

May combine two claims.

For GPT-4o and GPT-3.5-turbo fine-tuned over Yes/No question answering, ACC is increased and ✓→✗(%) is decreased on three complex tasks.

“Table 5 shows the three complex task performance of GPT-4o and GPT-3.5-turbofine-tuned over Yes/No question answering, where ACC is increased and ✓ → ✗(%) is decreased”

May combine two claims.

For some cases, ✗→✓(%) is larger than ✓→✗(%), but the overall ACC always drops.

“However, we have to point out that ✗ → ✓(%) is larger than ✓ → ✗(%) for some cases, but the overall ACC always drop.”

Classified as probably not a substantial claim.

The Refinement prompt has 2000+ tokens compared to 9 tokens in Yes/No question answering.

“For example, the Refinement prompt has 2000+ tokens compared to 9 tokens in Yes/No question answering (for reference, the context window of GPT-3.5-turbo is 4191).”

Probably not a substantial claim: may not pass S3 (load-bearing).

In reasoning task, the refinement prompt has 1000+ tokens.

“In this example, the refinement prompt has 1000+ tokens.”

May combine two claims.

The alleviation strategies increase model's accuracy and reduce overturned correct answers.

“Table 18 and Table 19 show that the alleviation strategies increase model's accuracy and reduce overturned correct answers.”

May combine two claims.

Self-correction failure is prevalent across different self-correction prompts in Yes/No questions, and is more severe for some prompts.

“To demonstrate the prevalence of self-correction failure in Yes/No questions, we conduct experiments on 4 other self-correction prompts from recent published papers (Xie et al., 2023;Sharma et al., 2023;Huang et al., 2024), and the results show that the self-correction failure is still prevalent across different prompts, even more severe for some prompts.”

• Also stated in: § C Failure of more self-correction prompts in Yes/No questions · plain

  “To conclude, we find that self-correction failure is still prevalent across different prompts, even more severe for some prompts.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

For the top left example in Figure 8, the original question contains 4 green tokens out of 10, while all tokens in refinement prompt are green.

“Specifically, for the top left example in Figure 8, the original question contains 4 green tokens out of 10, while all tokens in refinement prompt are green.”

Probably not a substantial claim: may not pass S3 (load-bearing).

For the top right example in Figure 8, the original question contains 10 green tokens out of 11, while the refinement prompt contains only 4 green tokens out of 9.

“Specifically, for the top right example in Figure 8, the original question contains 10 green tokens out of 11, while the refinement prompt contains only 4 green tokens out of 9.”

May combine two claims.

In Initial response generation, the confidence score of correct answer increases with deeper layers; after Feedback and Refinement, the internal answer wavers and results in a wrong final answer.

“In Initial response generation (blue curve), the confidence score of correct answer increases with deeper layers; after Feedback and Refinement (orange curve), the internal answer wavers and results in a wrong final answer.”

Probably not a substantial claim: may not pass S1 (asserted), S3 (load-bearing), S4 (based).

GPT-4o 'think' 7.4 times in failure cases compared to 2.6 times in successful cases.

“GPT-4o”

May combine two claims.

Question repeating increases accuracy in most cases; SFT increases accuracy in all cases.

“Question repeating increases ACC in most cases; SFT increases ACC in all cases.”

May combine two claims. The quote only partly states this claim.

In the Figure 16 programming example, the LLM initially correctly returned the (even, odd) tuple but then reversed the order because it tried to match the example 'assert candidate(-2) == (1, 0)', causing failure.

“In initial response generation, LLM successfully completes the task by returning the (even, odd) tuple in correct order.However, the LLMs want to make the answers closer to the example”

May combine two claims.

SFT further increases accuracy and reduces overturned correct answers; question repeating works in most cases while SFT works in all cases.

“Between the two alleviation strategies, SFT further increase accuracy and reduce overturned correct answers; question repeating works in most cases while SFT works in all cases.”

May combine two claims.

Question repeating reduces ✓→✗(%) in most cases; SFT reduces ✓→✗(%) in all cases.

“Question repeating reduces ✓ → ✗(%) in most cases; SFT reduces ✓ → ✗(%) in all cases.”

May combine two claims.

After alleviation, both question repeating and SFT reduce internal answer wavering during Feedback and Refinement, and correct ✓→✗ samples to ✓→✓ samples.

“After alleviation, both question repeating and SFT reduce the internal answer wavering during Feedback and Refinement, and correct the ✓ → ✗ samples to ✓ → ✓ samples.”

• AssumptionRests on an assumption: The internal answer decoded from intermediate layers reflects the model's internal answer or hesitation.

May combine two claims.

Prompt2, Prompt4, and Prompt5 make o1-preview easier to overturn a correct answer.

“For example, we find that Prompt2, Prompt4, and Prompt5 make o1-preview easier to overturn a correct answer, while prompt5 is supposed to be the most optimal since the reference answer and feedback are provided.”

May combine two claims.

After the 16th layer, the Initial response generation curve maintains a positive value and increases, indicating the model is able to give correct answer with increasing confidence; the Feedback and Refinement curve bounces up and down at 0, indicating the model hesitates to give correct or incorrect answers.

“After the 16th layer, the two curves differ.Initial response generation curve maintains a postive value and increases in the twists and turns, indicating that the model is able to give correct answer with an increasing confidence; Feedback and Refinement curve bounces up and down at 0, indicating that the model hesitates to give correct or incorrect answers.”

May combine two claims.

In complex tasks, overthinking takes up 17.6%, cognitive overload takes up 33.3%, and perfectionism bias takes up 49.0% of the three failure patterns of human-like cognitive bias.

“Overthinking takes up 17.6%, cognitive overload takes up 33.3%, and perfectionism bias takes up 49.0%.”

Probably not a substantial claim: may not pass S1 (asserted), S3 (load-bearing), S4 (based).

o1-mini 'think' 15.4 times in failure cases compared to 5.3 times in successful cases.

“o1-mini”

Classified as probably not a substantial claim.

The model tends to just modify its answers when meeting refinement-like prompts.

“In light of our hypothesis which model tends to just modify its answers when meeting refinement-like prompts”

May combine two claims.

Because the Yes/No question answering task contains no knowledge for complex tasks, the generalization finding supports the hypothesis that self-correction failure is due to the model's behavior of changing answers when meeting refinement-like prompts rather than lacking knowledge.

“Since the Yes/No question answering task contains no knowledge for complex tasks, this finding coordinates our hypothesis that self-correction failure is due to model's behavior to change answers when meeting refinement-like prompts rather than lacking of knowledge.”

• AssumptionRests on an assumption: If an LLM's initial response is correct, it has the related knowledge.

May combine two claims.

LLMs are biased toward the refinement prompt rather than the original question itself, leading to wrong answers.

“This indicates that LLMs are biased toward refinement prompt rather than the original question itself, leading to wrong answers.”

The quote only partly states this claim.

The behavior of focusing more on original questions after SFT rectifies the prompt bias leading to wrong answers.

“This behavior rectifies the prompt bias leading to wrong answer.”

May combine two claims.

LLMs focus more on the original question attached to the end of the refinement prompt, which eliminates the undesirable effects of self-correction.

“Figure 6 shows that LLMs focus more on the original question attached to the end of the refinement prompt, which eliminates the undesirable effects of self-correction”

May combine two claims.

SFT is a better alleviation method than question repeating, which can further alleviate the negative effects of Feedback and Refinement.

“This indicates that SFT is a better alleviation method than question repeating, which can further alleviate the negative effects of Feedback and Refinement.”

• AssumptionRests on an assumption: The internal answer decoded from intermediate layers reflects the model's internal answer or hesitation.

Classified as probably not a substantial claim.

Experiments on internal answer wavering were conducted only on open-sourced Llama because close-sourced ChatGPT does not provide hidden state information.

“The experiments are conducted on open-sourced Llama because close-sourced ChatGPT does not provide hidden state information.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The PACT method was adapted to be compatible with both open-sourced Llama and close-sourced ChatGPT.

“Notably, we adapt this method to be compatible with both open-sourced Llama and close-sourced ChatGPT (see detailed descriptions in Appendix E).”

May combine two claims.

After question repeating, LLMs focus more on the original question attached to the end of the refinement prompt, which eliminates the undesirable effects of self-correction.

“LLMs focus more on the original question attached to the end of the refinement prompt, which eliminates the undesirable effects of self-correction.”

• AssumptionThe authors state this as an assumption: Greener tokens indicate more positive contribution to the LLM's answer, and yellower tokens indicate more negative contribution.

May combine two claims. The quote only partly states this claim.

In decision making task, in Refinement, LLMs generate much more 'think' in order to take more caution than the first trial, and this behavior leads to failure by looping in 'think'.

“Nevertheless, in Refinement, LLMs generate much more”

May combine two claims.

After SFT in that Figure 24 example, the original question becomes greener (green tokens increase from 0 to 9) and the refinement prompt becomes yellower (yellow tokens increase from 4 to 7), meaning Llama-3.1-8B focuses more on the original question when answering, which is sufficient to give the second correct answer.

“After SFT, the original question becomes greener (i.e., green tokens increase from 0 to 9) and the refinement prompt becomes yellower (i.e., yello tokens increase from 4 to 7).This means Llama-3.1-8Bfocuses more on the original question when answering, which is sufficient to give the second correct answer.”

• AssumptionThe authors state this as an assumption: Greener tokens indicate more positive contribution to the LLM's answer, and yellower tokens indicate more negative contribution.

May combine two claims.

In decision making task, LLM wants to improve efficiency by picking up two pillows at the same time, and this behavior leads to failure because the environment restricts it from doing so.

“However, it wants to improve efficiency by picking up two pillows at the same time.This behavior instead leads to failure because the environment restricts it from doing so.”

May combine two claims.

After SFT, LLMs focus more on the original question rather than the refinement prompt, correcting the prompt bias which can lead to wrong answer.

“LLMs focus more on the original question rather than refinement prompt.This behavior corrects the prompt bias which can lead to wrong answer.”

• AssumptionThe authors state this as an assumption: Greener tokens indicate more positive contribution to the LLM's answer, and yellower tokens indicate more negative contribution.

May combine two claims.

In that Figure 24 example, all tokens in the original question are yellow while the refinement prompt contains 5 green tokens, indicating Llama-3.1-8B focuses more on the distracting refinement prompt rather than the original question when answering.

“Indeed, all tokens in the original question are yellow while the refinement prompt contains 5 green tokens.This means Llama-3.1-8Bfocuses more on the distracting refinement prompt rather than the original question when answering.”

• AssumptionThe authors state this as an assumption: Greener tokens indicate more positive contribution to the LLM's answer, and yellower tokens indicate more negative contribution.

Probably not a substantial claim: may not pass S3 (load-bearing).

Results of all 5 published self-correction prompts align with the paper's findings.

“Please note that results of all 5 published prompts align with our findings (see results of other prompts in Appendix C).”

May combine two claims.

Self-correction causes internal answer wavering which could further lead to wrong final answers.

“This indicates that self-correction causes internal answer wavering which could further leads to wrong final answers.”

Classified as probably not a substantial claim.

Feedback and Refinement were implemented regardless of the correctness of the initial response to avoid the unfair setting of only refining wrong responses.

“Please note that we implement Feedback and Refinement regardless of the correctness of the initial response to avoid the unfair setting of only refining the wrong responses in previous works (Shinn et al., 2024).”

• cites Shinn et al., 2024Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

In that Figure 23 example, the original question contains 8 yellow tokens while the refinement prompt contains 3 yellow tokens, indicating GPT-4o focuses more on the distracting refinement prompt rather than the original question when answering.

“Indeed, the original question contains 8 yellow tokens while the refinement prompt contains 3 yellow tokens.This means GPT-4o focuses more on the distracting refinement prompt rather than the original question when answering.”

• AssumptionThe authors state this as an assumption: Greener tokens indicate more positive contribution to the LLM's answer, and yellower tokens indicate more negative contribution.

May combine two claims.

After question repeating in that Figure 21 example, the original question becomes greener, meaning GPT-4o correctly focuses on the original question when answering, and the second response is correct.

“After question repeating, the original question, either in the beginning of the prompt for second response or in the refinement prompt, becomes greener.This means GPT-4o correctly focuses on the original question when answering.The second response is therefore correct.”

• AssumptionThe authors state this as an assumption: Greener tokens indicate more positive contribution to the LLM's answer, and yellower tokens indicate more negative contribution.

May combine two claims.

After SFT in that Figure 23 example, the original question becomes greener (green tokens increase from 8 to 11) and the refinement prompt becomes yellower (yellow tokens increase from 3 to 6), meaning GPT-4o focuses more on the original question when answering, which is sufficient to give the second correct answer.

“After SFT, the original question becomes greener (i.e., green tokens increase from 8 to 11) and the refinement prompt becomes yellower (i.e., yello tokens increase from 3 to 6).This means GPT-4o focuses more on the original question when answering, which is sufficient to give the second correct answer.”

• AssumptionThe authors state this as an assumption: Greener tokens indicate more positive contribution to the LLM's answer, and yellower tokens indicate more negative contribution.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Prompt5 is supposed to be the most optimal among the 5 prompts since the reference answer and feedback are provided.

“while prompt5 is supposed to be the most optimal since the reference answer and feedback are provided.”

May combine two claims.

In the Figure 17 programming example, the refinement prompt exceeded 1000 tokens, which increased cognitive load and caused the LLM to forget the 'less than n' condition, leading to task failure.

“When using self-correction to handle this task, the input prompts typically involve long contexts.For example, the refinement prompt exceeds 1000 tokens.This can increase cognitive load of LLMs, resulting in less understanding of the original question.In this example, the LLMs forget critical details from the original question (e.g.,”

• AssumptionRests on an assumption: Cognitive load theory applies to LLMs, such that long contexts increase cognitive load and reduce understanding.

May combine two claims.

The divergence between 'Are you sure?' and Initial response generation is higher than between 'Are you sure?' and 'You are wrong.', suggesting distinct internal behaviors when self-correction is not used.

“In contrast, the divergence between”

May combine two claims.

In reasoning task, LLMs want to concise the answer during self-correction, which fails because 'both Ruggero Deodato from Italy and Mexican Alejandro Springall are film directors' is not needed; only the answer 'Yes' is needed.

“However, LLMs want to concise the answer during self-correction, which fails because”