The DOI matches its Crossref record; 82 of the 82 references it cites were found in a registry or an index; none of the 41 sources its claims cite is checked for support yet.
Faith and Fate: Limits of Transformers on Compositionality
Open accessAbstract
Transformer large language models (LLMs) have sparked admiration for their exceptional performance on tasks that demand intricate multi-step reasoning. Yet, these models simultaneously show failures on surprisingly trivial problems. This begs the question: Are these errors incidental, or do they signal more substantial limitations? In an attempt to demystify transformer LLMs, we investigate the limits of these models across three representative compositional tasks -- multi-digit multiplication, logic grid puzzles, and a classic dynamic programming problem. These tasks require breaking problems down into sub-steps and synthesizing these steps into a precise answer. We formulate compositional tasks as computation graphs to systematically quantify the level of complexity, and break down reasoning steps into intermediate sub-procedures. Our empirical findings suggest that transformer LLMs solve compositional tasks by reducing multi-step compositional reasoning into linearized subgraph matching, without necessarily developing systematic problem-solving skills. To round off our empirical study, we provide theoretical arguments on abstract multi-step reasoning problems that highlight how autoregressive generations' performance can rapidly decay with\,increased\,task\,complexity.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
Transformer LLMs can achieve near-perfect in-domain accuracy on compositional tasks but fail sharply out-of-domain, often relying on linearized subgraph matching rather than systematic multi-step reasoning. The paper argues empirically and theoretically that error propagation makes autoregressive transformers increasingly unlikely to solve compositionally complex tasks as problem size grows.
Key findings
- Off-the-shelf ChatGPT and GPT4 achieve only 55% and 59% accuracy on 3-digit by 3-digit multiplication, and zero-/few-shot performance declines to near zero as problem size or graph parallelism increases.
- Fine-tuning GPT3 on task-specific question-answer or question-scratchpad data gives near-perfect in-domain performance but fails to generalize to out-of-domain deeper or wider computation graphs; grokking was not observed for multiplication after 420K question-answer steps or 30K scratchpad steps.
- Models appear to reduce multi-step reasoning to linearized subgraph matching: full computation subgraphs appear more frequently in training for correctly predicted test examples, and 82.3% of final correct answers for 4-digit by 2-digit multiplications had at least one error in the computation graph.
- Error analysis shows fully correct nodes decrease sharply with graph layer, propagation errors usually exceed local errors, and DP/puzzle tasks show high restoration errors; theoretical propositions show incorrect-prediction probability can converge exponentially to ≈1 as problem size increases under stated assumptions.
Methods
- The paper formulates compositional tasks as computation graphs and measures complexity via reasoning depth, reasoning width, average parallelism, and relative information gain.
- It studies three tasks—long-form multi-digit multiplication up to 5×5 digits, Einstein logic grid puzzles, and a dynamic-programming relaxation of maximum weighted independent set—evaluating GPT3, ChatGPT, GPT4, FlanT5, and LLaMA with zero-shot, few-shot, question-answer fine-tuning, and question-scratchpad fine-tuning; each task uses 500 test examples.
- It complements experiments with theoretical analysis of error accumulation for independent and iterated applications of estimated functions.
Limitations
- The paper states that scratchpad analysis can only establish correlation between model generation and preceding context because the authors lack access to model activations, and that alternative linearizations of reasoning may yield different performance.
- The paper also states that compute budget constraints and limited access to the largest language models prevented pushing empirical limits further.
- Evident from the text: fine-tuning is mainly on GPT3, grokking is tested only on multiplication, puzzle training uses a subset due to combinatorial explosion, and the theoretical results rely on assumptions such as independent errors, low collision, and c ≪ ϵ that may not hold generally.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
65 Own interpretation · 45 Own result · 17 Sources not found · 8 No basis given · 4 Own method
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Similarly, the prediction of the last digit (or two digits) of the output is observed to solely rely on the last digit (or two digits) of each input number.”
§ Faith and Fate: Limits of Transformers on Compositionality > 3.2.1 Information Gain Explains Where Transformers Partially Excel
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This observation holds for all tasks (See details in § B.4).”
§ Faith and Fate: Limits of Transformers on Compositionality > Limits of transformers with question-answer training
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In all settings, the ratio of fully correct nodes is almost perfect but sharply decreases toward zero with increasing graph layers.”
§ Faith and Fate: Limits of Transformers on Compositionality > 3.2.3 What Types of Errors do Transformers Make at Different Reasoning Depths?
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Moreover, the ratio of propagation errors is usually higher than the ratio of local errors.”
§ Faith and Fate: Limits of Transformers on Compositionality > 3.2.3 What Types of Errors do Transformers Make at Different Reasoning Depths?
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, the performance sharply declines when evaluating unseen splits during training, i.e., out-of-domain (OOD).”
§ Faith and Fate: Limits of Transformers on Compositionality > Limits of transformers with question-answer training
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“82.3% of the final correct answers for 4-digit by 2-digit multiplications (a setting unseen during training) had at least one error in the computation graph, but still produced correct answers.”
§ Faith and Fate: Limits of Transformers on Compositionality > 3.2.3 What Types of Errors do Transformers Make at Different Reasoning Depths?
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, this performance boost diminishes to zero as complexity increases.”
§ Faith and Fate: Limits of Transformers on Compositionality > Limits of transformers with question-answer training
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For multiplication, relative information gain shows that the first digit (two digits) of the output highly correlates with the first digit (two digits) of each input number (see §C.1).”
§ Faith and Fate: Limits of Transformers on Compositionality > 3.2.1 Information Gain Explains Where Transformers Partially Excel
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Similarly, prompting transformers with question-scratchpad pairs enhances the performance compared to the zero-shot setting (refer to § B.5).”
§ Faith and Fate: Limits of Transformers on Compositionality > Limits of transformers with question-answer training
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The trend remains the same for few-shot prompting (see §B.2).”
§ Faith and Fate: Limits of Transformers on Compositionality > 3.1 Testing the Limits of Transformers with Zero-shot, Few-shot and Finetuning
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- It may state two separate claims as one. 29
- The quote only partly states this claim. 4
- Probably not a substantial claim. 25
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The results, presented in Figure 5b, show that once again GPT3 achieves near-perfect performance on in-distribution, but fails entirely in generalizing to OOD cases-in particular, wider or deeper computation graphs.”
§ Faith and Fate: Limits of Transformers on Compositionality > Limits of transformers with question-answer training
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Figure 6 shows that full computation subgraphs appear significantly more frequently in the training data for correctly predicted test examples than for incorrectly predicted ones”
§ Faith and Fate: Limits of Transformers on Compositionality > Given a model-generated computation graph G A(x)
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one. The quote only partly states this claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Both the DP and the puzzle tasks have a high ratio of restoration errors, suggesting memorization since correct outputs are produced despite incorrect computations.”
§ Faith and Fate: Limits of Transformers on Compositionality > 3.2.3 What Types of Errors do Transformers Make at Different Reasoning Depths?
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Yet, off-the-shelf ChatGPT and GPT4 achieve only 55% and 59% accuracies on this task, respectively ( §3).”
§ Faith and Fate: Limits of Transformers on Compositionality > 1 Introduction
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Empirically, we verify that models indeed learn the patterns we predicted and other patterns as well (e.g., order of magnitude of the answer, number of trailing zeros for multiplication) in all the settings with and without scratchpad.”
§ Faith and Fate: Limits of Transformers on Compositionality > 3.2.1 Information Gain Explains Where Transformers Partially Excel
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Empirical results show that training on task-specific data leads to near-perfect performance on indomain instances and under low compositional complexity, but fails drastically on instances outside of this region.”
§ Faith and Fate: Limits of Transformers on Compositionality > 1 Introduction
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The three compositional tasks require breaking problems down into sub-steps and synthesizing these steps into a precise answer.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“task performances deteriorate significantly from near perfection to zero with increasing complexity”
§ Faith and Fate: Limits of Transformers on Compositionality > 3.1 Testing the Limits of Transformers with Zero-shot, Few-shot and Finetuning
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The quote only partly states this claim.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Among these variations, the scratchpad version depicted in Figure 9 ultimately produced the most favorable outcomes.”
§ Faith and Fate: Limits of Transformers on Compositionality > A Compositional Tasks
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“since GPT3 often predicts shorter output sequences than the required size”
§ Faith and Fate: Limits of Transformers on Compositionality > C.4 Empirical Surface Pattern Results for Dynamic Programming Task
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The results indicate a lack of generalization for out-of-domain (OOD) examples while showcasing near-perfect performance for in-domain examples.”
§ Faith and Fate: Limits of Transformers on Compositionality > B.3 Limits of Transformers with question-answer Training
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On all tasks, we can see that the model fails to generalize to OOD data while achieving perfect accuracy on in-domain data”
§ Faith and Fate: Limits of Transformers on Compositionality > B.3 Limits of Transformers with question-answer Training
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one. The quote only partly states this claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We perform fine-tuning on GPT3 (text-davinci-003) for the three tasks, observing faster convergence when training on question-scratchpad pairs rather than question-answer pairs.”
§ Faith and Fate: Limits of Transformers on Compositionality > B.1 Models
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Moreover, o i is highly predicted by both (a i-2 , a i-1 , a i ) and (a i , a i+1 , a i+2 )”
§ Faith and Fate: Limits of Transformers on Compositionality > C.1 Relative Information Gain Predictions for Multiplication
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The same applies for the rest of the models.”
§ Faith and Fate: Limits of Transformers on Compositionality > B.5 Limits of Transformers with Explicit Scratchpad Prompting
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Generally, we observe that larger batch sizes tend to yield better results for larger datasets.”
§ Faith and Fate: Limits of Transformers on Compositionality > B.1 Models
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In contrast to other tasks, the value of m is not fixed, and depends on the task instance, but using solvers we know that m increases with problem size.”
§ Faith and Fate: Limits of Transformers on Compositionality > Additional information
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As shown in Table 3, a i is a good predictor of o i , and this is especially true for a 1 and a n-1 , the first and last elements of the sequence.”
§ Faith and Fate: Limits of Transformers on Compositionality > Additional information
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Prop. 4.1's proof ( §D.1) shows the rate of convergence is exponential, thus concluding that transformers will rapidly fail with increasing n.”
§ Faith and Fate: Limits of Transformers on Compositionality > 4 Error Propagations: The Theoretical Limits
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This implies that P(f n ̸ = f n ) decreases exponentially as n increases, with lim inf n→+∞ P(f n ̸ = f n ) ≥ 1 -c.”
§ Faith and Fate: Limits of Transformers on Compositionality > 4 Error Propagations: The Theoretical Limits
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- The authors state this as an assumption: In Proposition 4.1, h_n is a perfect estimator of h_n.
- The authors state this as an assumption: In Proposition 4.1, h_n has low collision, with c_n being an upper bound of h_n's collision rate (c_n < c for all n, with c << 1).
- The authors state this as an assumption: In Proposition 4.1, g's errors are independent.
In the paper
“Shortcut learning [29] via pattern-matching may yield fast correct answers when similar compositional patterns are available during training but does not allow for robust generalization to uncommon or complex examples.”
§ Faith and Fate: Limits of Transformers on Compositionality > 1 Introduction
What the sources say
- Not matchedcites [29]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“All tasks analyzed empirically in this paper are instantiations of these abstractions ( §4).”
§ Faith and Fate: Limits of Transformers on Compositionality > 1 Introduction
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Rests on an assumption: Human problem-solving skills can be represented as a graph structure where vertices are partial solutions and edges are operators that modify those solutions.
- Rests on an assumption: Reasoning depth in a computation graph is a proxy for the maximum level of multi-hop reasoning required to solve the task.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This type of learning could be largely effective when the compositional complexity of tasks is low but it becomes less efficient when tasks are increasingly complex.”
§ Faith and Fate: Limits of Transformers on Compositionality > Correct Final Answer Incorrect Final Answer
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“If P(g ̸ = g) = ϵ > 0, then P(f n ̸ = f n ) decreases exponentially with n. Precisely, P(f n ̸ = f n ) ≥ 1 -(1 -ϵ -c) n-1 (1 -ϵ -c/(c + ϵ)), implying lim inf n→+∞ P(f n ̸ = f n ) ≥ 1 -c/(c + ϵ).”
§ Faith and Fate: Limits of Transformers on Compositionality > 4 Error Propagations: The Theoretical Limits
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- The authors state this as an assumption: In Proposition 4.2, the probability of recovering from a mistake due to the randomness of applying the estimator on an incorrect input is at most c.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Both phenomena suggest that models are able to correctly perform single-step reasoning, potentially due to memorizing such single-step operations during training, but fail to plan and compose several of these steps for an overall correct reasoning.”
§ Faith and Fate: Limits of Transformers on Compositionality > 3.2.3 What Types of Errors do Transformers Make at Different Reasoning Depths?
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
None of the sources cited for this claim could be matched to a paper in Crossref, so there is nothing to check it against.
In the paper
“Applying step-by-step reasoning is fundamental to human intelligence [69,68].”
§ Faith and Fate: Limits of Transformers on Compositionality > 1 Introduction
What the sources say
- Not matchedcites [69]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [68]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Both the DP and the puzzle tasks have a high ratio of restoration errors, suggesting memorization since correct outputs are produced despite incorrect computations.”
§ Faith and Fate: Limits of Transformers on Compositionality > 3.2.3 What Types of Errors do Transformers Make at Different Reasoning Depths?
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Note that Prop 4.1 and 4.2 apply to any high-performant estimator of reasoning tasks.”
§ Faith and Fate: Limits of Transformers on Compositionality > 4 Error Propagations: The Theoretical Limits
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.
- Rests on an assumption: The tasks evaluated are low collision settings.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our insights indicate that the current configuration of transformers, with their reliance on a greedy process for predicting the next word, constrains their error recovery capability and impedes the development of a comprehensive global understanding of the task.”
§ Faith and Fate: Limits of Transformers on Compositionality > Theoretical Findings and their Empirical Implications
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- The authors state this as an assumption: The theoretical proofs' assumptions hold for the abstract compositional tasks and autoregressive language models considered.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“All tasks evaluated in the present work can be seen as instances of the results just proven.”
§ Faith and Fate: Limits of Transformers on Compositionality > 4 Error Propagations: The Theoretical Limits
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Rests on an assumption: The tasks evaluated are low collision settings.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“However, because GPT3 often does not produce the correct output size, it hinders us from analyzing o n-1 .”
§ Faith and Fate: Limits of Transformers on Compositionality > C.4 Empirical Surface Pattern Results for Dynamic Programming Task
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“This limitation arises from our lack of access to the activations of the studied models.”
§ Faith and Fate: Limits of Transformers on Compositionality > 8 Limitations
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
In the paper
“Liu et al., [45] found that shallow transformers learn shortcuts during training, leading to poor OOD generalization.”
§ Faith and Fate: Limits of Transformers on Compositionality > Challenges of transformers in generalization
What the sources say
- Not matchedcites [45]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Not matched
None of the sources cited for this claim could be matched to a paper in Crossref, so there is nothing to check it against.
In the paper
“However, these models have also been shown to struggle across multiple domains [32], including algorithmic reasoning [78], commonsense reasoning [62,40], theory of mind [65], planning [76], logical reasoning [70], and ethical reasoning [36].”
§ Faith and Fate: Limits of Transformers on Compositionality > 6 Related Work
What the sources say
- Not matchedcites [32]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [78]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [62]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [40]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
None of the sources cited for this claim could be matched to a paper in Crossref, so there is nothing to check it against.
In the paper
“Recently, transformers [11,58,57,17,16,63,73,74] have demonstrated impressive reasoning abilities across a wide range of tasks, even outperforming humans in certain cases [79,28,15,85].”
§ Faith and Fate: Limits of Transformers on Compositionality > 6 Related Work
What the sources say
- Not matchedcites [11]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [58]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [57]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [17]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [16]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [63]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [73]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [74]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [79]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [28]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Not matched
In the paper
“Importantly, Hahn [30] shows that transformers cannot robustly model noncounter-free regular languages even when allowing infinite precision.”
§ Faith and Fate: Limits of Transformers on Compositionality > Grokking
What the sources say
- Not matchedcites [30]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Not matched
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“However, we acknowledge that due to our compute budget constraints as well as limited access to the largest language models such as GPT4, we are unable to push the empirical limits of transformers even further in terms of training data size and number of epochs.”
§ Faith and Fate: Limits of Transformers on Compositionality > Theoretical Findings and their Empirical Implications
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Should we choose to fine-tune GPT3 for 4 epochs on question-answer pairs, the cost would amount to $12 million and $700 million for question-scratchpad training.”
§ Faith and Fate: Limits of Transformers on Compositionality > B.3 Limits of Transformers with question-answer Training
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“Prop.4.2 directly applies to the recursive function of the dynamic programming task, as well as to m-by-1 digit multiplication, and to the puzzle through its elimination function (details in D.3).”
§ Faith and Fate: Limits of Transformers on Compositionality > 4 Error Propagations: The Theoretical Limits
What the sources say
- Not matchedcites D.3
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The tasks evaluated are low collision settings.
- Not matched
None of the sources cited for this claim could be matched to a paper in Crossref, so there is nothing to check it against.
In the paper
“Transformers perform fairly well in singlestep reasoning tasks [70], but face challenges when it comes to effectively combining multiple steps to solve compositionally complex problems [84,55,66,81].”
§ Faith and Fate: Limits of Transformers on Compositionality > Challenges of transformers in compositional tasks
What the sources say
- Not matchedcites [70]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [84]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [55]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [66]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [81]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Solving the puzzle task may be seen as f m for some m, where f is the elimination function”
§ Faith and Fate: Limits of Transformers on Compositionality > Additional information
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim could be matched to a paper in Crossref, so there is nothing to check it against.
In the paper
“When f is an affine transformation, the process falls simultaneously between two perspectives: time series [31] and dynamic programming and control [6].”
§ Faith and Fate: Limits of Transformers on Compositionality > Additional information
What the sources say
- Not matchedcites [31]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [6]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
In the paper
“In this latter literature, the focus is usually on the contractive regime in which accrued errors can be kept under control, and the subsequent convergence guarantees (e.g., [23]).”
§ Faith and Fate: Limits of Transformers on Compositionality > Additional information
What the sources say
- Not matchedcites [23]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Assume P(h n = h n ) = 1 and P(h n (X) = h n (Y ) | X ̸ = Y ) < c n , lim n→+∞ P(f n ̸ = f n ) = 1.”
§ Faith and Fate: Limits of Transformers on Compositionality > Additional information
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The quote only partly states this claim.
- The authors state this as an assumption: In Proposition D.1, P(h_n = h_n)=1.
- The authors state this as an assumption: In Proposition D.1, P(h_n(X)=h_n(Y) | X≠Y) < c_n.
- The authors state this as an assumption: In Proposition D.1, c_n ≤ β α^n for some α∈(0,1) and β>0.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In the DP task, the reconstruction of the desired subsequence given already computed dp values may be viewed as f m (x) for an input list of size m.”
§ Faith and Fate: Limits of Transformers on Compositionality > Additional information
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In the DP task, dp's computation may be viewed as f m-2 (x) for a list of size m”
§ Faith and Fate: Limits of Transformers on Compositionality > Additional information
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“where c n < c for some constant c ≪ 1 (i.e.h n perfectly estimates h n , and h n is almost injective).If P(g ̸ = g) = ϵ > 0 and errors in g are independent, lim inf n→+∞ P(f n ̸ = f n ) ≥ 1c.”
§ Faith and Fate: Limits of Transformers on Compositionality > Additional information
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Corollary D.3.The recursions for dynamic programming tasks, the m-by-1 digit multiplication, and the puzzle's elimination function are all tasks where there is a fixed reasoning step g being repeatedly applied.”
§ Faith and Fate: Limits of Transformers on Compositionality > C.4 Empirical Surface Pattern Results for Dynamic Programming Task
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“The process of repeatedly applying a noisy single operation or function f can be related to iterated random functions [24].”
§ Faith and Fate: Limits of Transformers on Compositionality > Additional information
What the sources say
- Not matchedcites [24]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Therefore, large m's (or alternatively, c ≪ ϵ) imply m m+1 will be close to 1.”
§ Faith and Fate: Limits of Transformers on Compositionality > Additional information
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper relies on this without support.
The three compositional tasks require breaking problems down into sub-steps and synthesizing these steps into a precise answer.
“These tasks require breaking problems down into sub-steps and synthesizing these steps into a precise answer.”
The paper relies on this without support.
Human problem-solving skills can be represented as a graph structure where vertices are partial solutions and edges are operators that modify those solutions.
“Human problem-solving skills can be conceptualized as a graph structure, where each vertex represents a partial solution and the edges represent operators that can be applied to modify these solutions.”
The paper relies on this without support.
Reasoning depth in a computation graph is a proxy for the maximum level of multi-hop reasoning required to solve the task.
“In computation graphs, reasoning depth is a proxy for the maximum level of multi-hop reasoning required to solve the task.”
The paper relies on this without support.
The three studied tasks are representative compositional tasks.
“we study three straightforward and flexible representative compositional tasks: long-form multiplication, logic grid puzzles (i.e., Einstein's puzzle [61]), and a classic dynamic programming problem.”
The paper relies on this without support.
As K and M increase, synthesizing different partial solutions that satisfy individual constraints becomes highly compositionally complex.
“As K and M increase, synthesizing different partial solutions that satisfy individual constraints becomes highly compositionally complex.”
The paper relies on this without support.
Algorithms designed to solve compositional tasks typically involve multiple independent applications of a function and/or iterated applications of the same function.
“Algorithms designed to solve compositional tasks typically involve multiple independent applications of a function and/or iterated applications of the same function.”
The paper relies on this without support.
A transformer executing such an algorithm acts as an estimator of these functions.
“A transformer executing such an algorithm acts as an estimator of these functions.”
The authors state this as an assumption.
In Proposition 4.1, h_n is a perfect estimator of h_n.
“Assume that h n is a perfect estimator of h n and that h n has low collision, with c n being an upper bound of h n 's collision rate (c n < c ∀n, with c ≪ 1).”
The authors state this as an assumption.
In Proposition 4.1, h_n has low collision, with c_n being an upper bound of h_n's collision rate (c_n < c for all n, with c << 1).
“Assume that h n is a perfect estimator of h n and that h n has low collision, with c n being an upper bound of h n 's collision rate (c n < c ∀n, with c ≪ 1).”
The authors state this as an assumption.
In Proposition 4.1, g's errors are independent.
“If P(g ̸ = g) = ϵ > 0 where g's errors are independent”
The authors state this as an assumption.
In Proposition 4.2, the probability of recovering from a mistake due to the randomness of applying the estimator on an incorrect input is at most c.
“Assume that the probability of recovering from a mistake due to the randomness of applying the estimator on an incorrect input has probability at most c.”
The authors state this as an assumption.
In Proposition 4.2's current form, there is a single valid reasoning for each input since g is a function.
“In Prop.4.2's current form, we implicitly assume that there is a single valid reasoning for each input since g is a function.”
The authors state this as an assumption.
When g has low collision, c << epsilon.
“It is reasonable to assume c ≪ ϵ when g has low collision, since c represents the probability of the estimator g(y) arriving at the correct output g(x) by chance when given the wrong input y ̸ = x.”
The paper relies on this without support.
The tasks evaluated are low collision settings.
“They are also all low collision settings.”
The authors state this as an assumption.
The theoretical proofs' assumptions hold for the abstract compositional tasks and autoregressive language models considered.
“The proofs presented in §4 show that, under reasonable assumptions, the probability of incorrect predictions converges exponentially to ≈ 1 for abstract compositional tasks.”
The authors state this as an assumption.
In Proposition D.1, P(h_n = h_n)=1.
“Assume P(h n = h n ) = 1”
The authors state this as an assumption.
In Proposition D.1, P(h_n(X)=h_n(Y) | X≠Y) < c_n.
“P(h n (X) = h n (Y ) | X ̸ = Y ) < c n”
The authors state this as an assumption.
In Proposition D.1, c_n ≤ β α^n for some α∈(0,1) and β>0.
“if c n ≤ βα n for some some α ∈ (0, 1) and β > 0”
The authors state this as an assumption.
In Lemma D.2, P(z(g(X))=1 | z(X)=0) ≤ c.
“Assume P(z( g(X)) = 1 | z(X) = 0) ≤ c”
The authors state this as an assumption.
In Lemma D.2, P(z(g(X))=0 | z(X)=1) = ε > 0 with c+ε<1.
“P(z( g(X)) = 0 | z(X) = 1) = ϵ > 0 with c + ϵ < 1”
The authors state this as an assumption.
In Proposition D.2, P(g(X)=g(Y) | X≠Y) ≤ c.
“Assume P(g(X) = g(Y ) | X ̸ = Y ) ≤ c”
The authors state this as an assumption.
In Proposition D.2, P(g≠g)=ε>0 with c+ε<1.
“If P(g ̸ = g) = ϵ > 0 with c + ϵ < 1”
The authors state this as an assumption.
The mathematical framework used is a simplified representation of how multi-step reasoning works.
“The mathematical framework used is a simplified representation of how multi-step reasoning works”
The authors state this as an assumption.
c ≪ ϵ when g has low collision.
“It is reasonable to assume c ≪ ϵ when g has low collision, since c represents the probability of the estimator g(y) arriving at the correct output g(x) by chance when given the wrong input y ̸ = x.”
The authors state this as an assumption.
Assuming approximately uniform errors, c ≈ ϵ/|Im(g)|, which implies c ≪ ϵ when g has low collision.
“Assuming approximately uniform errors, c ≈ ϵ/|Im(g)|, which in turn implies c ≪ ϵ since g being low collision implies |Im(g)| is large.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This may be due to the increased complexity of the task (Dziri et al., 2023) where the ranking of N generations may become much more difficult as N gets larger (since the task complexity increases).”
From Benchmarking Cognitive Biases in Large Language Models as Evaluators (2024) · cites this paper as Dziri et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Frontier reasoning-tuned language models still fail on deductive tasks at depth, and the cost of improved performance through extended internal reasoning scales poorly.”
From PrologMCP: A Standardized Prolog Tool Interface for LLM Agents (2026) · cites this paper as [6]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“First, accuracy degrades on long deductive chains, even for strong reasoning-tuned models.”
From PrologMCP: A Standardized Prolog Tool Interface for LLM Agents (2026) · cites this paper as [6]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“we agree with Dziri et al. (2023) that there is a fundamental”
From What Algorithms can Transformers Learn? A Study in Length Generalization (2023) · cites this paper as Dziri et al. (2023)
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Limitations in compositionality, entity binding, long-context evidence use, and numerical grounding are therefore especially relevant, but are rarely examined within one unified setting [11][12][13][14][15][16][17][18].”
From Diagnosing the limits of large language models for structured scientific evidence extraction (2026) · cites this paper as [11]
Experimental — accuracy not yet measured
The cited source says what the claim attributes to it.
In the citing paper
“Earlier analyses of compositional tasks [6] and template-based variants of standard mathematics benchmarks [7] attribute such failures to pattern matching over training-distribution structure rather than genuine multi-step deduction, with accuracy degrading as the number of required reasoning steps grows.”
From PrologMCP: A Standardized Prolog Tool Interface for LLM Agents (2026) · cites this paper as [6]
The cited source says what the claim attributes to it.
In the citing paper
“Dziri et al. (2023) represent reasoning tasks as computation graphs and find that full computation subgraphs appear much more frequently in training data for correct predictions than incorrect ones.”
From GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models (2024) · cites this paper as Dziri et al. (2023)
The cited source says what the claim attributes to it.
In the citing paper
“the reasoning performance of GPT-4 negatively correlates with the depth of the problem's computation graph (Dziri et al., 2023).”
From The expressive power of transformers with chain of thought (2023) · cites this paper as Dziri et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Studies on compositional generalization show that models often struggle to combine familiar components into systematically correct novel structures [11,47,48].”
From Diagnosing the limits of large language models for structured scientific evidence extraction (2026) · cites this paper as [11]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Faith and Fate: Limits of Transformers on Compositionality > Additional information
“lim n→+∞ P(f n = f n ) = 1. Recall that P(f n = f n ) < αβ n + (1 -ϵ) n • (1 -αβ n ).”
§ Faith and Fate: Limits of Transformers on Compositionality > Additional information
“0 ≤ lim inf n→+∞ P(f n = f n ) ≤ lim sup n→+∞ P(f n = f n ) ≤ lim sup n→+∞ βα n + (1 -ϵ) n • (1 -βα n ) = 0”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Faith and Fate: Limits of Transformers on Compositionality > 4 Error Propagations: The Theoretical Limits
“This implies that P(f n ̸ = f n ) decreases exponentially as n increases, with lim inf n→+∞ P(f n ̸ = f n ) ≥ 1 -c.”
§ Faith and Fate: Limits of Transformers on Compositionality > 4 Error Propagations: The Theoretical Limits
“Prop. 4.1's proof ( §D.1) shows the rate of convergence is exponential, thus concluding that transformers will rapidly fail with increasing n.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Faith and Fate: Limits of Transformers on Compositionality > Limits of transformers with question-answer training
“We separately finetune GPT3 models on ∼1.8M multiplication pairs, ∼142K DP pairs, and ∼41K puzzle pairs”
§ Faith and Fate: Limits of Transformers on Compositionality > B.3 Limits of Transformers with question-answer Training
“As for the puzzle task, the model was fine-tuned on puzzles of sizes ranging from 2x2 to 4x4 resulting in a total of 142k pairs.Additionally, for the DP task, the model was fine-tuned on problems with a sequence length of 5 resulting in 41K pairs.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Faith and Fate: Limits of Transformers on Compositionality > 4 Error Propagations: The Theoretical Limits
“If P(g ̸ = g) = ϵ > 0, then P(f n ̸ = f n ) decreases exponentially with n.”
§ Faith and Fate: Limits of Transformers on Compositionality > 4 Error Propagations: The Theoretical Limits
“Note that if c ≪ ϵ then lim inf n→+∞ P(f n ̸ = f n ) ≈ 1.”
Citations
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 5×
“5-turbo) [40], GPT3 (text-davinci-003) [8], FlanT5 [? ] and LLaMa [57].”
BackgroundMethodInfluentialScaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · doi:10.48550/arxiv.2001.08361 · Cited by 9,550
de Moura, L., & Bjørner, N. (2008) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-78800-3_24 · Cited by 6,595 · cited 1×
“Graph Construction Algorithm To solve the complex compositional reasoning process for a logical grid puzzle, we use existing puzzle solvers [21] to generate the computation graph.”
MethodFeigelson, E. (2003) · Statistics for Environmental Science and Management · doi:10.1007/978-3-319-03762-2_13 · Cited by 6,344 · cited 2×
“When f is an affine transformation, the process falls simultaneously between two perspectives: time series [26] and dynamic programming and control [5].”
BackgroundLLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602 · cited 3×
“[57] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample.”
BackgroundChain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505 · cited 3×
“[61] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou.”
BackgroundMethodThe Architecture of Complexity
Simon, H. A. (1991) · Springer eBooks · doi:10.1007/978-1-4899-0718-9_31 · Cited by 4,479 · cited 1×
“Applying step-by-step reasoning is fundamental to human intelligence [51, 50].”
BackgroundPaLM: Scaling Language Modeling with Pathways
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.02311 · Cited by 2,590 · cited 1×
“Reasoning abilities in transformer LLMs Recently, transformers [11, 58, 57, 17, 16, 63, 73, 74] have demonstrated impressive reasoning abilities across a wide range of tasks, even outperforming humans in certain cases [79, 28, 15, 85].”
BackgroundSelf-Refine: Iterative Refinement with Self-Feedback
Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye, S., Yang, Y., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.17651 · Cited by 2,314 · cited 2×
“[36] Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Sean Welleck, Bodhisattwa Prasad Majumder, Shashank Gupta, Amir Yazdanbakhsh, and Peter Clark.”
BackgroundMethodSelf-citationShortcut learning in deep neural networks
Geirhos, R., Jacobsen, J.-H., Michaelis, C., Zemel, R., Brendel, W., Bethge, M., & Wichmann, F. A. (2020) · Nature Machine Intelligence · doi:10.1038/s42256-020-00257-z · Cited by 2,224 · cited 2×
“[25] Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann.”
Background
Research works printed but not linked: 6
- Cem Anil, Yuhuai Wu, Anders Andreassen, Aitor Lewkowycz, Vedant Misra, Vinay Ramasesh, Ambrose Slone, Guy Gur-Ari, Ethan Dyer, and Behnam Neyshabur. Exploring length general- ization in large language models. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 38546-38556.
- Arpit Bansal, Avi Schwarzschild, Eitan Borgnia, Zeyad Emam, Furong Huang, Micah Goldblum, and Tom Goldstein. End-to-end algorithm synthesis with recurrent networks: Extrapolation without overthinking. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 20232-20242.
- Yoshua Bengio, Réjean Ducharme, and Pascal Vincent. A neural probabilistic language model. In Advances in Neural Information Processing Systems, volume 13. MIT Press, 2000.
- D. Bertsekas. Abstract Dynamic Programming: 3rd Edition. Athena scientific optimization and computation series. Athena Scientific., 2022.
- Bingbin Liu, Jordan T. Ash, Surbhi Goel, Akshay Krishnamurthy, and Cyril Zhang. Trans- formers learn shortcuts to automata. In International Conference on Learning Representations, 2023.
- Karthik Valmeekam, Alberto Olmo, Sarath Sreedharan, and Subbarao Kambhampati. Large language models still can't plan (a benchmark for LLMs on planning and reasoning about change). In NeurIPS 2022 Foundation Models for Decision Making Workshop, 2022.
Cited by
Sorted by citations
Evaluation and mitigation of the limitations of large language models in clinical decision-making
Hager, P., Jungmann, F., Holland, R., Bhagat, K., Hubrecht, I., Knauer, M., Vielhauer, J., Makowski, M., Braren, R., Kaissis, G., et al. (2024) · Nature Medicine · doi:10.1038/s41591-024-03097-1 · Cited by 798
“The subset was evenly split between the four target pathologies with 20 patients randomly selected from each pathology and matching the makeup of the full dataset (Supplementary Section H).”
Dissociating language and thought in large language models
Mahowald, K., Ivanova, A. A., Blank, I. A., Kanwisher, N., Tenenbaum, J. B., & Fedorenko, E. (2024) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2024.01.011 · Cited by 345
“One solution to overcome these limitations has been to augment an LLM with a symbolic tracker of entity states and character beliefs (Sclar et al., 2023), an approach that mirrors the separation between language and theory of mind processing in humans.”
Generative AI for economic research: use cases and implications for economists
Korinek, A. (2023) · Journal of Economic Literature · doi:10.1257/jel.20231736 · Cited by 293
“Dziri et al. (2023) show that LLMs also have difficulties in handling tasks that demand intricate multistep reasoning.”
BackgroundFrom text to insight: large language models for chemical data extraction
Schilling-Wilhelmi, M., Ríos-García, M., Shabih, S., Gil, M. V., Miret, S., Koch, C. T., Marquez, J. A., & Jablonka, K. M. (2024) · Chemical Society Reviews · doi:10.1039/d4cs00913d · Cited by 178
Markovitch, D. G., Stough, R. A., & Huang, D. (2024) · Journal of Retailing and Consumer Services · doi:10.1016/j.jretconser.2024.103847 · Cited by 100
McCoy, R. T., Yao, S., Friedman, D., Hardy, M. D., & Griffiths, T. L. (2024) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2322420121 · Cited by 96
Exploring the role of large language models in the scientific method: from hypothesis to discovery
Zhang, Y., Khan, S. A., Mahmud, A., Yang, H., Lavin, A., Levin, M. G., Frey, J. G., Dunnmon, J., Evans, J., Bundy, A., et al. (2025) · npj Artificial Intelligence · doi:10.1038/s44387-025-00019-5 · Cited by 160
“This suggeststhatLLMsmayrelymoreonpatternmatchingthanonreasoning, contrarytowhatmanyassume 168 – 170 .”
Can Large Language Models Really Improve by Self-critiquing Their Own Plans?
Valmeekam, K., Marquez, M., & Kambhampati, S. (2023) · arXiv · doi:10.48550/arxiv.2310.08118 · Cited by 90
Generating meaning: active inference and the scope and limits of passive AI
Pezzulo, G., Parr, T., Cisek, P., Clark, A., & Friston, K. (2024) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2023.10.002 · Cited by 80
“In addition, LLMs struggle with causal reasoning and multi-step compositional reasoning [110], and sometimes 'hallucinate'ratherthan reporting factualinformationand show 'self-delusions'(i.e.,they taketheir predictions as evidence that the predicted circumstance is true [78]; cf circular inference…”
Evaluating Correctness and Faithfulness of Instruction-Following Models for Question Answering
Adlakha, V., BehnamGhader, P., Lu, X. H., Meade, N., & Reddy, S. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00667 · Cited by 74
“The model can potentially provide accurate information using its parametric knowledge.”
BackgroundSelf-citationChatGPT is no Stochastic Parrot. But it also Claims that 1 is Greater than 1
Arkoudas, K. (2023) · Philosophy & Technology · doi:10.1007/s13347-023-00619-6 · Cited by 80
Benchmarking Cognitive Biases in Large Language Models as Evaluators
Koo, R., Lee, M., Raheja, V., Park, J. I., Kim, Z. M., & Kang, D. (2024) · Findings of the Association for Computational Linguistics ACL 2024 · doi:10.18653/v1/2024.findings-acl.29 · Cited by 70
Generative Artificial Intelligence in Qualitative Data Analysis: Analyzing—Or Just Chatting?
Nguyen, D. C., & Welch, C. (2025) · Organizational Research Methods · doi:10.1177/10944281251377154 · Cited by 81
“…method on which all current GenAI tools are built, and cannot be mitigated through technical improvements, prompt engineering, or scaling efforts; that is, increasing the size and training data of a model (Dziri et al., 2023; LeCun, 2022; Lin et al., 2021; Marcus, 2018; Sahoo et al., 2024).”
Olausson, T., Gu, A., Lipkin, B., Zhang, C., Solar-Lezama, A., Tenenbaum, J., & Levy, R. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.313 · Cited by 57
“However, this reasoning ability appears to be unreliable for tasks that require reasoning out of domain (Liang et al., 2022; Saparov et al., 2023), understanding negation (Anil et al., 2022), and following long reasoning chains (Dziri et al., 2023).”
BackgroundInfluentialGSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models
Mirzadeh, I., Alizadeh, K., Shahrokhi, H., Tuzel, O., Bengio, S., & Farajtabar, M. (2024) · arXiv · doi:10.48550/arxiv.2410.05229 · Cited by 714
Implications of Large Language Models for Quality and Efficiency of Neurologic Care
Moura, L. M. V. R., Jones, D. T., Sheikh, I., Murphy, S. N., Kalfin, M., Kummer, B. R., Weathers, A. L., Grinspan, Z. M., Silsbee, H. M., Jones, L. K., et al. (2024) · Neurology · doi:10.1212/wnl.0000000000209497 · Cited by 53
Shojaee, P., Mirzadeh, I., Alizadeh, K., Horton, M., Bengio, S., & Farajtabar, M. (2025) · arXiv · doi:10.48550/arxiv.2506.06941 · Cited by 45
Large language model for knowledge synthesis and AI-enhanced biomanufacturing
Li, W., Mao, Z., Xiao, Z., Liao, X., Koffas, M. A. G., Chen, Y., Ma, H., & Tang, Y. (2025) · Trends in biotechnology · doi:10.1016/j.tibtech.2025.02.008 · Cited by 48
Peng, Y., Han, J., Zhang, Z., Fan, L., Liu, T., Qi, S., Feng, X., Ma, Y., Wang, Y., & Zhu, S. C. (2023) · Engineering · doi:10.1016/j.eng.2023.07.006 · Cited by 46
Visual Programming for Zero-Shot Open-Vocabulary 3D Visual Grounding
Yuan, Z., Ren, J., Feng, C.-M., Zhao, H., Cui, S. R., & Li, Z. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01949 · Cited by 51
“Second, mathematical calculation is a common weakness of LLMs but is necessary for 3DVG [11].”
BackgroundBeyond Preferences in AI Alignment
Zhi‐Xuan, T., Carroll, M., Franklin, M., & Ashton, H. (2024) · Philosophical Studies · doi:10.1007/s11098-024-02249-w · Cited by 82
“Still, LLMs remain limited in their ability to represent and reason with compositional concepts (Dziri et al., 2023; Mahowald et al., 2024; Ramesh et al., 2024), and would function as poor models of humans on their own.”
BackgroundA survey on knowledge-enhanced multimodal learning
Lymperaiou, M., & Stamou, G. B. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10825-z · Cited by 45
Generating In-Context, Personalized Feedback for Intelligent Tutors with Large Language Models
Reddig, J. M., Arora, A., & MacLellan, C. J. (2025) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-025-00505-6 · Cited by 44
“…shown they are limited in their ability to reason (Berglund et al., 2023; Gendron et al., 2023), answer mathematics questions (Frieder et al., 2024; Dziri et al., 2024), or create a plan (Valmeekam et al., 2022, 2023), often generating incorrect or made-up facts (i.e., hallucinating) (Lee et al.,…”
Alam, M. F., Lentsch, A., Yu, N., Barmack, S., Kim, S., Acemoğlu, D., Hart, A. J., Johnson, S., & Ahmed, F. (2024) · doi:10.21428/e4baedd9.e39b392d · Cited by 31
“A major limitation of current Gen-AI technologies is the lack of reasoning capabilities (J. Huang et al. 2023; Dziri et al. 2023; Turpin et al. 2023; Shi et al. 2023).”
BackgroundA Proposal for a Language Model Based Cognitive Architecture
Knowles, K., Witbrock, M., Dobbie, G., & Yogarajan, V. (2024) · Proceedings of the AAAI Symposium Series · doi:10.1609/aaaiss.v2i1.27691 · Cited by 23
“…et al. 2023); struggle with pure causal reasoning (Jin et al. 2023); fail to understand simple identifier swaps in Python (Miceli Barone et al. 2023); often perform poorly on tasks that require multi-step reasoning and compositionality (Dziri et al. 2023); and produce confabulations (OpenAI 2023).”
BackgroundNezhurina, M., Cipolina-Kun, L., Cherti, M., & Jitsev, J. (2024) · arXiv · doi:10.48550/arxiv.2406.02061 · Cited by 21
Beyond Chatbot-K: On Large Language Models, “Generative AI,” and Rise of Chatbots—An Introduction
Goodlad, L. M. E., & Stone, M. (2024) · Critical AI · doi:10.1215/2834703x-11205147 · Cited by 33
Caution Ahead: Numerical Reasoning and Look‐Ahead Bias in AI Models
LEVY, B. (2026) · Journal of Accounting Research · doi:10.1111/1475-679x.70058 · Cited by 20
Transforming agency: On the mode of existence of large language models
Barandiaran, X. E., & Almendros, L. S. (2025) · Phenomenology and the Cognitive Sciences · doi:10.1007/s11097-025-10094-3 · Cited by 35
“( 7) The sampled token is added to the original input (returning to step 1), and the procedure repeats auto-regressively until an end-of-sequence token is produced or the maximum length is reached.”
Learning by thinking in natural and artificial minds
Lombrozo, T. (2024) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2024.07.007 · Cited by 24
On the Unexpected Abilities of Large Language Models
Nolfi, S. (2024) · Adaptive Behavior · doi:10.1177/10597123241256754 · Cited by 19
“The former group stresses the limitations of current LLMs, for example, their dif fi culty with causal and multi-step compositional reasoning (Dziri et al., 2023), their limited sensitivity to affordances with respect to humans (Jones et al., 2022), and their inability to distinguish their own…”
The Two Word Test as a semantic benchmark for large language models
Riccardi, N., Yang, X., & Desai, R. H. (2024) · Scientific Reports · doi:10.1038/s41598-024-72528-3 · Cited by 23
“For example, Dziri et al. 28 tested LLMs on three compositional tasks (multi-digit multiplication, logic grid puzzles, and dynamic programming).”
BackgroundTowards A Holistic Landscape of Situated Theory of Mind in Large Language Models
Ma, Z., Sansom, J., Peng, R., & Chai, J. Y. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.72 · Cited by 44
Fodor, J., Deyne, S. D., & Suzuki, S. (2024) · Computational Linguistics · doi:10.1162/coli_a_00536 · Cited by 16
Leveraging Structured Information for Explainable Multi-hop Question Answering and Reasoning
Li, R., & Du, X. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.452 · Cited by 12
LLMs Can't Plan, But Can Help Planning in LLM-Modulo Frameworks
Kambhampati, S., Valmeekam, K., Guan, L., Verma, M., Stechly, K., Bhambri, S., Saldyt, L., & Murthy, A. (2024) · arXiv · doi:10.48550/arxiv.2402.01817 · Cited by 237
“…of studies questioning the robustness of such behaviors–be it planning Valmeekam et al. (2023c); Kambhampati (2023), simple arithmetic and logic Dziri et al. (2023), theory of mind abilities Ullman (2023); Verma et al. (2024), or general mathematical and abstract benchmarks McCoy et al.…”
BackgroundOut-of-distribution generalization via composition: A lens through induction heads in Transformers
Song, J., Xu, Z., & Zhong, Y. (2025) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2417182122 · Cited by 25
“Tokens in blue are the target outputs for prediction.”
A brain-inspired agentic architecture to improve planning with LLMs
Webb, T. W., Mondal, S. S., & Momennejad, I. (2025) · Nature Communications · doi:10.1038/s41467-025-63804-5 · Cited by 36
“These results also align with studies that have found poor performance in tasks that involve other formsof extended multi-step reasoning, such as arithmetic 9 .”
MaaSDB: Spatial Databases in the Era of Large Language Models (Vision Paper)
Qi, J., Li, Z., & Tanin, E. (2023) · ACM International Conference on Advances in Geographic Information Systems (SIGSPATIAL) · doi:10.1145/3589132.3625597 · Cited by 10
“While prompting, training, and fine-tuning strategies have been proposed for this issue, achieving such advanced reasoning capabilities remains an open challenge [3].”
BackgroundTan, S., Shen, Y., Chen, Z., Courville, A. C., & Gan, C. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.12 · Cited by 11
What Algorithms can Transformers Learn? A Study in Length Generalization
Zhou, H., Bradley, A., Littwin, E., Razin, N., Saremi, O., Susskind, J., Bengio, S., & Nakkiran, P. (2023) · arXiv · doi:10.48550/arxiv.2310.16028 · Cited by 10
“There has been significant interest recently in understanding and improving the reasoning abilities of Transformer-based models (Cobbe et al., 2021; Magister et al., 2023; Wei et al., 2022; Dziri et al., 2023; Suzgun et al., 2022; Lewkowycz et al., 2022a; Press et al., 2023).”
BackgroundMethodResultInfluentialLanguage-Models-as-a-Service: Overview of a New Paradigm and its Challenges
La Malfa, E., Petrov, A., Frieder, S., Weinhuber, C., Burnell, R., Nazar, R., Cohn, A., Shadbolt, N., & Wooldridge, M. (2024) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.15865 · Cited by 9
“Parallel to that, the research community has focused on evaluating LMs and their zero-and few-shot behavior (Xian et al., 2018; Chang et al., 2023) as well their capability (and limitations) to solve tasks that require compositional reasoning (Dziri et al., 2023; McCoy et al., 2023b).”
BackgroundMachine Translation Meta Evaluation through Translation Accuracy Challenge Sets
Moghe, N., Fazla, A., Amrhein, C., Kocmi, T., Steedman, M. J., Birch, A., Sennrich, R., & Guillou, L. (2024) · Computational Linguistics · doi:10.1162/coli_a_00537 · Cited by 15
Präger, L., Woytowicz, J., Reitberger, R., & Lang, W. (2025) · Building and Environment · doi:10.1016/j.buildenv.2025.113502 · Cited by 12
Mixture of Soft Prompts for Controllable Data Generation
Chen, D., Lee, C. C., Lu, Y., Rosati, D., & Zhou, Y. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.988 · Cited by 22
On compositional generalization of transformer-based neural machine translation
Yin, Y., Fu, L., Li, Y., & Zhang, Y. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102491 · Cited by 10
Generating Meaning: Active Inference and the Scope and Limits of Passive AI
Pezzulo, G., Parr, T., Cisek, P., Clark, A., & Friston, K. (2023) · doi:10.31234/osf.io/8xgzv · Cited by 8
Derivational morphology reveals analogical generalization in large language models
Hofmann, V., Weissweiler, L., Mortensen, D. R., Schütze, H., & Pierrehumbert, J. B. (2025) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2423232122 · Cited by 21
Dynamic Multimodal Process Knowledge Graphs: A Neurosymbolic Framework for Compositional Reasoning
Venkataramanan, R., Shyalika, C., & Sheth, A. (2025) · IEEE Internet Computing · doi:10.1109/mic.2024.3520366 · Cited by 5
“3,4 However, looking under the hood, transformers seem to solve compositional tasks by reducing multistep reasoning into pattern matching or reciting responses from memory.”
Quantifying artificial intelligence through algorithmic generalization
Ito, T., Campbell, M. S., Lior Horesh, C., Klinger, T., & Ram, P. (2025) · Nature Machine Intelligence · doi:10.1038/s42256-025-01092-w · Cited by 10
“While important and insightful, most of those studies primarily focus on a restricted set of algebraic problems, which we will illustrate later 18,19,21,24,39 .”
InfluentialChatGPT: Inside and Impact on Business Automation
Huang, K. L., & Xing, C. (2023) · Future of business and finance · doi:10.1007/978-3-031-45282-6_2 · Cited by 5
Capraro, V., Paolo, R. D., & Pizziol, V. (2023) · doi:10.31234/osf.io/8ackv · Cited by 9
Khan, G., Sanford, C., & Trinh, C. T. (2025) · Biotechnology Advances · doi:10.1016/j.biotechadv.2025.108738 · Cited by 4
3VL: Using Trees to Improve Vision-Language Models’ Interpretability
Yellinek, N., Karlinsky, L., & Giryes, R. (2025) · IEEE Transactions on Image Processing · doi:10.1109/tip.2024.3523801 · Cited by 4
“Another work has shown that there is a fundamental challenge for transformers with compositionality [67].”
BackgroundLarge Language Models and Knowledge Graphs: A State-of-the-Art Exploration
Gallardo-Delgado, A. A., Nolazco‐Flores, J. A., Sánchez, B. P., & Buscaldi, D. (2025) · SN Computer Science · doi:10.1007/s42979-025-04277-7 · Cited by 4
“But these models simultaneously show failures on surprisingly trivial problems [29].”
Capraro, V., Paolo, R. D., & Pizziol, V. (2025) · Scientific Reports · doi:10.1038/s41598-025-01715-7 · Cited by 4
de Wynter, A. (2024) · IEEE Transactions on Games · doi:10.1109/tg.2024.3497601 · Cited by 9
“Other studies evaluating LLMs in generalised planning likewise found that the models evaluated are acceptable, but not effective, at this task, when working by themselves [10].”
The expressive power of transformers with chain of thought
Merrill, W., & Sabharwal, A. (2023) · arXiv · doi:10.48550/arxiv.2310.07923 · Cited by 4
On the Self-Verification Limitations of Large Language Models on Reasoning and Planning Tasks
Stechly, K., Valmeekam, K., & Kambhampati, S. (2024) · arXiv · doi:10.48550/arxiv.2402.08115 · Cited by 120
“…anecdotal accounts of performance on reasoning tasks [4] spurred much research into the capabilities of LLMs on various reasoning domains, from a menagerie of benchmarks covering basic problems [19] to planning [32], logic and arithmetic [9], analogical reasoning [33], and even math puzzles [37].”
BackgroundReliable Reasoning Beyond Natural Language
Borazjanizadeh, N., & Piantadosi, S. T. (2024) · arXiv · doi:10.48550/arxiv.2407.11373 · Cited by 3
“These limitations likely stem from the autoregressive architecture of transformers, which enforces the solution to the problems sequentially: the models’ reliance on a greedy process for predicting the next word constrains their backtracking and error recovery capability [13].”
Abdelkarim, S., Lu, D., Flores, D., Jaeggi, S. M., & Baldi, P. (2025) · Computers in Human Behavior Artificial Humans · doi:10.1016/j.chbah.2025.100170 · Cited by 6
“Transformers optimize next-token likelihood and hence favor pattern matching over rule discovery (Dziri et al., 2023).”
Sheth, A. P., Roy, K., Venkataramanan, R., Nadimuthu, V., & Shyalika, C. (2025) · IEEE Internet Computing · doi:10.1109/mic.2025.3570554 · Cited by 3
“11,12 In such high-stakes domains, incorrect recommendations can be harmful, so the model must explain the reasons and also trace its reasoning to trusted knowledge sources.”
Experiments with LLMs for Converting Language to Logic
Tammet, T., Järv, P., Verrev, M., & Draheim, D. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-71170-1_24 · Cited by 3
Lawsen, A. (2025) · arXiv · doi:10.48550/arxiv.2506.09250 · Cited by 3
“005% This type of “statistical inevitability” argument has in fact been put forward in the literature as a fundamental limitation of LLM scaling [3], yet it assumes models cannot recognize and adapt to their own limitations, an assumption contradicted by the evidence above.”
From Explicit CoT to Implicit CoT: Learning to Internalize CoT Step by Step
Deng, Y., Choi, Y., & Shieber, S. (2024) · arXiv · doi:10.48550/arxiv.2405.14838 · Cited by 3
Self-citationAssessing Logical Reasoning Capabilities of Encoder-Only Transformer Models
Pirozelli, P., José, M. M., Filho, P. D. T. P. D. C., Brandão, A. A. F., & Cozman, F. G. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-71167-1_2 · Cited by 5
Wang, Q., Wu, L., Zeng, Y., et al. (2025) · doi:10.1145/3746275.3762212 · Cited by 2
Towards a Mechanistic Interpretation of Multi-Step Reasoning Capabilities of Language Models
Hou, Y., Li, J., Fei, Y., Stolfo, A., Zhou, W., Zeng, G., Bosselut, A., & Sachan, M. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.299 · Cited by 42
Towards Equipping Transformer with the Ability of Systematic Compositionality
Chen, H., Qin, P., Lei, W., & Lv, J. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i16.29788 · Cited by 3
Evidence from formal logical reasoning reveals that the language of thought is not natural language
Kean, H., Fung, A., Jaggers, P., Chen, J., Rule, J. S., Benn, Y., Tenenbaum, J. B., Piantadosi, S. T., Varley, R. A., & Fedorenko, E. (2026) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2520095123 · Cited by 2
Guarding against the uncertain perils of AI
(2023) · Nature Biomedical Engineering · doi:10.1038/s41551-023-01064-8 · Cited by 2
Harnessing Dataset Cartography for Improved Compositional Generalization in Transformers
İnce, O. B., Zeraati, T., Yagcioglu, S., Yaghoobzadeh, Y., Erdem, E., & Erdem, A. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.867 · Cited by 2
Iterative improvements from feedback for language modelsPreprint
Li, Y. (2023) · doi:10.14293/pr2199.000220.v1 · Cited by 3
Reflexivity and Duplicability in Set Theory
Manca, V. (2025) · Mathematics · doi:10.3390/math13040678 · Cited by 1
Toward Enhancing Reasoning Capabilities of LLMs: An Approach via Synthetic Logic Corpus
Morishita, T., Morio, G., Yamaguchi, A., & Sogawa, Y. (2025) · Journal of Natural Language Processing · doi:10.5715/jnlp.32.520 · Cited by 1
The limits of large language models and the necessity of human cognition in K-12 education
Yoo, J., & Scribner, C. F. (2025) · Theory Into Practice · doi:10.1080/00405841.2025.2528553 · Cited by 2
Political Bias in LLMs: Unaligned Moral Values in Agent-centric Simulations
Münker, S. (2025) · LDV-Forum/Journal for language technology and computational linguistics · doi:10.21248/jlcl.38.2025.289 · Cited by 2
“While we disagree with a (naive) anthropomorphism and current research questions the language understanding capabilities (Dziri et al., 2024), we argue that when utilizing LLMs as human simulacra (Shanahan, 2024), we must assume human-like qualities to a certain degree.”
LGMT: Logic-Grounded Metamorphic Testing for evaluating the reasoning reliability of LLMs
Zhou, Z., Li, M., Fang, X., Zhou, X., Lin, W., & Zheng, Z. (2026) · Knowledge-Based Systems · doi:10.1016/j.knosys.2026.116324 · Cited by 1
“This perpetual "cat-and-mouse" game between benchmarks and model training incurs an unsustainable maintenance cost, further exacerbating the oracle problem [12].”
Augmenting Large Language Models with Causal Risk Patterns for AI Deployment Risk Assessment
McConomy, G., Bond, R., Glass, D., Liu, J., & Toner, H. (2026) · IEEE International Conference on AI and Data Analytics (ICAD) · doi:10.1109/icad69378.2026.11608435 · Cited by 1
“Dziri et al. [11] showed that transformers struggle with compositional generalisation, that is, combining learned components in novel ways not seen during training.”
The compositionality continuum as a principle for studying the neural basis of intelligence
Riveland, R., Pouget, A., & Driscoll, L. N. (2026) · Nature Neuroscience · doi:10.1038/s41593-026-02382-1 · Cited by 3
“Transformers and other architectures develop emergent compositional structures, such as attention heads that perform similar operations across multiple tasks (context 1, context 2), effectively modularizing computation with syntactic structure 47,48,52 . g , Mixture of experts.”
Exploring Continual Learning of Compositional Generalization in NLI
Fu, X., & Frank, A. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00680 · Cited by 4
“…the order of continual learning by observing dependencies and degrees of difficulty of primitive and compositional inference types, for compositional NLI performance. v) We control for data leakage using pseudo data and demonstrate that the C 2 Gen NLI challenge persists for LLMs such as Llama.”
MethodChou, C.-H., & Wu, C.-H. (2024) · doi:10.1109/aixheart62327.2024.00025 · Cited by 1
Benchmarking VLMs' Reasoning About Persuasive Atypical Images
Malakouti, S., Aghazadeh, A., Khandelwal, A., & Kovashka, A. (2025) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv61041.2025.00469 · Cited by 1
“Recent studies [5, 15, 26, 51] have explored LLMs for reasoning tasks, including chain-of-thought reasoning [44].”
Loquacity and Visible Emotion: ChatGPT as a Policy AdvisorPreprint
Biancotti, C., & Camassa, C. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4533699 · Cited by 4
Artificial Intelligence: A Kalam and Sufi PerspectivePreprint
Khalili, A. (2024) · Preprints.org · doi:10.20944/preprints202306.0363.v5 · Cited by 1
Whither developmental psycholinguistics?Preprint
Gomes, V. (2024) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/zm57t · Cited by 1
Transformers Can Navigate Mazes With Multi-Step PredictionPreprint
Nolte, N. S. (2024) · Qeios · doi:10.32388/3q1xzw · Cited by 10
PrologMCP: A Standardized Prolog Tool Interface for LLM Agents
Mensfelt, A., Prabhakaran, A., Haret, A., Trencsenyi, V., & Stathis, K. (2026) · arXiv · doi:10.48550/arxiv.2606.14935 · Cited by 1
Mittal, S. (2026) · arXiv · doi:10.48550/arxiv.2609.01660 · Cited by 1
Language models need inductive biases to count inductively
Chang, Y., & Bisk, Y. (2024) · arXiv · doi:10.48550/arxiv.2405.20131
““Difficulty in generalizing to longer instances” is a recurring theme in the discussion of Transformer limitations, regardless of the task domain (Dziri et al., 2023; Saparov et al., 2023; Zhang et al., 2023; Del’etang et al., 2022; Liu et al., 2022; Bhattamishra et al., 2020).”
BackgroundArtificial Intelligence: A Kalam and Sufi PerspectivePreprint
Khalili, A. M. (2023) · Preprints.org · doi:10.20944/preprints202306.0363.v2
Artificial Intelligence: A Kalam and Sufi PerspectivePreprint
Khalili, A. M. (2023) · Preprints.org · doi:10.20944/preprints202306.0363.v3
Withdrawn: Artificial Intelligence: A Kalam and Sufi PerspectivePreprint
Khalili, A. M. (2023) · Preprints.org · doi:10.20944/preprints202306.0363.v4
Artificial Intelligence: A Kalam and Sufi PerspectivePreprint
Khalili, A. (2024) · Preprints.org · doi:10.20944/preprints202306.0363.v6
Necib, A. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4817804
The Philosophical Proofs of the Existence of God: It’s Implication for Christian Faith and Practice
Ndzi, L. (2024) · Greener Journal of Social Sciences · doi:10.15580/gjss.2024.2.102524154
How Domain Adaptation of BERT Improves Syntactic Parsing of Math Text
Yoshida, R., & Matsuzaki, T. (2024) · Journal of Natural Language Processing · doi:10.5715/jnlp.31.1691
No Consciousness? No Meaning (and no AGI!)Preprint
Masi, M. (2024) · Qeios · doi:10.32388/dn232y
No Consciousness? No Meaning (and no AGI)!Preprint
Masi, M. (2025) · Qeios · doi:10.32388/dn232y.2
No Consciousness? No Meaning (and no AGI)!Preprint
Masi, M. (2025) · Qeios · doi:10.32388/dn232y.3
Metadata
- Authors
- Dziri, N., Lu, X., Sclar, M., Li, X. (., Jiang, L., Lin, B. Y., Welleck, S., West, P., Bhagavatula, C., Bras, R. L., Hwang, J., Sanyal, S., Ren, X., Ettinger, A., Harchaoui, Z., & Choi, Y.
- Venue
- Advances in Neural Information Processing Systems 36
- Published
- 2023
- Pages
- 70293-70332
- ISSN
- 1049-5258 (ISSN-L)
- OpenAlex
- W7133191918
- Cited by
- 335
- References
- 82
- Citations by year
- 2026: 106 · 2025: 77 · 2024: 92 · 2023: 60
- FWCI
- 610.22
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- neural information processing systems (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (2134012)
- Allen Institute for Artificial Intelligence
- Defense Advanced Research Projects Agency (N66001-19-2-4031, N66001-19)
- Naval Information Warfare Center Pacific
- Topics
- Social Sciences › Arts and Humanities › Language and Linguistics › Syntax, Semantics, Linguistic Variation
- Social Sciences › Arts and Humanities › Philosophy › Philosophy and Theoretical Science
- Social Sciences › Arts and Humanities › History and Philosophy of Science › Cybernetics and Technology in Society
- Keywords
- transformer models, compositional generalization, compositional reasoning, large language models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-23, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-23 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
