The DOI matches its Crossref record; 82 of the 82 references it cites were found in a registry or an index; none of the 41 sources its claims cite is checked for support yet.
Faith and Fate: Limits of Transformers on Compositionality
Open accessAbstract
Transformer large language models (LLMs) have sparked admiration for their exceptional performance on tasks that demand intricate multi-step reasoning. Yet, these models simultaneously show failures on surprisingly trivial problems. This begs the question: Are these errors incidental, or do they signal more substantial limitations? In an attempt to demystify transformer LLMs, we investigate the limits of these models across three representative compositional tasks -- multi-digit multiplication, logic grid puzzles, and a classic dynamic programming problem. These tasks require breaking problems down into sub-steps and synthesizing these steps into a precise answer. We formulate compositional tasks as computation graphs to systematically quantify the level of complexity, and break down reasoning steps into intermediate sub-procedures. Our empirical findings suggest that transformer LLMs solve compositional tasks by reducing multi-step compositional reasoning into linearized subgraph matching, without necessarily developing systematic problem-solving skills. To round off our empirical study, we provide theoretical arguments on abstract multi-step reasoning problems that highlight how autoregressive generations' performance can rapidly decay with\,increased\,task\,complexity.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
Transformer LLMs can achieve near-perfect in-domain accuracy on compositional tasks but fail sharply out-of-domain, often relying on linearized subgraph matching rather than systematic multi-step reasoning. The paper argues empirically and theoretically that error propagation makes autoregressive transformers increasingly unlikely to solve compositionally complex tasks as problem size grows.
Key findings
- Off-the-shelf ChatGPT and GPT4 achieve only 55% and 59% accuracy on 3-digit by 3-digit multiplication, and zero-/few-shot performance declines to near zero as problem size or graph parallelism increases.
- Fine-tuning GPT3 on task-specific question-answer or question-scratchpad data gives near-perfect in-domain performance but fails to generalize to out-of-domain deeper or wider computation graphs; grokking was not observed for multiplication after 420K question-answer steps or 30K scratchpad steps.
- Models appear to reduce multi-step reasoning to linearized subgraph matching: full computation subgraphs appear more frequently in training for correctly predicted test examples, and 82.3% of final correct answers for 4-digit by 2-digit multiplications had at least one error in the computation graph.
- Error analysis shows fully correct nodes decrease sharply with graph layer, propagation errors usually exceed local errors, and DP/puzzle tasks show high restoration errors; theoretical propositions show incorrect-prediction probability can converge exponentially to ≈1 as problem size increases under stated assumptions.
Methods
- The paper formulates compositional tasks as computation graphs and measures complexity via reasoning depth, reasoning width, average parallelism, and relative information gain.
- It studies three tasks—long-form multi-digit multiplication up to 5×5 digits, Einstein logic grid puzzles, and a dynamic-programming relaxation of maximum weighted independent set—evaluating GPT3, ChatGPT, GPT4, FlanT5, and LLaMA with zero-shot, few-shot, question-answer fine-tuning, and question-scratchpad fine-tuning; each task uses 500 test examples.
- It complements experiments with theoretical analysis of error accumulation for independent and iterated applications of estimated functions.
Limitations
- The paper states that scratchpad analysis can only establish correlation between model generation and preceding context because the authors lack access to model activations, and that alternative linearizations of reasoning may yield different performance.
- The paper also states that compute budget constraints and limited access to the largest language models prevented pushing empirical limits further.
- Evident from the text: fine-tuning is mainly on GPT3, grokking is tested only on multiplication, puzzle training uses a subset due to combinatorial explosion, and the theoretical results rely on assumptions such as independent errors, low collision, and c ≪ ϵ that may not hold generally.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
65 Own interpretation · 45 Own result · 17 Sources not found · 8 No basis given · 4 Own method
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Similarly, the prediction of the last digit (or two digits) of the output is observed to solely rely on the last digit (or two digits) of each input number.”
§ Faith and Fate: Limits of Transformers on Compositionality > 3.2.1 Information Gain Explains Where Transformers Partially Excel
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This observation holds for all tasks (See details in § B.4).”
§ Faith and Fate: Limits of Transformers on Compositionality > Limits of transformers with question-answer training
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In all settings, the ratio of fully correct nodes is almost perfect but sharply decreases toward zero with increasing graph layers.”
§ Faith and Fate: Limits of Transformers on Compositionality > 3.2.3 What Types of Errors do Transformers Make at Different Reasoning Depths?
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Moreover, the ratio of propagation errors is usually higher than the ratio of local errors.”
§ Faith and Fate: Limits of Transformers on Compositionality > 3.2.3 What Types of Errors do Transformers Make at Different Reasoning Depths?
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, the performance sharply declines when evaluating unseen splits during training, i.e., out-of-domain (OOD).”
§ Faith and Fate: Limits of Transformers on Compositionality > Limits of transformers with question-answer training
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“82.3% of the final correct answers for 4-digit by 2-digit multiplications (a setting unseen during training) had at least one error in the computation graph, but still produced correct answers.”
§ Faith and Fate: Limits of Transformers on Compositionality > 3.2.3 What Types of Errors do Transformers Make at Different Reasoning Depths?
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, this performance boost diminishes to zero as complexity increases.”
§ Faith and Fate: Limits of Transformers on Compositionality > Limits of transformers with question-answer training
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For multiplication, relative information gain shows that the first digit (two digits) of the output highly correlates with the first digit (two digits) of each input number (see §C.1).”
§ Faith and Fate: Limits of Transformers on Compositionality > 3.2.1 Information Gain Explains Where Transformers Partially Excel
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Similarly, prompting transformers with question-scratchpad pairs enhances the performance compared to the zero-shot setting (refer to § B.5).”
§ Faith and Fate: Limits of Transformers on Compositionality > Limits of transformers with question-answer training
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The trend remains the same for few-shot prompting (see §B.2).”
§ Faith and Fate: Limits of Transformers on Compositionality > 3.1 Testing the Limits of Transformers with Zero-shot, Few-shot and Finetuning
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- It may state two separate claims as one. 29
- The quote only partly states this claim. 4
- Probably not a substantial claim. 25
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The results, presented in Figure 5b, show that once again GPT3 achieves near-perfect performance on in-distribution, but fails entirely in generalizing to OOD cases-in particular, wider or deeper computation graphs.”
§ Faith and Fate: Limits of Transformers on Compositionality > Limits of transformers with question-answer training
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Figure 6 shows that full computation subgraphs appear significantly more frequently in the training data for correctly predicted test examples than for incorrectly predicted ones”
§ Faith and Fate: Limits of Transformers on Compositionality > Given a model-generated computation graph G A(x)
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one. The quote only partly states this claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Both the DP and the puzzle tasks have a high ratio of restoration errors, suggesting memorization since correct outputs are produced despite incorrect computations.”
§ Faith and Fate: Limits of Transformers on Compositionality > 3.2.3 What Types of Errors do Transformers Make at Different Reasoning Depths?
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Yet, off-the-shelf ChatGPT and GPT4 achieve only 55% and 59% accuracies on this task, respectively ( §3).”
§ Faith and Fate: Limits of Transformers on Compositionality > 1 Introduction
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Empirically, we verify that models indeed learn the patterns we predicted and other patterns as well (e.g., order of magnitude of the answer, number of trailing zeros for multiplication) in all the settings with and without scratchpad.”
§ Faith and Fate: Limits of Transformers on Compositionality > 3.2.1 Information Gain Explains Where Transformers Partially Excel
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Empirical results show that training on task-specific data leads to near-perfect performance on indomain instances and under low compositional complexity, but fails drastically on instances outside of this region.”
§ Faith and Fate: Limits of Transformers on Compositionality > 1 Introduction
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The three compositional tasks require breaking problems down into sub-steps and synthesizing these steps into a precise answer.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“task performances deteriorate significantly from near perfection to zero with increasing complexity”
§ Faith and Fate: Limits of Transformers on Compositionality > 3.1 Testing the Limits of Transformers with Zero-shot, Few-shot and Finetuning
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The quote only partly states this claim.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Among these variations, the scratchpad version depicted in Figure 9 ultimately produced the most favorable outcomes.”
§ Faith and Fate: Limits of Transformers on Compositionality > A Compositional Tasks
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“since GPT3 often predicts shorter output sequences than the required size”
§ Faith and Fate: Limits of Transformers on Compositionality > C.4 Empirical Surface Pattern Results for Dynamic Programming Task
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The results indicate a lack of generalization for out-of-domain (OOD) examples while showcasing near-perfect performance for in-domain examples.”
§ Faith and Fate: Limits of Transformers on Compositionality > B.3 Limits of Transformers with question-answer Training
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On all tasks, we can see that the model fails to generalize to OOD data while achieving perfect accuracy on in-domain data”
§ Faith and Fate: Limits of Transformers on Compositionality > B.3 Limits of Transformers with question-answer Training
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one. The quote only partly states this claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We perform fine-tuning on GPT3 (text-davinci-003) for the three tasks, observing faster convergence when training on question-scratchpad pairs rather than question-answer pairs.”
§ Faith and Fate: Limits of Transformers on Compositionality > B.1 Models
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Moreover, o i is highly predicted by both (a i-2 , a i-1 , a i ) and (a i , a i+1 , a i+2 )”
§ Faith and Fate: Limits of Transformers on Compositionality > C.1 Relative Information Gain Predictions for Multiplication
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The same applies for the rest of the models.”
§ Faith and Fate: Limits of Transformers on Compositionality > B.5 Limits of Transformers with Explicit Scratchpad Prompting
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Generally, we observe that larger batch sizes tend to yield better results for larger datasets.”
§ Faith and Fate: Limits of Transformers on Compositionality > B.1 Models
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In contrast to other tasks, the value of m is not fixed, and depends on the task instance, but using solvers we know that m increases with problem size.”
§ Faith and Fate: Limits of Transformers on Compositionality > Additional information
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As shown in Table 3, a i is a good predictor of o i , and this is especially true for a 1 and a n-1 , the first and last elements of the sequence.”
§ Faith and Fate: Limits of Transformers on Compositionality > Additional information
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Prop. 4.1's proof ( §D.1) shows the rate of convergence is exponential, thus concluding that transformers will rapidly fail with increasing n.”
§ Faith and Fate: Limits of Transformers on Compositionality > 4 Error Propagations: The Theoretical Limits
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This implies that P(f n ̸ = f n ) decreases exponentially as n increases, with lim inf n→+∞ P(f n ̸ = f n ) ≥ 1 -c.”
§ Faith and Fate: Limits of Transformers on Compositionality > 4 Error Propagations: The Theoretical Limits
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- The authors state this as an assumption: In Proposition 4.1, h_n is a perfect estimator of h_n.
- The authors state this as an assumption: In Proposition 4.1, h_n has low collision, with c_n being an upper bound of h_n's collision rate (c_n < c for all n, with c << 1).
- The authors state this as an assumption: In Proposition 4.1, g's errors are independent.
In the paper
“Shortcut learning [29] via pattern-matching may yield fast correct answers when similar compositional patterns are available during training but does not allow for robust generalization to uncommon or complex examples.”
§ Faith and Fate: Limits of Transformers on Compositionality > 1 Introduction
What the sources say
- Not matchedcites [29]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“All tasks analyzed empirically in this paper are instantiations of these abstractions ( §4).”
§ Faith and Fate: Limits of Transformers on Compositionality > 1 Introduction
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Rests on an assumption: Human problem-solving skills can be represented as a graph structure where vertices are partial solutions and edges are operators that modify those solutions.
- Rests on an assumption: Reasoning depth in a computation graph is a proxy for the maximum level of multi-hop reasoning required to solve the task.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This type of learning could be largely effective when the compositional complexity of tasks is low but it becomes less efficient when tasks are increasingly complex.”
§ Faith and Fate: Limits of Transformers on Compositionality > Correct Final Answer Incorrect Final Answer
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“If P(g ̸ = g) = ϵ > 0, then P(f n ̸ = f n ) decreases exponentially with n. Precisely, P(f n ̸ = f n ) ≥ 1 -(1 -ϵ -c) n-1 (1 -ϵ -c/(c + ϵ)), implying lim inf n→+∞ P(f n ̸ = f n ) ≥ 1 -c/(c + ϵ).”
§ Faith and Fate: Limits of Transformers on Compositionality > 4 Error Propagations: The Theoretical Limits
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- The authors state this as an assumption: In Proposition 4.2, the probability of recovering from a mistake due to the randomness of applying the estimator on an incorrect input is at most c.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Both phenomena suggest that models are able to correctly perform single-step reasoning, potentially due to memorizing such single-step operations during training, but fail to plan and compose several of these steps for an overall correct reasoning.”
§ Faith and Fate: Limits of Transformers on Compositionality > 3.2.3 What Types of Errors do Transformers Make at Different Reasoning Depths?
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
None of the sources cited for this claim could be matched to a paper in Crossref, so there is nothing to check it against.
In the paper
“Applying step-by-step reasoning is fundamental to human intelligence [69,68].”
§ Faith and Fate: Limits of Transformers on Compositionality > 1 Introduction
What the sources say
- Not matchedcites [69]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [68]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Both the DP and the puzzle tasks have a high ratio of restoration errors, suggesting memorization since correct outputs are produced despite incorrect computations.”
§ Faith and Fate: Limits of Transformers on Compositionality > 3.2.3 What Types of Errors do Transformers Make at Different Reasoning Depths?
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Note that Prop 4.1 and 4.2 apply to any high-performant estimator of reasoning tasks.”
§ Faith and Fate: Limits of Transformers on Compositionality > 4 Error Propagations: The Theoretical Limits
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.
- Rests on an assumption: The tasks evaluated are low collision settings.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our insights indicate that the current configuration of transformers, with their reliance on a greedy process for predicting the next word, constrains their error recovery capability and impedes the development of a comprehensive global understanding of the task.”
§ Faith and Fate: Limits of Transformers on Compositionality > Theoretical Findings and their Empirical Implications
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- The authors state this as an assumption: The theoretical proofs' assumptions hold for the abstract compositional tasks and autoregressive language models considered.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“All tasks evaluated in the present work can be seen as instances of the results just proven.”
§ Faith and Fate: Limits of Transformers on Compositionality > 4 Error Propagations: The Theoretical Limits
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Rests on an assumption: The tasks evaluated are low collision settings.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“However, because GPT3 often does not produce the correct output size, it hinders us from analyzing o n-1 .”
§ Faith and Fate: Limits of Transformers on Compositionality > C.4 Empirical Surface Pattern Results for Dynamic Programming Task
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“This limitation arises from our lack of access to the activations of the studied models.”
§ Faith and Fate: Limits of Transformers on Compositionality > 8 Limitations
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
In the paper
“Liu et al., [45] found that shallow transformers learn shortcuts during training, leading to poor OOD generalization.”
§ Faith and Fate: Limits of Transformers on Compositionality > Challenges of transformers in generalization
What the sources say
- Not matchedcites [45]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Not matched
None of the sources cited for this claim could be matched to a paper in Crossref, so there is nothing to check it against.
In the paper
“However, these models have also been shown to struggle across multiple domains [32], including algorithmic reasoning [78], commonsense reasoning [62,40], theory of mind [65], planning [76], logical reasoning [70], and ethical reasoning [36].”
§ Faith and Fate: Limits of Transformers on Compositionality > 6 Related Work
What the sources say
- Not matchedcites [32]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [78]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [62]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [40]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
None of the sources cited for this claim could be matched to a paper in Crossref, so there is nothing to check it against.
In the paper
“Recently, transformers [11,58,57,17,16,63,73,74] have demonstrated impressive reasoning abilities across a wide range of tasks, even outperforming humans in certain cases [79,28,15,85].”
§ Faith and Fate: Limits of Transformers on Compositionality > 6 Related Work
What the sources say
- Not matchedcites [11]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [58]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [57]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [17]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [16]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [63]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [73]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [74]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [79]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [28]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Not matched
In the paper
“Importantly, Hahn [30] shows that transformers cannot robustly model noncounter-free regular languages even when allowing infinite precision.”
§ Faith and Fate: Limits of Transformers on Compositionality > Grokking
What the sources say
- Not matchedcites [30]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Not matched
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“However, we acknowledge that due to our compute budget constraints as well as limited access to the largest language models such as GPT4, we are unable to push the empirical limits of transformers even further in terms of training data size and number of epochs.”
§ Faith and Fate: Limits of Transformers on Compositionality > Theoretical Findings and their Empirical Implications
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Should we choose to fine-tune GPT3 for 4 epochs on question-answer pairs, the cost would amount to $12 million and $700 million for question-scratchpad training.”
§ Faith and Fate: Limits of Transformers on Compositionality > B.3 Limits of Transformers with question-answer Training
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“Prop.4.2 directly applies to the recursive function of the dynamic programming task, as well as to m-by-1 digit multiplication, and to the puzzle through its elimination function (details in D.3).”
§ Faith and Fate: Limits of Transformers on Compositionality > 4 Error Propagations: The Theoretical Limits
What the sources say
- Not matchedcites D.3
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The tasks evaluated are low collision settings.
- Not matched
None of the sources cited for this claim could be matched to a paper in Crossref, so there is nothing to check it against.
In the paper
“Transformers perform fairly well in singlestep reasoning tasks [70], but face challenges when it comes to effectively combining multiple steps to solve compositionally complex problems [84,55,66,81].”
§ Faith and Fate: Limits of Transformers on Compositionality > Challenges of transformers in compositional tasks
What the sources say
- Not matchedcites [70]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [84]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [55]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [66]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [81]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Solving the puzzle task may be seen as f m for some m, where f is the elimination function”
§ Faith and Fate: Limits of Transformers on Compositionality > Additional information
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim could be matched to a paper in Crossref, so there is nothing to check it against.
In the paper
“When f is an affine transformation, the process falls simultaneously between two perspectives: time series [31] and dynamic programming and control [6].”
§ Faith and Fate: Limits of Transformers on Compositionality > Additional information
What the sources say
- Not matchedcites [31]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites [6]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
In the paper
“In this latter literature, the focus is usually on the contractive regime in which accrued errors can be kept under control, and the subsequent convergence guarantees (e.g., [23]).”
§ Faith and Fate: Limits of Transformers on Compositionality > Additional information
What the sources say
- Not matchedcites [23]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Assume P(h n = h n ) = 1 and P(h n (X) = h n (Y ) | X ̸ = Y ) < c n , lim n→+∞ P(f n ̸ = f n ) = 1.”
§ Faith and Fate: Limits of Transformers on Compositionality > Additional information
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The quote only partly states this claim.
- The authors state this as an assumption: In Proposition D.1, P(h_n = h_n)=1.
- The authors state this as an assumption: In Proposition D.1, P(h_n(X)=h_n(Y) | X≠Y) < c_n.
- The authors state this as an assumption: In Proposition D.1, c_n ≤ β α^n for some α∈(0,1) and β>0.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In the DP task, the reconstruction of the desired subsequence given already computed dp values may be viewed as f m (x) for an input list of size m.”
§ Faith and Fate: Limits of Transformers on Compositionality > Additional information
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In the DP task, dp's computation may be viewed as f m-2 (x) for a list of size m”
§ Faith and Fate: Limits of Transformers on Compositionality > Additional information
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“where c n < c for some constant c ≪ 1 (i.e.h n perfectly estimates h n , and h n is almost injective).If P(g ̸ = g) = ϵ > 0 and errors in g are independent, lim inf n→+∞ P(f n ̸ = f n ) ≥ 1c.”
§ Faith and Fate: Limits of Transformers on Compositionality > Additional information
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Corollary D.3.The recursions for dynamic programming tasks, the m-by-1 digit multiplication, and the puzzle's elimination function are all tasks where there is a fixed reasoning step g being repeatedly applied.”
§ Faith and Fate: Limits of Transformers on Compositionality > C.4 Empirical Surface Pattern Results for Dynamic Programming Task
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“The process of repeatedly applying a noisy single operation or function f can be related to iterated random functions [24].”
§ Faith and Fate: Limits of Transformers on Compositionality > Additional information
What the sources say
- Not matchedcites [24]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Therefore, large m's (or alternatively, c ≪ ϵ) imply m m+1 will be close to 1.”
§ Faith and Fate: Limits of Transformers on Compositionality > Additional information
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper relies on this without support.
The three compositional tasks require breaking problems down into sub-steps and synthesizing these steps into a precise answer.
“These tasks require breaking problems down into sub-steps and synthesizing these steps into a precise answer.”
The paper relies on this without support.
Human problem-solving skills can be represented as a graph structure where vertices are partial solutions and edges are operators that modify those solutions.
“Human problem-solving skills can be conceptualized as a graph structure, where each vertex represents a partial solution and the edges represent operators that can be applied to modify these solutions.”
The paper relies on this without support.
Reasoning depth in a computation graph is a proxy for the maximum level of multi-hop reasoning required to solve the task.
“In computation graphs, reasoning depth is a proxy for the maximum level of multi-hop reasoning required to solve the task.”
The paper relies on this without support.
The three studied tasks are representative compositional tasks.
“we study three straightforward and flexible representative compositional tasks: long-form multiplication, logic grid puzzles (i.e., Einstein's puzzle [61]), and a classic dynamic programming problem.”
The paper relies on this without support.
As K and M increase, synthesizing different partial solutions that satisfy individual constraints becomes highly compositionally complex.
“As K and M increase, synthesizing different partial solutions that satisfy individual constraints becomes highly compositionally complex.”
The paper relies on this without support.
Algorithms designed to solve compositional tasks typically involve multiple independent applications of a function and/or iterated applications of the same function.
“Algorithms designed to solve compositional tasks typically involve multiple independent applications of a function and/or iterated applications of the same function.”
The paper relies on this without support.
A transformer executing such an algorithm acts as an estimator of these functions.
“A transformer executing such an algorithm acts as an estimator of these functions.”
The authors state this as an assumption.
In Proposition 4.1, h_n is a perfect estimator of h_n.
“Assume that h n is a perfect estimator of h n and that h n has low collision, with c n being an upper bound of h n 's collision rate (c n < c ∀n, with c ≪ 1).”
The authors state this as an assumption.
In Proposition 4.1, h_n has low collision, with c_n being an upper bound of h_n's collision rate (c_n < c for all n, with c << 1).
“Assume that h n is a perfect estimator of h n and that h n has low collision, with c n being an upper bound of h n 's collision rate (c n < c ∀n, with c ≪ 1).”
The authors state this as an assumption.
In Proposition 4.1, g's errors are independent.
“If P(g ̸ = g) = ϵ > 0 where g's errors are independent”
The authors state this as an assumption.
In Proposition 4.2, the probability of recovering from a mistake due to the randomness of applying the estimator on an incorrect input is at most c.
“Assume that the probability of recovering from a mistake due to the randomness of applying the estimator on an incorrect input has probability at most c.”
The authors state this as an assumption.
In Proposition 4.2's current form, there is a single valid reasoning for each input since g is a function.
“In Prop.4.2's current form, we implicitly assume that there is a single valid reasoning for each input since g is a function.”
The authors state this as an assumption.
When g has low collision, c << epsilon.
“It is reasonable to assume c ≪ ϵ when g has low collision, since c represents the probability of the estimator g(y) arriving at the correct output g(x) by chance when given the wrong input y ̸ = x.”
The paper relies on this without support.
The tasks evaluated are low collision settings.
“They are also all low collision settings.”
The authors state this as an assumption.
The theoretical proofs' assumptions hold for the abstract compositional tasks and autoregressive language models considered.
“The proofs presented in §4 show that, under reasonable assumptions, the probability of incorrect predictions converges exponentially to ≈ 1 for abstract compositional tasks.”
The authors state this as an assumption.
In Proposition D.1, P(h_n = h_n)=1.
“Assume P(h n = h n ) = 1”
The authors state this as an assumption.
In Proposition D.1, P(h_n(X)=h_n(Y) | X≠Y) < c_n.
“P(h n (X) = h n (Y ) | X ̸ = Y ) < c n”
The authors state this as an assumption.
In Proposition D.1, c_n ≤ β α^n for some α∈(0,1) and β>0.
“if c n ≤ βα n for some some α ∈ (0, 1) and β > 0”
The authors state this as an assumption.
In Lemma D.2, P(z(g(X))=1 | z(X)=0) ≤ c.
“Assume P(z( g(X)) = 1 | z(X) = 0) ≤ c”
The authors state this as an assumption.
In Lemma D.2, P(z(g(X))=0 | z(X)=1) = ε > 0 with c+ε<1.
“P(z( g(X)) = 0 | z(X) = 1) = ϵ > 0 with c + ϵ < 1”
The authors state this as an assumption.
In Proposition D.2, P(g(X)=g(Y) | X≠Y) ≤ c.
“Assume P(g(X) = g(Y ) | X ̸ = Y ) ≤ c”
The authors state this as an assumption.
In Proposition D.2, P(g≠g)=ε>0 with c+ε<1.
“If P(g ̸ = g) = ϵ > 0 with c + ϵ < 1”
The authors state this as an assumption.
The mathematical framework used is a simplified representation of how multi-step reasoning works.
“The mathematical framework used is a simplified representation of how multi-step reasoning works”
The authors state this as an assumption.
c ≪ ϵ when g has low collision.
“It is reasonable to assume c ≪ ϵ when g has low collision, since c represents the probability of the estimator g(y) arriving at the correct output g(x) by chance when given the wrong input y ̸ = x.”
The authors state this as an assumption.
Assuming approximately uniform errors, c ≈ ϵ/|Im(g)|, which implies c ≪ ϵ when g has low collision.
“Assuming approximately uniform errors, c ≈ ϵ/|Im(g)|, which in turn implies c ≪ ϵ since g being low collision implies |Im(g)| is large.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This may be due to the increased complexity of the task (Dziri et al., 2023) where the ranking of N generations may become much more difficult as N gets larger (since the task complexity increases).”
From Benchmarking Cognitive Biases in Large Language Models as Evaluators (2024) · cites this paper as Dziri et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Frontier reasoning-tuned language models still fail on deductive tasks at depth, and the cost of improved performance through extended internal reasoning scales poorly.”
From PrologMCP: A Standardized Prolog Tool Interface for LLM Agents (2026) · cites this paper as [6]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“First, accuracy degrades on long deductive chains, even for strong reasoning-tuned models.”
From PrologMCP: A Standardized Prolog Tool Interface for LLM Agents (2026) · cites this paper as [6]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“we agree with Dziri et al. (2023) that there is a fundamental”
From What Algorithms can Transformers Learn? A Study in Length Generalization (2023) · cites this paper as Dziri et al. (2023)
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Limitations in compositionality, entity binding, long-context evidence use, and numerical grounding are therefore especially relevant, but are rarely examined within one unified setting [11][12][13][14][15][16][17][18].”
From Diagnosing the limits of large language models for structured scientific evidence extraction (2026) · cites this paper as [11]
Experimental — accuracy not yet measured
The cited source says what the claim attributes to it.
In the citing paper
“Earlier analyses of compositional tasks [6] and template-based variants of standard mathematics benchmarks [7] attribute such failures to pattern matching over training-distribution structure rather than genuine multi-step deduction, with accuracy degrading as the number of required reasoning steps grows.”
From PrologMCP: A Standardized Prolog Tool Interface for LLM Agents (2026) · cites this paper as [6]
The cited source says what the claim attributes to it.
In the citing paper
“Dziri et al. (2023) represent reasoning tasks as computation graphs and find that full computation subgraphs appear much more frequently in training data for correct predictions than incorrect ones.”
From GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models (2024) · cites this paper as Dziri et al. (2023)
The cited source says what the claim attributes to it.
In the citing paper
“the reasoning performance of GPT-4 negatively correlates with the depth of the problem's computation graph (Dziri et al., 2023).”
From The expressive power of transformers with chain of thought (2023) · cites this paper as Dziri et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Studies on compositional generalization show that models often struggle to combine familiar components into systematically correct novel structures [11,47,48].”
From Diagnosing the limits of large language models for structured scientific evidence extraction (2026) · cites this paper as [11]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Faith and Fate: Limits of Transformers on Compositionality > Additional information
“lim n→+∞ P(f n = f n ) = 1. Recall that P(f n = f n ) < αβ n + (1 -ϵ) n • (1 -αβ n ).”
§ Faith and Fate: Limits of Transformers on Compositionality > Additional information
“0 ≤ lim inf n→+∞ P(f n = f n ) ≤ lim sup n→+∞ P(f n = f n ) ≤ lim sup n→+∞ βα n + (1 -ϵ) n • (1 -βα n ) = 0”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Faith and Fate: Limits of Transformers on Compositionality > 4 Error Propagations: The Theoretical Limits
“This implies that P(f n ̸ = f n ) decreases exponentially as n increases, with lim inf n→+∞ P(f n ̸ = f n ) ≥ 1 -c.”
§ Faith and Fate: Limits of Transformers on Compositionality > 4 Error Propagations: The Theoretical Limits
“Prop. 4.1's proof ( §D.1) shows the rate of convergence is exponential, thus concluding that transformers will rapidly fail with increasing n.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Faith and Fate: Limits of Transformers on Compositionality > Limits of transformers with question-answer training
“We separately finetune GPT3 models on ∼1.8M multiplication pairs, ∼142K DP pairs, and ∼41K puzzle pairs”
§ Faith and Fate: Limits of Transformers on Compositionality > B.3 Limits of Transformers with question-answer Training
“As for the puzzle task, the model was fine-tuned on puzzles of sizes ranging from 2x2 to 4x4 resulting in a total of 142k pairs.Additionally, for the DP task, the model was fine-tuned on problems with a sequence length of 5 resulting in 41K pairs.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Faith and Fate: Limits of Transformers on Compositionality > 4 Error Propagations: The Theoretical Limits
“If P(g ̸ = g) = ϵ > 0, then P(f n ̸ = f n ) decreases exponentially with n.”
§ Faith and Fate: Limits of Transformers on Compositionality > 4 Error Propagations: The Theoretical Limits
“Note that if c ≪ ϵ then lim inf n→+∞ P(f n ̸ = f n ) ≈ 1.”
Citations
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 5×
“5-turbo) [40], GPT3 (text-davinci-003) [8], FlanT5 [? ] and LLaMa [57].”
BackgroundMethodInfluentialScaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · doi:10.48550/arxiv.2001.08361 · Cited by 9,550
de Moura, L., & Bjørner, N. (2008) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-78800-3_24 · Cited by 6,595 · cited 1×
“Graph Construction Algorithm To solve the complex compositional reasoning process for a logical grid puzzle, we use existing puzzle solvers [21] to generate the computation graph.”
MethodFeigelson, E. (2003) · Statistics for Environmental Science and Management · doi:10.1007/978-3-319-03762-2_13 · Cited by 6,344 · cited 2×
“When f is an affine transformation, the process falls simultaneously between two perspectives: time series [26] and dynamic programming and control [5].”
BackgroundLLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602 · cited 3×
“[57] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample.”
BackgroundChain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505 · cited 3×
“[61] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou.”
BackgroundMethodThe Architecture of Complexity
Simon, H. A. (1991) · Springer eBooks · doi:10.1007/978-1-4899-0718-9_31 · Cited by 4,479 · cited 1×
“Applying step-by-step reasoning is fundamental to human intelligence [51, 50].”
BackgroundPaLM: Scaling Language Modeling with Pathways
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.02311 · Cited by 2,590 · cited 1×
“Reasoning abilities in transformer LLMs Recently, transformers [11, 58, 57, 17, 16, 63, 73, 74] have demonstrated impressive reasoning abilities across a wide range of tasks, even outperforming humans in certain cases [79, 28, 15, 85].”
BackgroundSelf-Refine: Iterative Refinement with Self-Feedback
Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye, S., Yang, Y., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.17651 · Cited by 2,314 · cited 2×
“[36] Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Sean Welleck, Bodhisattwa Prasad Majumder, Shashank Gupta, Amir Yazdanbakhsh, and Peter Clark.”
BackgroundMethodSelf-citationShortcut learning in deep neural networks
Geirhos, R., Jacobsen, J.-H., Michaelis, C., Zemel, R., Brendel, W., Bethge, M., & Wichmann, F. A. (2020) · Nature Machine Intelligence · doi:10.1038/s42256-020-00257-z · Cited by 2,224 · cited 2×
“[25] Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann.”
Background
Research works printed but not linked: 6
- Cem Anil, Yuhuai Wu, Anders Andreassen, Aitor Lewkowycz, Vedant Misra, Vinay Ramasesh, Ambrose Slone, Guy Gur-Ari, Ethan Dyer, and Behnam Neyshabur. Exploring length general- ization in large language models. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 38546-38556.
- Arpit Bansal, Avi Schwarzschild, Eitan Borgnia, Zeyad Emam, Furong Huang, Micah Goldblum, and Tom Goldstein. End-to-end algorithm synthesis with recurrent networks: Extrapolation without overthinking. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 20232-20242.
- Yoshua Bengio, Réjean Ducharme, and Pascal Vincent. A neural probabilistic language model. In Advances in Neural Information Processing Systems, volume 13. MIT Press, 2000.
- D. Bertsekas. Abstract Dynamic Programming: 3rd Edition. Athena scientific optimization and computation series. Athena Scientific., 2022.
- Bingbin Liu, Jordan T. Ash, Surbhi Goel, Akshay Krishnamurthy, and Cyril Zhang. Trans- formers learn shortcuts to automata. In International Conference on Learning Representations, 2023.
- Karthik Valmeekam, Alberto Olmo, Sarath Sreedharan, and Subbarao Kambhampati. Large language models still can't plan (a benchmark for LLMs on planning and reasoning about change). In NeurIPS 2022 Foundation Models for Decision Making Workshop, 2022.
Cited by
Sorted by citations
Items 101–141 of 141
Ex2Vec: Enhancing assembly code semantics with end-to-end execution-aware embeddings
Gong, X., Yang, X., Zhang, S., & He, C. (2025) · Neural Networks · doi:10.1016/j.neunet.2025.107506 · Cited by 1
No Consciousness? No Meaning (and no AGI)!Preprint
Masi, M. (2025) · Qeios · doi:10.32388/dn232y.4
Assessing LLM Reasoning with Subtask Variation: Chess and DREAMS
Olea, C., Karns, A., & White, J. (2025) · IEEE International Conference on Cognitive Machine Intelligence (CogMI) · doi:10.1109/cogmi67134.2025.00014
Compositionality as a mechanism of intelligence in brains and machines
Zerroug, A. (2023) · theses.fr (ABES) · doi:10.70675/cd39b7d4z0eafz464az87caz2c10307cb623
Intelligibility of Recurrent Neural Networks via Finite State Machines
Marzouk, M. R. (2024) · theses.fr (ABES) · doi:10.70675/67c57743zf027z4a51zbd70zddd2e546358f
AGEL-Comp: A Neuro-Symbolic Framework for Compositional Generalization in Interactive Agents
Shahid, M., & Rothe, H. (2026) · Lecture notes in networks and systems · doi:10.1007/978-3-032-32732-1_19
A Diagnostic Framework for Compositional Generalization via NL-to-FOL Translation
Deveci, İ. E. (2026) · Journal of Logic Language and Information · doi:10.1007/s10849-026-09480-0
He, G., Chengmu, W., & Yang, Y. (2026) · Research Square · doi:10.21203/rs.3.rs-9656547/v1
Böhmer, A., Dillig, M., Andronache, D. C., et al. (2026) · Computers and Education Artificial Intelligence · doi:10.1016/j.caeai.2026.100636
Beyond Price: A Technical Quality Framework for AI Antitrust
Wu, P. Y. (2026) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3805689.3812338
On Locality and Length Generalization in Visual Reasoning
Madan, P., Haresh, S., Ebrahimi, R., Panchal, S., Bhattacharyya, A., & Memisevic, R. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37595-7_9
Gómez-Gutiérrez, O. A., Imbrognio, F., la Torre, O. A. D., Vegas, D. H., & Fabiani, M. A. (2026) · Annals of Vascular Surgery · doi:10.1016/j.avsg.2026.07.004
Hallucination Detection and Reduction in Large Language Models: A Practical Study
Rajput, K., H, P. A. B., Smitha, G., V, M, H. K., M, M. K. Y., & Kanbargi, M. (2026) · International Conference on Emerging Trends in Mobile Computing and Sustainable Informatics (ICEMCSI) · doi:10.1109/icemcsi67638.2026.11603013
“Dziri et al. [2] showed that such models often struggle with logical reasoning and compositional understanding, which can lead to inconsistent or incorrect outputs.”
Basu, S., Sheth, P., Muralidharan, B., Morgan, J., & Batniji, R. (2026) · AI · doi:10.3390/ai7070267 · Cited by 1
“…transformers are contained in the circuit class TC0 and cannot express certain compositional problems regardless of scale [12], a chain of thought relaxes but does not remove this for a bounded number of steps [13], and compositional accuracy decays empirically with problem depth [14].”
Pant, Y. R., Adarbah, H. Y., Afzel Noore, B., Che, D., Ahmed, A., & Ayala, R. (2026) · AI · doi:10.3390/ai7080280
“This tendency of LLMs to prioritize surface-level coherence over mathematical precision has been documented as a core limitation in multi-step reasoning tasks [41].”
Keep Adaptation Simple: Implicit vs. Explicit Adaptation for LLM-Based CBR
Regulagedda, R. M., Krupashankar, N. B., & Leake, D. B. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-33865-5_18
Break Out the Silverware: Semantic Understanding of Stored Household Items
Levi-Richter, M., Mirsky, R., & Glickman, O. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-31930-2_22
Layered Evidence-Centric Graph Construction for Explainable Multi-hop Question Answering
Zeng, W., Xie, H., Chao, Z., Wu, Q., Liu, H., Li, S., Qu, J., & Li, Z. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-34374-1_10
Why Large Language Models Cannot Be Certified for Safety-Critical Systems
Sayakov, A. (2026) · American Impact Review · doi:10.66308/air.e2026066
Semantic Raft: A Fault-Tolerant Multi-Agent Inference Framework for Reliable LLM Services
Su, Y., Wang, Y., Dong, Y., & Xu, Z. (2026) · International Workshop on Quality of Service · doi:10.1109/iwqos70441.2026.11661187
Wang, Z., Zhang, H., Hou, J., & Zheng, S. (2026) · International Conference on Electronics and Communication, Network and Computer Technology (ECNCT) · doi:10.1109/ecnct70535.2026.11661501
“The limits are documented too: Dziri et al. [14] show Transformers reduce multi-step compositional tasks to pattern matching rather than systematic computation, and Valmeekam et al. [15] find LLMs plan poorly alone (GPT-4 around 12% success) but improve when an external verifier checks their output.”
Do large language models stimulate student motivation? A meta-analysis of experimental studies
Han, X., & Liu, M. (2026) · Educational Technology Research and Development · doi:10.1007/s11423-026-10687-y
Zhao, C., Wang, W., Hu, H., Roy, N., & Zhang, B. (2026) · Journal of Loss Prevention in the Process Industries · doi:10.1016/j.jlp.2026.106159
Zhu, B., & Pan, R. (2026) · International Conference on Cyber Security, Artificial Intelligence and Digital Economy (CSAIDE) · doi:10.1145/3819836.3819899
A Transformation-Based Benchmark for Evaluating the Robustness of LLMs in Generating OCL
Attarwala, H., Chouchen, M., Alam, O., et al. (2026) · ACM/IEEE International Conference on Model Driven Engineering Languages and Systems (MODELS) · doi:10.1145/3822455.3830333
The recipe for intelligence in natural and artificial systems
Summerfield, C., & Stachenfeld, K. (2026) · Neuron · doi:10.1016/j.neuron.2026.08.013 · Cited by 1
Temporal grounding of Video Large Language Model via holistic event perception and matching
Nie, J., An, W., Zhang, G., Xu, Y., Tan, Y.-P., Kot, A. C., & Lu, S. (2026) · Pattern Recognition · doi:10.1016/j.patcog.2026.114930 · Cited by 1
COMPARATIVE ANALYSIS OF ENGINEERING DOCUMENTATION USING DETERMINISTIC AND LLM-BASED METHODS
Myhovych, A. V., & Kunytsia, A. (2026) · Journal of Rocket-Space Technology · doi:10.15421/452571
Atılan, A. U., & Çetin, N. (2026) · Cutaneous and Ocular Toxicology · doi:10.1080/15569527.2026.2733364 · Cited by 1
Camlet, A., Kusiak, A., & Świetlik, D. (2026) · Journal of Clinical Medicine · doi:10.3390/jcm15197387
“LLMs tend to underperform in compositional reasoning tasks that require solving multi-step problems [73–75].”
PROMISE: Process Reward Models for Unlocking Test-Time Scaling Laws in Generative Recommendations
Guo, C., Cai, K., Zhou, Y., Luo, Q., Tang, R., Li, H., Gai, K., & Zhou, G. (2026) · ACM Conference on Recommender Systems (RecSys) · doi:10.1145/3773078.3831741 · Cited by 20
Fan, J., Rao, J., Wei, L., Li, G., Yuan, J., & Qi, Y. (2026) · Expert Systems with Applications · doi:10.1016/j.eswa.2026.134549
Maass, W., Liddle, S. W., & Storey, V. C. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-40377-3_4
Tian, Z., Cao, X., Ma, X., et al. (2026) · 2022 IEEE 18th International Conference on Automation Science and Engineering (CASE) · doi:10.1109/case69030.2026.11704252
“The inherently probabilistic and autoregressive nature of LLMs [6] makes them fundamentally ill-equipped to directly generate numerical artifacts that require 100% strict constraint satisfaction, particularly in highly constrained spaces where the feasible region is vanishingly small [7], [8].”
Xu, S., & Asadinia, M. (2026) · IEEE International Conference on Omni-layer Intelligent Systems (COINS) · doi:10.1109/coins69995.2026.11715614
“Prior work shows that compositional generalization can degrade as proof depth increases [8], while longer rationales and internal reasoning traces increase autoregressive generation cost and key-value cache pressure [9], [10].”
E$$^3$$: Unified Architecture for Hallucination-Resistant Language Generation
Wang, G., & Nixon, M. S. (2026) · Communications in computer and information science · doi:10.1007/978-3-032-28469-3_19
Practical Mimicry and Modal Integration: Know-How, Large Language Models, and Human–AI Collaboration
Varghese, J. (2026) · Philosophy & Technology · doi:10.1007/s13347-026-01210-5
“Dziri et al. ( 2023) show that transformer LLMs solve compositional tasks such as multi-digit multiplication through what they term linearized subgraph matching, with performance decaying sharply once problem complexity moves outside previously seen sub-computations, the signature practical mimicry…”
Zheng, C., Peng, C., Liang, S., Cai, Z., Peng, Z., & Chen, Z. (2026) · Research Square · doi:10.21203/rs.3.rs-11029574/v1
Iterative Decomposition and Unified Hybrid Symbolic Execution for Small Language Models
Kakoty, D., Singh, K., & Nancy, P. (2026) · doi:10.1109/icic3s72305.2026.11737191
Beyond Statistical Learning: Exact Learning Is Essential for General Intelligence
György, A., Lattimore, T., Lazić, N., & Szepesvári, C. (2025) · arXiv · doi:10.48550/arxiv.2506.23908
Diagnosing the limits of large language models for structured scientific evidence extraction
Tan, Z., D’Souza, J., & Auer, S. (2026) · International Journal on Digital Libraries · doi:10.1007/s00799-026-00449-9
Metadata
- Authors
- Dziri, N., Lu, X., Sclar, M., Li, X. (., Jiang, L., Lin, B. Y., Welleck, S., West, P., Bhagavatula, C., Bras, R. L., Hwang, J., Sanyal, S., Ren, X., Ettinger, A., Harchaoui, Z., & Choi, Y.
- Venue
- Advances in Neural Information Processing Systems 36
- Published
- 2023
- Pages
- 70293-70332
- ISSN
- 1049-5258 (ISSN-L)
- OpenAlex
- W7133191918
- Cited by
- 335
- References
- 82
- Citations by year
- 2026: 106 · 2025: 77 · 2024: 92 · 2023: 60
- FWCI
- 610.22
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- neural information processing systems (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (2134012)
- Allen Institute for Artificial Intelligence
- Defense Advanced Research Projects Agency (N66001-19-2-4031, N66001-19)
- Naval Information Warfare Center Pacific
- Topics
- Social Sciences › Arts and Humanities › Language and Linguistics › Syntax, Semantics, Linguistic Variation
- Social Sciences › Arts and Humanities › Philosophy › Philosophy and Theoretical Science
- Social Sciences › Arts and Humanities › History and Philosophy of Science › Cybernetics and Technology in Society
- Keywords
- transformer models, compositional generalization, compositional reasoning, large language models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-23, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-23 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
