The DOI matches its DataCite record; 34 of the 34 references it cites were found in a registry or an index; 7 of the 28 sources its claims cite were checked for support.
Looped Transformers for Length Generalization
Open accessAbstract
Recent work has shown that Transformers trained from scratch can successfully solve various arithmetic and algorithmic tasks, such as adding numbers and computing parity. While these Transformers generalize well on unseen inputs of the same length, they struggle with length generalization, i.e., handling inputs of unseen lengths. In this work, we demonstrate that looped Transformers with an adaptive number of steps significantly improve length generalization. We focus on tasks with a known iterative solution, involving multiple iterations of a RASP-L operation - a length-generalizable operation that can be expressed by a finite-sized Transformer. We train looped Transformers using our proposed learning algorithm and observe that they learn highly length-generalizable solutions for various tasks.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
Looped Transformers with an adaptive number of looped steps and step-dependent supervision improve length generalization on algorithmic tasks with known iterative n-RASP-L solutions. Trained only on end-to-end input-output pairs plus the number of steps, they outperform NTP baselines on Parity, Copy, Addition, and related tasks.
Key findings
- Looped Transformers significantly improve length generalization: for Parity, the model generalizes to more than 40 digits near perfectly when trained with lengths up to 20 digits.
- For Addition and Copy, where next-token prediction failed when tested at maximum training length +10, the looped model still performs almost perfectly.
- Variants of NTP can improve generalization—NTP-Loop helps on Addition, Copy, and Multiplication; NTP-Pause helps slightly on Binary Sum and Unique Set—but all fall behind the adaptive-depth looped method.
- The maximum-confidence stopping criterion reaches lowest cross-entropy when generalization is near perfect; per-sample stopping works almost as well for converging tasks (Addition, Copy, Multiplication, Unique Set) but worse for non-converging tasks (Parity, Binary Sum).
Methods
- The paper defines n-RASP-L problems: tasks solvable by T(n) sequential applications of a RASP-L operation P′, with pre- and post-processing, and gives n-RASP-L solutions for Parity, Copy, and Addition.
- It trains looped decoder-only GPT-2 models with NoPE, input injection, and step-dependent supervision on final answers only, without intermediate-step supervision; training data include (x, y) pairs and the pre-defined number of steps T(n).
- Experiments cover Parity, Copy, Binary Addition, Binary Sum, Binary Multiplication, and Unique Set, with curriculum learning, AdamW, batch size 64, 100k gradient steps, evaluation on 6400 random samples, and mean exact match accuracy from five seeds; baselines include vanilla NTP, NTP with pause tokens, and NTP with weight-tied layers.
Limitations
- The paper states that n-RASP-L does not support tasks requiring multiple loops followed by each other, and extending to multiple loops is future work.
- Direct looped training can be computationally demanding when the number of looped steps is large; stopping gradient tracking for earlier steps may help but could trade off performance.
- The authors state they train for a limited number of steps and lengths due to limited computing resources, use NoPE for simplicity, and require the ground-truth number of steps in training data, an additional requirement compared with normal end-to-end training.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
20 Own interpretation · 16 Own result · 12 Sources not found · 9 No basis given · 5 Partly supported · 4 Not checked yet · 1 Supported · 1 Own method · accuracy not yet measured
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“However, even the largest and latest LLMs like Achiam et al. (2023) trained on much of the existing text on the Internet, still struggle with length generalization (Wu et al., 2023;Anil et al., 2022;Lee et al., 2024).”
§ LOOPED TRANSFORMERS FOR LENGTH GENERALIZA-TION > 1 INTRODUCTION
What the sources say
- Not matchedcites Wu et al., 2023
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Partly supportedcites Anil et al., 2022 sourceExperimental — accuracy not yet measured
The Conclusion (p. 9) says length generalization remains challenging for language models up to the 100B-parameter scale, and Section 3.1 (p. 4) reports weak extrapolation in the models tested. The paper does not establish the claim for the latest models cited in the sentence or for models trained on much of the Internet’s text; it also reports substantial improvements with few-shot scratchpad prompting. (LLM, conf=0.86)
- Not matchedcites Lee et al., 2024
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In this work, we demonstrate that looped Transformers with an adaptive number of steps significantly improve length generalization.”
- Also stated in: § 1 INTRODUCTION · plain
“Empirically, we show that looped Transformers with an adaptive number of steps can successfully length-generalize to longer lengths simply by appropriately adapting the number of loops at inference time, indicating that our approach encourages the model to implicitly learn the necessary steps to solve a task.”
- Also stated in: § 1 INTRODUCTION · boosted
“(3) Empirically, we show that our proposed method outperforms the baseline approaches in terms of length generalization performance (Section 6).”
§ LOOPED TRANSFORMERS FOR LENGTH GENERALIZA-TION > Abstract
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 1 INTRODUCTION · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We train looped Transformers using our proposed learning algorithm and observe that they learn highly length-generalizable solutions for various tasks.”
§ LOOPED TRANSFORMERS FOR LENGTH GENERALIZA-TION > Abstract
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Compared with vanilla NTP, we observe that NTP-Loop could lead to improved generalization in tasks like Addition, Copy and Multiplication.”
§ LOOPED TRANSFORMERS FOR LENGTH GENERALIZA-TION > 6.2 LENGTH GENERALIZATION RESULTS
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“(3) Empirically, we show that our proposed method outperforms the baseline approaches in terms of length generalization performance (Section 6).”
§ LOOPED TRANSFORMERS FOR LENGTH GENERALIZA-TION > 1 INTRODUCTION
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“we find input injection useful to further improve the performance (see details in Section 6.3).”
§ LOOPED TRANSFORMERS FOR LENGTH GENERALIZA-TION > 5 RELATED WORK
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“we also find 2) to be an effective stopping criterion in Section 6.4.”
§ LOOPED TRANSFORMERS FOR LENGTH GENERALIZA-TION > 6.1.3 TRAINING AND EVALUATION SETUP
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Proposition 3.2.(Parity.)There exists a n-RASP-L program with T (n) = n that solves the n-bit parity check task:”
§ LOOPED TRANSFORMERS FOR LENGTH GENERALIZA-TION > 3 n-RASP-L
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For example, for Parity, it can generalize to more than 40 digits near perfectly when trained with up to 20 digits.”
§ LOOPED TRANSFORMERS FOR LENGTH GENERALIZA-TION > 6.2 LENGTH GENERALIZATION RESULTS
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Looped Transformers help with length generalization.Our looped training significantly improves the length generalization performance.”
- Also stated in: § F EXPERIMENTAL DETAILS · plain
“Our looped Transformer model with adaptive depth generalized better than NTP methods across studied tasks, including the variants with pause tokens and weight-tied layers.”
§ LOOPED TRANSFORMERS FOR LENGTH GENERALIZA-TION > 6.2 LENGTH GENERALIZATION RESULTS
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § F EXPERIMENTAL DETAILS · plain
- It may state two separate claims as one. 18
- Probably not a substantial claim. 27
- The quote only partly states this claim. 1
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“It has been shown that copy with unique tokens could be easily solved by inductive head (Olsson et al., 2022), but copy with repeated tokens (e.g., binary) does not length-generalize with vanilla NTP training (Zhou et al., 2024a).”
§ LOOPED TRANSFORMERS FOR LENGTH GENERALIZA-TION > Copy (with repeated tokens).
What the sources say
- Supportedcites Olsson et al., 2022 source
The Olsson et al. citation supports the first clause of the citing sentence. In “Induction Heads” (p. 5), the paper defines induction heads by their ability to use an earlier [A][B] pair to increase the likelihood of [B] when [A] recurs in a random-token sequence. The repeated-token length-generalization claim is cited to Zhou et al., not to this source, and is not evaluated here. (LLM, conf=0.86)
- Not matchedcites Zhou et al., 2024a
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Positional embeddings have been shown to greatly affect Transformers' ability to generalize to longer lengths (Press et al., 2021;Kazemnejad et al., 2024;Ruoss et al., 2023;Su et al., 2024;Li et al., 2023;Cho et al., 2024;Sabbaghi et al., 2024;McLeish et al., 2024;Golovneva et al., 2024).”
§ LOOPED TRANSFORMERS FOR LENGTH GENERALIZA-TION > 5 RELATED WORK
What the sources say
- Supportedcites Press et al., 2021 source
The Introduction (p. 1) says weak length extrapolation with sinusoidal position embeddings is caused by the position embedding method. The Conclusion (p. 9) says changing the position method enables Transformers to extrapolate to longer inputs, supporting the citing sentence’s broad claim. (LLM, conf=0.94)
- Not matchedcites Kazemnejad et al., 2024
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Supportedcites Ruoss et al., 2023 source
The paper’s “This work” section reports that changing positional encodings significantly improved Transformers’ length generalization on algorithmic reasoning tasks. Its Conclusion likewise says the proposed encodings significantly outperformed prior approaches, supporting the cited statement that positional embeddings affect this ability. (LLM, conf=0.92)
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“To learn a model that can effectively generalize to longer problems, it is important to consider architectures that can adaptively adjust the computational budget to the difficulty of the tasks (Anil et al., 2022;Du et al., 2022;2024).”
§ LOOPED TRANSFORMERS FOR LENGTH GENERALIZA-TION > 1 INTRODUCTION
What the sources say
- Partly supportedcites Anil et al., 2022 sourceExperimental — accuracy not yet measured
Section 2 (p. 3) says longer problem instances require more compute, and Section 5.1 (p. 7) describes scratchpad solutions with a step for each operation. But the paper does not recommend architectures that adaptively adjust computational budgets; its Conclusion (p. 9) emphasizes a length-generalization approach that does not require changing model architecture. (LLM, conf=0.82)
- Not matchedcites Du et al., 2022;2024
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“One approach to achieve this is to explicitly generate intermediate output tokens, similar to writing down a scratchpad, which improves LLMs' capability for solving harder problems (Nye et al., 2021).”
§ LOOPED TRANSFORMERS FOR LENGTH GENERALIZA-TION > 1 INTRODUCTION
What the sources say
- Supportedcites Nye et al., 2021 source
The Introduction describes generating intermediate tokens as a scratchpad before the final answer. The Conclusion reports that this approach improved performance on challenging algorithmic tasks, including long addition, polynomial evaluation, and Python code execution. (LLM, conf=0.92)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“As discussed in Goyal et al. (2023), pause tokens mainly introduce parallel but not sequential compute, which is less powerful than adaptive depth.”
§ LOOPED TRANSFORMERS FOR LENGTH GENERALIZA-TION > 6.3 ABLATION STUDIES
What the sources say
- Partly supportedcites Goyal et al. (2023) sourceExperimental — accuracy not yet measured
Section I.2 says pause tokens are processed in parallel and do not add sequential computation. The discussion of adaptive compute says some methods increase serial computation, but it does not establish that pause tokens are generally less powerful than adaptive depth; the paper also identifies tasks where parallel computation is advantageous. (LLM, conf=0.86)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Partly supported
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Similarly, NTP-pause could introduce slight improvement in Binary Sum and Unique Set.”
§ LOOPED TRANSFORMERS FOR LENGTH GENERALIZA-TION > 6.2 LENGTH GENERALIZATION RESULTS
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, they all fall behind compared with our method.”
- Also stated in: § 6.2 LENGTH GENERALIZATION RESULTS · hedged
“Variants of NTP could improve generalization but not as effectively as our adaptive-depth model.”
§ LOOPED TRANSFORMERS FOR LENGTH GENERALIZA-TION > 6.2 LENGTH GENERALIZATION RESULTS
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § 6.2 LENGTH GENERALIZATION RESULTS · hedged
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The convergence nature of the tasks also affects the performance of the per-sample-based stopping criterion (B = 1) in Figure 6 (Appendix B): the per-sample stopping criterion performs almost as well as when using B = N test to decide for converging tasks, but the performance is worse in non-converging tasks.”
- Also stated in: § B RESULTS FROM THE ADAPTIVE STOPPING CRITERIA · plain
“For tasks with converging behaviors (Addition, Copy, Multiplication, Unique Set), per sampler stopping criterion works almost as well as using B = N test , while for tasks without the converging behavior (Parity and Binary Sum), per sampler stopping criterion does not work as well as using B = N test .”
§ LOOPED TRANSFORMERS FOR LENGTH GENERALIZA-TION > 6.4 THE STOPPING CRITERION AND VISUALIZATIONS
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § B RESULTS FROM THE ADAPTIVE STOPPING CRITERIA · plain
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The loss curves for these tasks are also smoother than those without convergence behaviors.”
§ LOOPED TRANSFORMERS FOR LENGTH GENERALIZA-TION > 6.4 THE STOPPING CRITERION AND VISUALIZATIONS
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Additionally, non-converging tasks are less tolerant when choosing which step to stop, and we find using the exponential moving average (EMA) of model parameters helpful for Parity and Binary Sum with a factor 0.9999.”
§ Input example: 1 1 0 1 > # # # # Output example: * * * * 1 # # # # * indicates ignored token, > is EOQ, and # is EOS.def parity_step(partial_ans_seq, seq): # align the last digit with the answer location seq = shift_right(seq, 1) # calculate XOR partial_ans_seq = ( partial_ans_seq | seq) & (~(partial_ans_seq & seq) ) return partial_ans_seq, seq def parity_loop(seq, num_step): # get the question in the prompt prompt_mask = 1 -has_seen(seq, full(seq, EOQ)) seq = mask(seq, prompt_mask) # init answer seq with 0 partial_ans_seq = full(seq, 0) # generate EOS seq after EOQ end_seq = where(prompt_mask==1, full(seq, 0), full(seq, EOS)) # perform parity steps for i in range(num_step): partial_ans_seq, seq = parity_step(partial_ans_seq, seq) # get answer with EOS ans_seq = partial_ans_seq end_seq = shift_right(end_seq, 1) ans_seq = where(end_seq==EOS, end_seq, ans_seq) return ans_seq Listing 1: Parity.# Input example: 0 1 0 1 1 > # # # # # # # Output example: * * * * * 0 1 0 1 1 # # # * indicates ignored token, > is EOQ, and # is EOS.def copy_step(seq, end_seq): seq = shift_right(seq, 1) end_seq = shift_right(end_seq, 1) return seq, end_seq def copy_loop(seq, num_step): # generate EOS seq after EOQ end_mask = has_seen(seq, full(seq, EOQ)) end_seq = where(end_mask==0, full(seq, 0), full(seq, EOS)) # perform copy steps for i in range(num_step): seq, end_seq = copy_step(seq, end_seq) # get answer with EOS seq = where(end_seq==EOS, end_seq, seq) return seq Listing 2: Copy.# Input example: 0 0 1 + 1 1 1 > # # # # # # # Output example: * * * * * * * 1 0 0 0 # # # # * indicates ignored token, > is EOQ, and # is EOS.def addition_step(seq1, seq2, end_seq): end_seq = shift_right(end_seq, 1) seq1 = np.array(seq1,dtype=bool) seq2 = np.array(seq2,dtype=bool) carry_on = seq1 & seq2 # A XOR B = (A OR B) AND (NOT (A AND B)) in_place = ((seq1 | seq2) & (~(seq1 & seq2))) in_place = shift_right(in_place, 1) seq1 = np.array(in_place,dtype=int) seq2 = np.array(carry_on,dtype=int) return seq1, seq2, end_seq def addition_preprocess(seq): # generate EOS seq after EOQ end_mask = has_seen(seq, full(seq, EOQ)) end_seq = where(end_mask==0, full(seq, 0), full(seq, EOS)) # generate masks for the first and second summands seen_tok0 = has_seen(seq, full(seq, ADD_SIGN)) seen_tok1 = has_seen(seq, full(seq, EOQ)) mask1 = ~seen_tok0 mask2 = seen_tok0 & (~seen_tok1) mask2 = mask2 & shift_right(mask2, 1) # get the first and second summands seq1 = mask(seq, mask1) seq2 = mask(seq, mask2) # align the first summand with the second induct_num1 = cumsum(mask1) induct_num2 = cumsum(mask2) target_index = firsts(induct_num1, induct_num2, default=0) seq1 = index_select(seq1, target_index) seq1 = mask(seq1, mask2) return seq1, seq2, end_seq def addition_loop(seq, num_step): seq1, seq2, end_seq = addition_preprocess(seq) # perform addition steps for i in range(num_step): seq1, seq2, end_seq = addition_step(seq1, seq2, end_seq) # get answer with EOS ans = seq1 ans = where(end_seq==EOS, end_seq, ans) return ans Listing 3: Addition (in forward order).import numpy as np def full(x, const): return np.full_like(x, const, dtype=int) def indices(x): return np.arange(len(x), dtype=int) def select(k, q, pred, causal=True): # compute attention matrix s = len(k) A = np.zeros((s,s), dtype=bool) for qi in range(s): for kj in (range(qi+1) if causal else range(s)): # k_index <= q_index if causal A[qi, kj] = pred(k[kj], q[qi]) return A def sel_width(A): return np.dot(A, np.ones(len(A))).astype(int)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Additionally, non-converging tasks are less tolerant when choosing which step to stop, and we find using the exponential moving average (EMA) of model parameters helpful for Parity and Binary Sum with a factor 0.9999.”
§ Input example: 1 1 0 1 > # # # # Output example: * * * * 1 # # # # * indicates ignored token, > is EOQ, and # is EOS.def parity_step(partial_ans_seq, seq): # align the last digit with the answer location seq = shift_right(seq, 1) # calculate XOR partial_ans_seq = ( partial_ans_seq | seq) & (~(partial_ans_seq & seq) ) return partial_ans_seq, seq def parity_loop(seq, num_step): # get the question in the prompt prompt_mask = 1 -has_seen(seq, full(seq, EOQ)) seq = mask(seq, prompt_mask) # init answer seq with 0 partial_ans_seq = full(seq, 0) # generate EOS seq after EOQ end_seq = where(prompt_mask==1, full(seq, 0), full(seq, EOS)) # perform parity steps for i in range(num_step): partial_ans_seq, seq = parity_step(partial_ans_seq, seq) # get answer with EOS ans_seq = partial_ans_seq end_seq = shift_right(end_seq, 1) ans_seq = where(end_seq==EOS, end_seq, ans_seq) return ans_seq Listing 1: Parity.# Input example: 0 1 0 1 1 > # # # # # # # Output example: * * * * * 0 1 0 1 1 # # # * indicates ignored token, > is EOQ, and # is EOS.def copy_step(seq, end_seq): seq = shift_right(seq, 1) end_seq = shift_right(end_seq, 1) return seq, end_seq def copy_loop(seq, num_step): # generate EOS seq after EOQ end_mask = has_seen(seq, full(seq, EOQ)) end_seq = where(end_mask==0, full(seq, 0), full(seq, EOS)) # perform copy steps for i in range(num_step): seq, end_seq = copy_step(seq, end_seq) # get answer with EOS seq = where(end_seq==EOS, end_seq, seq) return seq Listing 2: Copy.# Input example: 0 0 1 + 1 1 1 > # # # # # # # Output example: * * * * * * * 1 0 0 0 # # # # * indicates ignored token, > is EOQ, and # is EOS.def addition_step(seq1, seq2, end_seq): end_seq = shift_right(end_seq, 1) seq1 = np.array(seq1,dtype=bool) seq2 = np.array(seq2,dtype=bool) carry_on = seq1 & seq2 # A XOR B = (A OR B) AND (NOT (A AND B)) in_place = ((seq1 | seq2) & (~(seq1 & seq2))) in_place = shift_right(in_place, 1) seq1 = np.array(in_place,dtype=int) seq2 = np.array(carry_on,dtype=int) return seq1, seq2, end_seq def addition_preprocess(seq): # generate EOS seq after EOQ end_mask = has_seen(seq, full(seq, EOQ)) end_seq = where(end_mask==0, full(seq, 0), full(seq, EOS)) # generate masks for the first and second summands seen_tok0 = has_seen(seq, full(seq, ADD_SIGN)) seen_tok1 = has_seen(seq, full(seq, EOQ)) mask1 = ~seen_tok0 mask2 = seen_tok0 & (~seen_tok1) mask2 = mask2 & shift_right(mask2, 1) # get the first and second summands seq1 = mask(seq, mask1) seq2 = mask(seq, mask2) # align the first summand with the second induct_num1 = cumsum(mask1) induct_num2 = cumsum(mask2) target_index = firsts(induct_num1, induct_num2, default=0) seq1 = index_select(seq1, target_index) seq1 = mask(seq1, mask2) return seq1, seq2, end_seq def addition_loop(seq, num_step): seq1, seq2, end_seq = addition_preprocess(seq) # perform addition steps for i in range(num_step): seq1, seq2, end_seq = addition_step(seq1, seq2, end_seq) # get answer with EOS ans = seq1 ans = where(end_seq==EOS, end_seq, ans) return ans Listing 3: Addition (in forward order).import numpy as np def full(x, const): return np.full_like(x, const, dtype=int) def indices(x): return np.arange(len(x), dtype=int) def select(k, q, pred, causal=True): # compute attention matrix s = len(k) A = np.zeros((s,s), dtype=bool) for qi in range(s): for kj in (range(qi+1) if causal else range(s)): # k_index <= q_index if causal A[qi, kj] = pred(k[kj], q[qi]) return A def sel_width(A): return np.dot(A, np.ones(len(A))).astype(int)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim have been checked yet.
In the paper
“Training with pause tokens in FOP could sometimes boost the generalization performance compared to vanilla FOP, but not as effective as our method with looped steps and adaptive depth.”
§ LOOPED TRANSFORMERS FOR LENGTH GENERALIZA-TION > 6.3 ABLATION STUDIES
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“Their results show that RNNs and Transformers fail to generalize on non-regular tasks, LSTMs can solve regular and counter-language tasks, and only networks augmented with structured memory (such as a stack or memory tape) can successfully generalize on some context-free and context-sensitive tasks.”
§ LOOPED TRANSFORMERS FOR LENGTH GENERALIZA-TION > 5 RELATED WORK
What the sources say
- Not matchedcites Delétang et al. (2022)
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Generally, it is hard to expect models to be trained on inputs with all possible lengths, and we need to rely on length generalization.”
§ LOOPED TRANSFORMERS FOR LENGTH GENERALIZA-TION > 1 INTRODUCTION
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On the other hand, architectures such as the Looped Transformer have external loops embedded in the architecture which naturally provides adaptive depth.”
§ LOOPED TRANSFORMERS FOR LENGTH GENERALIZA-TION > 3 n-RASP-L
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
In the paper
“Their results show that RNNs and Transformers fail to generalize on non-regular tasks, LSTMs can solve regular and counter-language tasks, and only networks augmented with structured memory (such as a stack or memory tape) can successfully generalize on some context-free and context-sensitive tasks.”
§ LOOPED TRANSFORMERS FOR LENGTH GENERALIZA-TION > 5 RELATED WORK
What the sources say
- Not matchedcites Delétang et al. (2022)
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As a result, we expect the model to learn length-generalizable intermediate steps that could potentially generalize to more steps at test time.”
§ LOOPED TRANSFORMERS FOR LENGTH GENERALIZA-TION > 4.2.1 ARCHITECTURE OF THE LOOPED TRANSFORMERS
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one. The quote only partly states this claim.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“While Transformers can accept a variable length of inputs (that can be processed in parallel), they usually have a fixed depth.”
§ LOOPED TRANSFORMERS FOR LENGTH GENERALIZA-TION > 1 INTRODUCTION
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“Notice that these approaches still use fixed-depth models.While these approaches help solve more complex reasoning tasks, they are still far from achieving near-perfect length generalization for simple algorithmic tasks.”
§ LOOPED TRANSFORMERS FOR LENGTH GENERALIZA-TION > 1 INTRODUCTION
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The number of steps needed in the loop depends on the problem's complexity, i.e., more difficult problems could potentially require more steps to solve.”
§ LOOPED TRANSFORMERS FOR LENGTH GENERALIZA-TION > 1 INTRODUCTION
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Also, if a model can length-generalize, it means the model has truly learned the correct algorithmic solution to the task, not just a spurious solution that works only for certain lengths.”
§ LOOPED TRANSFORMERS FOR LENGTH GENERALIZA-TION > 1 INTRODUCTION
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
In the paper
“For instance, Lee et al. applied CoT for arithmetic tasks but observed that Transformers cannot length generalize even for simple addition tasks (Lee et al., 2024).”
§ LOOPED TRANSFORMERS FOR LENGTH GENERALIZA-TION > 1 INTRODUCTION
What the sources say
- Not matchedcites Lee et al., 2024
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, such input representations are highly dependent on the specific problems and might not necessarily exist in general.”
§ LOOPED TRANSFORMERS FOR LENGTH GENERALIZA-TION > Input representation.
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
In the paper
“In Zhou et al. (2024a), they show algorithmic tasks that can be written as a RASP-L program can be easily learned by a Transformer in a length-generalizable way with next-token prediction.”
§ LOOPED TRANSFORMERS FOR LENGTH GENERALIZA-TION > 2.1 RASP-L
What the sources say
- Not matchedcites Zhou et al. (2024a)
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This might be sufficient for certain tasks, but not always.”
§ LOOPED TRANSFORMERS FOR LENGTH GENERALIZA-TION > 1 INTRODUCTION
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds. It may state two separate claims as one.
In the paper
“Their results show that RNNs and Transformers fail to generalize on non-regular tasks, LSTMs can solve regular and counter-language tasks, and only networks augmented with structured memory (such as a stack or memory tape) can successfully generalize on some context-free and context-sensitive tasks.”
§ LOOPED TRANSFORMERS FOR LENGTH GENERALIZA-TION > 5 RELATED WORK
What the sources say
- Not matchedcites Delétang et al. (2022)
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Not matched
None of the sources cited for this claim have been checked yet.
In the paper
“With input injection, the model can maintain a strong connection to the original input, preventing information loss with improved performance (Bai et al., 2019;Yang et al., 2024).”
§ LOOPED TRANSFORMERS FOR LENGTH GENERALIZA-TION > 4.2.1 ARCHITECTURE OF THE LOOPED TRANSFORMERS
What the sources say
- Not checked — no full text of the sourcecites Bai et al., 2019 source
- Not matchedcites Yang et al., 2024
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In our paper, the Looped Transformer architecture also has augmented memory and the recurrent structure but is potentially more powerful since each iteration contains an operation of the whole sequence.”
§ LOOPED TRANSFORMERS FOR LENGTH GENERALIZA-TION > 5 RELATED WORK
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
In the paper
“However, they also showed that for algorithmic tasks whose RASP-L program representation is not known to exist, such as addition, parity, and copying the input sequence, it is hard to learn in a length-generalizable way.”
§ LOOPED TRANSFORMERS FOR LENGTH GENERALIZA-TION > 2.1 RASP-L
What the sources say
- Not matchedcites Zhou et al. (2024a)
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Besides, CoT is normally constrained to fixed-depth models, while we study a more general and powerful way to use adaptive compute at inference time.”
§ LOOPED TRANSFORMERS FOR LENGTH GENERALIZA-TION > Input representation.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The length of an input often correlates with the difficulty of the problem instance.For example, the longer the input, the more difficult the problem tends to be.”
§ LOOPED TRANSFORMERS FOR LENGTH GENERALIZA-TION > 1 INTRODUCTION
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For n-RASP-L problems, if we can learn these lengthindependent steps, we can utilize an adaptive number of steps to achieve length generalization.”
§ LOOPED TRANSFORMERS FOR LENGTH GENERALIZA-TION > 1 INTRODUCTION
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
It may state two separate claims as one.
In the paper
“Second, not all CoT steps are helpful.If CoT steps introduce additional complexity or require operations not easily expressible in RASP-L, then CoT may hinder length generalization, as shown in Zhou et al. (2024a).”
§ LOOPED TRANSFORMERS FOR LENGTH GENERALIZA-TION > Input representation.
What the sources say
- Not matchedcites Zhou et al. (2024a)
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For the first stopping criterion where we know a(n ′ ), our inference time would be O(ka(n ′ )n ′2 ), and NTP (with KV cache) and FOP will be O(Cn ′2 ).”
§ LOOPED TRANSFORMERS FOR LENGTH GENERALIZA-TION > D VISUALIZATION OF USING PAUSE TOKENS IN NTP AND FOP
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- The authors state this as an assumption: The maximum length of the training set is n, the number of steps needed is T(n), and the number of layers in each step is k.
- The authors state this as an assumption: NTP and FOP use a fixed number of layers C.
- The authors state this as an assumption: The test length is n'.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“We only train the looped Transformers for a limited number of steps and lengths due to a lack of computing resources.”
§ LOOPED TRANSFORMERS FOR LENGTH GENERALIZA-TION > 7 LIMITATIONS AND CONCLUSION
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In NTP and FOP, we use some C ≈ kT (n) in our experiments such that they use similar compute during training.Our inference time is then slightly longer than NTP with KV cache and FOP since we use more steps than the fixed-depth models.”
§ LOOPED TRANSFORMERS FOR LENGTH GENERALIZA-TION > D VISUALIZATION OF USING PAUSE TOKENS IN NTP AND FOP
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Moreover, our step-dependent supervision requires the ground-truth number of steps in the training data, which is an additional requirement compared with normal end-to-end training.”
§ LOOPED TRANSFORMERS FOR LENGTH GENERALIZA-TION > 7 LIMITATIONS AND CONCLUSION
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For the second criterion, we need to specify the maximum number of steps in order to find the step with maximum confidence.So our inference time would be O(kN ′ n ′2 ), where N ′ is the maximum number of steps.”
§ LOOPED TRANSFORMERS FOR LENGTH GENERALIZA-TION > D VISUALIZATION OF USING PAUSE TOKENS IN NTP AND FOP
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- The authors state this as an assumption: The maximum length of the training set is n, the number of steps needed is T(n), and the number of layers in each step is k.
- The authors state this as an assumption: NTP and FOP use a fixed number of layers C.
- The authors state this as an assumption: The test length is n'.
The paper relies on this without support.
Intermediate supervision and CoT training data can be difficult to collect.
“However, such intermediate supervision can be difficult to get, just like collecting helping CoT steps could be difficult.”
“First, CoT training data could be hard to collect.”
The paper relies on this without support.
CoT steps that could convert the next token prediction task to RASP-L programs might not always exist.
“Moreover, CoT steps that could convert the next token prediction task to RASP-L programs might not always exist.”
The authors state this as an assumption.
The maximum length of the training set is n, the number of steps needed is T(n), and the number of layers in each step is k.
“Assume that the maximum length of the training set is n, the number of steps needed is T (n), and the number of layers in each step is k.”
The authors state this as an assumption.
NTP and FOP use a fixed number of layers C.
“Assume that NTP and FOP are using a fixed number of layers C.”
The authors state this as an assumption.
The test length is n'.
“And we test on length n ′ .”
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ LOOPED TRANSFORMERS FOR LENGTH GENERALIZA-TION > Binary Multiplication.
“The output is in reversed order and the length is the sum of the lengths of two numbers, without removing leading 0.”
§ LOOPED TRANSFORMERS FOR LENGTH GENERALIZA-TION > Binary Multiplication.
“1 1 × 1 1 0 > # # # # # , example output: * * * * * * 0 1 0 0 1 0 # .”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ LOOPED TRANSFORMERS FOR LENGTH GENERALIZA-TION > Binary Multiplication.
“1 1 × 1 1 0 > # # # # # , example output: * * * * * * 0 1 0 0 1 0 # .”
§ LOOPED TRANSFORMERS FOR LENGTH GENERALIZA-TION > 2.2 NEXT-TOKEN PREDICTION AND FULL-OUTPUT PREDICTION
“During both training and inference time, the input given is just the query part, and the rest of the locations are filled with multiple EOS tokens to keep the input and the output to be the same length.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ LOOPED TRANSFORMERS FOR LENGTH GENERALIZA-TION > Binary Addition.
“Example input: 1 0 + 1 1 > # # # , example output: * * * * * 1 0 1 # # .”
§ LOOPED TRANSFORMERS FOR LENGTH GENERALIZA-TION > 2.2 NEXT-TOKEN PREDICTION AND FULL-OUTPUT PREDICTION
“During both training and inference time, the input given is just the query part, and the rest of the locations are filled with multiple EOS tokens to keep the input and the output to be the same length.”
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 4
- Yilun Du, Shuang Li, Joshua Tenenbaum, and Igor Mordatch. Learning iterative reasoning through energy minimization. In International Conference on Machine Learning, pp. 5570-5582. PMLR, 2022.
- Yilun Du, Jiayuan Mao, and Joshua B Tenenbaum. Learning iterative reasoning through energy diffusion. In Forty-first International Conference on Machine Learning, 2024.
- Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019.
- Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Cited by
Sorted by citations
Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computation
Bae, S., Kim, Y., Bayat, R., et al. (2025) · Neural Information Processing Systems · doi:10.48550/arxiv.2507.10524 · Cited by 168
“Looped Transformers shown to be effective, can act as programmable computers [33], learn iterative data-fitting algorithms [100], generalize to much longer inputs on algorithmic tasks [23], and illuminate few-shot learning by mimicking multi-step optimizers [27].”
Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRAPreprint
Bae, S., Fisch, A., Harutyunyan, H., Ji, Z., Kim, S., & Schuster, T. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2410.20672 · Cited by 41
“The examples utilize models with six layers, where identical colors represent shared weights.”
LoopFormer: Elastic-Depth Looped Transformers for Latent Reasoning via Shortcut Modulation
Jeddi, A., Ciccone, M., & Taati, B. (2026) · arXiv.org · doi:10.48550/arxiv.2602.11451 · Cited by 77
“Once trained, their representations collapse when evaluated at shorter or longer depths, since those settings are off-distribution Bae et al. (2024); Fan et al. (2024).”
Teaching Pretrained Language Models to Think Deeper with Retrofitted RecurrencePreprint
McLeish, S., Li, A., Kirchenbauer, J., Kalra, D. S., Bartoldson, B. R., Kailkhura, B., Schwarzschild, A., Geiping, J., Goldstein, T., & Goldblum, M. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2511.07384 · Cited by 55
“Recurrent mechanisms are shown to learn generalizable solutions to problems using ResNet (He et al., 2015) based architectures (Schwarzschild et al., 2021; Bansal et al., 2022; Anil et al., 2022; Schwarzschild, 2023; Bear et al., 2024).”
Loop, Think, & Generalize: Implicit Reasoning in Recurrent-Depth TransformersPreprint
Kohli, H., Parthasarathy, S., Sun, H., & Yao, Y. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2604.07822 · Cited by 30
“Fan et al. (2025) show that such looped transformers offer superior length generalization on algorithmic tasks such as parity and binary addition.”
Metadata
- Authors
- Fan, Y., Du, Y., Ramchandran, K., & Lee, K.
- Affiliations
- University of Wisconsin–Madison (US)
- Massachusetts Institute of Technology (US)
- University of California, Berkeley (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2024
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2409.15647
- OpenAlex
- W4403788968
- Semantic Scholar
- 408295dd8cb6140179fe075ca04a9d80f064bd36
- Cited by
- 0
- References
- 0
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (2339978, 2023239, CAREER)
- Topics
- Physical Sciences › Computer Science › Computational Theory and Mathematics › semigroups and automata theory
- Physical Sciences › Computer Science › Computational Theory and Mathematics › Computability, Logic, AI Algorithms
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Keywords
- length generalization, iterative computation, arithmetic exercises, algorithmic tasks
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-23, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-23 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
