The DOI matches its DataCite record; 34 of the 34 references it cites were found in a registry or an index; 7 of the 28 sources its claims cite were checked for support.
Looped Transformers for Length Generalization
Open accessAbstract
Recent work has shown that Transformers trained from scratch can successfully solve various arithmetic and algorithmic tasks, such as adding numbers and computing parity. While these Transformers generalize well on unseen inputs of the same length, they struggle with length generalization, i.e., handling inputs of unseen lengths. In this work, we demonstrate that looped Transformers with an adaptive number of steps significantly improve length generalization. We focus on tasks with a known iterative solution, involving multiple iterations of a RASP-L operation - a length-generalizable operation that can be expressed by a finite-sized Transformer. We train looped Transformers using our proposed learning algorithm and observe that they learn highly length-generalizable solutions for various tasks.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
Looped Transformers with an adaptive number of looped steps and step-dependent supervision improve length generalization on algorithmic tasks with known iterative n-RASP-L solutions. Trained only on end-to-end input-output pairs plus the number of steps, they outperform NTP baselines on Parity, Copy, Addition, and related tasks.
Key findings
- Looped Transformers significantly improve length generalization: for Parity, the model generalizes to more than 40 digits near perfectly when trained with lengths up to 20 digits.
- For Addition and Copy, where next-token prediction failed when tested at maximum training length +10, the looped model still performs almost perfectly.
- Variants of NTP can improve generalization—NTP-Loop helps on Addition, Copy, and Multiplication; NTP-Pause helps slightly on Binary Sum and Unique Set—but all fall behind the adaptive-depth looped method.
- The maximum-confidence stopping criterion reaches lowest cross-entropy when generalization is near perfect; per-sample stopping works almost as well for converging tasks (Addition, Copy, Multiplication, Unique Set) but worse for non-converging tasks (Parity, Binary Sum).
Methods
- The paper defines n-RASP-L problems: tasks solvable by T(n) sequential applications of a RASP-L operation P′, with pre- and post-processing, and gives n-RASP-L solutions for Parity, Copy, and Addition.
- It trains looped decoder-only GPT-2 models with NoPE, input injection, and step-dependent supervision on final answers only, without intermediate-step supervision; training data include (x, y) pairs and the pre-defined number of steps T(n).
- Experiments cover Parity, Copy, Binary Addition, Binary Sum, Binary Multiplication, and Unique Set, with curriculum learning, AdamW, batch size 64, 100k gradient steps, evaluation on 6400 random samples, and mean exact match accuracy from five seeds; baselines include vanilla NTP, NTP with pause tokens, and NTP with weight-tied layers.
Limitations
- The paper states that n-RASP-L does not support tasks requiring multiple loops followed by each other, and extending to multiple loops is future work.
- Direct looped training can be computationally demanding when the number of looped steps is large; stopping gradient tracking for earlier steps may help but could trade off performance.
- The authors state they train for a limited number of steps and lengths due to limited computing resources, use NoPE for simplicity, and require the ground-truth number of steps in training data, an additional requirement compared with normal end-to-end training.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
20 Own interpretation · 16 Own result · 12 Sources not found · 9 No basis given · 5 Partly supported · 4 Not checked yet · 1 Supported · 1 Own method · accuracy not yet measured
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“However, even the largest and latest LLMs like Achiam et al. (2023) trained on much of the existing text on the Internet, still struggle with length generalization (Wu et al., 2023;Anil et al., 2022;Lee et al., 2024).”
§ LOOPED TRANSFORMERS FOR LENGTH GENERALIZA-TION > 1 INTRODUCTION
What the sources say
- Not matchedcites Wu et al., 2023
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Partly supportedcites Anil et al., 2022 sourceExperimental — accuracy not yet measured
The Conclusion (p. 9) says length generalization remains challenging for language models up to the 100B-parameter scale, and Section 3.1 (p. 4) reports weak extrapolation in the models tested. The paper does not establish the claim for the latest models cited in the sentence or for models trained on much of the Internet’s text; it also reports substantial improvements with few-shot scratchpad prompting. (LLM, conf=0.86)
- Not matchedcites Lee et al., 2024
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In this work, we demonstrate that looped Transformers with an adaptive number of steps significantly improve length generalization.”
- Also stated in: § 1 INTRODUCTION · plain
“Empirically, we show that looped Transformers with an adaptive number of steps can successfully length-generalize to longer lengths simply by appropriately adapting the number of loops at inference time, indicating that our approach encourages the model to implicitly learn the necessary steps to solve a task.”
- Also stated in: § 1 INTRODUCTION · boosted
“(3) Empirically, we show that our proposed method outperforms the baseline approaches in terms of length generalization performance (Section 6).”
§ LOOPED TRANSFORMERS FOR LENGTH GENERALIZA-TION > Abstract
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 1 INTRODUCTION · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We train looped Transformers using our proposed learning algorithm and observe that they learn highly length-generalizable solutions for various tasks.”
§ LOOPED TRANSFORMERS FOR LENGTH GENERALIZA-TION > Abstract
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Compared with vanilla NTP, we observe that NTP-Loop could lead to improved generalization in tasks like Addition, Copy and Multiplication.”
§ LOOPED TRANSFORMERS FOR LENGTH GENERALIZA-TION > 6.2 LENGTH GENERALIZATION RESULTS
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“(3) Empirically, we show that our proposed method outperforms the baseline approaches in terms of length generalization performance (Section 6).”
§ LOOPED TRANSFORMERS FOR LENGTH GENERALIZA-TION > 1 INTRODUCTION
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“we find input injection useful to further improve the performance (see details in Section 6.3).”
§ LOOPED TRANSFORMERS FOR LENGTH GENERALIZA-TION > 5 RELATED WORK
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“we also find 2) to be an effective stopping criterion in Section 6.4.”
§ LOOPED TRANSFORMERS FOR LENGTH GENERALIZA-TION > 6.1.3 TRAINING AND EVALUATION SETUP
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Proposition 3.2.(Parity.)There exists a n-RASP-L program with T (n) = n that solves the n-bit parity check task:”
§ LOOPED TRANSFORMERS FOR LENGTH GENERALIZA-TION > 3 n-RASP-L
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For example, for Parity, it can generalize to more than 40 digits near perfectly when trained with up to 20 digits.”
§ LOOPED TRANSFORMERS FOR LENGTH GENERALIZA-TION > 6.2 LENGTH GENERALIZATION RESULTS
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Looped Transformers help with length generalization.Our looped training significantly improves the length generalization performance.”
- Also stated in: § F EXPERIMENTAL DETAILS · plain
“Our looped Transformer model with adaptive depth generalized better than NTP methods across studied tasks, including the variants with pause tokens and weight-tied layers.”
§ LOOPED TRANSFORMERS FOR LENGTH GENERALIZA-TION > 6.2 LENGTH GENERALIZATION RESULTS
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § F EXPERIMENTAL DETAILS · plain
- It may state two separate claims as one. 18
- Probably not a substantial claim. 27
- The quote only partly states this claim. 1
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“It has been shown that copy with unique tokens could be easily solved by inductive head (Olsson et al., 2022), but copy with repeated tokens (e.g., binary) does not length-generalize with vanilla NTP training (Zhou et al., 2024a).”
§ LOOPED TRANSFORMERS FOR LENGTH GENERALIZA-TION > Copy (with repeated tokens).
What the sources say
- Supportedcites Olsson et al., 2022 source
The Olsson et al. citation supports the first clause of the citing sentence. In “Induction Heads” (p. 5), the paper defines induction heads by their ability to use an earlier [A][B] pair to increase the likelihood of [B] when [A] recurs in a random-token sequence. The repeated-token length-generalization claim is cited to Zhou et al., not to this source, and is not evaluated here. (LLM, conf=0.86)
- Not matchedcites Zhou et al., 2024a
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Positional embeddings have been shown to greatly affect Transformers' ability to generalize to longer lengths (Press et al., 2021;Kazemnejad et al., 2024;Ruoss et al., 2023;Su et al., 2024;Li et al., 2023;Cho et al., 2024;Sabbaghi et al., 2024;McLeish et al., 2024;Golovneva et al., 2024).”
§ LOOPED TRANSFORMERS FOR LENGTH GENERALIZA-TION > 5 RELATED WORK
What the sources say
- Supportedcites Press et al., 2021 source
The Introduction (p. 1) says weak length extrapolation with sinusoidal position embeddings is caused by the position embedding method. The Conclusion (p. 9) says changing the position method enables Transformers to extrapolate to longer inputs, supporting the citing sentence’s broad claim. (LLM, conf=0.94)
- Not matchedcites Kazemnejad et al., 2024
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Supportedcites Ruoss et al., 2023 source
The paper’s “This work” section reports that changing positional encodings significantly improved Transformers’ length generalization on algorithmic reasoning tasks. Its Conclusion likewise says the proposed encodings significantly outperformed prior approaches, supporting the cited statement that positional embeddings affect this ability. (LLM, conf=0.92)
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“To learn a model that can effectively generalize to longer problems, it is important to consider architectures that can adaptively adjust the computational budget to the difficulty of the tasks (Anil et al., 2022;Du et al., 2022;2024).”
§ LOOPED TRANSFORMERS FOR LENGTH GENERALIZA-TION > 1 INTRODUCTION
What the sources say
- Partly supportedcites Anil et al., 2022 sourceExperimental — accuracy not yet measured
Section 2 (p. 3) says longer problem instances require more compute, and Section 5.1 (p. 7) describes scratchpad solutions with a step for each operation. But the paper does not recommend architectures that adaptively adjust computational budgets; its Conclusion (p. 9) emphasizes a length-generalization approach that does not require changing model architecture. (LLM, conf=0.82)
- Not matchedcites Du et al., 2022;2024
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“One approach to achieve this is to explicitly generate intermediate output tokens, similar to writing down a scratchpad, which improves LLMs' capability for solving harder problems (Nye et al., 2021).”
§ LOOPED TRANSFORMERS FOR LENGTH GENERALIZA-TION > 1 INTRODUCTION
What the sources say
- Supportedcites Nye et al., 2021 source
The Introduction describes generating intermediate tokens as a scratchpad before the final answer. The Conclusion reports that this approach improved performance on challenging algorithmic tasks, including long addition, polynomial evaluation, and Python code execution. (LLM, conf=0.92)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“As discussed in Goyal et al. (2023), pause tokens mainly introduce parallel but not sequential compute, which is less powerful than adaptive depth.”
§ LOOPED TRANSFORMERS FOR LENGTH GENERALIZA-TION > 6.3 ABLATION STUDIES
What the sources say
- Partly supportedcites Goyal et al. (2023) sourceExperimental — accuracy not yet measured
Section I.2 says pause tokens are processed in parallel and do not add sequential computation. The discussion of adaptive compute says some methods increase serial computation, but it does not establish that pause tokens are generally less powerful than adaptive depth; the paper also identifies tasks where parallel computation is advantageous. (LLM, conf=0.86)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Partly supported
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Similarly, NTP-pause could introduce slight improvement in Binary Sum and Unique Set.”
§ LOOPED TRANSFORMERS FOR LENGTH GENERALIZA-TION > 6.2 LENGTH GENERALIZATION RESULTS
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, they all fall behind compared with our method.”
- Also stated in: § 6.2 LENGTH GENERALIZATION RESULTS · hedged
“Variants of NTP could improve generalization but not as effectively as our adaptive-depth model.”
§ LOOPED TRANSFORMERS FOR LENGTH GENERALIZA-TION > 6.2 LENGTH GENERALIZATION RESULTS
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § 6.2 LENGTH GENERALIZATION RESULTS · hedged
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The convergence nature of the tasks also affects the performance of the per-sample-based stopping criterion (B = 1) in Figure 6 (Appendix B): the per-sample stopping criterion performs almost as well as when using B = N test to decide for converging tasks, but the performance is worse in non-converging tasks.”
- Also stated in: § B RESULTS FROM THE ADAPTIVE STOPPING CRITERIA · plain
“For tasks with converging behaviors (Addition, Copy, Multiplication, Unique Set), per sampler stopping criterion works almost as well as using B = N test , while for tasks without the converging behavior (Parity and Binary Sum), per sampler stopping criterion does not work as well as using B = N test .”
§ LOOPED TRANSFORMERS FOR LENGTH GENERALIZA-TION > 6.4 THE STOPPING CRITERION AND VISUALIZATIONS
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § B RESULTS FROM THE ADAPTIVE STOPPING CRITERIA · plain
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The loss curves for these tasks are also smoother than those without convergence behaviors.”
§ LOOPED TRANSFORMERS FOR LENGTH GENERALIZA-TION > 6.4 THE STOPPING CRITERION AND VISUALIZATIONS
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Additionally, non-converging tasks are less tolerant when choosing which step to stop, and we find using the exponential moving average (EMA) of model parameters helpful for Parity and Binary Sum with a factor 0.9999.”
§ Input example: 1 1 0 1 > # # # # Output example: * * * * 1 # # # # * indicates ignored token, > is EOQ, and # is EOS.def parity_step(partial_ans_seq, seq): # align the last digit with the answer location seq = shift_right(seq, 1) # calculate XOR partial_ans_seq = ( partial_ans_seq | seq) & (~(partial_ans_seq & seq) ) return partial_ans_seq, seq def parity_loop(seq, num_step): # get the question in the prompt prompt_mask = 1 -has_seen(seq, full(seq, EOQ)) seq = mask(seq, prompt_mask) # init answer seq with 0 partial_ans_seq = full(seq, 0) # generate EOS seq after EOQ end_seq = where(prompt_mask==1, full(seq, 0), full(seq, EOS)) # perform parity steps for i in range(num_step): partial_ans_seq, seq = parity_step(partial_ans_seq, seq) # get answer with EOS ans_seq = partial_ans_seq end_seq = shift_right(end_seq, 1) ans_seq = where(end_seq==EOS, end_seq, ans_seq) return ans_seq Listing 1: Parity.# Input example: 0 1 0 1 1 > # # # # # # # Output example: * * * * * 0 1 0 1 1 # # # * indicates ignored token, > is EOQ, and # is EOS.def copy_step(seq, end_seq): seq = shift_right(seq, 1) end_seq = shift_right(end_seq, 1) return seq, end_seq def copy_loop(seq, num_step): # generate EOS seq after EOQ end_mask = has_seen(seq, full(seq, EOQ)) end_seq = where(end_mask==0, full(seq, 0), full(seq, EOS)) # perform copy steps for i in range(num_step): seq, end_seq = copy_step(seq, end_seq) # get answer with EOS seq = where(end_seq==EOS, end_seq, seq) return seq Listing 2: Copy.# Input example: 0 0 1 + 1 1 1 > # # # # # # # Output example: * * * * * * * 1 0 0 0 # # # # * indicates ignored token, > is EOQ, and # is EOS.def addition_step(seq1, seq2, end_seq): end_seq = shift_right(end_seq, 1) seq1 = np.array(seq1,dtype=bool) seq2 = np.array(seq2,dtype=bool) carry_on = seq1 & seq2 # A XOR B = (A OR B) AND (NOT (A AND B)) in_place = ((seq1 | seq2) & (~(seq1 & seq2))) in_place = shift_right(in_place, 1) seq1 = np.array(in_place,dtype=int) seq2 = np.array(carry_on,dtype=int) return seq1, seq2, end_seq def addition_preprocess(seq): # generate EOS seq after EOQ end_mask = has_seen(seq, full(seq, EOQ)) end_seq = where(end_mask==0, full(seq, 0), full(seq, EOS)) # generate masks for the first and second summands seen_tok0 = has_seen(seq, full(seq, ADD_SIGN)) seen_tok1 = has_seen(seq, full(seq, EOQ)) mask1 = ~seen_tok0 mask2 = seen_tok0 & (~seen_tok1) mask2 = mask2 & shift_right(mask2, 1) # get the first and second summands seq1 = mask(seq, mask1) seq2 = mask(seq, mask2) # align the first summand with the second induct_num1 = cumsum(mask1) induct_num2 = cumsum(mask2) target_index = firsts(induct_num1, induct_num2, default=0) seq1 = index_select(seq1, target_index) seq1 = mask(seq1, mask2) return seq1, seq2, end_seq def addition_loop(seq, num_step): seq1, seq2, end_seq = addition_preprocess(seq) # perform addition steps for i in range(num_step): seq1, seq2, end_seq = addition_step(seq1, seq2, end_seq) # get answer with EOS ans = seq1 ans = where(end_seq==EOS, end_seq, ans) return ans Listing 3: Addition (in forward order).import numpy as np def full(x, const): return np.full_like(x, const, dtype=int) def indices(x): return np.arange(len(x), dtype=int) def select(k, q, pred, causal=True): # compute attention matrix s = len(k) A = np.zeros((s,s), dtype=bool) for qi in range(s): for kj in (range(qi+1) if causal else range(s)): # k_index <= q_index if causal A[qi, kj] = pred(k[kj], q[qi]) return A def sel_width(A): return np.dot(A, np.ones(len(A))).astype(int)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Additionally, non-converging tasks are less tolerant when choosing which step to stop, and we find using the exponential moving average (EMA) of model parameters helpful for Parity and Binary Sum with a factor 0.9999.”
§ Input example: 1 1 0 1 > # # # # Output example: * * * * 1 # # # # * indicates ignored token, > is EOQ, and # is EOS.def parity_step(partial_ans_seq, seq): # align the last digit with the answer location seq = shift_right(seq, 1) # calculate XOR partial_ans_seq = ( partial_ans_seq | seq) & (~(partial_ans_seq & seq) ) return partial_ans_seq, seq def parity_loop(seq, num_step): # get the question in the prompt prompt_mask = 1 -has_seen(seq, full(seq, EOQ)) seq = mask(seq, prompt_mask) # init answer seq with 0 partial_ans_seq = full(seq, 0) # generate EOS seq after EOQ end_seq = where(prompt_mask==1, full(seq, 0), full(seq, EOS)) # perform parity steps for i in range(num_step): partial_ans_seq, seq = parity_step(partial_ans_seq, seq) # get answer with EOS ans_seq = partial_ans_seq end_seq = shift_right(end_seq, 1) ans_seq = where(end_seq==EOS, end_seq, ans_seq) return ans_seq Listing 1: Parity.# Input example: 0 1 0 1 1 > # # # # # # # Output example: * * * * * 0 1 0 1 1 # # # * indicates ignored token, > is EOQ, and # is EOS.def copy_step(seq, end_seq): seq = shift_right(seq, 1) end_seq = shift_right(end_seq, 1) return seq, end_seq def copy_loop(seq, num_step): # generate EOS seq after EOQ end_mask = has_seen(seq, full(seq, EOQ)) end_seq = where(end_mask==0, full(seq, 0), full(seq, EOS)) # perform copy steps for i in range(num_step): seq, end_seq = copy_step(seq, end_seq) # get answer with EOS seq = where(end_seq==EOS, end_seq, seq) return seq Listing 2: Copy.# Input example: 0 0 1 + 1 1 1 > # # # # # # # Output example: * * * * * * * 1 0 0 0 # # # # * indicates ignored token, > is EOQ, and # is EOS.def addition_step(seq1, seq2, end_seq): end_seq = shift_right(end_seq, 1) seq1 = np.array(seq1,dtype=bool) seq2 = np.array(seq2,dtype=bool) carry_on = seq1 & seq2 # A XOR B = (A OR B) AND (NOT (A AND B)) in_place = ((seq1 | seq2) & (~(seq1 & seq2))) in_place = shift_right(in_place, 1) seq1 = np.array(in_place,dtype=int) seq2 = np.array(carry_on,dtype=int) return seq1, seq2, end_seq def addition_preprocess(seq): # generate EOS seq after EOQ end_mask = has_seen(seq, full(seq, EOQ)) end_seq = where(end_mask==0, full(seq, 0), full(seq, EOS)) # generate masks for the first and second summands seen_tok0 = has_seen(seq, full(seq, ADD_SIGN)) seen_tok1 = has_seen(seq, full(seq, EOQ)) mask1 = ~seen_tok0 mask2 = seen_tok0 & (~seen_tok1) mask2 = mask2 & shift_right(mask2, 1) # get the first and second summands seq1 = mask(seq, mask1) seq2 = mask(seq, mask2) # align the first summand with the second induct_num1 = cumsum(mask1) induct_num2 = cumsum(mask2) target_index = firsts(induct_num1, induct_num2, default=0) seq1 = index_select(seq1, target_index) seq1 = mask(seq1, mask2) return seq1, seq2, end_seq def addition_loop(seq, num_step): seq1, seq2, end_seq = addition_preprocess(seq) # perform addition steps for i in range(num_step): seq1, seq2, end_seq = addition_step(seq1, seq2, end_seq) # get answer with EOS ans = seq1 ans = where(end_seq==EOS, end_seq, ans) return ans Listing 3: Addition (in forward order).import numpy as np def full(x, const): return np.full_like(x, const, dtype=int) def indices(x): return np.arange(len(x), dtype=int) def select(k, q, pred, causal=True): # compute attention matrix s = len(k) A = np.zeros((s,s), dtype=bool) for qi in range(s): for kj in (range(qi+1) if causal else range(s)): # k_index <= q_index if causal A[qi, kj] = pred(k[kj], q[qi]) return A def sel_width(A): return np.dot(A, np.ones(len(A))).astype(int)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim have been checked yet.
In the paper
“Training with pause tokens in FOP could sometimes boost the generalization performance compared to vanilla FOP, but not as effective as our method with looped steps and adaptive depth.”
§ LOOPED TRANSFORMERS FOR LENGTH GENERALIZA-TION > 6.3 ABLATION STUDIES
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“Their results show that RNNs and Transformers fail to generalize on non-regular tasks, LSTMs can solve regular and counter-language tasks, and only networks augmented with structured memory (such as a stack or memory tape) can successfully generalize on some context-free and context-sensitive tasks.”
§ LOOPED TRANSFORMERS FOR LENGTH GENERALIZA-TION > 5 RELATED WORK
What the sources say
- Not matchedcites Delétang et al. (2022)
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Generally, it is hard to expect models to be trained on inputs with all possible lengths, and we need to rely on length generalization.”
§ LOOPED TRANSFORMERS FOR LENGTH GENERALIZA-TION > 1 INTRODUCTION
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On the other hand, architectures such as the Looped Transformer have external loops embedded in the architecture which naturally provides adaptive depth.”
§ LOOPED TRANSFORMERS FOR LENGTH GENERALIZA-TION > 3 n-RASP-L
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
In the paper
“Their results show that RNNs and Transformers fail to generalize on non-regular tasks, LSTMs can solve regular and counter-language tasks, and only networks augmented with structured memory (such as a stack or memory tape) can successfully generalize on some context-free and context-sensitive tasks.”
§ LOOPED TRANSFORMERS FOR LENGTH GENERALIZA-TION > 5 RELATED WORK
What the sources say
- Not matchedcites Delétang et al. (2022)
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As a result, we expect the model to learn length-generalizable intermediate steps that could potentially generalize to more steps at test time.”
§ LOOPED TRANSFORMERS FOR LENGTH GENERALIZA-TION > 4.2.1 ARCHITECTURE OF THE LOOPED TRANSFORMERS
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one. The quote only partly states this claim.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“While Transformers can accept a variable length of inputs (that can be processed in parallel), they usually have a fixed depth.”
§ LOOPED TRANSFORMERS FOR LENGTH GENERALIZA-TION > 1 INTRODUCTION
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“Notice that these approaches still use fixed-depth models.While these approaches help solve more complex reasoning tasks, they are still far from achieving near-perfect length generalization for simple algorithmic tasks.”
§ LOOPED TRANSFORMERS FOR LENGTH GENERALIZA-TION > 1 INTRODUCTION
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The number of steps needed in the loop depends on the problem's complexity, i.e., more difficult problems could potentially require more steps to solve.”
§ LOOPED TRANSFORMERS FOR LENGTH GENERALIZA-TION > 1 INTRODUCTION
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Also, if a model can length-generalize, it means the model has truly learned the correct algorithmic solution to the task, not just a spurious solution that works only for certain lengths.”
§ LOOPED TRANSFORMERS FOR LENGTH GENERALIZA-TION > 1 INTRODUCTION
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
In the paper
“For instance, Lee et al. applied CoT for arithmetic tasks but observed that Transformers cannot length generalize even for simple addition tasks (Lee et al., 2024).”
§ LOOPED TRANSFORMERS FOR LENGTH GENERALIZA-TION > 1 INTRODUCTION
What the sources say
- Not matchedcites Lee et al., 2024
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, such input representations are highly dependent on the specific problems and might not necessarily exist in general.”
§ LOOPED TRANSFORMERS FOR LENGTH GENERALIZA-TION > Input representation.
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
In the paper
“In Zhou et al. (2024a), they show algorithmic tasks that can be written as a RASP-L program can be easily learned by a Transformer in a length-generalizable way with next-token prediction.”
§ LOOPED TRANSFORMERS FOR LENGTH GENERALIZA-TION > 2.1 RASP-L
What the sources say
- Not matchedcites Zhou et al. (2024a)
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This might be sufficient for certain tasks, but not always.”
§ LOOPED TRANSFORMERS FOR LENGTH GENERALIZA-TION > 1 INTRODUCTION
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds. It may state two separate claims as one.
In the paper
“Their results show that RNNs and Transformers fail to generalize on non-regular tasks, LSTMs can solve regular and counter-language tasks, and only networks augmented with structured memory (such as a stack or memory tape) can successfully generalize on some context-free and context-sensitive tasks.”
§ LOOPED TRANSFORMERS FOR LENGTH GENERALIZA-TION > 5 RELATED WORK
What the sources say
- Not matchedcites Delétang et al. (2022)
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Not matched
None of the sources cited for this claim have been checked yet.
In the paper
“With input injection, the model can maintain a strong connection to the original input, preventing information loss with improved performance (Bai et al., 2019;Yang et al., 2024).”
§ LOOPED TRANSFORMERS FOR LENGTH GENERALIZA-TION > 4.2.1 ARCHITECTURE OF THE LOOPED TRANSFORMERS
What the sources say
- Not checked — no full text of the sourcecites Bai et al., 2019 source
- Not matchedcites Yang et al., 2024
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In our paper, the Looped Transformer architecture also has augmented memory and the recurrent structure but is potentially more powerful since each iteration contains an operation of the whole sequence.”
§ LOOPED TRANSFORMERS FOR LENGTH GENERALIZA-TION > 5 RELATED WORK
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
In the paper
“However, they also showed that for algorithmic tasks whose RASP-L program representation is not known to exist, such as addition, parity, and copying the input sequence, it is hard to learn in a length-generalizable way.”
§ LOOPED TRANSFORMERS FOR LENGTH GENERALIZA-TION > 2.1 RASP-L
What the sources say
- Not matchedcites Zhou et al. (2024a)
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Besides, CoT is normally constrained to fixed-depth models, while we study a more general and powerful way to use adaptive compute at inference time.”
§ LOOPED TRANSFORMERS FOR LENGTH GENERALIZA-TION > Input representation.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The length of an input often correlates with the difficulty of the problem instance.For example, the longer the input, the more difficult the problem tends to be.”
§ LOOPED TRANSFORMERS FOR LENGTH GENERALIZA-TION > 1 INTRODUCTION
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For n-RASP-L problems, if we can learn these lengthindependent steps, we can utilize an adaptive number of steps to achieve length generalization.”
§ LOOPED TRANSFORMERS FOR LENGTH GENERALIZA-TION > 1 INTRODUCTION
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
It may state two separate claims as one.
In the paper
“Second, not all CoT steps are helpful.If CoT steps introduce additional complexity or require operations not easily expressible in RASP-L, then CoT may hinder length generalization, as shown in Zhou et al. (2024a).”
§ LOOPED TRANSFORMERS FOR LENGTH GENERALIZA-TION > Input representation.
What the sources say
- Not matchedcites Zhou et al. (2024a)
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For the first stopping criterion where we know a(n ′ ), our inference time would be O(ka(n ′ )n ′2 ), and NTP (with KV cache) and FOP will be O(Cn ′2 ).”
§ LOOPED TRANSFORMERS FOR LENGTH GENERALIZA-TION > D VISUALIZATION OF USING PAUSE TOKENS IN NTP AND FOP
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- The authors state this as an assumption: The maximum length of the training set is n, the number of steps needed is T(n), and the number of layers in each step is k.
- The authors state this as an assumption: NTP and FOP use a fixed number of layers C.
- The authors state this as an assumption: The test length is n'.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“We only train the looped Transformers for a limited number of steps and lengths due to a lack of computing resources.”
§ LOOPED TRANSFORMERS FOR LENGTH GENERALIZA-TION > 7 LIMITATIONS AND CONCLUSION
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In NTP and FOP, we use some C ≈ kT (n) in our experiments such that they use similar compute during training.Our inference time is then slightly longer than NTP with KV cache and FOP since we use more steps than the fixed-depth models.”
§ LOOPED TRANSFORMERS FOR LENGTH GENERALIZA-TION > D VISUALIZATION OF USING PAUSE TOKENS IN NTP AND FOP
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Moreover, our step-dependent supervision requires the ground-truth number of steps in the training data, which is an additional requirement compared with normal end-to-end training.”
§ LOOPED TRANSFORMERS FOR LENGTH GENERALIZA-TION > 7 LIMITATIONS AND CONCLUSION
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For the second criterion, we need to specify the maximum number of steps in order to find the step with maximum confidence.So our inference time would be O(kN ′ n ′2 ), where N ′ is the maximum number of steps.”
§ LOOPED TRANSFORMERS FOR LENGTH GENERALIZA-TION > D VISUALIZATION OF USING PAUSE TOKENS IN NTP AND FOP
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- The authors state this as an assumption: The maximum length of the training set is n, the number of steps needed is T(n), and the number of layers in each step is k.
- The authors state this as an assumption: NTP and FOP use a fixed number of layers C.
- The authors state this as an assumption: The test length is n'.
The paper relies on this without support.
Intermediate supervision and CoT training data can be difficult to collect.
“However, such intermediate supervision can be difficult to get, just like collecting helping CoT steps could be difficult.”
“First, CoT training data could be hard to collect.”
The paper relies on this without support.
CoT steps that could convert the next token prediction task to RASP-L programs might not always exist.
“Moreover, CoT steps that could convert the next token prediction task to RASP-L programs might not always exist.”
The authors state this as an assumption.
The maximum length of the training set is n, the number of steps needed is T(n), and the number of layers in each step is k.
“Assume that the maximum length of the training set is n, the number of steps needed is T (n), and the number of layers in each step is k.”
The authors state this as an assumption.
NTP and FOP use a fixed number of layers C.
“Assume that NTP and FOP are using a fixed number of layers C.”
The authors state this as an assumption.
The test length is n'.
“And we test on length n ′ .”
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ LOOPED TRANSFORMERS FOR LENGTH GENERALIZA-TION > Binary Multiplication.
“The output is in reversed order and the length is the sum of the lengths of two numbers, without removing leading 0.”
§ LOOPED TRANSFORMERS FOR LENGTH GENERALIZA-TION > Binary Multiplication.
“1 1 × 1 1 0 > # # # # # , example output: * * * * * * 0 1 0 0 1 0 # .”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ LOOPED TRANSFORMERS FOR LENGTH GENERALIZA-TION > Binary Multiplication.
“1 1 × 1 1 0 > # # # # # , example output: * * * * * * 0 1 0 0 1 0 # .”
§ LOOPED TRANSFORMERS FOR LENGTH GENERALIZA-TION > 2.2 NEXT-TOKEN PREDICTION AND FULL-OUTPUT PREDICTION
“During both training and inference time, the input given is just the query part, and the rest of the locations are filled with multiple EOS tokens to keep the input and the output to be the same length.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ LOOPED TRANSFORMERS FOR LENGTH GENERALIZA-TION > Binary Addition.
“Example input: 1 0 + 1 1 > # # # , example output: * * * * * 1 0 1 # # .”
§ LOOPED TRANSFORMERS FOR LENGTH GENERALIZA-TION > 2.2 NEXT-TOKEN PREDICTION AND FULL-OUTPUT PREDICTION
“During both training and inference time, the input given is just the query part, and the rest of the locations are filled with multiple EOS tokens to keep the input and the output to be the same length.”
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 4
- Yilun Du, Shuang Li, Joshua Tenenbaum, and Igor Mordatch. Learning iterative reasoning through energy minimization. In International Conference on Machine Learning, pp. 5570-5582. PMLR, 2022.
- Yilun Du, Jiayuan Mao, and Joshua B Tenenbaum. Learning iterative reasoning through energy diffusion. In Forty-first International Conference on Machine Learning, 2024.
- Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019.
- Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Cited by
Sorted by citations
Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computation
Bae, S., Kim, Y., Bayat, R., et al. (2025) · Neural Information Processing Systems · doi:10.48550/arxiv.2507.10524 · Cited by 168
“Looped Transformers shown to be effective, can act as programmable computers [33], learn iterative data-fitting algorithms [100], generalize to much longer inputs on algorithmic tasks [23], and illuminate few-shot learning by mimicking multi-step optimizers [27].”
Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRAPreprint
Bae, S., Fisch, A., Harutyunyan, H., Ji, Z., Kim, S., & Schuster, T. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2410.20672 · Cited by 41
“The examples utilize models with six layers, where identical colors represent shared weights.”
LoopFormer: Elastic-Depth Looped Transformers for Latent Reasoning via Shortcut Modulation
Jeddi, A., Ciccone, M., & Taati, B. (2026) · arXiv.org · doi:10.48550/arxiv.2602.11451 · Cited by 77
“Once trained, their representations collapse when evaluated at shorter or longer depths, since those settings are off-distribution Bae et al. (2024); Fan et al. (2024).”
Teaching Pretrained Language Models to Think Deeper with Retrofitted RecurrencePreprint
McLeish, S., Li, A., Kirchenbauer, J., Kalra, D. S., Bartoldson, B. R., Kailkhura, B., Schwarzschild, A., Geiping, J., Goldstein, T., & Goldblum, M. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2511.07384 · Cited by 55
“Recurrent mechanisms are shown to learn generalizable solutions to problems using ResNet (He et al., 2015) based architectures (Schwarzschild et al., 2021; Bansal et al., 2022; Anil et al., 2022; Schwarzschild, 2023; Bear et al., 2024).”
Loop, Think, & Generalize: Implicit Reasoning in Recurrent-Depth TransformersPreprint
Kohli, H., Parthasarathy, S., Sun, H., & Yao, Y. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2604.07822 · Cited by 30
“Fan et al. (2025) show that such looped transformers offer superior length generalization on algorithmic tasks such as parity and binary addition.”
Xu, K., & Sato, I. (2024) · International Conference on Machine Learning · doi:10.48550/arxiv.2410.01405 · Cited by 40
“In other words, TF = (id + FF ) ◦ (id + Attn) , where id denotes the identity mapping, where we omit the domain of definition for simplicity.”
Fixed-Point Reasoners: Stable and Adaptive Deep Looped TransformersPreprint
Movahedi, S., Milovanović, V., Feigin, S. L., Theus, A., Hofmann, T., Boeva, V., Rusch, T. K., & Orvieto, A. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2606.18206 · Cited by 31
“…show that Looped Transformers can improve length generalization, learn algorithmic structure, and approximate much deeper untied models on reasoning tasks (Yang et al., 2024; Fan et al., 2025; Saunshi et al., 2025; Kapl et al., 2026; Geiping et al., 2026; Giannou et al., 2023; Kohli et al., 2026).”
Self-Improving Transformers Overcome Easy-to-Hard and Length Generalization ChallengesPreprint
Lee, N., Cai, Z., Schwarzschild, A., Lee, K., & Papailiopoulos, D. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2502.01612 · Cited by 17
“Other methods focus on architectural changes, such as introducing looping mechanisms (Fan et al., 2024) or incorporating hand-crafted bias corrections in attention score matrices (Duan et al., 2023).”
Self-citationCoevolutionary Continuous Discrete Diffusion: Make Your Diffusion Language Model a Latent ReasonerPreprint
Zhou, C., Yang, C., Hu, Y., Wang, C., Zhang, C., Zhang, M., Mackey, L. W., Jaakkola, T., Bates, S., & Zhang, D. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2510.03206 · Cited by 25
“…universal transformers (Dehghani et al., 2018a) theoretically demonstrates that when a block of middle layers of a fixed transformer model is repeated for a variable number of times, its expressivity can be significantly improved (Saunshi et al., 2025; Merrill & Sabharwal, 2025; Fan et al., 2024).”
Parallel Loop Transformer for Efficient Test-Time Computation ScalingPreprint
Wu, B., Chen, M., Luo, X., Yan, S., Yu, Q., Xia, F., Zhang, T., Zhan, H., Zhong, Z., Zhou, X., et al. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2510.24824 · Cited by 26
“Meanwhile, Fan et al. [13], Yang et al. [43] demonstrated its parameter efficiency in data-fitting settings, and superior length generalization on RASP-L tasks.”
Encode, Think, Decode: Scaling test-time reasoning with recursive latent thoughts
Koishekenov, Y., Lipani, A., & Cancedda, N. (2025) · arXiv.org · doi:10.48550/arxiv.2510.07358 · Cited by 31
“Other works offered theoretical and small-scale analyses of looped transformers (Giannou et al., 2023; Gatmiry et al., 2024; Yang et al., 2023; Fan et al., 2024).”
Stabilizing Recurrent Dynamics for Test-Time Scalable Latent Reasoning in Looped Language ModelsPreprint
Yang, X.-W., Han, Z., Zhang, X.-H., Wei, W.-D., Shao, J.-J., Guo, L.-Z., & Li, Y.-F. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2605.26733 · Cited by 17
“Subsequent research (Giannou et al., 2023; Yang et al., 2023; Fan et al., 2024) have investigated their potential benefits through theoretical and small-scale empirical analyses.”
Enhancing Auto-regressive Chain-of-Thought through Loop-Aligned Reasoning
Yu, Q., He, Z., Li, S., et al. (2025) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.48550/arxiv.2502.08482 · Cited by 22
“Several preliminary results (Fan et al., 2024) show that the looped Transformer model has better length generalization capabilities, partially because the increase in problem complexity (e.g., problem length) is not as significant as in the Chain-of-Thought steps.”
Skip a Layer or Loop it? Test-Time Depth Adaptation of Pretrained LLMs
Li, Z.-Y., Li, Y., & Zhou, T. (2025) · arXiv.org · doi:10.48550/arxiv.2507.07996 · Cited by 20
“[9] demonstrate that a Looped Transformer – a single Transformer block applied repeatedly – can achieve much better length generalization on algorithmic tasks by adjusting the number of loops during inference.”
Solve the Loop: Attractor Models for Language and Reasoning
Fein-Ashley, J., & Rashidinejad, P. (2026) · arXiv.org · doi:10.48550/arxiv.2605.12466 · Cited by 19
“…or algorithmic procedures (Yang et al., 2024b; Giannou et al., 2023a), emulate additional depth through weight sharing (Dehghani et al., 2019; Zhu et al., 2025c), reduce the context-length costs of token-level reasoning, and improve downstream generalization (Labovich, 2026b; Fan et al., 2025).”
A Formal Comparison Between Chain-of-Thought and Latent Thought
Xu, K., & Sato, I. (2025) · arXiv.org · doi:10.48550/arxiv.2509.25239 · Cited by 18
“January 30, 2026.”
Training Large Language Models to Reason in a Continuous Latent Space
Hao, S., Sukhbaatar, S., Su, D., Li, X., Hu, Z., Weston, J., & Tian, Y. (2024) · arXiv · doi:10.48550/arxiv.2412.06769 · Cited by 18
“Recently, looped transformers (Giannou et al., 2023; Fan et al., 2024) have been proposed to solve algorithmic tasks, which have some similarities to the computing process of continuous thoughts, but we focus on common reasoning tasks and aim at investigating latent reasoning in comparison to…”
Training-Free Looped Transformers
Chen, L.-Z., Li, J., Liang, C., Lao, N., & Liu, Q. (2026) · arXiv.org · doi:10.48550/arxiv.2605.23872 · Cited by 21
“Naively looping g for K rounds, i.e. x ← g ( x ) applied K times [27, 47, 35, 28, 77, 95, 33, 102, 88, 1, 58, 32, 12], is a K -step forward Euler integration of the window residual field with step size h =1 , which approximates x ( t = K ) .”
InfluentialMemory-Efficient Looped Transformer: Decoupling Compute from Memory in Looped Language ModelsPreprint
Vendrell, V. C., Masdemont, A. P., Grillo, N., Ros-Giralt, J., Behboodi, A., & Massoli, F. V. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2605.07721 · Cited by 16
“Existing studies on simplified setups have shown that, compared to similar-sized vanilla transformers, looped transformers show superior capacity at multi-hop reasoning [8, 9], lenght generalization [10, 9] and learning algorithms [11].”
LoopViT: Scaling Visual ARC with Looped Transformers
Shu, W., Qiu, X.-R., Zhu, R., et al. (2026) · arXiv.org · doi:10.48550/arxiv.2602.02156 · Cited by 16
“Recent work on “thinking mod-els” [1, 26, 45, 31, 43] further suggests that recurrence supports a latent Chain-of-Thought, enabling models to adapt their computation based on task complexity.”
Two-Scale Latent Dynamics for Recurrent-Depth TransformersPreprint
Pappone, F., Crisostomi, D., & Rodolà, E. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2509.23314 · Cited by 12
“Universal Transformers share parameters across depth and learn to halt (often via ACT) [2]; depth-adaptive and early-exit variants similarly modulate per-token compute by stopping when intermediate predictions stabilize (e.g., entropy/confidence heuristics) [3, 11].”
Bridging the Gap Between Latent and Explicit Reasoning with Looped Transformers
Fan, Y., Svete, A., & Lee, K. (2026) · arXiv.org · doi:10.48550/arxiv.2606.31779 · Cited by 13
“This is a finite-unroll, input-injected recurrence over a looped Transformer [Dehghani et al., 2019, Fan et al., 2025], with a fixed unroll depth R and gradients propagated through all R iterations.”
Self-citationSpiralFormer: Looped Transformers Can Learn Hierarchical Dependencies via Multi-Resolution Recursion
Yu, C., Shu, X., Wang, Y.-D., et al. (2026) · arXiv.org · doi:10.48550/arxiv.2602.11698 · Cited by 13
“…(Elbayad et al., 2019; Li & Li, 2021; Takase & Kiyono, 2021; Shen et al., 2022; Tan et al., 2023; Giannou et al., 2023; Yang et al., 2023; Zhang et al., 2024; Fan et al., 2024; Hay & Wolf, 2024; Chen et al., 2025b; Nguyen & Lin, 2025; Li et al., 2025a; Aleksandrov et al., 2025; Gong et al., 2025).”
Extrapolation by Association: Length Generalization Transfer in Transformers
Cai, Z.-Y., Lee, N., Schwarzschild, A., et al. (2025) · Neural Information Processing Systems · doi:10.48550/arxiv.2506.09251 · Cited by 13
“…Li et al., 2023, Ruoss et al., 2023, Kazemnejad et al., 2024, Sabbaghi et al., 2024, Cho et al., 2024, Zhou et al., 2024, McLeish et al., 2024], looping [Fan et al., 2024], novel attention mechanisms [Duan et al., 2023, Li et al., 2025], and input format augmentation [Zhou et al., 2023, 2024].”
Skip a Layer or Loop It? Learning Program-of-Layers in LLMs
Li, Z.-Y., Li, Y., & Zhou, T. (2026) · arXiv.org · doi:10.48550/arxiv.2606.06574 · Cited by 12
“Fan et al. (2024) demonstrate that a Looped Transformer – a single Transformer block applied repeatedly – can achieve much better length generalization on algorithmic tasks by adjusting the number of loops during inference.”
ELT: Elastic Looped Transformers for Visual Generation
Goyal, S., Agrawal, S., Anil, G., et al. (2026) · arXiv.org · doi:10.48550/arxiv.2604.09168 · Cited by 12
“While looping of transformers was popularized by Universal Transformers [9] and has recently empowered language models with stronger reasoning capabilities [62, 76], its potential for high-fidelity visual generation remains largely untapped.”
InfluentialMeSH: Memory-as-State-Highways for Recursive TransformersPreprint
Yu, C., Shu, X., Wang, Y., Zhang, Y., Wu, H., Li, J., Long, R., Chen, Z., Xu, Y. C., Su, W., et al. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2510.07739 · Cited by 6
“…looped Transformers (Tan et al., 2023; Giannou et al., 2023; Li & Li, 2021; Takase & Kiyono, 2021; Elbayad et al., 2019; Yang et al., 2023; Zhang et al., 2024; Fan et al., 2024; Hay & Wolf, 2024; Chen et al., 2025b; Nguyen & Lin, 2025; Li et al., 2025a; Aleksandrov et al., 2025; Bae et al., 2025).”
DiffusionBlocks: Block-wise Neural Network Training via Diffusion Interpretation
Shing, M., Koyama, M., & Akiba, T. (2025) · arXiv.org · doi:10.48550/arxiv.2506.14202 · Cited by 12
“This scheme is also used in the recent development of recurrent-depth models (Dehghani et al., 2019; Fan et al., 2025; Geiping et al., 2025), which apply the same network parameters θ recursively K times: However, these methods suffer from the expensive backpropagation through time (BPTT) , and…”
LT2: Linear-Time Looped Transformers
Deng, C., Zhang, Y.-Z., Zhu, R., et al. (2026) · arXiv.org · doi:10.48550/arxiv.2605.20670 · Cited by 11
“…are Turing-complete under mild assumptions [49] and can be programmed to emulate iterative algorithms such as multi-step gradient descent [23, 69, 21, 20, 18], which has been formalized as a “latent thought” view of looping where each pass refines an internal computation [53, 25, 6, 8].”
Loop the Loopies!Preprint
Gao, Z., Chen, Y., Xiao, Y. H., Yang, X., Tao, R., Zhou, J., & Dai, B. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2607.16051 · Cited by 22
“…programs, optimization procedures, graph algorithms, and structured algorithmic processors, often improving length or depth extrapolation when the loop count adapts to problem size (Giannou et al., 2023; Back de Luca and Fountoulakis, 2024; Gao et al., 2025a; Fan et al., 2025; Xu and Sato, 2025a).”
From Growing to Looping: A Unified View of Iterative Computation in LLMs
Kapl, F., Angelis, E., Maile, K., et al. (2026) · arXiv.org · doi:10.48550/arxiv.2602.16490 · Cited by 8
“They improve in-context learning-to-learn [42] and algorithmic length generalization [13] and enable latent reasoning [33, 45].”
Quantitative Bounds for Length Generalization in Transformers
Izzo, Z., Nichani, E., & Lee, J. D. (2025) · arXiv.org · doi:10.48550/arxiv.2510.27015 · Cited by 8
“Most recently, architectural modifications such as looping [6] or recurrence [13] have led to length generalization improvements.”
SCOUT: Teaching Pre-trained Language Models to Enhance Reasoning via Flow Chain-of-Thought
Li, G.-H., Jiang, W., Chen, M.-F., et al. (2025) · Neural Information Processing Systems · doi:10.48550/arxiv.2505.24181 · Cited by 8
“Recurrent computation has been explored in various transformer designs, such as Universal Transformers [21], Looped Transformers [22, 23], and Sparse Recurrent Trans-formers [24].”
Ke, Y.-K., Li, X.-Y., Liang, Y.-Y., et al. (2024) · arXiv.org · doi:10.48550/arxiv.2410.11279 · Cited by 8
When Does Recurrence Become an Algorithm? Convergence Selection in Weight-Tied Looped Transformers
Zhang, T.-Y., Hu, J.-H., Peng, Y., et al. (2026) · arXiv.org · doi:10.48550/arxiv.2607.20594 · Cited by 7
“Looped transformers provably length-generalize on n -RASP- L tasks (Fan et al., 2024) and can emulate iterative solvers (Yang et al., 2023; Gatmiry et al., 2024); Gatmiry et al. (2024) show the global minimizer for in-context regression implements preconditioned gradient descent—an inductive bias…”
Xu, R.-H., Gao, Y.-T., Wang, L., Li, J.-N., Chen, W., Guo, Q.-P., Yang, M., & Zhang, S.-L. (2026) · arXiv.org · doi:10.48550/arxiv.2602.09080 · Cited by 7
“Looped Transform-ers exemplify this paradigm: they can simulate programmable computation [15], learn iterative optimization algorithms [13], generalize to longer sequences on algorithmic tasks [12], and model data-fitting dynamics [38].”
Self-Supervised Transformers as Iterative Solution Improvers for Constraint Satisfaction
Xu, Y. W., Li, W.-H., Sanner, S., et al. (2025) · International Conference on Machine Learning · doi:10.48550/arxiv.2502.15794 · Cited by 7
“Furthermore, recurrence has been shown to enhance the generalization abilities of Transformers (McLeish et al., 2024; Fan et al., 2025), reinforcing their suitability for our setting.”
InfluentialYao, X., Ren, R., Liao, Y., Ding, L., & Liu, Y. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2502.04667 · Cited by 17
“Loop transformers [24, 8, 15] have been proposed to solve algorithmic tasks.”
One Step Forward and K Steps Back: Better Reasoning with Denoising Recursion Models
Cameron, C., Wang, W., Ivanov, N., et al. (2026) · arXiv.org · doi:10.48550/arxiv.2604.18839 · Cited by 6
“Empirically, looped models have shown improved parameter efficiency, sample efficiency, and length generalization on reasoning tasks (Saunshi et al., 2025; Geiping et al., 2025a; Fan et al., 2025; Zhu et al., 2025b).”
Step-resolved data attribution for looped transformers
Kaissis, G., Mildenberger, D., Gómez, J., et al. (2026) · arXiv.org · doi:10.48550/arxiv.2602.10097 · Cited by 6
“The methodology here is identical to Fan et al. (2024).”
InfluentialStabilizing Extrapolation in Looped Transformers via Learned Stochastic Stopping
Kuo, H.-Y., Chayti, E. M., Reizinger, P., et al. (2026) · arXiv.org · doi:10.48550/arxiv.2606.29983 · Cited by 5
“In our experiments, we follow the looped architecture used by Fan et al. [11].”
InfluentialNeural Scaling Universality: If Exponents Are Fixed, Time to Understand CoefficientsPreprint
Liu, Y., & Gore, J. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2606.25008 · Cited by 5
“Mixture of experts [44, 45], looped transformers [46, 47], and attention residual networks [48] should not alter the three scaling laws [15–17] if the underlying mechanisms are unchanged, but may change the coefficients, especially c ℓ , which governs transformation.”
Recursive Scaling in Masked Diffusion ModelsPreprint
Carballo-Castro, A., Piskorz, J., Rauba, P., van der Schaar, M., & Frossard, P. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2606.18022 · Cited by 3
“This is the comparison regime used throughout our experiments.”
Looped Diffusion Language Models
Lee, S.-H., Hong, C., Kim, S., et al. (2026) · arXiv.org · doi:10.48550/arxiv.2605.26106 · Cited by 5
“This perspective has been used to improve length generalization on algorithmic tasks [18], to implement token-level variants of chain-of-thought-style computation [45], and to study the theoretical power of repeated transformer computation [63].”
Learning Modal-Mixed Chain-of-Thought Reasoning with Latent Embeddings
Shao, Y.-F., Zhou, K., Xu, Z.-M., et al. (2026) · arXiv.org · doi:10.48550/arxiv.2602.00574 · Cited by 5
“…2023; Pfau et al., 2024), knowledge distillation (Deng et al., 2023; 2024; Yu et al., 2024), or architectural modifications (Giannou et al., 2023; Fan et al., 2024; Barrault et al., 2024) aim to strengthen this capability by improving the expressivity of the transformer network and designing new…”
Transformers as Intrinsic Optimizers: Forward Inference through the Energy PrinciplePreprint
Ren, R., Ouyang, S., Tang, H., & Liu, Y. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2511.00907 · Cited by 5
“Next, we show how different attention forms (including unnormalized linear attention, standard softmax attention, etc.) can be derived from the above framework by making specific choices.”
To Infinity and Beyond: Tool-Use Unlocks Length Generalization in State Space ModelsPreprint
Malach, E., Saremi, O., Williamson, S., Bradley, A., Lotfi, A., Abbe, E. A., Susskind, J., & Littwin, E. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2510.14826 · Cited by 1
“So, to solve long-form generation tasks we can either use Transformers and suffer quadratic scaling of compute, or use SSMs and suffer performance degradation.”
InfluentialAllocating Recurrent Compute in Looped Language Models
Lin, R.-H., Guo, Y.-Y., Zhu, R.-J., et al. (2026) · doi:10.48550/arxiv.2608.18230 · Cited by 4
“Additional methods allocate variable steps across tokens, layers, or compute budgets, including SkipLayer, LoopFormer, Think-at-Hard, and stability-regularized recurrence [12–16].”
Lu, H.-Y., Wei, Z., Zhang, Q.-X., et al. (2026) · arXiv.org · doi:10.48550/arxiv.2606.18208 · Cited by 4
“Fan et al. (2024) demonstrated that looped transformers with adaptive stopping significantly improve length generalisation.”
The Imitation Game: Turing Machine Imitator is Length Generalizable Reasoner
Hua, Z., Zhang, W.-W., Lyu, C., et al. (2025) · arXiv.org · doi:10.48550/arxiv.2507.13332 · Cited by 4
“As shown in Figure 3, TAIL substantially enhances the long-sequence reasoning capabilities of the 7B model, enabling it to outperform DeepSeek-R1 on the majority of tasks.”
Intra-Layer Recurrence in Transformers for Language Modeling
Nguyen, A., & Lin, W. (2025) · Canadian AI · doi:10.48550/arxiv.2505.01855 · Cited by 4
“Looped transformers [4–6] extend this idea by applying recurrence to enhance algorithmic reasoning and length generalization.”
Right Direction, Wrong Step: Geometric Analysis of Finite-Step Failure in Looped Transformers
Guo, Z.-H., Wu, Z.-H., Du, H.-Z., et al. (2026) · doi:10.48550/arxiv.2609.16665 · Cited by 3
“Shared recurrent computation supports depth and length generalisation (Schwarzschild et al., 2021; Bansal et al., 2022; Fan et al., 2025; Saunshi et al., 2025), while adaptive computation methods learn how much processing to allocate to each input (Graves, 2017; Banino et al., 2021; Elbayad et al.,…”
Exploration of Fast-Slow Latent Recurrence for Train-Short, Test-Long GeneralizationPreprint
Takashiro, S., Koyama, M., Miyato, T., Iwasawa, Y., Matsuo, Y., & Hayashi, K. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2604.01577 · Cited by 2
“Our models also outperform baselines such as Looped Transformer (Fan et al., 2025) on a limited-vision maze navigation task, showing that rules learned in small environments can transfer to larger ones.”
InfluentialExpert-Choice Routing Enables Adaptive Computation in Diffusion Language Models
Zhang, S., Zhuang, C., Cui, C., et al. (2026) · arXiv.org · doi:10.48550/arxiv.2604.01622 · Cited by 3
“…early exit and confidence-based halting (Schuster et al., 2022; Elhoushi et al., 2024), Mixture-of-Depths routing (Raposo et al., 2024; Bae et al., 2025b), and looped/recursive transformers (Lan et al., 2020; Hutchins et al., 2022; Fan et al., 2025; Bae et al., 2025a; Geiping et al., 2025).”
Large Neighborhood Search meets Iterative Neural Constraint Heuristics
Xu, W., Li, W., Sanner, S., et al. (2026) · Integration of AI and OR Techniques in Constraint Programming · doi:10.48550/arxiv.2603.20801 · Cited by 3
“This includes masked diffusion models [30], recurrent Transformers [11, 40], and single-step iterative Transformers [39].”
On the Spatiotemporal Dynamics of Generalization in Neural NetworksPreprint
Wei, Z. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2602.01651 · Cited by 3
“Looped Transformers [9] and Universal Transformers [6] simulate recurrence by reusing layer weights, breaking the fixed depth limit.”
Aligning transformer circuit mechanisms to neural representations in relational reasoning
Hearne, L., Robinson, C., Cocchi, L., et al. (2026) · bioRxiv · doi:10.1101/2025.10.29.685457 · Cited by 2
“Second, although standard implementations of MLPs and LSTMs failed to generalize on the LST, it is possible that non-standard training set ups, learning regimes, or architectural variants (e.g., graph 74 or looped transformers 75 ) may confer improved model performance or brain-model alignment.”
Monitoring and Discovering Reward Hacking with Internal Representations during LLM EvaluationsPreprint
Bergen, L., Bhalla, U., Lee, A., Widawsky, B., Nasvytis, L., Watts, C., Boppana, S., Baskaran, S., Hazra, D., Byun, M. J., et al. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.19101 · Cited by 3
“Such obfuscation may arise from architectural choices that move computation away from explicit outputs, such as looped transformers (Fan et al., 2025) or latent chain-of-thought methods (Hao et al., 2025; Deng et al., 2023), or as a consequence of optimization pressure that incentivizes compressed…”
T-LoopFormer: Token-Level Elastic-Depth Looped Transformers with Dynamic Routing
Yu, M.-Q., Zhang, W.-P., Cui, S.-B., et al. (2026) · doi:10.48550/arxiv.2609.15160 · Cited by 2
“Besides, there are some works that explore looped transformers as programmable computers (Giannou et al., 2023) and mechanisms for algorithmic length generalization (Fan et al., 2025; Jeddi et al., 2026).”
LoopCTR: Unlocking the Loop Scaling Power for Click-Through Rate Prediction
Tang, J.-K., Zhang, R.-F., Wang, W., et al. (2026) · arXiv.org · doi:10.48550/arxiv.2604.19550 · Cited by 2
“Recent work has established theoretical foundations [Xu and Sato, 2024, Saunshi et al., 2025] and demonstrated practical benefits for length generalization [Fan et al., 2024].”
Turbo Connection: Reasoning as Information Flow from Higher to Lower LayersPreprint
Tang, M., & Lu, S. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2602.17993 · Cited by 2
“Parity (Fan et al., 2025a; Banino et al., 2021; Graves, 2017): Given a sequence of 0s and 1s, the task is to determine whether there is an odd number of 1s.”
Hartl, B., Pio-Lopez, L., Fields, C., et al. (2026) · Physics of Life Reviews · doi:10.48550/arxiv.2601.14096 · Cited by 2
The Illusion of Diminishing Returns: Measuring Long Horizon Execution in LLMs
Sinha, A., Arun, A., Goel, S., Staab, S., & Geiping, J. (2025) · arXiv · doi:10.48550/arxiv.2509.09677 · Cited by 1
“Our observations on scaling could hold for the related problem of length-generalization—training models to succeed on tasks longer than those seen during training (Fan et al., 2024; Cai et al., 2025).”
Scaling to Tens of Thousands of Test-Time Iterations with Loop-Native Attention Residuals
Li, P.-X., Muhtar, D., He, D., et al. (2026) · doi:10.48550/arxiv.2610.11570 · Cited by 1
“Looped Transformers have also been studied as learners of algorithms (Yang et al., 2024) and for generalization to longer input sequences (Fan et al., 2025).”
Looping Beyond Twice: A Scalable Recipe for Looped Mixture-of-ExpertsPreprint
He, D., Li, P., Chang, D., Meng, Q., Yin, L., & Liu, S. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.01153 · Cited by 3
“Recent studies demonstrate that recurrent depth improves algorithmic computation (Giannou et al., 2023), length generalization (Fan et al., 2025), and latent reasoning (Saunshi et al., 2025; Geiping et al., 2025; Hao et al., 2024), establishing looped computation as a promising direction for LLM…”
What Limits Recursive Reasoning Models: Optimization, Architecture and Test-Time Scaling
Shakhvalieva, Y., Kharchev, D., Безруков, В. Е., et al. (2026) · doi:10.48550/arxiv.2609.39967 · Cited by 1
“Looped Transformers demonstrate this principle on algorithmic reasoning and length generalization (Saunshi et al., 2025; Fan et al., 2025), while recurrent-depth language models extend it to test-time computation in larger language models (Geiping et al., 2025).”
Beyond the Training Horizon: Mechanisms and Limits of Length Generalization in Looped Transformers
Liang, J., Jin, X., & Pan, L.-M. (2026) · doi:10.48550/arxiv.2609.33144 · Cited by 1
“Fan et al. (2025) demonstrate extrapolation to unseen input lengths on iterative algorithmic tasks, while Kohli et al. (2026) show that increasing recurrent depth at inference time can enable deeper implicit compositions.”
InfluentialPrograms-of-Layers in LLMs through the Lens of Cortical AreasPreprint
Westerhoff, J., Olbrich, S., Oraby, H., Larkum, M. E., & Gers, F. A. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.31360 · Cited by 1
“…them: Universal Transformers (Dehghani et al., 2019) apply one shared block recurrently with a per-token halting policy, looped transformers (Fan et al., 2025) repeat a block for a fixed or learned number of iterations to improve length generalization, MoEUT (Csordás et al., 2024)…”
Arrabal-Campos, F., Fernández, I., Montoya, F. G., et al. (2026) · doi:10.48550/arxiv.2608.24319 · Cited by 1
“…—Adaptive Computation Time (Graves, 2016), PonderNet (Banino et al., 2021), Universal Transformers (Dehghani et al., 2019), looped transformers (Fan et al., 2025), depth-adaptive and early-exit decoding (Elbayad et al., 2020; Schwartz et al., 2020), and token-level compute routing (Raposo et…”
Length Generalization with Log-Depth Recurrent UnitsPreprint
Pert, C., Alrajeh, D., & Russo, A. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2605.26035 · Cited by 1
“However, recurrent transformers trade parallelism for improved length generalization (Soulos et al., 2024; Fan et al., 2025).”
Do Transformers Use their Depth Adaptively? Evidence from a Relational Reasoning TaskPreprint
Curth, A., Lawrence, R., Karmalkar, S., & Prasad, N. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2604.12426
“…because they equip models with additional effective computational depth (Merrill & Sabharwal, 2023), and has spurred investigations into recurrent transformer architectures that allow models to adaptively determine their own depth (Dehghani et al., 2018; Saunshi et al., 2025; Fan et al., 2024).”
Thinking into the Future: Latent Lookahead Training for Transformers
Noci, L., Bachmann, G., Moosavi-Dezfooli, S.-M., et al. (2026) · arXiv.org · doi:10.48550/arxiv.2603.20219 · Cited by 1
“Another direction employs looped transformers (Giannou et al., 2023; Fan et al., 2024), which recursively update hidden states à la recurrent networks.”
VersatileFFN: Achieving Parameter Efficiency in LLMs via Adaptive Wide-and-Deep ReusePreprint
Nie, Y., Han, K., Li, H., Zhou, H., Guo, T., Wu, E., Chen, X., & Wang, Y. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2512.14531
SAI: Latency-Aware Satellite Edge LAM Inference with Looped Transformer
Yuan, H., Wang, Z., Jiang, Y., Liu, X., Shi, Y., & Wang, T. (2025) · International Conference on Communications · doi:10.1109/icc52391.2025.11161072 · Cited by 1
“…and retaining less than 10% of the parameters of standard layered trans-formers, have drawn significant attention for their superior performance in the in-context problem-solving paradigm as well as their enhanced ability to emulate iterative algorithms more effectively than vanilla LAMs [17]–[19].”
PAR: Progress-Aligned Recurrence for Length Generalization in State Tracking
Wang, P., Hu, Y., Dong, G.-Y., et al. (2026) · doi:10.48550/arxiv.2610.27628
“Prior work demonstrates length generalisation by tying the loop count to task length [7], and shows that intermediate-state supervision can teach a pretrained model to perform successive rule lookups [28].”
Spherical Fixed-Point Reasoners: Adaptive Loop Halting on the Hypersphere
Zela, A., Maranjyan, A., Puigdemont, P., et al. (2026) · doi:10.48550/arxiv.2610.25646
“…et al., 2024), emulate programmable computation (Giannou et al., 2023), generalize in length when the loop matches the algorithm’s re-cursion (Fan et al., 2025), scale to billions of parameters (Geiping et al., 2025; Prairie et al., 2026), and supply a scaling axis for masked diffusion…”
Copying Under Repetition: From Prefix Matching to Position HashingPreprint
Guo, A., Huang, T., Yang, C., & Li, Z. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.26722
“Fan et al. (2025) implement copying by iterating a fixed block for n steps, and Kraus et al. (2026) use a growing signpost alphabet to address copying and retrieval barriers in chain-of-thought reasoning.”
Looped GNNs Beyond Fixed Depth: Graph Size Generalization and Parameter-Efficient Adaptation
Wang, Z.-Y., Luo, Z.-S., & Wang, Y.-S. (2026) · doi:10.48550/arxiv.2610.24112
“Related work on looped Transformers studies shared-block computation for learning iterative algorithms [10] and length generalization on tasks with iterative solutions [11].”
Inside the Loop: A Mechanistic Study of Weight-Tied Transformers on Depth-Bound Algorithmic Tasks
Sesodia, M., Sock, J., Torr, P., et al. (2026) · doi:10.48550/arxiv.2610.24135
“Beyond parameter efficiency (Lan et al., 2020), looping can turn even a shallow transformer into a programmable computer (Giannou et al., 2023) and may bias learning toward iterative algorithms (Fan et al., 2025; Yang et al., 2024).”
Learning to Compute Is Not Learning to Retain or Reuse in Looped Transformers
Zhou, P.-L., Zhang, H.-T., Huang, Y.-L., et al. (2026) · doi:10.48550/arxiv.2610.19768
“…connect shared parameters to adaptive computation, programmable execution, and length generalization (Dehghani et al., 2019; Giannou et al., 2023; Fan et al., 2025); recurrent-depth language models extend this motivation to latent test-time compute (Geiping et al., 2025 (Geiger et al., 2021;…”
Looped Reasoning Finishes Earlier Than You Think
Park, S., & Rusch, T. K. (2026) · doi:10.48550/arxiv.2610.18174
“Looped reasoners trade architectural depth for recurrent computation (Dehghani et al., 2019; Saunshi et al., 2025; Fan et al., 2025).”
Learning to Extrapolate in Depth with Looped Transformers
Yang, H.-T., Huang, J.-H., Wang, Z.-X., et al. (2026) · doi:10.48550/arxiv.2610.14456
“A natural approach is to train with a few loops on tractable instances, then tackle harder ones by running the block for more loops [Schwarzschild et al., 2021, Fan et al., 2025].”
Recurrent In-Context Learning for Tabular Foundation ModelsPreprint
Vaidhya, T., Sahu, G., & Rish, I. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.14998
“…that reasoning benefits from depth without a proportional increase in distinct parameters, looped transformers generalize to longer inputs than they were trained on (Fan et al., 2025), and recurrent networks extrapolate from easy to hard problems by iterating longer (Schwarzschild et al., 2021).”
Running Fewer Loops Removes Refusal Behaviour in Recurrent-Depth Language ModelsPreprint
Azizi, S., Potraghloo, E. B., Fayyazi, A., Shokoufa, S., & Pedram, M. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.15218
“…(Dehghani et al., 2018) and cross-layer parameter sharing in ALBERT (Lan et al., 2019) through analyses of what a looped block can express and learn (Giannou et al., 2023; Yang et al., 2023; Fan et al., 2024; Xu & Sato, 2024; Gatmiry et al., 2024; Saunshi et al., 2025; Xu & Sato, 2025).”
ResidualQuant: KV Cache Quantization for Looped Transformers with 2-Bit Residuals
Kim, H.-J., Lee, J., Cho, S., et al. (2026) · doi:10.48550/arxiv.2610.10381
“Such repeated computation supports length generalization and latent reasoning without intermediate token generation (Fan et al., 2025; Saunshi et al., 2025), with HRM, TRM, and Looped Flows exploring iterative refinement of internal states (Wang et al., 2025; Jolicoeur-Martineau, 2025; Suleymanzade…”
What Matters for Latent Reasoning with Flow Matching
Ouali, Y., Bulat, A., & Tzimiropoulos, G. (2026) · doi:10.48550/arxiv.2610.06666
“Looped or recurrent methods (Dehghani et al., 2019; Giannou et al., 2023; Fan et al., 2025; Saunshi et al., 2025; Geiping et al., 2025; Zhu et al., 2025c) keep the number of slots fixed and apply the same layers over them several times, so the reasoning becomes an iterative computation within the…”
Decoding Looped Transformers Better for (Almost) Free
Liu, W., Zheng, H., Chen, T.-R., et al. (2026) · doi:10.48550/arxiv.2610.02185
“By iteratively applying shared parameters, models can implicitly unroll reasoning steps in the latent space to solve complex logical tasks and generalize to longer sequences (Giannou et al., 2023; Yang et al., 2024; Fan et al., 2025; Saunshi et al., 2025; Zhu et al., 2025b; Liu et al., 2026).”
Mohamud, J., & Ravanelli, M. (2026) · doi:10.48550/arxiv.2610.00541
“…algorithms, length generalization, parameter-efficient language modeling, latent reasoning, and adaptive computational depth (Yang et al., 2024; Fan et al., 2025; Bae et al., 2025a; Geiping et al., 2025; Chen et al., 2025; Bae et al., 2025b; Zhu et al., 2026; Sapunov, 2026; Jeddi et al., 2026).”
Rotate the Loop: Blockwise Conformality for Looped Transformers
Zemtsov, A., Klepach, A., Lomshakov, V., et al. (2026) · doi:10.48550/arxiv.2610.14030
“…and has become an axis of test-time compute: k layers looped L times match kL distinct layers on iterative reasoning and can simulate chain-of-thought steps in latent space (Saunshi et al., 2025); an adaptive trip count generalizes in length on copying, parity and addition (Fan et al., 2025); a 3 .”
Transformers Stop Thinking Too Early, and a Tiny LoRA Fixes It
Jin, Z.-H., Deng, R.-X., & Wang, J.-R. (2026) · doi:10.48550/arxiv.2609.36585
“Recurrent-depth models make inference computation adjustable (Dehghani et al., 2019; Gi-annou et al., 2023; Yang et al., 2024a; Saunshi et al., 2025; Fan et al., 2025).”
Looped Actor: Depth-Recurrent Reasoning Models for Reinforcement Learning
Rusch, T. K., Seyde, T., Boyer, J., et al. (2026) · doi:10.48550/arxiv.2609.37432
“This approach has since been extended to various algorithmic and arithmetic tasks (Giannou et al., 2023; Kohli et al., 2026; Yang et al., 2024; Fan et al., 2025).”
Looped Transformers as OptimizersPreprint
Huang, Y., Jiang, C., Zhou, Z., Zhang, H., Li, T., He, T., Zhang, X., & Cheng, B. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.37379
“Later work studies length generalization with adaptive loop counts (Fan et al., 2025).”
LoopVL: Recurrent Visual IntelligencePreprint
Qian, Z., Gong, Z., Xu, Z., Li, H., Wang, Z., Liu, H., Qian, L., Pei, X., Xu, P., Luo, F., et al. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.38426 · Cited by 2
“(Fan et al., 2025; Saunshi et al., 2025) Abstract reasoning tasks make this distinction concrete because a model must infer a transformation or structure from limited task evidence.”
LoopICL: Looping a single transformer block to solve tabular tasks
Balef, A. R., & Eggensperger, K. (2026) · doi:10.48550/arxiv.2609.36108
“This effect is particularly pronounced for the recurrent model with α = 1 / L , consistent with the findings of (Fan et al., 2025), who show that looped Transformers with an adaptive number of steps can substantially improve length generalization.”
Distance generalization in transformers: why bother with positional encoding?Preprint
Nevermann, D. H., & Gros, C. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.11913
“In an effort to improve length generalization abilities, recent studies use several alternative approaches, including specialized positional encoding [18], specific training protocols [5], scratchpad prompting [1] and knowledge transfer from related tasks [2].”
Planning with Transformers: Chain of Computation and Structured Context Windows
Futuhi, E., & Sturtevant, N. R. (2026) · arXiv.org · doi:10.48550/arxiv.2607.17710
“…2023; Jiang et al., 2026; Li et al., 2024; Hou et al., 2025), as well as their ability to learn algorithms with guarantees, often through the lens of length generalization (Giannou et al., 2023; Zhou et al., 2024; Hua et al., 2026; Malach et al., 2026; Chen et al., 2026; Fan et al., 2025).”
Repeated Shared Access Enables Grokking, but Edit Propagation Depends on an Addressable MemoryPreprint
Niu, Y. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2606.20737
“On the theoretical side, looped transformers have been shown to length-generalize on algorithmic tasks by emulating iterative computation [8], and Saunshi et al. [9] show that looped models carry an inductive bias toward reasoning —matching far deeper iso-FLOP baselines on reasoning primitives…”
Anti Mode-Collapse in Mean-Field Transformer via Auxiliary Variables
Imaizumi, M., Koyama, M., Isobe, N., et al. (2026) · arXiv.org · doi:10.48550/arxiv.2605.30229
“In particular, the interaction kernel is reused along depth/time, so the model should be read as a shared-weight or looped-block surrogate [FDRL24, XS24, ACLY25, Che26] rather than as a full untied-depth transformer.”
InfluentialGeneralization in LLM Problem Solving: The Case of the Shortest Path
Tong, Y., Ye, J.-Y., Borovykh, A., et al. (2026) · arXiv.org · doi:10.48550/arxiv.2604.15306
“It has been widely discussed as a central challenge (Dubois et al., 2019; Newman et al., 2020; Cai et al., 2025; Fan et al., 2024; Jelassi et al., 2023; Anil et al., 2022), and is sometimes framed as a form of recursive composition or extrapolation Kim & Linzen (2020); Hupkes et al. (2020); Dziri…”
On the Mirage of Long-Range Dependency, with an Application to Integer MultiplicationPreprint
Wei, Z. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2603.29069
“Other work attempts to restore time depth through recurrent architectures [32], [33], [34], but the spatial geometry remains unchanged.”
Metadata
- Authors
- Fan, Y., Du, Y., Ramchandran, K., & Lee, K.
- Affiliations
- University of Wisconsin–Madison (US)
- Massachusetts Institute of Technology (US)
- University of California, Berkeley (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2024
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2409.15647
- OpenAlex
- W4403788968
- Semantic Scholar
- 408295dd8cb6140179fe075ca04a9d80f064bd36
- Cited by
- 0
- References
- 0
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (2339978, 2023239, CAREER)
- Topics
- Physical Sciences › Computer Science › Computational Theory and Mathematics › semigroups and automata theory
- Physical Sciences › Computer Science › Computational Theory and Mathematics › Computability, Logic, AI Algorithms
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Keywords
- length generalization, iterative computation, arithmetic exercises, algorithmic tasks
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-23, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-23 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
