Looped Transformers for Length Generalization

“Looped Transformers shown to be effective, can act as programmable computers [33], learn iterative data-fitting algorithms [100], generalize to much longer inputs on algorithmic tasks [23], and illuminate few-shot learning by mimicking multi-step optimizers [27].”

“The examples utilize models with six layers, where identical colors represent shared weights.”

“Once trained, their representations collapse when evaluated at shorter or longer depths, since those settings are off-distribution Bae et al. (2024); Fan et al. (2024).”

“Recurrent mechanisms are shown to learn generalizable solutions to problems using ResNet (He et al., 2015) based architectures (Schwarzschild et al., 2021; Bansal et al., 2022; Anil et al., 2022; Schwarzschild, 2023; Bear et al., 2024).”

“Fan et al. (2025) show that such looped transformers offer superior length generalization on algorithmic tasks such as parity and binary addition.”

“In other words, TF = (id + FF ) ◦ (id + Attn) , where id denotes the identity mapping, where we omit the domain of definition for simplicity.”

“…show that Looped Transformers can improve length generalization, learn algorithmic structure, and approximate much deeper untied models on reasoning tasks (Yang et al., 2024; Fan et al., 2025; Saunshi et al., 2025; Kapl et al., 2026; Geiping et al., 2026; Giannou et al., 2023; Kohli et al., 2026).”

“Other methods focus on architectural changes, such as introducing looping mechanisms (Fan et al., 2024) or incorporating hand-crafted bias corrections in attention score matrices (Duan et al., 2023).”

“…universal transformers (Dehghani et al., 2018a) theoretically demonstrates that when a block of middle layers of a fixed transformer model is repeated for a variable number of times, its expressivity can be significantly improved (Saunshi et al., 2025; Merrill & Sabharwal, 2025; Fan et al., 2024).”

“Meanwhile, Fan et al. [13], Yang et al. [43] demonstrated its parameter efficiency in data-fitting settings, and superior length generalization on RASP-L tasks.”

“Other works offered theoretical and small-scale analyses of looped transformers (Giannou et al., 2023; Gatmiry et al., 2024; Yang et al., 2023; Fan et al., 2024).”

“Subsequent research (Giannou et al., 2023; Yang et al., 2023; Fan et al., 2024) have investigated their potential benefits through theoretical and small-scale empirical analyses.”

“Several preliminary results (Fan et al., 2024) show that the looped Transformer model has better length generalization capabilities, partially because the increase in problem complexity (e.g., problem length) is not as significant as in the Chain-of-Thought steps.”

“[9] demonstrate that a Looped Transformer – a single Transformer block applied repeatedly – can achieve much better length generalization on algorithmic tasks by adjusting the number of loops during inference.”

“…or algorithmic procedures (Yang et al., 2024b; Giannou et al., 2023a), emulate additional depth through weight sharing (Dehghani et al., 2019; Zhu et al., 2025c), reduce the context-length costs of token-level reasoning, and improve downstream generalization (Labovich, 2026b; Fan et al., 2025).”

“January 30, 2026.”

“Recently, looped transformers (Giannou et al., 2023; Fan et al., 2024) have been proposed to solve algorithmic tasks, which have some similarities to the computing process of continuous thoughts, but we focus on common reasoning tasks and aim at investigating latent reasoning in comparison to…”

“Naively looping g for K rounds, i.e. x ← g ( x ) applied K times [27, 47, 35, 28, 77, 95, 33, 102, 88, 1, 58, 32, 12], is a K -step forward Euler integration of the window residual field with step size h =1 , which approximates x ( t = K ) .”

“Existing studies on simplified setups have shown that, compared to similar-sized vanilla transformers, looped transformers show superior capacity at multi-hop reasoning [8, 9], lenght generalization [10, 9] and learning algorithms [11].”

“Recent work on “thinking mod-els” [1, 26, 45, 31, 43] further suggests that recurrence supports a latent Chain-of-Thought, enabling models to adapt their computation based on task complexity.”

“Universal Transformers share parameters across depth and learn to halt (often via ACT) [2]; depth-adaptive and early-exit variants similarly modulate per-token compute by stopping when intermediate predictions stabilize (e.g., entropy/confidence heuristics) [3, 11].”

“This is a finite-unroll, input-injected recurrence over a looped Transformer [Dehghani et al., 2019, Fan et al., 2025], with a fixed unroll depth R and gradients propagated through all R iterations.”

“…(Elbayad et al., 2019; Li & Li, 2021; Takase & Kiyono, 2021; Shen et al., 2022; Tan et al., 2023; Giannou et al., 2023; Yang et al., 2023; Zhang et al., 2024; Fan et al., 2024; Hay & Wolf, 2024; Chen et al., 2025b; Nguyen & Lin, 2025; Li et al., 2025a; Aleksandrov et al., 2025; Gong et al., 2025).”

“…Li et al., 2023, Ruoss et al., 2023, Kazemnejad et al., 2024, Sabbaghi et al., 2024, Cho et al., 2024, Zhou et al., 2024, McLeish et al., 2024], looping [Fan et al., 2024], novel attention mechanisms [Duan et al., 2023, Li et al., 2025], and input format augmentation [Zhou et al., 2023, 2024].”

“Fan et al. (2024) demonstrate that a Looped Transformer – a single Transformer block applied repeatedly – can achieve much better length generalization on algorithmic tasks by adjusting the number of loops during inference.”

“While looping of transformers was popularized by Universal Transformers [9] and has recently empowered language models with stronger reasoning capabilities [62, 76], its potential for high-fidelity visual generation remains largely untapped.”

“…looped Transformers (Tan et al., 2023; Giannou et al., 2023; Li & Li, 2021; Takase & Kiyono, 2021; Elbayad et al., 2019; Yang et al., 2023; Zhang et al., 2024; Fan et al., 2024; Hay & Wolf, 2024; Chen et al., 2025b; Nguyen & Lin, 2025; Li et al., 2025a; Aleksandrov et al., 2025; Bae et al., 2025).”

“This scheme is also used in the recent development of recurrent-depth models (Dehghani et al., 2019; Fan et al., 2025; Geiping et al., 2025), which apply the same network parameters θ recursively K times: However, these methods suffer from the expensive backpropagation through time (BPTT) , and…”

“…are Turing-complete under mild assumptions [49] and can be programmed to emulate iterative algorithms such as multi-step gradient descent [23, 69, 21, 20, 18], which has been formalized as a “latent thought” view of looping where each pass refines an internal computation [53, 25, 6, 8].”

“…programs, optimization procedures, graph algorithms, and structured algorithmic processors, often improving length or depth extrapolation when the loop count adapts to problem size (Giannou et al., 2023; Back de Luca and Fountoulakis, 2024; Gao et al., 2025a; Fan et al., 2025; Xu and Sato, 2025a).”

“They improve in-context learning-to-learn [42] and algorithmic length generalization [13] and enable latent reasoning [33, 45].”

“Most recently, architectural modifications such as looping [6] or recurrence [13] have led to length generalization improvements.”

“Recurrent computation has been explored in various transformer designs, such as Universal Transformers [21], Looped Transformers [22, 23], and Sparse Recurrent Trans-formers [24].”

“Looped transformers provably length-generalize on n -RASP- L tasks (Fan et al., 2024) and can emulate iterative solvers (Yang et al., 2023; Gatmiry et al., 2024); Gatmiry et al. (2024) show the global minimizer for in-context regression implements preconditioned gradient descent—an inductive bias…”

“Looped Transform-ers exemplify this paradigm: they can simulate programmable computation [15], learn iterative optimization algorithms [13], generalize to longer sequences on algorithmic tasks [12], and model data-fitting dynamics [38].”

“Furthermore, recurrence has been shown to enhance the generalization abilities of Transformers (McLeish et al., 2024; Fan et al., 2025), reinforcing their suitability for our setting.”

“Loop transformers [24, 8, 15] have been proposed to solve algorithmic tasks.”

“Empirically, looped models have shown improved parameter efficiency, sample efficiency, and length generalization on reasoning tasks (Saunshi et al., 2025; Geiping et al., 2025a; Fan et al., 2025; Zhu et al., 2025b).”

“The methodology here is identical to Fan et al. (2024).”

“In our experiments, we follow the looped architecture used by Fan et al. [11].”

“Mixture of experts [44, 45], looped transformers [46, 47], and attention residual networks [48] should not alter the three scaling laws [15–17] if the underlying mechanisms are unchanged, but may change the coefficients, especially c ℓ , which governs transformation.”

“This is the comparison regime used throughout our experiments.”

“This perspective has been used to improve length generalization on algorithmic tasks [18], to implement token-level variants of chain-of-thought-style computation [45], and to study the theoretical power of repeated transformer computation [63].”

“…2023; Pfau et al., 2024), knowledge distillation (Deng et al., 2023; 2024; Yu et al., 2024), or architectural modifications (Giannou et al., 2023; Fan et al., 2024; Barrault et al., 2024) aim to strengthen this capability by improving the expressivity of the transformer network and designing new…”

“Next, we show how different attention forms (including unnormalized linear attention, standard softmax attention, etc.) can be derived from the above framework by making specific choices.”

“So, to solve long-form generation tasks we can either use Transformers and suffer quadratic scaling of compute, or use SSMs and suffer performance degradation.”

“Additional methods allocate variable steps across tokens, layers, or compute budgets, including SkipLayer, LoopFormer, Think-at-Hard, and stability-regularized recurrence [12–16].”

“Fan et al. (2024) demonstrated that looped transformers with adaptive stopping significantly improve length generalisation.”

“As shown in Figure 3, TAIL substantially enhances the long-sequence reasoning capabilities of the 7B model, enabling it to outperform DeepSeek-R1 on the majority of tasks.”

“Looped transformers [4–6] extend this idea by applying recurrence to enhance algorithmic reasoning and length generalization.”

“Shared recurrent computation supports depth and length generalisation (Schwarzschild et al., 2021; Bansal et al., 2022; Fan et al., 2025; Saunshi et al., 2025), while adaptive computation methods learn how much processing to allocate to each input (Graves, 2017; Banino et al., 2021; Elbayad et al.,…”

“Our models also outperform baselines such as Looped Transformer (Fan et al., 2025) on a limited-vision maze navigation task, showing that rules learned in small environments can transfer to larger ones.”

“…early exit and confidence-based halting (Schuster et al., 2022; Elhoushi et al., 2024), Mixture-of-Depths routing (Raposo et al., 2024; Bae et al., 2025b), and looped/recursive transformers (Lan et al., 2020; Hutchins et al., 2022; Fan et al., 2025; Bae et al., 2025a; Geiping et al., 2025).”

“This includes masked diffusion models [30], recurrent Transformers [11, 40], and single-step iterative Transformers [39].”

“Looped Transformers [9] and Universal Transformers [6] simulate recurrence by reusing layer weights, breaking the fixed depth limit.”

“Second, although standard implementations of MLPs and LSTMs failed to generalize on the LST, it is possible that non-standard training set ups, learning regimes, or architectural variants (e.g., graph 74 or looped transformers 75 ) may confer improved model performance or brain-model alignment.”

“Such obfuscation may arise from architectural choices that move computation away from explicit outputs, such as looped transformers (Fan et al., 2025) or latent chain-of-thought methods (Hao et al., 2025; Deng et al., 2023), or as a consequence of optimization pressure that incentivizes compressed…”

“Besides, there are some works that explore looped transformers as programmable computers (Giannou et al., 2023) and mechanisms for algorithmic length generalization (Fan et al., 2025; Jeddi et al., 2026).”

“Recent work has established theoretical foundations [Xu and Sato, 2024, Saunshi et al., 2025] and demonstrated practical benefits for length generalization [Fan et al., 2024].”

“Parity (Fan et al., 2025a; Banino et al., 2021; Graves, 2017): Given a sequence of 0s and 1s, the task is to determine whether there is an odd number of 1s.”

“Our observations on scaling could hold for the related problem of length-generalization—training models to succeed on tasks longer than those seen during training (Fan et al., 2024; Cai et al., 2025).”

“Looped Transformers have also been studied as learners of algorithms (Yang et al., 2024) and for generalization to longer input sequences (Fan et al., 2025).”

“Recent studies demonstrate that recurrent depth improves algorithmic computation (Giannou et al., 2023), length generalization (Fan et al., 2025), and latent reasoning (Saunshi et al., 2025; Geiping et al., 2025; Hao et al., 2024), establishing looped computation as a promising direction for LLM…”

“Looped Transformers demonstrate this principle on algorithmic reasoning and length generalization (Saunshi et al., 2025; Fan et al., 2025), while recurrent-depth language models extend it to test-time computation in larger language models (Geiping et al., 2025).”

“Fan et al. (2025) demonstrate extrapolation to unseen input lengths on iterative algorithmic tasks, while Kohli et al. (2026) show that increasing recurrent depth at inference time can enable deeper implicit compositions.”

“…them: Universal Transformers (Dehghani et al., 2019) apply one shared block recurrently with a per-token halting policy, looped transformers (Fan et al., 2025) repeat a block for a fixed or learned number of iterations to improve length generalization, MoEUT (Csordás et al., 2024)…”

“…—Adaptive Computation Time (Graves, 2016), PonderNet (Banino et al., 2021), Universal Transformers (Dehghani et al., 2019), looped transformers (Fan et al., 2025), depth-adaptive and early-exit decoding (Elbayad et al., 2020; Schwartz et al., 2020), and token-level compute routing (Raposo et…”

“However, recurrent transformers trade parallelism for improved length generalization (Soulos et al., 2024; Fan et al., 2025).”

“…because they equip models with additional effective computational depth (Merrill & Sabharwal, 2023), and has spurred investigations into recurrent transformer architectures that allow models to adaptively determine their own depth (Dehghani et al., 2018; Saunshi et al., 2025; Fan et al., 2024).”

“Another direction employs looped transformers (Giannou et al., 2023; Fan et al., 2024), which recursively update hidden states à la recurrent networks.”

“…and retaining less than 10% of the parameters of standard layered trans-formers, have drawn significant attention for their superior performance in the in-context problem-solving paradigm as well as their enhanced ability to emulate iterative algorithms more effectively than vanilla LAMs [17]–[19].”

“Prior work demonstrates length generalisation by tying the loop count to task length [7], and shows that intermediate-state supervision can teach a pretrained model to perform successive rule lookups [28].”

“…et al., 2024), emulate programmable computation (Giannou et al., 2023), generalize in length when the loop matches the algorithm’s re-cursion (Fan et al., 2025), scale to billions of parameters (Geiping et al., 2025; Prairie et al., 2026), and supply a scaling axis for masked diffusion…”

“Fan et al. (2025) implement copying by iterating a fixed block for n steps, and Kraus et al. (2026) use a growing signpost alphabet to address copying and retrieval barriers in chain-of-thought reasoning.”

“Related work on looped Transformers studies shared-block computation for learning iterative algorithms [10] and length generalization on tasks with iterative solutions [11].”

“Beyond parameter efficiency (Lan et al., 2020), looping can turn even a shallow transformer into a programmable computer (Giannou et al., 2023) and may bias learning toward iterative algorithms (Fan et al., 2025; Yang et al., 2024).”

“…connect shared parameters to adaptive computation, programmable execution, and length generalization (Dehghani et al., 2019; Giannou et al., 2023; Fan et al., 2025); recurrent-depth language models extend this motivation to latent test-time compute (Geiping et al., 2025 (Geiger et al., 2021;…”

“Looped reasoners trade architectural depth for recurrent computation (Dehghani et al., 2019; Saunshi et al., 2025; Fan et al., 2025).”

“A natural approach is to train with a few loops on tractable instances, then tackle harder ones by running the block for more loops [Schwarzschild et al., 2021, Fan et al., 2025].”

“…that reasoning benefits from depth without a proportional increase in distinct parameters, looped transformers generalize to longer inputs than they were trained on (Fan et al., 2025), and recurrent networks extrapolate from easy to hard problems by iterating longer (Schwarzschild et al., 2021).”

“…(Dehghani et al., 2018) and cross-layer parameter sharing in ALBERT (Lan et al., 2019) through analyses of what a looped block can express and learn (Giannou et al., 2023; Yang et al., 2023; Fan et al., 2024; Xu & Sato, 2024; Gatmiry et al., 2024; Saunshi et al., 2025; Xu & Sato, 2025).”

“Such repeated computation supports length generalization and latent reasoning without intermediate token generation (Fan et al., 2025; Saunshi et al., 2025), with HRM, TRM, and Looped Flows exploring iterative refinement of internal states (Wang et al., 2025; Jolicoeur-Martineau, 2025; Suleymanzade…”

“Looped or recurrent methods (Dehghani et al., 2019; Giannou et al., 2023; Fan et al., 2025; Saunshi et al., 2025; Geiping et al., 2025; Zhu et al., 2025c) keep the number of slots fixed and apply the same layers over them several times, so the reasoning becomes an iterative computation within the…”

“By iteratively applying shared parameters, models can implicitly unroll reasoning steps in the latent space to solve complex logical tasks and generalize to longer sequences (Giannou et al., 2023; Yang et al., 2024; Fan et al., 2025; Saunshi et al., 2025; Zhu et al., 2025b; Liu et al., 2026).”

“…algorithms, length generalization, parameter-efficient language modeling, latent reasoning, and adaptive computational depth (Yang et al., 2024; Fan et al., 2025; Bae et al., 2025a; Geiping et al., 2025; Chen et al., 2025; Bae et al., 2025b; Zhu et al., 2026; Sapunov, 2026; Jeddi et al., 2026).”

“…and has become an axis of test-time compute: k layers looped L times match kL distinct layers on iterative reasoning and can simulate chain-of-thought steps in latent space (Saunshi et al., 2025); an adaptive trip count generalizes in length on copying, parity and addition (Fan et al., 2025); a 3 .”

“Recurrent-depth models make inference computation adjustable (Dehghani et al., 2019; Gi-annou et al., 2023; Yang et al., 2024a; Saunshi et al., 2025; Fan et al., 2025).”

“This approach has since been extended to various algorithmic and arithmetic tasks (Giannou et al., 2023; Kohli et al., 2026; Yang et al., 2024; Fan et al., 2025).”

“Later work studies length generalization with adaptive loop counts (Fan et al., 2025).”

“(Fan et al., 2025; Saunshi et al., 2025) Abstract reasoning tasks make this distinction concrete because a model must infer a transformation or structure from limited task evidence.”

“This effect is particularly pronounced for the recurrent model with α = 1 / L , consistent with the findings of (Fan et al., 2025), who show that looped Transformers with an adaptive number of steps can substantially improve length generalization.”

“In an effort to improve length generalization abilities, recent studies use several alternative approaches, including specialized positional encoding [18], specific training protocols [5], scratchpad prompting [1] and knowledge transfer from related tasks [2].”

“…2023; Jiang et al., 2026; Li et al., 2024; Hou et al., 2025), as well as their ability to learn algorithms with guarantees, often through the lens of length generalization (Giannou et al., 2023; Zhou et al., 2024; Hua et al., 2026; Malach et al., 2026; Chen et al., 2026; Fan et al., 2025).”

“On the theoretical side, looped transformers have been shown to length-generalize on algorithmic tasks by emulating iterative computation [8], and Saunshi et al. [9] show that looped models carry an inductive bias toward reasoning —matching far deeper iso-FLOP baselines on reasoning primitives…”

“In particular, the interaction kernel is reused along depth/time, so the model should be read as a shared-weight or looped-block surrogate [FDRL24, XS24, ACLY25, Che26] rather than as a full untied-depth transformer.”

“It has been widely discussed as a central challenge (Dubois et al., 2019; Newman et al., 2020; Cai et al., 2025; Fan et al., 2024; Jelassi et al., 2023; Anil et al., 2022), and is sometimes framed as a form of recursive composition or extrapolation Kim & Linzen (2020); Hupkes et al. (2020); Dziri…”

“Other work attempts to restore time depth through recurrent architectures [32], [33], [34], but the spatial geometry remains unchanged.”