Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its DataCite record; 52 of the 52 references it cites were found in a registry or an index; none of the 55 sources its claims cite is checked for support yet.

TransformerFAM: Feedback attention is working memory

Open access

Abstract

While Transformers have revolutionized deep learning, their quadratic attention complexity hinders their ability to process infinitely long inputs. We propose Feedback Attention Memory (FAM), a novel Transformer architecture that leverages a feedback loop to enable the network to attend to its own latent representations. This design fosters the emergence of working memory within the Transformer, allowing it to process indefinitely long sequences. TransformerFAM requires no additional weights, enabling seamless integration with pre-trained models. Our experiments show that TransformerFAM significantly improves Transformer performance on long-context tasks across various model sizes (1B, 8B, and 24B). These results showcase the potential to empower Large Language Models (LLMs) to process sequences of unlimited length.

Overview

CoChat Summary

TL;DR

TransformerFAM adds a feedback attention memory (FAM) loop to block sliding-window attention, letting each Transformer layer attend to its own latent representations as working memory without adding new weights. It improves long-context performance over TransformerBSWA across 1B, 8B, and 24B Flan-PaLM models, though the paper notes gains are not yet substantial.

Key findings

  • TransformerFAM outperformed TransformerBSWA on long-context tasks across model sizes; for 8B models, ScrollsQasper rose from 12.4 to 18.5 and NarrativeQA from 18.4 to 19.3, while 24B models improved ScrollsQasper from 28.0 to 29.4 and ScrollsQuality from 55.4 to 58.0.
  • On PassKey retrieval, TransformerFAM solved the task with filler contexts up to 260k tokens, whereas TransformerBSWA saturated after M2 and M12 dropped significantly after 20k tokens.
  • On GPT-3 tasks, FAM marginally surpassed BSWA: 1B rank 61.0 vs 60.2 and generation 34.7 vs 33.9; 8B rank 74.0 vs 72.8 and generation 54.9 vs 54.3; 24B rank 78.5 vs 78.2 and generation 63.4 vs 62.6.
  • Ablations found FAM length 64 performed best, with performance declining above 64, and Random State Passing probability 0.8 gave the best PG-19 accuracy.

Methods

  • Architecture: TransformerFAM builds on Block Sliding Window Attention (BSWA), appending a feedback attention memory that is updated block-wise and propagated recursively; default block size 1024, FAM length 64, memory segment 3, with inference complexity O(L) and memory O(1).
  • Training: Reused 1B, 8B, and 24B Flan-PaLM checkpoints, applied LoRA rank 64 to attention and FF layers, fine-tuned for 50k steps on Flan instruction data packed to 8.5k tokens using Adafactor and TPUv5 cores (32 for 1B, 64 for 8B, 128 for 24B) with minibatch size 128.
  • Evaluation: Compared TransformerFAM against TransformerBSWA on PassKey retrieval, long-context tasks (NarrativeQA, Scrolls-Qasper, Scrolls-Quality, XLSum, PG-19, Isabelle), and GPT-3 tasks, with additional comparisons to TransformerXL, RMT, and AutoCompressors.

Limitations

  • The paper states that performance improvements on long-context tasks are “not substantial,” indicating room for further enhancement of working memory mechanisms.
  • The authors state they could not find training infrastructure or data suitable for training memory due to the “curse of IID,” so they used Flan instruction data “as a last resort” rather than ideal long continuous documents.
  • The paper notes that full fine-tuning caused catastrophic forgetting, so LoRA was used; it also states that using more resources and a larger minibatch might have yielded better results than reported.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2404.09173

Cites

0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Figure 1 Comparison of query-key attention masks for Sliding Window Attention (SWA) variants. (a) Sliding Window Attention: Attention is restricted to the current window = 3 . (b) Block Sliding Window Attention (BSWA) (block size = 2 , memory segment = 1 ): Attention is allowed to previous blocks within the memory segment. (c) BSWA (block size = 2 , memory segment = 2 ): The memory segment is expanded, allowing attention to a larger past context. (d) Illustrates the receptive field of BSWA (block size = 2 , memory segment = 1 , depth = 4 ): The region within the curly braces represents the receptive field.
Figure 1 Comparison of query-key attention masks for Sliding Window Attention (SWA) variants. (a) Sliding Window Attention: Attention is restricted to the current window = 3 . (b) Block Sliding Window Attention (BSWA) (block size = 2 , memory segment = 1 ): Attention is allowed to previous blocks within the memory segment. (c) BSWA (block size = 2 , memory segment = 2 ): The memory segment is expanded, allowing attention to a larger past context. (d) Illustrates the receptive field of BSWA (block size = 2 , memory segment = 1 , depth = 4 ): The region within the curly braces represents the receptive field.CC-BY · From the PDF
Figure 2 Comparison of attention patterns in Transformer layer. (a) TransformerBSWA: Input query attends to the current block and two memory segments, providing past context. (b) TransformerFAM: Input query attends to the current block, memory segments, and past FAM (green lines). FAM query (copied from previous FAM, blue dash arrow) compresses the current block to update FAM. This feedback loop enables information compression and propagation over indefinite horizon, which is working memory. Fig. 4 shows in detail how the dynamic process occurs over time.
Figure 2 Comparison of attention patterns in Transformer layer. (a) TransformerBSWA: Input query attends to the current block and two memory segments, providing past context. (b) TransformerFAM: Input query attends to the current block, memory segments, and past FAM (green lines). FAM query (copied from previous FAM, blue dash arrow) compresses the current block to update FAM. This feedback loop enables information compression and propagation over indefinite horizon, which is working memory. Fig. 4 shows in detail how the dynamic process occurs over time.CC-BY · From the PDF
Image 3
CC-BY · From the PDF
Figure 4 Visualization of self-attention during inference over time. (A) Self-attention pattern of TransformerBSWA layer with memory segment size of 1. (B) Self-attention pattern with FAM added.
Figure 4 Visualization of self-attention during inference over time. (A) Self-attention pattern of TransformerBSWA layer with memory segment size of 1. (B) Self-attention pattern with FAM added.CC-BY · From the PDF
Figure 5 PG-19 accuracy for various ablation studies such as RPO (Random Position Offset), RSP (Random State Passing) and Prefix FAM tuning over different base frequency.
Figure 5 PG-19 accuracy for various ablation studies such as RPO (Random Position Offset), RSP (Random State Passing) and Prefix FAM tuning over different base frequency.CC-BY · From the PDF
Figure 6 PG-19 accuracy for different probability of Random State Passing.
Figure 6 PG-19 accuracy for different probability of Random State Passing.CC-BY · From the PDF
Figure 7 PassKey Accuracy: FAM maintains performance with long sequences, outperforming BSWA, RMT and AutoCompressors.
Figure 7 PassKey Accuracy: FAM maintains performance with long sequences, outperforming BSWA, RMT and AutoCompressors.CC-BY · From the PDF
Figure 8 Convert TransformerBSWA to TransformerFM
Figure 8 Convert TransformerBSWA to TransformerFMCC-BY · From the PDF
Figure 9 PassKey Retrieval Accuracy Using BSWA (M1,2,4), Feedback Memory Segments (FM1,2,4), Static Summary Tokens (SUM FAM) and FAM
Figure 9 PassKey Retrieval Accuracy Using BSWA (M1,2,4), Feedback Memory Segments (FM1,2,4), Static Summary Tokens (SUM FAM) and FAMCC-BY · From the PDF
Figure 11 Selected attention map of TransformerFAM with block size of 256 and FAM length of 8. The vertical axis of the attention map represents the Query, and the horizontal axis represents the Key. The FAM is prepended, and is located at the bottom left corner.
Figure 11 Selected attention map of TransformerFAM with block size of 256 and FAM length of 8. The vertical axis of the attention map represents the Query, and the horizontal axis represents the Key. The FAM is prepended, and is located at the bottom left corner.CC-BY · From the PDF

Verification

Claims the paper makes

0 Valid · 7 Not valid · 33 Not checked · 114 Own results and interpretations

76 more claims

Create a free account to see all 154 claims and every source verdict.

See all claims — free account
  • It may state two separate claims as one. 42
  • Probably not a substantial claim. 48

Create a free account to see these claims.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

2 of 2

Citations

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 2

  1. Xiong, Ruibin u.a.(2020): On layer normalization in the transformer architectureIn: Interna- tional Conference on Machine Learning10524-10533.
  2. Katharopoulos, Angelos / Vyas, Apoorv / Pappas, Nikolaos / Fleuret, François(2020): Trans- formers are rnns: Fast autoregressive transformers with linear attentionIn: International conference on machine learning5156-5165.

Cited by

See all 26

Metadata

Authors
Hwang, D., Wang, W., Huo, Z., Sim, K. C., & Mengibar, P. M.
Affiliations
Google (United States) (US)
Venue
arXiv
Publisher
arXiv
Published
2024
ISSN
2331-8422 (ISSN-L)
OpenAlex
W4394867741
Cited by
26
References
0
Citations by year
2026: 10 · 2025: 8 · 2024: 8
Type
posted content
Language
en
Open access
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
Keywords
transformer models, working memory, long-context modeling, feedback loops, self-attention, quadratic complexity, long sequences, large language models

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-11-14, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-11-14 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: TransformerFAM: Feedback attention is working memory. https://app.cochat.ai/papers/10.48550/arxiv.2404.09173
The paper
Hwang, D., Wang, W., Huo, Z., Sim, K. C., & Mengibar, P. M. (2024). TransformerFAM: Feedback attention is working memory. arXiv. https://doi.org/10.48550/arxiv.2404.09173

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.