The DOI matches its DataCite record; 52 of the 52 references it cites were found in a registry or an index; none of the 55 sources its claims cite is checked for support yet.
TransformerFAM: Feedback attention is working memory
Open accessAbstract
While Transformers have revolutionized deep learning, their quadratic attention complexity hinders their ability to process infinitely long inputs. We propose Feedback Attention Memory (FAM), a novel Transformer architecture that leverages a feedback loop to enable the network to attend to its own latent representations. This design fosters the emergence of working memory within the Transformer, allowing it to process indefinitely long sequences. TransformerFAM requires no additional weights, enabling seamless integration with pre-trained models. Our experiments show that TransformerFAM significantly improves Transformer performance on long-context tasks across various model sizes (1B, 8B, and 24B). These results showcase the potential to empower Large Language Models (LLMs) to process sequences of unlimited length.
Overview
CoChat Summary
TL;DR
TransformerFAM adds a feedback attention memory (FAM) loop to block sliding-window attention, letting each Transformer layer attend to its own latent representations as working memory without adding new weights. It improves long-context performance over TransformerBSWA across 1B, 8B, and 24B Flan-PaLM models, though the paper notes gains are not yet substantial.
Key findings
- TransformerFAM outperformed TransformerBSWA on long-context tasks across model sizes; for 8B models, ScrollsQasper rose from 12.4 to 18.5 and NarrativeQA from 18.4 to 19.3, while 24B models improved ScrollsQasper from 28.0 to 29.4 and ScrollsQuality from 55.4 to 58.0.
- On PassKey retrieval, TransformerFAM solved the task with filler contexts up to 260k tokens, whereas TransformerBSWA saturated after M2 and M12 dropped significantly after 20k tokens.
- On GPT-3 tasks, FAM marginally surpassed BSWA: 1B rank 61.0 vs 60.2 and generation 34.7 vs 33.9; 8B rank 74.0 vs 72.8 and generation 54.9 vs 54.3; 24B rank 78.5 vs 78.2 and generation 63.4 vs 62.6.
- Ablations found FAM length 64 performed best, with performance declining above 64, and Random State Passing probability 0.8 gave the best PG-19 accuracy.
Methods
- Architecture: TransformerFAM builds on Block Sliding Window Attention (BSWA), appending a feedback attention memory that is updated block-wise and propagated recursively; default block size 1024, FAM length 64, memory segment 3, with inference complexity O(L) and memory O(1).
- Training: Reused 1B, 8B, and 24B Flan-PaLM checkpoints, applied LoRA rank 64 to attention and FF layers, fine-tuned for 50k steps on Flan instruction data packed to 8.5k tokens using Adafactor and TPUv5 cores (32 for 1B, 64 for 8B, 128 for 24B) with minibatch size 128.
- Evaluation: Compared TransformerFAM against TransformerBSWA on PassKey retrieval, long-context tasks (NarrativeQA, Scrolls-Qasper, Scrolls-Quality, XLSum, PG-19, Isabelle), and GPT-3 tasks, with additional comparisons to TransformerXL, RMT, and AutoCompressors.
Limitations
- The paper states that performance improvements on long-context tasks are “not substantial,” indicating room for further enhancement of working memory mechanisms.
- The authors state they could not find training infrastructure or data suitable for training memory due to the “curse of IID,” so they used Flan instruction data “as a last resort” rather than ideal long continuous documents.
- The paper notes that full fine-tuning caused catastrophic forgetting, so LoRA was used; it also states that using more resources and a larger minibatch might have yielded better results than reported.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures










Verification
Claims the paper makes
0 Valid · 7 Not valid · 33 Not checked · 114 Own results and interpretations
None of the sources cited for this claim have been checked yet.
In the paper
“While this modification might seem to burden training memory and computation, it does not significantly impact performance in practice.”
§ TransformerFAM: Feedback attention is working memory > Algorithm 1
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In Fig. 5, FAM shows significantly better PG-19 accuracy than FAM-RSP.”
§ TransformerFAM: Feedback attention is working memory > B.4.5 Random State Passing
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“FAM solved the PassKey retrieval task”
- Also stated in: § H Broader Impacts · plain
“TransformerFAM successfully solves the task.”
§ TransformerFAM: Feedback attention is working memory > B.4.7 TransformerXL vs. TransformerBSWA
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § H Broader Impacts · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The performance of M12 also drops significantly after 20k tokens.”
§ TransformerFAM: Feedback attention is working memory > H Broader Impacts
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“It did not help at all even when the FAM length was 256 or longer.”
§ TransformerFAM: Feedback attention is working memory > C.4 Diversity Loss
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
76 more claims
Create a free account to see all 154 claims and every source verdict.
- It may state two separate claims as one. 42
- Probably not a substantial claim. 48
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ TransformerFAM: Feedback attention is working memory > Abstract
“TransformerFAM requires no additional weights, enabling seamless integration with pre-trained models.”
§ TransformerFAM: Feedback attention is working memory > B.1 FAM initialization
“In summary, we learned the initial FAM using prompt tuning, which only adds a very small number of weights of FAM length to the entire model.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ TransformerFAM: Feedback attention is working memory > B.4.1 FAM length
“Interestingly, performance declined when FAM length exceeded 64, suggesting that information compression is more effective with limited space.”
§ TransformerFAM: Feedback attention is working memory > Additional information
“| 64 | 7.2 | 27.9 | 15.9 | | 256 | 5.1 | 26.5 | 16.0 | | 1024 | 5.3 | 26.3 | 16.0 |”
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 2
- Xiong, Ruibin u.a.(2020): On layer normalization in the transformer architectureIn: Interna- tional Conference on Machine Learning10524-10533.
- Katharopoulos, Angelos / Vyas, Apoorv / Pappas, Nikolaos / Fleuret, François(2020): Trans- formers are rnns: Fast autoregressive transformers with linear attentionIn: International conference on machine learning5156-5165.
Cited by
Generative Artificial Intelligence in Anatomic Pathology
Brodsky, V., Ullah, E., Bychkov, A., Song, A. H., Walk, E. E., Louis, P., Rasool, G., Singh, R. S., Mahmood, F., Bui, M. M., et al. (2025) · Archives of Pathology & Laboratory Medicine · doi:10.5858/arpa.2024-0215-ra · Cited by 48
Needle in the Haystack for Memory Based Large Language ModelsPreprint
Nelson, E., Κόλλιας, Γ., Das, P., Chaudhury, S., & Dan, S. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2407.01437 · Cited by 61
End-to-End Speech-to-Text Translation: A Survey
Sethiya, N., & Maurya, C. K. (2024) · Computer Speech & Language · doi:10.1016/j.csl.2024.101751 · Cited by 48
InfiniPot: Infinite Context Processing on Memory-Constrained LLMs
Kim, M., Shim, K., Choi, J., & Chang, S. (2024) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2024.emnlp-main.897 · Cited by 39
Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention
Munkhdalai, T., Faruqui, M., & Gopal, S. (2024) · arXiv · doi:10.48550/arxiv.2404.07143 · Cited by 23
Metadata
- Authors
- Hwang, D., Wang, W., Huo, Z., Sim, K. C., & Mengibar, P. M.
- Affiliations
- Google (United States) (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2024
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2404.09173
- OpenAlex
- W4394867741
- Cited by
- 26
- References
- 0
- Citations by year
- 2026: 10 · 2025: 8 · 2024: 8
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- transformer models, working memory, long-context modeling, feedback loops, self-attention, quadratic complexity, long sequences, large language models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-11-14, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-14 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: TransformerFAM: Feedback attention is working memory. https://app.cochat.ai/papers/10.48550/arxiv.2404.09173
- The paper
- Hwang, D., Wang, W., Huo, Z., Sim, K. C., & Mengibar, P. M. (2024). TransformerFAM: Feedback attention is working memory. arXiv. https://doi.org/10.48550/arxiv.2404.09173
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.