The DOI matches its DataCite record, with a small difference in title or year; the references it cites are not checked yet; none of the 43 sources its claims cite is checked for support yet.
Memformer: A Memory-Augmented Transformer for Sequence Modeling
Open accessAbstract
Transformers have reached remarkable success in sequence modeling. However, these models have efficiency issues as they need to store all the history token-level representations as memory. We present Memformer, an efficient neural network for sequence modeling, that utilizes an external dynamic memory to encode and retrieve past information. Our model achieves linear time complexity and constant memory space complexity when processing long sequences. We also propose a new optimization scheme, memory replay back-propagation (MRBP), which promotes long-range back-propagation through time with a significantly reduced memory requirement. Experimental results show that Memformer has achieved comparable performance compared to the baselines by using 8.1x less memory space and 3.2x faster on inference. Analysis of the attention pattern shows that our external memory slots can encode and retain important information through timesteps.
Checked 2026-11-14 · 4 references not checked yet
Overview
CoChat Summary
TL;DR
Memformer is a memory-augmented Transformer that uses fixed-size external dynamic memory and recurrence to model long sequences with linear time and constant memory complexity. It also introduces memory replay backpropagation (MRBP) to train recurrent memory with reduced memory cost, achieving comparable performance to Transformer-XL and Compressive Transformer on image generation and language modeling.
Key findings
- Memformer achieved comparable or better performance with much lower cost: on autoregressive MNIST image generation, Memformer (4 encoder + 8 decoder) reached perplexity 1.555 with 5.0B FLOPs, versus Transformer-XL with memory=784 at perplexity 1.611 with 49.1B FLOPs.
- On WikiText-103 language modeling, Memformer (4 encoder + 16 decoder, memory=1,024) reached perplexity 22.74 with 54B FLOPs, versus Transformer-XL memory=1,024 at perplexity 23.67 with 168B FLOPs.
- The paper reports Memformer uses 8.1x less memory space and is 3.2x faster on inference in its efficiency comparisons.
- MRBP reduced training GPU memory to 7,229 MB versus 16,177 MB for BPTT and 9,885 MB for gradient checkpointing, with relative speed x0.90 versus x1.00 for BPTT and x0.48 for gradient checkpointing.
- Memory writer analysis found that around 60% to 80% of memory slots in the middle of a document attended mainly to themselves, while only 5% to 10% attended completely to input tokens later; one slot had forgetting bias magnitude 3.20 versus 1.15 for others.
Methods
- Memformer processes sequences at the segment level with a Transformer encoder-decoder: the encoder reads from and writes to a fixed-size external dynamic memory, and the decoder predicts the next segment conditioned on the current segment and memory.
- Memory reading uses cross-attention from input queries to memory keys/values; memory writing uses slot attention where each memory slot attends only to itself and encoder outputs, plus a Biased Memory Normalization forgetting mechanism.
- Training uses Memory Replay Back-Propagation (MRBP), which replays memory at each timestep to reduce activation storage during backpropagation through time.
- Experiments cover MNIST image generation (28×28 images reshaped to 784 tokens, 256 vocabulary) and WikiText-103 language modeling (28K articles, average 3.6K tokens), with results reported as median from three trials.
Limitations
- The paper states that due to limited computational resources, the authors were unable to experiment on the more recent PG19 dataset.
- The paper notes there is no official implementation of Compressive Transformer, so they re-implemented a version of the same size for comparison.
- Evident from the text: evaluation is limited to MNIST image generation and WikiText-103 language modeling, and the memory writer analysis is qualitative/visual rather than a broad quantitative probe.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
0 Valid · 4 Not valid · 28 Not checked · 100 Own results and interpretations
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“the forgetting vector's bias for m 355 had a larger magnitude (3.20) compared to some other slots (1.15)”
§ Memformer: A Memory-Augmented Transformer for Sequence Modeling > 4.3.1 Memory Writer Analysis
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our Memformer with 4 layers of encoder and 8 layers of decoder achieved the best performance (1.555)”
§ Memformer: A Memory-Augmented Transformer for Sequence Modeling > 4.2 Autoregressive Image Generation
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“and achieved better efficiency than Transformer-XL.”
§ Memformer: A Memory-Augmented Transformer for Sequence Modeling > 4.1 Computation and Memory Cost
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“the performance dropped after the memory size is reduced to 512.”
§ Memformer: A Memory-Augmented Transformer for Sequence Modeling > 4.3 Language Modeling
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“If we reduce the number of decoder layers in Memformer (4 encoder+4 decoder), the performance dropped as shown (1.628).”
§ Memformer: A Memory-Augmented Transformer for Sequence Modeling > 4.2 Autoregressive Image Generation
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
73 more claims
Create a free account to see all 132 claims and to verify the 12 references still open.
- It may state two separate claims as one. 29
- The quote only partly states this claim. 2
- Probably not a substantial claim. 31
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Memformer: A Memory-Augmented Transformer for Sequence Modeling > 4.2 Autoregressive Image Generation
“For the baselines, LSTM had 4 layers and 512 hidden size.”
§ Memformer: A Memory-Augmented Transformer for Sequence Modeling > 4.2 Autoregressive Image Generation
“All the models had the same 128 hidden size, 4 attention heads, 32 head size, and 256 feedforward size.”
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 16
- Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Ben- gio. 2015. Neural machine translation by jointly learning to align and translate. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
- Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. CoRR, abs/2004.05150. · arXiv
- Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. 2016. Training deep nets with sublinear memory cost. CoRR, abs/1604.06174. · arXiv
- Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamás Sar- lós, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, David Belanger, Lucy Colwell, and Adrian Weller. 2020. Rethinking attention with per- formers. CoRR, abs/2009.14794. · arXiv
- Junyoung Chung, Caglar Gulcehre, Kyunghyun Cho, and Yoshua Bengio. 2014. Empirical evaluation of gated recurrent neural networks on sequence mod- eling. In NIPS 2014 Workshop on Deep Learning, December 2014.
- Alex Graves, Greg Wayne, and Ivo Danihelka. 2014. Neural turing machines. CoRR, abs/1410.5401. · arXiv
- Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pap- pas, and François Fleuret. 2020. Transformers are rnns: Fast autoregressive transformers with linear attention. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 5156-5165.
- PMLR.
- Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. 2020. Reformer: The efficient transformer. In 8th International Conference on Learning Representa- tions, ICLR 2020, Addis Ababa, Ethiopia, April 26- 30, 2020. OpenReview.net.
- Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2017. Pointer sentinel mixture mod- els. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. Open- Review.net.
- Jack W. Rae, Anna Potapenko, Siddhant M. Jayakumar, Chloe Hillier, and Timothy P. Lillicrap. 2020. Com- pressive transformers for long-range sequence mod- elling. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
- David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams. 1988. Learning Representations by Back- Propagating Errors, page 696-699. MIT Press, Cam- bridge, MA, USA.
- Jos van der Westhuizen and Joan Lasenby. 2018. The unreasonable effectiveness of the forget gate. CoRR, abs/1804.04849. · arXiv
- Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Pro- cessing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 4-9 December 2017, Long Beach, CA, USA, pages 5998-6008.
- Sinong Wang, Belinda Z. Li, Madian Khabsa, Han Fang, and Hao Ma. 2020. Linformer: Self-attention with linear complexity. CoRR, abs/2006.04768. · arXiv
- Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago On- tanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. 2020. Big bird: Transformers for longer sequences. Advances in Neural Information Processing Systems, 33.
Cited by
Lin, T., Wang, Y., Liu, X., & Qiu, X. (2022) · AI Open · doi:10.1016/j.aiopen.2022.10.001 · Cited by 1,422
Resource-efficient Algorithms and Systems of Foundation Models: A Survey
Xu, M., Cai, D., Yin, W., Wang, S., Jin, X., & Liu, X. (2024) · ACM Computing Surveys · doi:10.1145/3706418 · Cited by 256
LLM Inference Unveiled: Survey and Roofline Model InsightsPreprint
Yuan, Z., Shang, Y., Zhou, Y., Dong, Z., Zhou, Z., Xue, C., Wu, B., Li, Z., Gu, Q., Lee, Y. J., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2402.16363 · Cited by 246
GENA-LM: a family of open-source foundational DNA language models for long sequences
Fishman, V., Kuratov, Y., Shmelev, A., Petrov, M., Penzar, D., Shepelin, D., Chekanov, N., Kardymon, O. L., & Burtsev, M. (2024) · Nucleic Acids Research · doi:10.1093/nar/gkae1310 · Cited by 171
Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive SurveyPreprint
Huang, Y.-P., Xu, J., Lai, J., Jiang, Z., Chen, T., Li, Z., Yao, Y., Ma, X., Yang, L., Chen, H., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2311.12351 · Cited by 152
A Survey on Long Text Modeling with TransformersPreprint
Dong, Z., Tang, T., Li, J., & Zhao, W. X. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2302.14502 · Cited by 81
Recursively summarizing enables long-term dialogue memory in large language models
Wang, Q., Fu, Y., Cao, Y., Wang, S., Tian, Z., & Ding, L. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.130193 · Cited by 55
Bulatov, A., Kuratov, Y., & Burtsev, M. S. (2022) · arXiv · doi:10.48550/arxiv.2207.06881 · Cited by 55
Wittkuhn, L., Chien, S., Hall-McMaster, S., & Schuck, N. W. (2021) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2021.08.002 · Cited by 50
Pawar, S., Towhidul, S. M., Zaman, S. M. M., Jain, V., Chadha, A. R., & Das, A. (2024) · arXiv (Cornell University) · doi:10.13140/rg.2.2.16805.06884 · Cited by 44
MambaEVT: Event Stream-Based Visual Object Tracking Using State Space Model
Wang, X., Wang, C., Wang, S., Wang, X., Zhao, Z., Zhu, L., & Jiang, B. (2025) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2025.3588533 · Cited by 35
P3IV: Probabilistic Procedure Planning from Instructional Videos with Weak Supervision
Zhao, H., Hadji, I., Dvornik, N., Derpanis, K. G., Wildes, R. P., & Jepson, A. D. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00295 · Cited by 34
HMT: Hierarchical Memory Transformer for Efficient Long Context Language Processing
He, Z., Cao, Y., Qin, Z., Prakriya, N., Sun, Y., & Cong, J. (2025) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2025.naacl-long.410 · Cited by 30
Relative Positional Encoding for Transformers with Linear ComplexityPreprint
Antoine Liutkus, A., Cífka, O., Wu, S.-L., Şimşekli, U., Yang, Y., & Gaël Richard, A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.08399 · Cited by 28
Anatomically accurate cardiac segmentation using Dense Associative Networks
Ullah, Z., & Kim, J. (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2025.112742 · Cited by 27
Titans: Learning to Memorize at Test Time
Behrouz, A., Zhong, P., & Mirrokni, V. (2025) · arXiv · doi:10.48550/arxiv.2501.00663 · Cited by 24
“To address this issue, various studies discuss the importance of a memory so it can help the model to transfer information across chunks [94–99, 98, 100, 101, 96, 97].”
Recurrent Action Transformer with MemoryPreprint
Cherepanov, E., Staroverov, A., Kovalev, A. K., & Panov, A. I. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2306.09459 · Cited by 19
Liang, R., Li, Y., Yi, Y., Zhou, J., & Li, X. (2025) · Pattern Recognition · doi:10.1016/j.patcog.2025.111789 · Cited by 17
Omidi, P., Huang, X., Laborieux, A., Nikpour, B., Shi, T., & Eshaghi, A. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2508.10824 · Cited by 15
A Roadmap for Big ModelPreprint
Yuan, S., Zhao, H., Zhao, S., Leng, J., Liang, Y., Wang, X., Yu, J., Lv, X., Shao, Z., He, J., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2203.14101 · Cited by 15
Self-citationRULER: What's the Real Context Size of Your Long-Context Language Models?
Hsieh, C.-P., Sun, S., Kriman, S., Acharya, S., Rekesh, D., Jia, F., Zhang, Y., & Ginsburg, B. (2024) · arXiv · doi:10.48550/arxiv.2404.06654 · Cited by 1,094
“This can be achieved by caching previous context using recurrence mechanism (Zhang et al., 2024a; Bulatov et al., 2023; Martins et al., 2022; Wu et al., 2022), retrieving relevant information from context (Xu et al., 2024a; Mohtashami & Jaggi, 2023; Wang et al., 2024; Tworkowski et al., 2024; Xiao…”
MethodBeyond Attention: Breaking the Limits of Transformer Context Length with Recurrent Memory
Bulatov, A., Kuratov, Y., Kapushev, Y., & Burtsev, M. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i16.29722 · Cited by 12
Evaluating Memory Structure in LLM AgentsPreprint
Shutova, A., Olenina, A., Vinogradov, I., & Sinitsin, A. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2602.11243 · Cited by 11
ELMUR: External Layer Memory with Update/Rewrite for Long-Horizon RL ProblemsPreprint
Cherepanov, E., Kovalev, A. K., & Panov, A. I. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2510.07151 · Cited by 9
Yu, Y., Zhang, Q., Qiao, L., Yin, D., Li, F., Wang, J., Chen, Z., Zheng, S., Liang, X., & Sun, X. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2504.04713 · Cited by 9
FashionNTM: Multi-turn Fashion Image Retrieval via Cascaded Memory
Pal, A., Wadhwa, S., Jaiswal, A., Zhang, X., Wu, Y., Chada, R., Natarajan, P., & Christensen, H. I. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01040 · Cited by 7
Diable: Efficient Dialogue State Tracking as Operations on Tables
Lesci, P., Fujinuma, Y., Hardalov, M., Shang, C., & Màrquez, L. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.615 · Cited by 7
The CAP Principle for LLM Serving: A Survey of Long-Context Large Language Model ServingPreprint
Zeng, P., Ning, Z., Zhao, J., Cui, W., Xu, M., Guo, L., Chen, X., & Shan, Y. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2405.11299 · Cited by 7
BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystack
Kuratov, Y., Bulatov, A., Anokhin, P., Rodkin, I., Sorokin, D., Sorokin, A., & Burtsev, M. (2024) · arXiv · doi:10.48550/arxiv.2406.10149 · Cited by 19
“Many different architectures adding recurrence to transformers have been proposed (Wu et al., 2022a; Lei et al., 2020; Fan et al., 2020).”
BackgroundHIINT: Historical, Intra- and Inter- personal Dynamics Modeling with Cross-person Memory Transformer
Kim, Y., Lee, D. W., Liang, P. P., Alghowinem, S., Breazeal, C., & Park, H. W. (2023) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3577190.3614122 · Cited by 6
Episodic Memory in AI Agents Poses Risks that Should be Studied and Mitigated
DeChant, C. (2025) · 2025 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML) · doi:10.1109/satml64287.2025.00024 · Cited by 5
Episodic memory in AI agents poses risks that should be studied and mitigated
DeChant, C. (2025) · arXiv · doi:10.48550/arxiv.2501.11739 · Cited by 5
Alternating Updates for Efficient Transformers
Baykal, C., Cutler, D. J., Dikkala, N., Ghosh, N., Panigrahy, R., & Wang, X. E. (2023) · neural information processing systems · doi:10.52202/075280-3352 · Cited by 5
A Lexical-aware Non-autoregressive Transformer-based ASR Model
Lin, C.-Y., & Chen, K. (2023) · Interspeech (USB) · doi:10.21437/interspeech.2023-863 · Cited by 5
Sound Source Separation Using Latent Variational Block-Wise Disentanglement
Helwani, K., Togami, M., Smaragdis, P., & Goodwin, M. M. (2024) · IEEE International Conference on Acoustics, Speech, and Signal Processing Workshops (ICASSPW) · doi:10.1109/icasspw62465.2024.10626303 · Cited by 4
MemATr: An Efficient and Lightweight Memory-augmented Transformer for Video Anomaly Detection
Chang, J., Zhen, P., Yan, X., Yang, Y., Gao, Z., & Chen, H. (2025) · ACM Transactions on Embedded Computing Systems · doi:10.1145/3719203 · Cited by 4
Explain My Surprise: Learning Efficient Long-Term Memory by Predicting Uncertain Outcomes
Sorokin, A., Buzun, N., Pugachev, L., & Burtsev, M. (2022) · neural information processing systems · doi:10.52202/068431-2672 · Cited by 4
Multi-Modal Retrieval For Large Language Model Based Speech Recognition
Gourav, A., Kolehmainen, J., Shivakumar, P. G., Gu, Y., Strimel, G. P., Gandhe, A., Rastrow, A., & Bulyko, I. (2024) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2024.findings-acl.262 · Cited by 4
Explainable Hallucination Mitigation in Large Language Models: A Survey
Deng, W., Li, J., Zhang, H., Li, J., Deng, Z., Cheng, D., Feng, Z., & Zeng, H. (2026) · Wiley Interdisciplinary Reviews Data Mining and Knowledge Discovery · doi:10.1002/widm.70110 · Cited by 3
Token Turing Machines are Efficient Vision Models
Jajal, P., Eliopoulos, N., Chou, B. S.-H., Thiravathukal, G. K., Davis, J. C., & Lu, Y.-H. (2025) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv61041.2025.00767 · Cited by 3
You Only Use Reactive Attention Slice For Long Context RetrievalPreprint
Soh, Y. J., Huang, H., Tian, Y., & Zhao, J. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2409.13695 · Cited by 3
Vision encoders should be image size agnostic and task drivenPreprint
Prisadnikov, N., Paudel, D. P., Fu, Y., & Gool, L. V. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2508.16317 · Cited by 3
An, S., Sung, J., Park, W., Park, C., & Seo, P. H. (2025) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2025.naacl-long.524 · Cited by 2
Yu, Y., Zhang, Q., Qiao, L., Yin, D., Li, F., Wang, J., Xi, C., Zheng, S., Liang, X., & Sun, X. (2025) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2025.emnlp-main.1497 · Cited by 2
Egocentric Online Action Segmentation via Parametric Context Memory Learning
Jiang, X., Xu, X., Liu, C., Wang, Z., Song, J., Sun, Z., Cichocki, A. S., & Shen, H. T. (2026) · IEEE Transactions on Image Processing · doi:10.1109/tip.2026.3714835 · Cited by 2
HiFormer: Hierarchical transformer for grounded situation recognition
Pan, Y. (2024) · Applied and Computational Engineering · doi:10.54254/2755-2721/49/20241075 · Cited by 2
Mastering Long-Context Multi-Task Reasoning with Transformers and Recurrent Memory
Bulatov, A., Kuratov, Y., & Burtsev, M. (2024) · Optical Memory and Neural Networks · doi:10.3103/s1060992x24700735 · Cited by 2
Sokol, O. (2025) · Technologies and Engineering · doi:10.30857/2786-5371.2025.1.3 · Cited by 2
Combining Transformers with Natural Language ExplanationsPreprint
Ruggeri, F., Lippi, M., & Torroni, P. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4531749 · Cited by 2
A Framework for Inference Inspired by Human Memory MechanismsPreprint
Zeng, X., Lin, J., Hu, P., Huang, R., & Zhang, Z. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.09297 · Cited by 2
Toward generalizable learning of all (linear) first-order methods via memory augmented TransformersPreprint
Dutta, S., & Sra, S. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2410.07263 · Cited by 2
MemAgent: Reshaping Long-Context LLM with Multi-Conv RL-based Memory Agent
Yu, H., Chen, T., Feng, J., Chen, J., Dai, W., Yu, Q., Zhang, Y.-Q., Ma, W.-Y., Liu, J., Wang, M., et al. (2025) · arXiv · doi:10.48550/arxiv.2507.02259 · Cited by 1
Position: Episodic Memory is the Missing Piece for Long-Term LLM Agents
Pink, M., Wu, Q., Vo, V. A., Turek, J., Mu, J., Huth, A., & Toneva, M. (2025) · arXiv · doi:10.48550/arxiv.2502.06975 · Cited by 1
Self-citationUncertainty Guided Global Memory Improves Multi-Hop Question Answering
Sagirova, A., & Burtsev, M. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.262 · Cited by 1
Learnable Temporal Sparse Memory iTransformer: Revisiting Sparsity and Memory in Transformers
Yakoi, P. S., Meng, X., Yu, C., Odeh, V. A., Suleman, D., Zhang, Y., & Zhang, X. (2025) · IEEE Access · doi:10.1109/access.2025.3649490 · Cited by 1
Improving Autoregressive NLP Tasks via Modular Linearized Attention
Agostinelli, V., & Chen, L. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-43421-1_6 · Cited by 1
Yakoi, P. S., Meng, X., Yu, C., & Zhang, X. (2025) · International Conference on Computing, Networks and Internet of Things (CNIOT) · doi:10.1109/cniot65435.2025.11070699 · Cited by 1
Heterogenous Memory Augmented Neural NetworksPreprint
Qiu, Z., Liu, Z., Yan, S., Zhang, S., & Fu, J. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.10909 · Cited by 1
Goal-Based Vision-Language DrivingPreprint
Patapati, S., & Srinivasan, T. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2507.23042 · Cited by 1
Filipek, A. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2510.03561 · Cited by 1
Mixture of Chapters: Scaling Learnt Memory in TransformersPreprint
Tibrewal, T. P., Saha, P., Meda, A., Singh, K., & Moturi, P. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2603.21096 · Cited by 1
Deep learning methods for music style transfer
Cífka, O. (2021) · theses.fr (ABES) · doi:10.70675/fb591beeza958z4897z936dz180fad412ad9 · Cited by 1
Beyond a Million Tokens: Benchmarking and Enhancing Long-Term Memory in LLMs
Tavakoli, M., Salemi, A., Ye, C., Abdalla, M., Zamani, H., & Mitchell, J. R. (2025) · arXiv · doi:10.48550/arxiv.2510.27246
The Power of External Memory in Increasing Predictive Model CapacityPreprint
Baykal, C., Cutler, D. J., Dikkala, N., Ghosh, N., Panigrahy, R., & Wang, X. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2302.00003
Abstract2Appendix: Academic Reviews Enhance LLM Long-Context Capabilities
Li, S., Kampa, K., Lin, R., Li, B., & Pei, S. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2411.05232
IN-CONTEXT LEARNING AS GENERAL-PURPOSE LEARNING: A COMPREHENSIVE SURVEY AND NEW PERSPECTIVESPreprint
Wang, F., Bo, Y., Shao, P., Chen, Z., Chou, H.-C., Kang, L., Shao, S., Xia, X., Ding, N., Xiong, H., et al. (2025) · doi:10.36227/techrxiv.175492111.15449662/v1
Pecora, A. E., Strada, F., & Bottino, A. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-97778-7_8
In-Context Learning as General-Purpose Learning: a Comprehensive Survey and New PerspectivesPreprint
Wang, F., Bo, Y., Shao, P., Chen, Z., Chou, H.-C., Kang, L., Shao, S., Xia, X., Ding, N., Xiong, H., et al. (2025) · doi:10.36227/techrxiv.175492111.15449662/v2
Multimodal Ocean Forecast Correction Method Based on Transformer
Zhang, L. (2025) · IEEE International Conference on Computer and Communication Engineering Technology (CCET) · doi:10.1109/ccet66260.2025.11199487
UAMF: U-Net with Adaptive Memory Fusion For Infrared Small Target Detection
Cui, Y., Li, S., Liang, Y., & Zhang, J. (2025) · International Conference on Internet of Things, Artificial Intelligence and Mechanical Automation (IoTAIMA) · doi:10.1109/iotaima66468.2025.11212769
You Only Use Reactive Attention Slice When Retrieving From Long Context
Soh, Y. J., Huang, H., Tian, Y., & Zhao, J. (2025) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2025.findings-emnlp.1125
MAMFI: Learning Motion at All Scales via Memory-as-Motion Attention for Frame Interpolation
Jeong, M. W., & Rhee, C. E. (2025) · IEEE Access · doi:10.1109/access.2025.3635165
In-Context Learning as General-Purpose Learning: a Comprehensive Survey and New PerspectivesPreprint
Wang, F., Bo, Y., Shao, P., Chen, Z., Chou, H.-C., Kang, L., Shao, S., Xia, X., Ding, N., Xiong, H., et al. (2025) · doi:10.36227/techrxiv.175492111.15449662/v3
Three Tiers of Computation in Transformers and in Brain Architectures
Graham, E., & Granger, R. (2026) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3844612
Exploring Causal Attention Models in Transformers for Large Language Models
Terroa, J. F. G. V. (2024) · Portuguese National Funding Agency for Science, Research and Technology (RCAAP Project by FCT)
Kairov, M., Bulatov, A., & Kuratov, Y. (2025) · Optical Memory and Neural Networks · doi:10.3103/s1060992x25601733
Early Goal-Guided Multi-Scale Fusion for Real-Time Vision–Language Driving
Patapati, S., & Srinivasan, T. (2025) · IEEE Vehicular Technology Conference · doi:10.1109/vtc2025-fall65116.2025.11310131
MemoryINP: Memory Enhanced Neural Processes for Understanding-Based Informed Meta-Learning
Tsoumplekas, G., Ntampakis, N., Vasilakis, C., Papadopoulos, P., Sarigiannidis, P., & Argyriou, V. (2025) · IEEE International Conference on Cyber Humanities (IEEE-CH) · doi:10.1109/ieee-ch65308.2025.11279273
RAMTime: Retrieval-Augmented Memory for Time Series Forecasting
Wu, Y., & Li, J. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11462579
Hussain, R., Ma, Z., Khandelwal, R., Oltmanns, J. R., & Gupta, M. (2026) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2026.1760246
Beyond Episodic AI: Cognitive Field Networks for Biologically Inspired Persistent CognitionPreprint
Chae, B. G. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.16752
ReRead: Retrieval-Guided Efficient Reading for Long-Context Question Answering
Ling, J., Bao, W., Zhong, Z., Zhou, T., & Wang, J. (2026) · International Conference on Big Data and Information Analytics (BigDIA) · doi:10.1109/bigdia71496.2026.11684612
Chen, G., Zheng, J., Zhao, X., Wang, Z., Li, B., Zhao, J., & 吳政叡 (2026) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2026.116210
GENA-LM: A Family of Open-Source Foundational DNA Language Models for Long Sequences
Fishman, V., Kuratov, Y., Shmelev, A., Petrov, M., Penzar, D., Shepelin, D., Chekanov, N., Kardymon, O., & Burtsev, M. (2023) · doi:10.1101/2023.06.12.544594 · Cited by 137
“In the domain of recurrent models, inputs are segmented and sequentially processed, with inter-segment information relayed through prior hidden states [25, 26] or specialized memory [27–29].”
Metadata
- Authors
- Wu, Q., Lan, Z., Qian, K., Gu, J., Geramifard, A., & Yu, Z.
- Affiliations
- Columbia University (US)
- Westlake University (CN)
- University of California, Santa Cruz (US)
- Meta (United Kingdom) (GB)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2020
- Pages
- 308-318
- arXiv
- 2010.06891
- OpenAlex
- W4404783458
- Semantic Scholar
- 67ee20536c30a225b86902af2f091e28e5e19b40
- Cited by
- 80
- References
- 0
- Citations by year
- 2026: 10 · 2025: 34 · 2024: 14 · 2023: 14 · 2022: 5 · 2021: 3
- FWCI
- 7.02
- Type
- conference paper
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- memory-augmented transformer, external memory, sequence modeling, long sequences, linear time complexity, memory efficiency, attention patterns, backpropagation through time
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-11-14, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-14 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
