Theoretical Limitations of Self-Attention in Neural Sequence Models

Cited by (172)

Items 101–172 of 172

1. Algorithms for Acyclic Weighted Finite-State Automata with Failure Arcs
   Anej Svete, Benjamin Dayan, Ryan Cotterell, Tim Vieira, Jason M. Eisner · 2022 · Conference on Empirical Methods in Natural Language Processing (EMNLP)
2. Using Artificial Populations to Study Psychological Phenomena in Neural Models
   Jesse Roberts, Kyle Moore, Drew Wilenzick, Douglas H. Fisher · 2023 · AAAI Conference on Artificial Intelligence
3. Enhancing Generalizability in Biomedical Entity Recognition: Self-Attention PCA-CLS Model
   Rajesh Kumar Mundotiya, J. Likhitha Priya, Divya Kuwarbi, Teekam Singh · 2024 · IEEE Transactions on Computational Biology and Bioinformatics
4. Weather temperature prediction based on LSTM and transformer
   Hongye Wang · 2024 · International Conference on Electronics, Electrical and Information Engineering (ICEEIE)
5. Transformers as Transducers
   Lena Strobl, Dana Angluin, David Chiang, Jonathan Rawski, Ashish Sabharwal · 2025 · Transactions of the Association for Computational Linguistics
6. PERFUME
   Nicolaas Weideman, Virginia K. Felkner, Wei-Cheng Wu, Jonathan May, Christophe Hauser, Luis Garcia · 2021
7. Implicit n-grams Induced by Recurrence
   Xiaobing Sun, Wei Lu · 2022 · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT)
8. Can the Transformer Learn Nested Recursion with Symbol Masking?
   Jean-Philippe Bernardy, Adam Ek, Vladislav Maraev · 2021 · Findings of the Association for Computational Linguistics: ACL (Findings of ACL)
9. I-BERT: Inductive Generalization of Transformer to Arbitrary Context Lengths
   Hyoungwook Nam, Seung Byum Seo, Vikram Sharma Mailthody, Noor Michael, Lan Li · 2020 · arXiv (Cornell University)
10. Current Limitations of Language Models: What You Need is Retrieval
    Aran Komatsuzaki · 2020 · arXiv (Cornell University)
11. On the Dynamics of Training Attention Models
    Haoye Lu, Yongyi Mao, Amiya R. Nayak · 2020 · arXiv (Cornell University)
12. Transformer Working Memory Enables Regular Language Reasoning And Natural Language Length Extrapolation
    Ta-Chung Chi, Ting-Han Fan, Alexander I. Rudnicky, Peter J. Ramadge · 2023 · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP)
13. Large Visual Perception Network With Auxiliary Supervision for Surface Defect Detection
    Qingqing Huang, Yangyong Mu, Yan Han, Yan Zhang · 2025 · IEEE Transactions on Instrumentation and Measurement
14. Simulating Hard Attention Using Soft Attention
    Andy Yang, Lena Strobl, David Chiang, Dana Angluin · 2026 · Transactions of the Association for Computational Linguistics
15. Hybrid learning framework to enable recency-aware attention
    Min Chen, Amir Hossein Etemadi · 2026 · Neural Computing and Applications
16. On-device generative anomaly detection for unmanned aerial vehicle telemetry using memristor analog compute-in-memory
    Hyojung Ahn · 2026 · Engineering Applications of Artificial Intelligence
17. LLMs as a platform for studying constraint interaction: Motivation and challenges
    Ethan Wilcox, Elissa L. Newport · 2026 · Behavioral and Brain Sciences
18. A Memory-Driven Neural Attention Model for Aspect-Based Sentiment Classification
    Jonathan van de Ruitenbeek, Flavius Frăsincar, Gianni Brauwers · 2022 · Journal of Web Engineering
19. Formal Languages and the NLP Black Box
    William Merrill · 2023 · Lecture notes in computer science
20. Exploring the health care system’s representation in the media through hierarchical topic modeling
    Ravil I. Mukhamediev, Marina Yelis, Kirill Yakunin, Yelena Popova, Yan I. Kuchin, Адилхан Сымагулов, Nadiya Yunicheva, Elena Zaitseva, Vitaly Levashenko, Elena Muhamedijeva et al. · 2024 · Cogent Engineering
21. OASIS: Harnessing Diffusion Adversarial Network for Ocean Salinity Imputation using Sparse Drifter Trajectories
    Bo Li, Yingqi Feng, Ming Jin, Xin Zheng, Yufei Tang, Laurent Marcel Chérubin, Can Wang, Alan Wee‐Chung Liew, Qinghua Lu, Jingwei Yao et al. · 2025 · ACM International Conference on Information and Knowledge Management (CIKM)
22. Advances in reasoning by prompting large language models: A survey
    Ruixin Hong, Xinyu Pang, Changshui Zhang · 2026 · Cybernetics and Intelligence
23. Joint Translation and Unit Conversion for End-to-end Localization
    Georgiana Dinu, Prashant Mathur, Marcello Federico, Stanislas Lauly, Yaser Al-Onaizan · 2020 · International Conference on Spoken Language Translation (IWSLT)
24. Compositional Generalization via Parsing Tree Annotation
    Segwang Kim, Joonyoung Kim, Kyomin Jung · 2021 · IEEE Access
25. One for “All”: a unified model for fine-grained sentiment analysis under three tasks
    Hengyang Lu, Jun Yang, Cong Hu, Wei Hua Fang · 2021 · PeerJ Computer Science
26. On the Power of Saturated Transformers: A View from Circuit Complexity.
    William Merrill, Yoav Goldberg, Roy Schwartz, Noah A. Smith · 2021 · arXiv (Cornell University)
27. Training dynamics of neural language models
    Naomi Saphra · 2021 · ERA
28. Advanced Language Understanding with Syntax-Enhanced Transformer
    Wender Rine, Rodolfo Patel, Neo Steve · 2023 · Preprints.org
29. Exploring Parity Challenges in Reinforcement Learning Through Curriculum Learning with Noisy Labels
    Bei Zhou, Søren Kamaric Riis · 2024 · SSRN Electronic Journal
30. Inference Acceleration for Large Language Models on CPUs
    Ditto PS, Jithin VG · 2024 · Preprints.org
31. Synergizing Augmented Reality and Llms for Advanced Cognitive Support in Emergency Audio Communications
    Fang Xu, Tianyu Zhou, Tri Nguyen, Haohui Bao, Christine Lin, Jing Du · 2024 · SSRN Electronic Journal
32. Evaluating and Improving the Reasoning Abilities of Language Models
    Chadi Helwe · 2024 · theses.fr (ABES)
33. Language models need inductive biases to count inductively
    Chang, Yingshan, Bisk, Yonatan · 2024 · arXiv
34. How LSTM Encodes Syntax: Exploring Context Vectors and Semi-Quantization on Natural Text
    Chihiro Shibata, Kei Uchiumi, Daichi Mochihashi · 2020 · arXiv (Cornell University)
35. Abstraction, Reasoning and Deep Learning: A Study of the "Look and Say" Sequence
    Wlodek Zadrozny · 2021 · arXiv (Cornell University)
36. Enhancing Reinforcement Learning with discrete interfaces to learn the Dyck Language
    Florian Dietz, Dietrich Klakow · 2021 · arXiv (Cornell University)
37. Peer Review #3 of "One for “All”: a unified model for fine-grained sentiment analysis under three tasks (v0.1)"
    2021 · PeerJ Computer Science
38. Peer Review #2 of "One for “All”: a unified model for fine-grained sentiment analysis under three tasks (v0.1)"
    2021 · PeerJ Computer Science
39. Peer Review #1 of "One for “All”: a unified model for fine-grained sentiment analysis under three tasks (v0.1)"
    2021 · PeerJ Computer Science
40. Peer Review #3 of "One for “All”: a unified model for fine-grained sentiment analysis under three tasks (v0.2)"
    2021 · PeerJ Computer Science
41. Why Self-Attention Is Natural for Sequence-to-Sequence Problems? A Perspective from Symmetries
    Chao Ma, Lexing Ying B · 2023 · Journal of Machine Learning
42. t-ConvESN: Temporal Convolution-Readout for Random Recurrent Neural Networks
    Matthew Evanusa, Vaishnavi Patil, Michelle Girvan, Joel I. Goodman, Cornelia M. Fermuller, Yiannis Aloimonos · 2023 · Lecture notes in computer science
43. Bidirectional Masked Self-attention and N-gram Span Attention for Constituency Parsing
    Soohyeong Kim, Whanhee Cho, Minji Kim, Yong Choi · 2023 · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP)
44. Pushdown Layers: Encoding Recursive Structure in Transformer Language Models
    Shikhar Murty, Pratyusha Sharma, Jacob Andreas, Christopher D. Manning · 2023 · Conference on Empirical Methods in Natural Language Processing (EMNLP)
45. Simpler neural networks prefer subregular languages
    Charles Torres, Richard Futrell · 2023 · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP)
46. Unraveling Feature Extraction Mechanisms in Neural Networks
    Xiaobing Sun, Jiaxi Li, Wei Lü · 2023 · Conference on Empirical Methods in Natural Language Processing (EMNLP)
47. Improving Image Captioning Using Deep Convolutional Neural Network
    Fahimeh Basiri, Alireza Mohammadi, Ayman Amer, Mohammad Javad Ranjbar Naserabadi, Mohammad Mahdi Moghimi · 2023 · International Engineering Conference on Electrical, Energy, and Artificial Intelligence (EICEEAI)
48. GISA: Gradual Information Selection Attention for MCQ Difficulty Estimation
    Manikandan Ravikiran, Tarun Sharma, Arnav V. Bhavsar, Rohit Saluja · 2025 · Lecture notes in computer science
49. Adaptive Attention Mechanism Based on Dynamic Event Priority for Highway Surveillance
    Tong Liu, Zhigang Qi, Denan Kong, Yu Zhang, Yuxin Tian, Chao Lv · 2025 · Communications in computer and information science
50. Reframing Hallucination in Large Language Models: A Lifecycle-Based, Mechanism-Aligned, and Phenomenon-Consistent Definition
    Xinyi Fang, Hao Ran Yuan, Hanxia Li, Jieren Kou, Chuqiao Gu, Wuyang Zhang, Xiaoman Duan, Yajun Fang · 2024 · International Conference on Universal Village (UV)
51. Efficient Occupancy Prediction Guided Point Cloud Geometry Compression
    Xinjie Wang, Yifan Zhang, Ting Liu, Xinpu Liu, Ke Xu, Jianwei Wan, Yulan Guo, Hanyun Wang · 2025 · IEEE Transactions on Circuits and Systems for Video Technology
52. The Role of Logic and Automata in Understanding Transformers
    Anthony W. Lin, Pablo Barceló · 2025 · Lecture notes in computer science
53. Hey Pentti, We Did (More of) It!: A Vector-Symbolic Lisp With Residue Arithmetic
    Connor Hanley, Eilene Tomkins-Flanagan, Matthew A. Kelly · 2025 · International Joint Conference on Neural Networks (IJCNN)
54. Detecting and Correcting Hallucinations in LLMs via Substantive Uncertainty and Iterative Validation
    Zheng Chen, Yijie Cheng, Yuxin Gao · 2025 · Lecture notes in computer science
55. Hybrid self-attention aided generative adversarial network for hazy image restoration
    Wende Dong, Tianheng Xu, Muhan Ji, Gang Luo, Geng Li, Zhenzhen Zheng · 2025 · Optik
56. Three Tiers of Computation in Transformers and in Brain Architectures
    Emma Graham, Richard Granger · 2026 · ACM Transactions on Intelligent Systems and Technology
57. Crude Oil Leakage Simulation Method Based on Physical Processes
    Yuanfeng Lian, Zhenlin Xu, Guangyang Li, Lianen Ji, Shaohua Dong · 2023 · Journal of Computer-Aided Design & Computer Graphics
58. Reverse Mathematics for Neural Networks
    Yoshihiro Maruyama · 2026 · Lecture notes in computer science
59. Hierarchical Resolution Transformers: A Wavelet-Inspired Architecture for Multi-Scale Language Understanding
    Ayan Sar, Sampurna Roy, Kanav Gupta, Anurag Kaushish, Tanupriya Choudhury, Abhijit Kumar · 2025 · IEEE International Conference on Big Data (BigData)
60. Early detection of fake news through integrating hypergraph-based multi-scale temporal and structural information
    Chao Liu, Jiayi Chen · 2026 · Multimedia Systems
61. Resukan: an improved medical image segmentation method based on U-KAN architecture
    Yu Feng, Haomeng Xie, Chuanshen Zhao, He Zhao · 2026 · Multimedia Systems
62. Study of the abstraction capabilities of neural language models
    Bingzhi Li · 2023 · theses.fr (ABES)
63. Contributions to evaluating and improving the faithfulness for text generation
    Kun Zhang · 2025 · theses.fr (ABES)
64. On the Computational Hardness of Transformers
    Barna Saha, Yinzhan Xu, Christopher Ye, Hantao Yu · 2026 · ACM Symposium on Theory of Computing (STOC)
65. Algorithmic affective blunting quantifies the collapse curve of interpretative failure in large language models
    Ryan SangBaek Kim · 2026 · Discover Artificial Intelligence
66. Circuit Complexity of Hierarchical Knowledge Tracing
    N.-S. Liu, Richard G. Baraniuk, Shashank Sonkar · 2026 · Lecture notes in computer science
67. Beyond Price: A Technical Quality Framework for AI Antitrust
    Patrick Y. Wu · 2026 · ACM Conference on Fairness, Accountability, and Transparency (FAccT)
68. RAGnRoll: Learning to Iteratively Retrieve and Generate Attributable Answer Snippets
    Hanane Djeddal, Laure Soulier, Karen Pinel-Sauvagnat, Sophia Katrenko, Pierre Erbacher, Lynda Tamine · 2026 · ACM Transactions on Information Systems
69. Self-attention in large language models as adaptive stochastic mixing
    Chikoo Oosawa · 2026 · Physica A Statistical Mechanics and its Applications
70. PeNorm: Enhancing positional discriminability in transformers through normalization-based encoding optimization
    Xilong Zhang, Ruochen Liu, Xuefeng Liang · 2026 · Neural Networks
71. Training an LLM for Grammaticality Judgements in Romanian
    Universitatea din București, Șerban Hartular · 2026 · Studii și cercetări lingvistice
72. Going Beyond Linear Transformers with Recurrent Fast Weight Programmers
    Irie, Kazuki, Schlag, Imanol, Csordás, Róbert, Schmidhuber, Jürgen · 2021 · arXiv