The DOI matches its Crossref record; 4 of the 4 references it cites were found in a registry or an index.
Exploring Transformer Extrapolation
Open accessAbstract
Length extrapolation has attracted considerable attention recently since it allows transformers to be tested on longer sequences than those used in training. Previous research has shown that this property can be attained by using carefully designed Relative Positional Encodings (RPEs). While these methods perform well on a variety of corpora, the conditions for length extrapolation have yet to be investigated. This paper attempts to determine what types of RPEs allow for length extrapolation through a thorough mathematical and empirical analysis. We discover that a transformer is certain to possess this property as long as the series that corresponds to the RPE's exponential converges. Two practices are derived from the conditions and examined in language modeling tasks on a variety of corpora. As a bonus from the conditions, we derive a new Theoretical Receptive Field (TRF) to measure the receptive field of RPEs without taking any training steps. Extensive experiments are conducted on the Wikitext-103, Books, Github, and WikiBook datasets to demonstrate the viability of our discovered conditions. We also compare TRF to Empirical Receptive Field (ERF) across different models, showing consistently matched trends on these datasets. Code is released at: https://github.com/OpenNLPLab/Rpe.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.11929 · Cited by 47,472
AST: Audio Spectrogram Transformer
Gong, Y., Chung, Y.-A., & Glass, J. (2021) · Interspeech 2021 · doi:10.21437/interspeech.2021-698 · Cited by 1,159
Flagged first, then by citations
Research works printed but not linked: 30
- Akbari, H.; Yuan, L.; Qian, R.; Chuang, W.-H.; Chang, S.-F.; Cui, Y.; and Gong, B. 2021. Vatt: Transformers for multi- modal self-supervised learning from raw video, audio and text. In arXiv preprint arXiv:2104.11178. · arXiv
- Peters, M. E.; and Cohan, A. 2020. Longformer: The Long-Document Transformer. In arXiv:2004.05150. · arXiv
- Chi, T.-C.; Fan, T.-H.; Ramadge, P. J.; and Rudnicky, A. I. 2022. KERPLE: Kernelized Relative Positional Embedding for Length Extrapolation. ArXiv, abs/2205.09921. · arXiv
- Chi, T.-C.; Fan, T.-H.; and Rudnicky, A. I. 2022. Recep- tive Field Alignment Enables Transformer Length Extrapo- lation. ArXiv, abs/2212.10356. · arXiv
- Cho, K.; van Merriënboer, B.; Gulcehre, C.; Bahdanau, D.; Bougares, F.; Schwenk, H.; and Bengio, Y. 2014. Learn- ing Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Lan- guage Processing (EMNLP), 1724-1734. Doha, Qatar: As- sociation for Computational Linguistics.
- Gao, L.; Biderman, S.; Black, S.; Golding, L.; Hoppe, T.; Foster, C.; Phang, J.; He, H.; Thite, A.; Nabeshima, N.; Presser, S.; and Leahy, C. 2020. The Pile: An 800GB Dataset of Diverse Text for Language Modeling. In arXiv preprint arXiv:2101.00027. · arXiv
- Gehring, J.; Auli, M.; Grangier, D.; Yarats, D.; and Dauphin, Y. N. 2017. Convolutional sequence to sequence learning. In International Conference on Machine Learning, 1243- 1252.
- PMLR.
- Hao, D.; Mao, Y.; He, B.; Han, X.; Dai, Y.; and Zhong, Y. 2024. Improving Audio-Visual Segmentation with Bidirec- tional Generation. In Proceedings of the AAAI Conference on Artificial Intelligence.
- Hershey, S.; Chaudhuri, S.; Ellis, D. P. W.; Gemmeke, J. F.; Jansen, A.; Moore, R. C.; Plakal, M.; Platt, D.; Saurous, R. A.; Seybold, B.; Slaney, M.; Weiss, R. J.; and Wilson, K. W. 2016. CNN architectures for large-scale audio classi- fication. 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 131-135.
- Kim, Y. 2014. Convolutional Neural Networks for Sentence Classification. In Conference on Empirical Methods in Nat- ural Language Processing.
- Liu, Y.; Ott, M.; Goyal, N.; Du, J.; Joshi, M.; Chen, D.; Levy, O.; Lewis, M.; Zettlemoyer, L.; and Stoyanov, V. 2019. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprint arXiv:1907.11692. · arXiv
- Liu, Z.; Li, D.; Lu, K.; Qin, Z.; Sun, W.; Xu, J.; and Zhong, Y. 2022. Neural architecture search on efficient transformers and beyond. arXiv preprint arXiv:2207.13955. · arXiv
- Liutkus, A.; Cífka, O.; Wu, S.-L.; Simsekli, U.; Yang, Y.- H.; and Richard, G. 2021. Relative positional encoding for transformers with linear complexity. In International Con- ference on Machine Learning, 7067-7079. PMLR.
- Lu, K.; Liu, Z.; Wang, J.; Sun, W.; Qin, Z.; Li, D.; Shen, X.; Deng, H.; Han, X.; Dai, Y.; and Zhong, Y. 2022. Lin- ear video transformer with feature fixation. arXiv preprint arXiv:2210.08164. · arXiv
- Merity, S.; Xiong, C.; Bradbury, J.; and Socher, R. 2016. Pointer Sentinel Mixture Models. In arXiv:1609.07843. · arXiv
- Ott, M.; Edunov, S.; Baevski, A.; Fan, A.; Gross, S.; Ng, N.; Grangier, D.; and Auli, M. 2019. fairseq: A fast, extensible toolkit for sequence modeling. arXiv preprint arXiv:1904.01038. · arXiv
- Press, O.; Smith, N.; and Lewis, M. 2022. Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation. In International Conference on Learning Representations.
- Qin, Z.; Han, X.; Sun, W.; Li, D.; Kong, L.; Barnes, N.; and Zhong, Y. 2022a. The Devil in Linear Transformer. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, 7025-7041. Abu Dhabi, United Arab Emirates: Association for Computational Lin- guistics.
- Qin, Z.; Li, D.; Sun, W.; Sun, W.; Shen, X.; Han, X.; Wei, Y.; Lv, B.; Yuan, F.; Luo, X.; Qiao, Y.; and Zhong, Y. 2023b. Scaling TransNormer to 175 Billion Parameters. In arXiv preprint 2307.14995. · arXiv
- Qin, Z.; Sun, W.; Deng, H.; Li, D.; Wei, Y.; Lv, B.; Yan, J.; Kong, L.; and Zhong, Y. 2022b. cosFormer: Rethink- ing Softmax In Attention. In International Conference on Learning Representations.
- Qin, Z.; Sun, W.; Lu, K.; Deng, H.; Li, D.; Han, X.; Dai, Y.; Kong, L.; and Zhong, Y. 2023c. Linearized Relative Posi- tional Encoding. arXiv preprint arXiv:2307.09270. · arXiv
- Qin, Z.; Yang, S.; and Zhong, Y. 2023. Hierarchically gated recurrent neural network for sequence modeling. NeurIPS.
- Qin, Z.; and Zhong, Y. 2023. Accelerating Toeplitz Neu- ral Network with Constant-time Inference Complexity. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. Association for Computa- tional Linguistics.
- Su, J.; Lu, Y.; Pan, S.; Wen, B.; and Liu, Y. 2021. Ro- former: Enhanced transformer with rotary position embed- ding. arXiv preprint arXiv:2104.09864. · arXiv
- Sun, J.; Zhong, G.; Zhou, D.; Li, B.; and Zhong, Y. 2022a. Locality Matters: A Locality-Biased Linear At- tention for Automatic Speech Recognition. arXiv preprint arXiv:2203.15609. · arXiv
- Sun, W.; Qin, Z.; Deng, H.; Wang, J.; Zhang, Y.; Zhang, K.; Barnes, N.; Birchfield, S.; Kong, L.; and Zhong, Y. 2022b. Vicinity Vision Transformer. IEEE Transactions on Pattern Analysis and Machine Intelligence, (01): 1-14.
- Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, Ł.; and Polosukhin, I. 2017. At- tention is all you need. Advances in neural information pro- cessing systems, 30.
- Wettig, A.; Gao, T.; Zhong, Z.; and Chen, D. 2022. Should You Mask 15% in Masked Language Modeling? In arXiv:2202.08005. · arXiv
- Zhu, Y.; Kiros, R.; Zemel, R.; Salakhutdinov, R.; Urtasun, R.; Torralba, A.; and Fidler, S. 2015. Aligning Books and Movies: Towards Story-Like Visual Explanations by Watch- ing Movies and Reading Books. In The IEEE International Conference on Computer Vision (ICCV). Wikitext-103 Length Sinusoidal Alibi Kerple-Log Kerple-Power Sandwich Type1 Type2 512 24.73 24.22 24.12 24.18 24.76 24.25 24.29 768 41.08 23.45 23.36 23.42 24.04 23.43 23.51 1024 62.71 23.06 22.93 22.98 23.63 23.03 23.09 1280 83.81 22.83 22.67 22.73 23.39 22.77 22.83 1536 102.28 22.66 22.45 22.54 23.21 22.55 22.63 1792 121.98 22.60 22.41 22.47 23.18 22.50 22.59 2048 138.17 22.52 22.28 22.37 23.08 22.38 22.48 3072 194.43 22.33 22.02 22.14 22.91 22.15 22.27 4096 259.55 22.26 21.97 22.08 22.96 22.09 22.21 5120 289.79 22.20 21.86 22.00 22.93 21.99 22.14 6144 337.46 22.17 21.87 21.96 23.06 21.97 22.11 7168 376.41 22.16 21.84 21.95 23.13 21.96 22.10 8192 406.95 22.14 21.82 21.94 23.20 21.95 22.08 9216 423.92 22.12 21.80 21.90 23.26 21.90 22.06 Books Length Sinusoidal Alibi Kerple-Log Kerple-Power Sandwich Type1 Type2 512 7.49 7.28 7.34 7.31 7.64 7.30 7.35 768 10.43 7.15 7.21 7.18 7.55 7.18 7.22 1024 13.32 7.09 7.15 7.11 7.49 7.11 7.15 1280 15.53 7.06 7.11 7.08
Cited by
Sorted by citations
Titans: Learning to Memorize at Test Time
Behrouz, A., Zhong, P., & Mirrokni, V. (2025) · arXiv · doi:10.48550/arxiv.2501.00663 · Cited by 24
Metadata
- Authors
- Qin, Z., Zhong, Y., & Deng, H.
- Affiliations
- Shanghai Artificial Intelligence Laboratory (CN)
- Northwestern Polytechnical University (CN)
- Venue
- Proceedings of the AAAI Conference on Artificial Intelligence
- Published
- 2024-03-24
- Volume
- 38
- Issue
- 17
- Pages
- 18897-18905
- ISSN
- 2159-5399 (ISSN-L)
- 2374-3468
- OpenAlex
- W4393147252
- Cited by
- 2
- References
- 4
- Citations by year
- 2024: 2
- FWCI
- 0.59
- Type
- conference paper
- Language
- en
- Open access
- diamond
- dx.doi.org
- Locations
- Proceedings of the AAAI Conference on Artificial Intelligence (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- length extrapolation, relative position encoding, exponential convergence, language modeling
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
