Position Information in Transformers: An Overview
Open accessAbstract
Transformers are arguably the main workhorse in recent natural language processing research. By definition, a Transformer is invariant with respect to reordering of the input. However, language is inherently sequential and word order is essential to the semantics and syntax of an utterance. In this article, we provide an overview and theoretical comparison of existing methods to incorporate position information into Transformer models. The objectives of this survey are to (1) showcase that position information in Transformer is a vibrant and extensive research area; (2) enable the reader to compare existing methods by providing a unified notation and systematization of different approaches along important model dimensions; (3) indicate what characteristics of an application should be taken into account when selecting a position encoding; and (4) provide stimuli for future research.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The order of concatenation does not matter due to the permutation invariance of transformers.
“It is worth noting that the order of concatenation does not matter due to the permutation invariance of transformers (Dufter et al., 2022).”
From Prompt Cache: Modular Attention Reuse for Low-Latency Inference (2023) · cites this paper as Dufter et al., 2022
The Introduction says a plain Transformer is invariant to reordering of its input, supporting the claim's underlying point. But §2.3 shows that reordering also reorders the outputs, and the paper explains that incorporating position information lets Transformers distinguish order. The claim omits those limits when saying concatenation order does not matter for transformers generally. (LLM, conf=0.88)
Experimental — accuracy not yet measuredTransformer-based language models integrate various methods to incorporate positional information.
“Consequently, Transformer-based language models integrate various methods to incorporate positional information (Wang et al. 2021; Dufter et al. 2022).”
From Large language models (LLMs): survey, technical frameworks, and future challenges (2024) · cites this paper as Dufter et al. 2022
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
The citation lists haven’t been fetched yet
References 78 · Cited by 175
Research works printed but not linked: 6
- Ba, Lei Jimmy, Jamie Ryan Kiros, and Geoffrey E. Hinton. 2016. Layer normalization. CoRR, abs/1607.06450. · arXiv
- Bahdanau, Dzmitry, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In 3rd International Conference on Learning Representations, ICLR 2015, Conference Track Proceedings.
- Radford, Alec, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9.
- Tay, Yi, Dara Bahri, Donald Metzler, Da-Cheng Juan, Zhe Zhao, and Che Zheng. 2021a. Synthesizer: Rethinking self-attention for transformer models. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, pages 10183-10192.
- Tay, Yi, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. 2021b. Long range arena: A benchmark for efficient transformers. In 9th International Conference on Learning Representations, ICLR 2021, OpenReview.net.
- Curran Associates, Inc., pages 5998-6008.
- Authors
- Dufter, P., Schmitt, M., & Schütze, H.
- Affiliations
- Ludwig-Maximilians-Universität München (DE)
- Corresponding author
- Dufter, Philipp
- Schmitt, Martin
- Schütze, Hinrich
- ORCID
- Schütze, Hinrich
- Venue
- Computational Linguistics
- Publisher
- MIT Press
- Published
- 2022
- Volume
- 48
- Issue
- 3
- Pages
- 733-763
- ISSN
- 0891-2017 (ISSN-L)
- 1530-9312
- OpenAlex
- W3132607382
- Cited by
- 175
- References
- 78
- Citations by year
- 2026: 25 · 2025: 45 · 2024: 44 · 2023: 35 · 2022: 14 · 2021: 11 · 2020: 1
- FWCI
- 11.35
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- Locations
- Computational Linguistics (publishedVersion, cc-by-nc-nd)
- arXiv (Cornell University) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Open access LMU (Ludwid Maxmilian's Universitat Munchen) (submittedVersion, cc-by)
- arXiv (Cornell University)
- Funders
- Bundesministerium für Bildung und Forschung (01IS18050)
- Studienstiftung des Deutschen Volkes
- H2020 European Research Council (740516)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- positional encoding, transformer models, word order, positional information, sequence modeling, natural language processing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
