CoChat
Sign Up
Cited by 1,422Topic ModelingAdvanced Neural Network ApplicationsDomain Adaptation and Few-Shot Learning

The DOI matches its Crossref record; 181 of the 183 references it cites were found in a registry or an index.

A survey of transformers

Open access
Lin, Tianyang, Wang, Yuxin, Liu, Xiangyang, Qiu, Xipeng·2022·AI Open
DOI 10.1016/j.aiopen.2022.10.001
Open this paper in CoChat — free accountOpen PDF

Abstract

Transformers have achieved great success in many artificial intelligence fields, such as natural language processing, computer vision, and audio processing. Therefore, it is natural to attract lots of interest from academic and industry researchers. Up to the present, a great variety of Transformer variants (a.k.a. X-formers) have been proposed, however, a systematic and comprehensive literature review on these Transformer variants is still missing. In this survey, we provide a comprehensive review of various X-formers. We first briefly introduce the vanilla Transformer and then propose a new taxonomy of X-formers. Next, we introduce the various X-formers from three perspectives: architectural modification, pre-training, and applications. Finally, we outline some potential directions for future research.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1016/j.aiopen.2022.10.001

Cites

181 of 181 references verified0 citation errors · 0 not checkable

Statistics

Figures

Image 1
CC-BY · From the PDF
Fig. 1 Overview of vanilla Transformer architecture.
Fig. 1 Overview of vanilla Transformer architecture.CC-BY · From the PDF
Fig. 2 Categorization of Transformer variants.
Fig. 2 Categorization of Transformer variants.CC-BY · From the PDF
Fig. 3 Taxonomy of Transformers.
Fig. 3 Taxonomy of Transformers.CC-BY · From the PDF
Fig. 4 Some representative atomic sparse attention patterns. The colored squares means corresponding attention scores are calculated and a blank square means the attention score is discarded.
Fig. 4 Some representative atomic sparse attention patterns. The colored squares means corresponding attention scores are calculated and a blank square means the attention score is discarded.CC-BY · From the PDF
Fig. 5 Some representative compound sparse attention patterns. The red boxes indicate sequence boundaries.
Fig. 5 Some representative compound sparse attention patterns. The red boxes indicate sequence boundaries.CC-BY · From the PDF
Fig. 6 Other types of sparse attentions. The red box indicates the query position, and the orange nodes/squares means corresponding tokens are attended to by the query.
Fig. 6 Other types of sparse attentions. The red box indicates the query position, and the orange nodes/squares means corresponding tokens are attended to by the query.CC-BY · From the PDF
Fig. 7 Illustration of complexity difference between standard self-attention and linearized self-attention.
Fig. 7 Illustration of complexity difference between standard self-attention and linearized self-attention.CC-BY · From the PDF
Fig. 8 Query prototyping and memory compression.
Fig. 8 Query prototyping and memory compression.CC-BY · From the PDF
Fig. 9 Attention with prior. This type of model fuse generated attention scores with prior attention scores, producing the final attention scores for attention computation.
Fig. 9 Attention with prior. This type of model fuse generated attention scores with prior attention scores, producing the final attention scores for attention computation.CC-BY · From the PDF
Fig. 10 Three types of span masking function 𝑚 ( 𝑥 ) . The horizontal axis represents distance 𝑥 and vertical axis the mask value.
Fig. 10 Three types of span masking function 𝑚 ( 𝑥 ) . The horizontal axis represents distance 𝑥 and vertical axis the mask value.CC-BY · From the PDF
Fig. 12 Illustrations of recurrent and hierarchical Transformers.
Fig. 12 Illustrations of recurrent and hierarchical Transformers.CC-BY · From the PDF

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 183 references

Cited by

Sorted by citations

See all 1,422

Related

Metadata

Authors
Lin, T., Wang, Y., Liu, X., & Qiu, X.
Affiliations
Fudan University (CN)
Corresponding author
Qiu, Xipeng
ORCID
Venue
AI Open
Publisher
Elsevier BV
Published
2022
Volume
3
Pages
111-132
ISSN
2666-6510 (ISSN-L)
DOI
10.1016/j.aiopen.2022.10.001
OpenAlex
W3171649327
Cited by
1,422
References
183
Citations by year
2026: 217 · 2025: 423 · 2024: 438 · 2023: 264 · 2022: 69
FWCI
98.23
Type
journal article
Language
en
Open access
Licence
Locations
  • AI Open (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • arXiv (Cornell University)
Funders
National Natural Science Foundation of China (62022027)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords
transformer models, architectural alterations, pre-training, natural language processing, computer vision, audio signal processing

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review