Image transformer
Open accessAbstract
Image generation has been successfully cast as an autoregressive sequence generation or transformation problem. Recent work has shown that self-attention is an effective way of modeling textual sequences. In this work, we generalize a recently proposed model architecture based on self-attention, the Transformer, to a sequence modeling formulation of image generation with a tractable likelihood. By restricting the self-attention mechanism to attend to local neighborhoods we significantly increase the size of images the model can process in practice, despite maintaining significantly larger receptive fields per layer than typical convolutional neural networks. While conceptually simple, our generative models significantly outperform the current state of the art in image generation on ImageNet, improving the best published negative log-likelihood on ImageNet from 3.83 to 3.77. We also present results on image super-resolution with a large magnification ratio, applying an encoder-decoder configuration of our architecture. In a human evaluation study, we find that images generated by our super-resolution model fool human observers three times more often than the previous state of the art.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Virtually all existing architectures for dense prediction are based on convolutional networks.
“Virtually all existing architectures for dense prediction are based on convolutional networks [6,31,34,42,49,50,53].”
From Vision Transformers for Dense Prediction (2021) · cites this paper as [6]
The Introduction (p. 1) says convolutional models such as PixelCNN had received much more attention for pixel-by-pixel image generation. It does not establish that virtually all architectures across the broader category of dense prediction are convolutional; it also discusses recurrent models and proposes a self-attention alternative (Introduction, p. 2). (LLM, conf=0.78)
Experimental — accuracy not yet measuredClosest to DETR are end-to-end set predictions for object detection and instance segmentation.
“Closest to our approach are end-to-end set predictions for object detection [43] and instance segmentation [41,30,36,42].”
From End-to-End Object Detection with Transformers (2020) · cites this paper as [30]
The Introduction (p. 1) describes predicting image pixels, and the Experiments (pp. 5–7) concern image generation and super-resolution. Searches for end-to-end set prediction, object detection, and instance segmentation found no passage in our copy addressing the attributed claim. (LLM, conf=0.77)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
No references could be resolved yet.
Research works printed but not linked: 24
- Ba, Jimmy Lei, Kiros, Jamie Ryan, and Hinton, Geoffrey E. Layer normalization. arXiv preprint arXiv:1607.06450 , 2016. · arXiv
- Bellemare, Marc G., Srinivasan, Sriram, Ostrovski, Georg, Schaul, Tom, Saxton, David, and Munos, R´ emi. Unifying count-based exploration and intrinsic motivation. CoRR , abs/1606.01868, 2016. URL http://arxiv.org/ abs/1606.01868 . · arXiv
- Bengio, Yoshua and Bengio, Samy. Modeling highdimensional discrete data with multi-layer neural networks. In Neural Information Processing Systems , pp. 400-406. MIT Press, 2000.
- CoRR , abs/1703.10717, 2017. URL http://arxiv. org/abs/1703.10717 . · arXiv
- Chen, Xi, Mishra, Nikhil, Rohaninejad, Mostafa, and Abbeel, Pieter. Pixelsnail: An improved autoregressive generative model. arXiv preprint arXiv:1712.09763 , 2017. · arXiv
- Cheng, Jianpeng, Dong, Li, and Lapata, Mirella. Long short-term memory-networks for machine reading. arXiv preprint arXiv:1601.06733 , 2016. · arXiv
- Dahl, Ryan, Norouzi, Mohammad, and Shlens, Jonathan. Pixel recursive super resolution. 2017. URL https: //arxiv.org/abs/1702.00783 .
- Kalchbrenner, Nal and Blunsom, Phil. Recurrent continuous translation models. In Proceedings EMNLP 2013 , pp. 1700-1709, 2013. URL http://nal.co/papers/ KalchbrennerBlunsom\_EMNLP13 .
- Kalchbrenner, Nal, van den Oord, A¨ aron, Simonyan, Karen, Danihelka, Ivo, Vinyals, Oriol, Graves, Alex, and Kavukcuoglu, Koray. Video pixel networks. CoRR , abs/1610.00527, 2016. URL http://arxiv.org/ abs/1610.00527 . · arXiv
- Kingma, Diederik and Ba, Jimmy. Adam: A method for stochastic optimization. In ICLR , 2015.
- Larochelle, Hugo and Murray, Iain. The neural autoregressive distribution estimator. In The Proceedings of the 14th International Conference on Artificial Intelligence and Statistics , volume 15 of JMLR: W&CP , pp. 29-37, 2011.
- Ledig, Christian, Theis, Lucas, Huszar, Ferenc, Caballero, Jose, Aitken, Andrew, Tejani, Alykhan, Totz, Johannes, Wang, Zehan, and Shi, Wenzhe. Photo-realistic single image super-resolution using a generative adversarial network. arXiv:1609.04802 , 2016. · arXiv
- Mansimov, Elman, Parisotto, Emilio, Ba, Lei Jimmy, and Salakhutdinov, Ruslan. Generating images from captions with attention. CoRR , abs/1511.02793, 2015. URL http://arxiv.org/abs/1511.02793 . · arXiv
- Metz, Luke, Poole, Ben, Pfau, David, and Sohl-Dickstein, Jascha. Unrolled generative adversarial networks. CoRR , abs/1611.02163, 2016. URL http://arxiv.org/ abs/1611.02163 . · arXiv
- In Empirical Methods in Natural Language Processing , 2016. URL https://arxiv.org/pdf/1606. 01933.pdf .
- Radford, Alec, Metz, Luke, and Chintala, Soumith. Unsupervised representation learning with deep convolutional generative adversarial networks. CoRR , abs/1511.06434, 2015. URL http://arxiv.org/ abs/1511.06434 . · arXiv
- Srivastava, Nitish, Hinton, Geoffrey E, Krizhevsky, Alex, Sutskever, Ilya, and Salakhutdinov, Ruslan. Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research , 15(1):1929-1958, 2014.
- Theis, Lucas and Bethge, Matthias. Generative image modeling using spatial lstms. In Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 2 , NIPS'15, pp. 1927-1935, Cambridge, MA, USA, 2015. MIT Press. URL http://dl.acm. org/citation.cfm?id=2969442.2969455 .
- van den Oord, A¨ aron and Schrauwen, Benjamin. The student-t mixture as a natural image patch prior with application to image compression. Journal of Machine Learning Research , 15:2061-2086, 2014. URL http://jmlr.org/papers/v15/ vandenoord14a.html .
- van den Oord, A¨ aron, Kalchbrenner, Nal, and Kavukcuoglu, Koray. Pixel recurrent neural networks. ICML , 2016a.
- van den Oord, A¨ aron, Kalchbrenner, Nal, Vinyals, Oriol, Espeholt, Lasse, Graves, Alex, and Kavukcuoglu, Koray. Conditional image generation with pixelcnn decoders. NIPS , 2016b.
- Vaswani, Ashish, Shazeer, Noam, Parmar, Niki, Uszkoreit, Jakob, Jones, Llion, Gomez, Aidan N., Kaiser, Lukasz, and Polosukhin, Illia. Attention is all you need. 2017. URL http://arxiv.org/abs/1706.03762 . · arXiv
- Vaswani, Ashish, Bengio, Samy, Brevdo, Eugene, Chollet, Francois, Gomez, Aidan N., Gouws, Stephan, Jones, Llion, Kaiser, Łukasz, Kalchbrenner, Nal, Parmar, Niki, Sepassi, Ryan, Shazeer, Noam, and Uszkoreit, Jakob. Tensor2tensor for neural machine translation. CoRR , abs/1803.07416, 2018. URL http://arxiv.org/ abs/1803.07416 . · arXiv
- Zhang, Han, Xu, Tao, Li, Hongsheng, Zhang, Shaoting, Huang, Xiaolei, Wang, Xiaogang, and Metaxas, Dimitris N. Stackgan: Text to photo-realistic image synthesis with stacked generative adversarial networks. CoRR , abs/1612.03242, 2016. URL http://arxiv.org/ abs/1612.03242 . · arXiv
Cited by
Sorted by citations
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.11929 · Cited by 21,793 · see citing works
“Parmar et al. (2018) applied the self-attention only in local neighborhoods for each query pixel instead of globally.”
Self-citationEnd-to-End Object Detection with Transformers
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58452-8_13 · Cited by 18,285 · see citing works
TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation
Chen, J., Lu, Y., Yu, Q., Luo, X., Adeli, E., Wang, Y., Lu, L., Yuille, A. L., & Zhou, Y. (2021) · arXiv · doi:10.48550/arxiv.2102.04306 · Cited by 4,206 · see citing works
“[12] applied the self-attention only in local neighborhoods for each query pixel instead of globally.”
MethodVision Transformers for Dense Prediction
Ranftl, R., Bochkovskiy, A., & Koltun, V. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01196 · Cited by 2,373 · see citing works
Scalable Diffusion Models with Transformers
Peebles, W., & Xie, S. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00387 · Cited by 2,284 · see citing works
Deformable DETR: Deformable Transformers for End-to-End Object Detection
Zhu, X., Su, W., Lu, L., Li, B., Wang, X., & Dai, J. (2020) · arXiv · doi:10.48550/arxiv.2010.04159 · Cited by 1,837 · see citing works
“In the image domain, the designs of efficient attention mechanism (e.g., Parmar et al. (2018) 2019) admit such approaches are much slower in implementation than traditional convolution with the same FLOPs (at least 3 × slower), due to the intrinsic limitation in memory access patterns.”
BackgroundIs Space-Time Attention All You Need for Video Understanding?
Bertasius, G., Wang, H., & Torresani, L. (2021) · arXiv · doi:10.48550/arxiv.2102.05095 · Cited by 1,663 · see citing works
Chen, X., Yan, B., Zhu, J., Wang, D., Yang, X., & Lu, H. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00803 · Cited by 1,588 · see citing works
Lin, T., Wang, Y., Liu, X., & Qiu, X. (2022) · AI Open · doi:10.1016/j.aiopen.2022.10.001 · Cited by 1,422 · see citing works
TrackFormer: Multi-Object Tracking with Transformers
Meinhardt, T., Kirillov, A., Leal-Taixe, L., & Feichtenhofer, C. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00864 · Cited by 1,081 · see citing works
“Waveglow: a flow-based generative network for speech synthesis,”
Prenger, R., Valle, R., & Catanzaro, B. (2019) · ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2019.8683143 · Cited by 1,004 · see citing works
Rethinking Attention with Performers
Choromanski, K., Likhosherstov, V., Dohan, D., Song, X., Gane, A., Sarlos, T., Hawkins, P., Davis, J., Mohiuddin, A., Kaiser, L., et al. (2020) · arXiv · doi:10.48550/arxiv.2009.14794 · Cited by 695 · see citing works
“Most approaches restrict the attention mechanism to attend to local neighborhoods (Parmar et al., 2018) or incorporate structural priors on attention such as sparsity (Child et al.”
BackgroundMethodInfluentialSelf-citationDo Vision Transformers See Like Convolutional Neural Networks?
Raghu, M., Unterthiner, T., Kornblith, S., Zhang, C., & Dosovitskiy, A. (2021) · arXiv · doi:10.48550/arxiv.2108.08810 · Cited by 663 · see citing works
Conditional Positional Encodings for Vision Transformers
Chu, X., Tian, Z., Zhang, B., Wang, X., & Shen, C. (2021) · arXiv · doi:10.48550/arxiv.2102.10882 · Cited by 489 · see citing works
MaskGIT: Masked Generative Image Transformer
Chang, H., Zhang, H., Jiang, L., Liu, C., & Freeman, W. T. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01103 · Cited by 446 · see citing works
Skeleton-based action recognition via spatial and temporal transformer networks
Plizzari, C., Cannici, M., & Matteucci, M. (2021) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2021.103219 · Cited by 409 · see citing works
Axial Attention in Multidimensional Transformers
Ho, J., Kalchbrenner, N., Weissenborn, D., & Salimans, T. (2019) · arXiv · doi:10.48550/arxiv.1912.12180 · Cited by 370 · see citing works
Pixel-Adaptive Convolutional Neural Networks
Su, H., Jampani, V., Sun, D., Gallo, O., Learned-Miller, E., & Kautz, J. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.01142 · Cited by 348 · see citing works
Vision Transformers Are Robust Learners
Paul, S., & Chen, P.-Y. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i2.20103 · Cited by 213 · see citing works
SinSR: Diffusion-Based Image Super-Resolution in a Single Step
Wang, Y., Yang, W., Chen, X., Wang, Y., Guo, L., Chau, L.-P., Liu, Z., Qiao, Y., Kot, A. C., & Wen, B. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.02437 · Cited by 196 · see citing works
MelNet: a generative model for audio in the frequency domain
Vasquez, S., & Lewis, M. (2019) · arXiv · doi:10.48550/arxiv.1906.01083 · Cited by 107 · see citing works
Wang, P., Zheng, W., Chen, T., & Wang, Z. (2022) · arXiv · doi:10.48550/arxiv.2203.05962 · Cited by 80 · see citing works
“…et al., 2020; Zhu et al., 2021; Zheng et al., 2021a; Sun et al., 2020), segmentation (Chen et al., 2021a; Wang et al., 2021), image generation (Parmar et al., 2018; Jiang et al., 2021a), video processing (Zhou et al., 2018; Arnab et al., 2021), and 3D instance processing (Guo et al., 2021;…”
MethodAutoregressive Unsupervised Image Segmentation
Ouali, Y., Hudelot, C., & Tami, M. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58571-6_9 · Cited by 75 · see citing works
Probabilistic simulation of quantum circuits using a deep-learning architecture
Carrasquilla, J., Luo, D., Pérez, F., Milsted, A., Clark, B. K., Volkovs, M., & Aolita, L. (2021) · Physical Review A · doi:10.1103/physreva.104.032610 · Cited by 53 · see citing works
Deepfakes on Demand: The rise of accessible non-consensual deepfake image generators
Hawkins, W., Mittelstadt, B., & Russell, C. (2025) · Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3715275.3732107 · Cited by 20 · see citing works
“2 Related Work 2.1 Text-to-Image Models T2I generation has rapidly advanced within the last decade, thanks to breakthroughs in techniques including generative adversarial networks (GANs), transformer-based image generation, and diffusion models [15, 38, 48].”
InfluentialRigAnything: Template-Free Autoregressive Rigging for Diverse 3D Assets
Liu, I., Xu, Z., Yifan, W., Tan, H., Xu, Z., Wang, X., Su, H., & Shi, Z. (2025) · ACM Transactions on Graphics · doi:10.1145/3731149 · Cited by 12 · see citing works
Mehta, D., Betz-Stablein, B., Nguyen, T. D., Gal, Y., Bowling, A., Haskett, M., Sashindranath, M., Bonnington, P., Mar, V., Soyer, H. P., et al. (2023) · arXiv · doi:10.48550/arxiv.2311.01009 · Cited by 3 · see citing works
“Each encoder layer in the Transformer includes a standard architecture comprising a multi-head self-attention module and a feed-forward network (FFN), further supplemented with fixed positional encoding [47, 48] that is added to the input of each attention layer.”
Method
- Authors
- Parmar, N., Vaswani, A., Uszkoreit, J., Kaiser, Ł., Shazeer, N., Ku, A., & Tran, D.
- Affiliations
- Google (United States) (US)
- University of California, Berkeley (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2018
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1802.05751
- OpenAlex
- W4295224299
- Cited by
- 1,085
- References
- 0
- Citations by year
- 2026: 2 · 2025: 83 · 2024: 191 · 2023: 260 · 2022: 238 · 2021: 216 · 2020: 54 · 2019: 34 · 2018: 6
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Image Processing Techniques
- Keywords
- image generation, self-attention, transformer models, local attention, image super-resolution, ImageNet, encoder-decoder architecture
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
