The DOI matches its DataCite record; 6 of the 6 references it cites were found in a registry or an index.
Generating Images with Sparse Representations
Open accessAbstract
The high dimensionality of images presents architecture and sampling-efficiency challenges for likelihood-based generative models. Previous approaches such as VQ-VAE use deep autoencoders to obtain compact representations, which are more practical as inputs for likelihood-based models. We present an alternative approach, inspired by common image compression methods like JPEG, and convert images to quantized discrete cosine transform (DCT) blocks, which are represented sparsely as a sequence of DCT channel, spatial location, and DCT coefficient triples. We propose a Transformer-based autoregressive architecture, which is trained to sequentially predict the conditional distribution of the next element in such sequences, and which scales effectively to high resolution images. On a range of image datasets, we demonstrate that our approach can generate high quality, diverse images, with sample metric scores competitive with state of the art methods. We additionally show that simple modifications to our method yield effective image colorization and super-resolution models.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Likelihood-based models still fall short in terms of visual sample quality compared to GANs.
“While these models capture more diversity and are typically easier to scale and train than GANs, they still fall short in terms of visual sample quality.”
From Diffusion Models Beat GANs on Image Synthesis (2021) · cites this paper as [42]
The Image generation benchmarks section reports that GANs had the strongest precision and FID scores, and that DCTransformer samples were less reliably coherent than GAN samples. But it calls the overall comparison mixed: DCTransformer led on spatial FID in three of five datasets, and the Introduction describes its sample quality as competitive with state-of-the-art GANs. The source therefore supports a qualified quality gap, not a blanket claim that likelihood-based models fall short. (LLM, conf=0.84)
Experimental — accuracy not yet measuredLikelihood-based models are typically easier to scale and train than GANs.
“While these models capture more diversity and are typically easier to scale and train than GANs, they still fall short in terms of visual sample quality.”
From Diffusion Models Beat GANs on Image Synthesis (2021) · cites this paper as [42]
The Introduction says likelihood-based training tends to be more stable than adversarial training, supporting the training comparison. It does not establish that likelihood-based image models are easier to scale; the Discussion instead says good results on complex, high-resolution datasets require substantial computational resources, while GANs achieve high quality with less. (LLM, conf=0.88)
Experimental — accuracy not yet measuredsFID better captures spatial relationships, rewarding image distributions with coherent high-level structure.
“They find that this metric better captures spatial relationships, rewarding image distributions with coherent high-level structure.”
From Diffusion Models Beat GANs on Image Synthesis (2021) · cites this paper as [42]
In §4, the authors say sFID uses intermediate spatial features because standard FID features compress spatial information, supporting the spatial-relationships part. But §4.1 says DCTransformer performs better on sFID despite producing samples that are less reliably coherent than GAN samples. The paper therefore does not support the claim that sFID rewards coherent high-level structure. (LLM, conf=0.86)
Experimental — accuracy not yet measuredGANs capture less diversity than state-of-the-art likelihood-based models.
“it has been shown that GANs capture less diversity than state-of-the-art likelihood-based models [51, 43, 42].”
From Diffusion Models Beat GANs on Image Synthesis (2021) · cites this paper as [42]
Some sample quality metrics do not fully capture diversity.
“However, some of these metrics do not fully capture diversity”
From Diffusion Models Beat GANs on Image Synthesis (2021) · cites this paper as [42]
Likelihood-based models capture more diversity than GANs.
“While these models capture more diversity and are typically easier to scale and train than GANs, they still fall short in terms of visual sample quality.”
From Diffusion Models Beat GANs on Image Synthesis (2021) · cites this paper as [42]
Much work has been done to achieve GAN-like sample quality with likelihood-based models.
“As a result, much work has been done to achieve GAN-like sample quality with likelihood-based models [51, 25, 42, 9].”
From Diffusion Models Beat GANs on Image Synthesis (2021) · cites this paper as [42]
sFID uses spatial features rather than the standard pooled features.
“Recently, sFID was proposed by Nash et al. [42] as a version of FID that uses spatial features rather than the standard pooled features.”
From Diffusion Models Beat GANs on Image Synthesis (2021) · cites this paper as [42]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Image quality assessment: from error visibility to structural similarity
Wang, Z., Bovik, A. C., Sheikh, H. R., & Simoncelli, E. P. (2004) · IEEE Transactions on Image Processing · doi:10.1109/tip.2003.819861 · Cited by 57,892
A Style-Based Generator Architecture for Generative Adversarial Networks
Karras, T., Laine, S., & Aila, T. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00453 · Cited by 11,714
Scaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · doi:10.48550/arxiv.2001.08361 · Cited by 9,550 · cited 1×
“Kaplan et al. (2020) show that for Transformer language models, performance improves reliably subject to constraints on model size, compute and dataset size.”
BackgroundGANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., & Hochreiter, S. (2017) · arXiv · doi:10.48550/arxiv.1706.08500 · Cited by 8,365
Large Scale GAN Training for High Fidelity Natural Image Synthesis
Brock, A., Donahue, J., & Simonyan, K. (2018) · arXiv · doi:10.48550/arxiv.1809.11096 · Cited by 3,717
Rethinking Attention with Performers
Choromanski, K., Likhosherstov, V., Dohan, D., Song, X., Gane, A., Sarlos, T., Hawkins, P., Davis, J., Mohiuddin, A., Kaiser, L., et al. (2020) · arXiv · doi:10.48550/arxiv.2009.14794 · Cited by 695 · cited 1×
“While memory-efficient Transformer variants (Child et al., 2019; Kitaev et al., 2020; Choromanski et al., 2021; Dhari-wal et al., 2020; Zaheer et al., 2021; Roy et al., 2020) make long-sequence training more practical, care is still required to optimize model hyperparameters against available memory…”
Background
Flagged first, then by citations
Research works printed but not linked: 31
- Ahmed, N., Natarajan, T., and Rao, K. R. Discrete cosine transform. IEEE transactions on Computers, 100(1):90- 93, 1974.
- Bachlechner, T., Majumder, B. P., Mao, H. H., Cottrell, G. W., and McAuley, J. J. Rezero is all you need: Fast convergence at large depth. CoRR, abs/2003.04887, 2020. · arXiv
- Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. Language models are few-shot learners. In NeurIPS, 2020.
- Chen, M., Radford, A., Child, R., Wu, J., Jun, H., Luan, D., and Sutskever, I. Generative pretraining from pixels. In ICML, volume 119 of Proceedings of Machine Learning Research, pp. 1691-1703. PMLR, 2020.
- Child, R., Gray, S., Radford, A., and Sutskever, I. Gener- ating long sequences with sparse transformers. CoRR, abs/1904.10509, 2019. · arXiv
- De, S. and Smith, S. L. Batch normalization biases residual blocks towards the identity function in deep networks. In NeurIPS, 2020.
- Du, Y. and Mordatch, I. Implicit generation and modeling with energy based models. In NeurIPS, 2019.
- Goodfellow, I. J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A. C., and Bengio, Y. Generative adversarial nets. In NIPS, pp. 2672-2680, 2014.
- Johnson, J., Hariharan, B., van der Maaten, L., Fei-Fei, L., Zitnick, C. L., and Girshick, R. B. CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning. In CVPR, pp. 1988-1997. IEEE Computer Society, 2017.
- Karras, T., Aila, T., Laine, S., and Lehtinen, J. Progres- sive growing of gans for improved quality, stability, and variation. In ICLR. OpenReview.net, 2018.
- Karras, T., Laine, S., Aittala, M., Hellsten, J., Lehtinen, J., and Aila, T. Analyzing and improving the image quality of stylegan. In CVPR, pp. 8107-8116. IEEE, 2020.
- Kersten, D. Predictability and redundancy of natural images. JOSA A, 4(12):2395-2400, 1987.
- Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. In ICLR (Poster), 2015.
- Kingma, D. P. and Welling, M. Auto-encoding variational bayes. In ICLR, 2014.
- Kitaev, N., Kaiser, L., and Levskaya, A. Reformer: The efficient transformer. In ICLR. OpenReview.net, 2020.
- Kuznetsova, A., Rom, H., Alldrin, N., Uijlings, J. R. R., Krasin, I., Pont-Tuset, J., Kamali, S., Popov, S., Mal- loci, M., Duerig, T., and Ferrari, V. The open images dataset V4: unified image classification, object detec- tion, and visual relationship detection at scale. CoRR, abs/1811.00982, 2018. · arXiv
- Kynkäänniemi, T., Karras, T., Laine, S., Lehtinen, J., and Aila, T. Improved precision and recall metric for assess- ing generative models. In NeurIPS, pp. 3929-3938, 2019.
- Mandava, S., Migacz, S., and Florea, A. F. Pay attention when required. CoRR, abs/2009.04534, 2020. · arXiv
- Menick, J. and Kalchbrenner, N. Generating high fidelity images with subscale pixel networks and multidimen- sional upscaling. CoRR, abs/1812.01608, 2018. URL http://arxiv.org/abs/1812.01608. · arXiv
- Oord, A. v. d., Vinyals, O., and Kavukcuoglu, K. Neu- ral discrete representation learning. arXiv preprint arXiv:1711.00937, 2017. · arXiv
- Parisotto, E., Song, H. F., Rae, J. W., Pascanu, R., Gülc ¸ehre, C ¸., Jayakumar, S. M., Jaderberg, M., Kaufman, R. L., Clark, A., Noury, S., Botvinick, M., Heess, N., and Had- sell, R. Stabilizing transformers for reinforcement learn- ing. In ICML, volume 119 of Proceedings of Machine Learning Research, pp. 7487-7498. PMLR, 2020.
- Razavi, A., van den Oord, A., and Vinyals, O. Generating diverse high-fidelity images with VQ-VAE-2. In NeurIPS, pp. 14837-14847, 2019.
- Rezende, D. J. and Mohamed, S. Variational inference with normalizing flows. In ICML, volume 37 of JMLR Workshop and Conference Proceedings, pp. 1530-1538. JMLR.org, 2015.
- Rezende, D. J., Mohamed, S., and Wierstra, D. Stochastic backpropagation and approximate inference in deep gen- erative models. In ICML, volume 32 of JMLR Workshop and Conference Proceedings, pp. 1278-1286. JMLR.org, 2014.
- Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M. S., Berg, A. C., and Li, F. Imagenet large scale visual recognition challenge. Int. J. Comput. Vis., 115(3):211- 252, 2015.
- Shannon, C. E. Prediction and entropy of printed english. Bell system technical journal, 30(1):50-64, 1951.
- Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., and Wojna, Z. Rethinking the inception architecture for computer vision. In CVPR, pp. 2818-2826. IEEE Computer Society, 2016.
- Van Oord, A., Kalchbrenner, N., and Kavukcuoglu, K. Pixel recurrent neural networks. In International Conference on Machine Learning, pp. 1747-1756. PMLR, 2016.
- Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention is all you need. arXiv preprint arXiv:1706.03762, 2017. · arXiv
- Wallace, G. K. The jpeg still picture compression standard. IEEE transactions on consumer electronics, 38(1):xviii- xxxiv, 1992.
- Yu, F., Zhang, Y., Song, S., Seff, A., and Xiao, J. Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv:1506.03365, 2015. · arXiv
Cited by
Sorted by citations
Diffusion Models Beat GANs on Image Synthesis
Dhariwal, P., & Nichol, A. (2021) · arXiv · doi:10.48550/arxiv.2105.05233 · Cited by 3,362
Scalable Diffusion Models with Transformers
Peebles, W., & Xie, S. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00387 · Cited by 2,284
MaskGIT: Masked Generative Image Transformer
Chang, H., Zhang, H., Jiang, L., Liu, C., & Freeman, W. T. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01103 · Cited by 446
Metadata
- Authors
- Nash, C., Menick, J., Dieleman, S., & Battaglia, P. W.
- Affiliations
- Google (United States) (US)
- Google DeepMind (United Kingdom) (GB)
- ORCID
- Battaglia, Peter W.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2021
- Pages
- 7958-7968
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2103.03841
- OpenAlex
- W3135058862
- Cited by
- 226
- References
- 6
- Citations by year
- 2026: 48 · 2025: 65 · 2024: 55 · 2023: 36 · 2022: 16 · 2021: 6
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- International Conference on Machine Learning
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Image Processing Techniques
- Keywords
- image generation, sparse representation, discrete cosine transform, autoregressive models, transformer models, image compression, image colorization, super-resolution, high-resolution image generation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
