CoChat
Sign Up
PreprintCited by 3,353Generative Adversarial Networks and Image SynthesisAdvanced Neural Network ApplicationsDigital Media Forensic Detection

Self-Attention Generative Adversarial Networks

Open access
Zhang, Han, Goodfellow, Ian, Metaxas, Dimitris, Odena, Augustus·2018·arXiv
DOI 10.48550/arxiv.1805.08318arXiv 1805.08318
Open this paper in CoChat — free accountOpen PDF

Abstract

In this paper, we propose the Self-Attention Generative Adversarial Network (SAGAN) which allows attention-driven, long-range dependency modeling for image generation tasks. Traditional convolutional GANs generate high-resolution details as a function of only spatially local points in lower-resolution feature maps. In SAGAN, details can be generated using cues from all feature locations. Moreover, the discriminator can check that highly detailed features in distant portions of the image are consistent with each other. Furthermore, recent work has shown that generator conditioning affects GAN performance. Leveraging this insight, we apply spectral normalization to the GAN generator and find that this improves training dynamics. The proposed SAGAN achieves the state-of-the-art results, boosting the best published Inception score from 36.8 to 52.52 and reducing Frechet Inception distance from 27.62 to 18.65 on the challenging ImageNet dataset. Visualization of the attention layers shows that the generator leverages neighborhoods that correspond to object shapes rather than local regions of fixed shape.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1805.08318

Cites

0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Recent works on adversarial perturbation attacks to fool DNN-based detectors make the deepfake detection task more difficult.

    “Recent works on adversarial perturbation attacks to fool DNN-based detectors make the deepfake detection task more difficult [157, 171–174].”

    From Deep learning for deepfakes creation and detection: A survey (2022) · cites this paper as [173]

    The Introduction (p. 1) and Related Work (p. 2) discuss GAN-based image synthesis, while the self-attention section (p. 3) describes a GAN discriminator. Searches for adversarial perturbation attacks, DNN-based detectors, and deepfake detection found no passage addressing whether such attacks make detection more difficult. (LLM, conf=0.76)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 49

  1. Arjovsky, M., Chintala, S., and Bottou, L. Wasserstein GAN. arXiv:1701.07875 , 2017. · arXiv
  2. Azadi, S., Olsson, C., Darrell, T., Goodfellow, I., and Odena, A. Discriminator rejection sampling. arXiv preprint arXiv:1810.06758 , 2018. · arXiv
  3. Bahdanau, D., Cho, K., and Bengio, Y. Neural machine translation by jointly learning to align and translate. arXiv:1409.0473 , 2014. · arXiv
  4. Brock, A., Donahue, J., and Simonyan, K. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv:1809.11096 , 2018. · arXiv
  5. Che, T., Li, Y., Jacob, A. P., Bengio, Y., and Li, W. Mode regularized generative adversarial networks. In ICLR , 2017.
  6. Chen, X., Mishra, N., Rohaninejad, M., and Abbeel, P. Pixelsnail: An improved autoregressive generative model. In ICML , 2018.
  7. Cheng, J., Dong, L., and Lapata, M. Long short-term memory-networks for machine reading. In EMNLP , 2016.
  8. Goodfellow, I. J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A. C., and Bengio, Y. Generative adversarial nets. In NIPS , 2014.
  9. Gregor, K., Danihelka, I., Graves, A., Rezende, D. J., and Wierstra, D. DRAW: A recurrent neural network for image generation. In ICML , 2015.
  10. Gulrajani, I., Ahmed, F., Arjovsky, M., Dumoulin, V., and Courville, A. C. Improved training of wasserstein GANs. In NIPS , 2017.
  11. Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, S. GANs trained by a two time-scale update rule converge to a local nash equilibrium. In NIPS , pp. 6629-6640, 2017.
  12. Hong, S., Yang, D., Choi, J., and Lee, H. Inferring semantic layout for hierarchical text-to-image synthesis. In CVPR , 2018.
  13. Isola, P., Zhu, J.-Y ., Zhou, T., and Efros, A. A. Image-toimage translation with conditional adversarial networks. In CVPR , 2017.
  14. Jolicoeur-Martineau, A. The relativistic discriminator: a key element missing from standard GAN. In ICLR , 2019.
  15. Karras, T., Aila, T., Laine, S., and Lehtinen, J. Progressive growing of GANs for improved quality, stability, and variation. In ICLR , 2018.
  16. Karras, T., Laine, S., and Aila, T. A style-based generator architecture for generative adversarial networks. In CVPR , 2019.
  17. Khrulkov, V. and Oseledets, I. Geometry score: A method for comparing generative adversarial networks. arXiv preprint arXiv:1802.02664 , 2018. · arXiv
  18. Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. In ICLR , 2015.
  19. Ledig, C., Theis, L., Huszar, F., Caballero, J., Aitken, A., Tejani, A., Totz, J., Wang, Z., and Shi, W. Photo-realistic single image super-resolution using a generative adversarial network. In CVPR , 2017.
  20. Lim, J. H. and Ye, J. C. Geometric GAN. arXiv:1705.02894 , 2017. · arXiv
  21. Liu, M. and Tuzel, O. Coupled generative adversarial networks. In NIPS , 2016.
  22. Metz, L., Poole, B., Pfau, D., and Sohl-Dickstein, J. Unrolled generative adversarial networks. In ICLR , 2017.
  23. Miyato, T. and Koyama, M. cGANs with projection discriminator. In ICLR , 2018.
  24. Miyato, T., Kataoka, T., Koyama, M., and Yoshida, Y. Spectral normalization for generative adversarial networks. In ICLR , 2018.
  25. Odena, A. Faster asynchronous sgd. arXiv preprint arXiv:1601.04033 , 2016. · arXiv
  26. Odena, A., Olah, C., and Shlens, J. Conditional image synthesis with auxiliary classifier GANs. In ICML , 2017.
  27. Odena, A., Buckman, J., Olsson, C., Brown, T. B., Olah, C., Raffel, C., and Goodfellow, I. Is generator conditioning causally related to GAN performance? In ICML , 2018.
  28. Olsson, C., Bhupatiraju, S., Brown, T., Odena, A., and Goodfellow, I. Skill rating for generative models. arXiv preprint arXiv:1808.04888 , 2018. · arXiv
  29. Parikh, A. P., T¨ ackstr¨ om, O., Das, D., and Uszkoreit, J. A decomposable attention model for natural language inference. In EMNLP , 2016.
  30. Park, T., Liu, M., Wang, T., and Zhu, J. Semantic image synthesis with spatially-adaptive normalization. In CVPR , 2019.
  31. Parmar, N., Vaswani, A., Uszkoreit, J., ukasz Kaiser, Shazeer, N., and Ku, A. Image transformer. arXiv:1802.05751 , 2018. · arXiv
  32. Radford, A., Metz, L., and Chintala, S. Unsupervised representation learning with deep convolutional generative adversarial networks. In ICLR , 2016.
  33. Reed, S., Akata, Z., Mohan, S., Tenka, S., Schiele, B., and Lee, H. Learning what and where to draw. In NIPS , 2016a.
  34. Reed, S., Akata, Z., Yan, X., Logeswaran, L., Schiele, B., and Lee, H. Generative adversarial text-to-image synthesis. In ICML , 2016b.
  35. Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A. C., and Fei-Fei, L. ImageNet large scale visual recognition challenge. IJCV , 2015.
  36. Salimans, T., Goodfellow, I. J., Zaremba, W., Cheung, V., Radford, A., and Chen, X. Improved techniques for training GANs. In NIPS , 2016.
  37. Salimans, T., Zhang, H., Radford, A., and Metaxas, D. N. Improving GANs using optimal transport. In ICLR , 2018.
  38. Snderby, C. K., Caballero, J., Theis, L., Shi, W., and Huszar, F. Amortised map inference for image super-resolution. In ICLR , 2017.
  39. Taigman, Y., Polyak, A., and Wolf, L. Unsupervised crossdomain image generation. In ICLR , 2017.
  40. Tran, D., Ranganath, R., and Blei, D. M. Deep and hierarchical implicit models. arXiv:1702.08896 , 2017. · arXiv
  41. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention is all you need. arXiv:1706.03762 , 2017. · arXiv
  42. Wang, X., Girshick, R., Gupta, A., and He, K. Non-local neural networks. In CVPR , 2018.
  43. Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A. C., Salakhutdinov, R., Zemel, R. S., and Bengio, Y. Show, attend and tell: Neural image caption generation with visual attention. In ICML , 2015.
  44. Xu, T., Zhang, P., Huang, Q., Zhang, H., Gan, Z., Huang, X., and He, X. AttnGAN: Fine-grained text to image generation with attentional generative adversarial networks. In CVPR , 2018.
  45. Xue, Y., Xu, T., Zhang, H., Long, L. R., and Huang, X. SegAN: Adversarial network with multi-scale L1 loss for medical image segmentation. Neuroinformatics , pp. 1-10, 2018.
  46. Yang, Z., He, X., Gao, J., Deng, L., and Smola, A. J. Stacked attention networks for image question answering. In CVPR , 2016.
  47. Zhang, H., Xu, T., Li, H., Zhang, S., Wang, X., Huang, X., and Metaxas, D. StackGAN: Text to photo-realistic image synthesis with stacked generative adversarial networks. In ICCV , 2017.
  48. Zhou, S., Gordon, M., Krishna, R., Narcomey, A., Morina, D., and Bernstein, M. S. HYPE: human eye perceptual evaluation of generative models. CoRR , abs/1904.01121, 2019. URL http://arxiv.org/ abs/1904.01121 . · arXiv
  49. Zhu, J.-Y., Park, T., Isola, P., and Efros, A. A. Unpaired image-to-image translation using cycle-consistent adversarial networks. In ICCV , 2017.

Cited by

Sorted by citations

Related

Authors
Zhang, H., Goodfellow, I., Metaxas, D., & Odena, A.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2018
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1805.08318
arXiv
1805.08318
OpenAlex
W2804078698
Cited by
3,353
References
0
Citations by year
2026: 8 · 2025: 193 · 2024: 353 · 2023: 492 · 2022: 525 · 2021: 814 · 2020: 596 · 2019: 332 · 2018: 40
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Digital Media Forensic Detection
Keywords
self-attention, generative adversarial networks, image generation, long-range dependency modeling, spectral normalization, attention visualization, ImageNet, Inception Score, Fréchet Inception Distance

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review