CoChat
Sign Up
PreprintCited by 38Generative Adversarial Networks and Image SynthesisMultimodal Machine Learning ApplicationsDomain Adaptation and Few-Shot Learning

LAFITE: Towards Language-Free Training for Text-to-Image Generation

Open access
Zhou, Yufan, Zhang, Ruiyi, Chen, Changyou, Li, Chunyuan, Tensmeyer, Chris, Yu, Tong, Gu, Jiuxiang, Xu, Jinhui +1 more·2021·arXiv
DOI 10.48550/arxiv.2111.13792arXiv 2111.13792
Open this paper in CoChat — free accountOpen PDF

Abstract

One of the major challenges in training text-to-image generation models is the need of a large number of high-quality image-text pairs. While image samples are often easily accessible, the associated text descriptions typically require careful human captioning, which is particularly time- and cost-consuming. In this paper, we propose the first work to train text-to-image generation models without any text data. Our method leverages the well-aligned multi-modal semantic space of the powerful pre-trained CLIP model: the requirement of text-conditioning is seamlessly alleviated via generating text features from image features. Extensive experiments are conducted to illustrate the effectiveness of the proposed method. We obtain state-of-the-art results in the standard text-to-image generation tasks. Importantly, the proposed language-free model outperforms most existing models trained with full image-text pairs. Furthermore, our method can be applied in fine-tuning pre-trained models, which saves both training time and cost in training text-to-image generation models. Our pre-trained model obtains competitive results in zero-shot text-to-image generation on the MS-COCO dataset, yet with around only 1% of the model size and training data size relative to the recently proposed large DALL-E model.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2111.13792

Cites

2 of 2 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • On the MS-COCO validation set, our model improves upon powerful autoregressive and GAN-based methods for text-to-image generation.

    “For quantitative analysis, we follow prior work and evaluate text-to-image generation on the MS-COCO [51] validation set, where our model improves upon powerful AR [17, 66] and GAN-based [109] methods, cf . Tab. 2.”

    From High-Resolution Image Synthesis with Latent Diffusion Models (2022) · cites this paper as [109]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 51

  1. [1][1] Harsh Agrawal, Karan Desai, Yufei Wang, Xinlei Chen, Rishabh Jain, Mark Johnson, Dhruv Batra, Devi Parikh, Ste- fan Lee, and Peter Anderson. nocaps: novel object caption- ing at scale. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8948–8957, 2019. 2
  2. [2][2] Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pages 2425– 2433, 2015. 2
  3. [3][3] Eungchun Cho. Inner product of random vectors. Interna- tional Journal of Pure and Applied Mathematics, 56(2):217– 221, 2009. 12
  4. [4][4] Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. In EMNLP, 2014. 6
  5. [5][5] Karan Desai and Justin Johnson. Virtex: Learning visual representations from textual annotations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11162–11173, 2021. 2
  6. [6][6] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018. 2 · arXiv
  7. [9][9] Rinon Gal, Or Patashnik, Haggai Maron, Gal Chechik, and Daniel Cohen-Or. Stylegan-nada: Clip-guided do- main adaptation of image generators. arXiv preprint arXiv:2108.00946, 2021. 2, 3 · arXiv
  8. [10][10] Federico A Galatolo, Mario GCA Cimino, and Gigliola Vaglini. Generating images from caption and vice versa via clip-guided generative latent space search. arXiv preprint arXiv:2102.01645, 2021. 2, 3 · arXiv
  9. [11][11] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014. 2
  10. [12][12] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilib- rium. Advances in neural information processing systems, 30, 2017. 6
  11. [13][13] Tobias Hinz, Stefan Heinrich, and Stefan Wermter. Semantic object accuracy for generative text-to-image synthesis. IEEE Transactions on Pattern Analysis and Machine Intelligence, pages 1–1, 2020. 7
  12. [14][14] Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition, pages 6700–6709, 2019. 2
  13. [15][15] Jongheon Jeong and Jinwoo Shin. Training gans with stronger augmentations via contrastive discriminator. In In- ternational Conference on Learning Representations, 2020. 4
  14. [17][17] Minguk Kang and Jaesik Park. Contragan: Contrastive learn- ing for conditional image generation. Advances in Neural Information Processing Systems, 33:21357–21369, 2020. 4
  15. [18][18] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. arXiv preprint arXiv:1710.10196, 2017. 2 · arXiv
  16. [19][19] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4401–4410, 2019. 6
  17. [20][20] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and im- proving the image quality of styleGAN. arXiv preprint arXiv:1912.04958, 2019. 2 · arXiv
  18. [21][21] Wonjae Kim, Bokyung Son, and Ildoo Kim. Vilt: Vision- and-language transformer without convolution or region su- pervision. arXiv preprint arXiv:2102.03334, 2021. 2 · arXiv
  19. [22][22] Diederik P Kingma and Max Welling. Auto-encoding varia- tional bayes. arXiv preprint arXiv:1312.6114, 2013. 3 · arXiv
  20. [23][23] Chunyuan Li, Hao Liu, Changyou Chen, Yuchen Pu, Liqun Chen, Ricardo Henao, and Lawrence Carin. ALICE: To- wards understanding adversarial learning for joint distribu- tion matching. In Advances in Neural Information Process- ing Systems, 2017. 2
  21. [24][24] Junnan Li, Ramprasaath R Selvaraju, Akhilesh Deepak Got- mare, Shafiq Joty, Caiming Xiong, and Steven Hoi. Align be- fore fuse: Vision and language representation learning with momentum distillation. arXiv preprint arXiv:2107.07651, 2021. 2 · arXiv
  22. [25][25] Wenbo Li, Pengchuan Zhang, Lei Zhang, Qiuyuan Huang, Xiaodong He, Siwei Lyu, and Jianfeng Gao. Object-driven text-to-image synthesis via adversarial training. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12174–12182, 2019. 7
  23. [26][26] Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al. Oscar: Object-semantics aligned pre-training for vision-language tasks. In European Conference on Computer Vision, pages 121–137. Springer, 2020. 2
  24. [27][27] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740–755. Springer, 2014. 1, 2
  25. [28][28] Ming-Yu Liu, Thomas Breuel, and Jan Kautz. Unsupervised image-to-image translation networks. In Advances in neural information processing systems, pages 700–708, 2017. 2
  26. [30][30] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettle- moyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019. 2, 8 · arXiv
  27. [31][31] Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolinguistic represen- tations for vision-and-language tasks. arXiv preprint arXiv:1908.02265, 2019. 2 · arXiv
  28. [32][32] Mehdi Mirza and Simon Osindero. Conditional generative adversarial nets. arXiv preprint arXiv:1411.1784, 2014. 2 · arXiv
  29. [33][33] Daniil Pakhomov, Sanchit Hira, Narayani Wagle, Kemar E Green, and Nassir Navab. Segmentation in style: Unsuper- vised semantic image segmentation with stylegan and clip. arXiv preprint arXiv:2107.12518, 2021. 2, 3 · arXiv
  30. [34][34] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An im- perative style, high-performance deep learning library. Ad- vances in neural information processing systems, 32:8026– 8037, 2019. 6
  31. [35][35] Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski. Styleclip: Text-driven manipulation of stylegan imagery. arXiv preprint arXiv:2103.17249, 2021. 2, 3 · arXiv
  32. [36][36] Jordi Pont-Tuset, Jasper Uijlings, Soravit Changpinyo, Radu Soricut, and Vittorio Ferrari. Connecting vision and lan- guage with localized narratives. In European Conference on Computer Vision, pages 647–664. Springer, 2020. 6
  33. [37][37] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learn- ing transferable visual models from natural language super- vision. arXiv preprint arXiv:2103.00020, 2021. 2 · arXiv
  34. [38][38] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. arXiv preprint arXiv:2102.12092, 2021. 1, 2, 6, 7 · arXiv
  35. [39][39] Ali Razavi, Aaron van den Oord, and Oriol Vinyals. Generat- ing diverse high-fidelity images with vq-vae-2. In Advances in neural information processing systems, pages 14866– 14876, 2019. 2
  36. [40][40] Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in neural information processing systems, 29:2234–2242, 2016. 6
  37. [41][41] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, im- age alt-text dataset for automatic image captioning. In ACL, 2018. 1, 5
  38. [42][42] Richard Socher and Li Fei-Fei. Connecting modalities: Semi-supervised segmentation and annotation of images us- ing unaligned text corpora. In 2010 IEEE Computer Soci- ety Conference on Computer Vision and Pattern Recognition, pages 966–973. IEEE, 2010. 2
  39. [43][43] Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. Vl-bert: Pre-training of generic visual- linguistic representations. arXiv preprint arXiv:1908.08530, 2019. 2 · arXiv
  40. [45][45] Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learning. In Proceedings of the 31st International Conference on Neural Information Processing Systems, pages 6309–6318, 2017. 2
  41. [46][46] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko- reit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, 2017. 2
  42. [48][48] Liwei Wang, Yin Li, Jing Huang, and Svetlana Lazebnik. Learning two-branch neural networks for image-text match- ing tasks. IEEE Transactions on Pattern Analysis and Ma- chine Intelligence, 41(2):394–407, 2018. 2
  43. [49][49] Liwei Wang, Yin Li, and Svetlana Lazebnik. Learning deep structure-preserving image-text embeddings. In Proceed- ings of the IEEE conference on computer vision and pattern recognition, pages 5005–5013, 2016. 2
  44. [50][50] Jason Weston, Samy Bengio, and Nicolas Usunier. Large scale image annotation: learning to rank with joint word- image embeddings. Machine learning, 81(1):21–35, 2010. 2
  45. [51][51] Zongze Wu, Dani Lischinski, and Eli Shechtman. Stylespace analysis: Disentangled controls for stylegan image genera- tion. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pages 12863–12872, 2021. 4
  46. [52][52] Weihao Xia, Yujiu Yang, Jing-Hao Xue, and Baoyuan Wu. Tedigan: Text-guided diverse face image generation and ma- nipulation. In IEEE Conference on Computer Vision and Pat- tern Recognition (CVPR), 2021. 6
  47. [54][54] Hui Ye, Xiulong Yang, Martin Takac, Rajshekhar Sunderra- man, and Shihao Ji. Improving text-to-image synthesis us- ing contrastive learning. arXiv preprint arXiv:2107.02423, 2021. 6 · arXiv
  48. [55][55] Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. From recognition to cognition: Visual commonsense reason- ing. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pages 6720–6731, 2019. 2
  49. [57][57] Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang, Yejin Choi, and Jianfeng Gao. Vinvl: Revisiting visual representations in vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5579– 5588, 2021. 2, 6
  50. [58][58] Yuhao Zhang, Hang Jiang, Yasuhide Miura, Christopher D Manning, and Curtis P Langlotz. Contrastive learning of medical visual representations from paired images and text. arXiv preprint arXiv:2010.00747, 2020. 2 · arXiv
  51. [59][59] Minfeng Zhu, Pingbo Pan, Wei Chen, and Yi Yang. Dm-gan: Dynamic memory generative adversarial networks for text- to-image synthesis. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 5802–5810, 2019. 1, 2, 6, 7 A. Appendix A.1. Theoretical Results Theorem 1. For a given threshold c > 0, the generated text feature by LAFITEG satisfies Sim(fimg(xi),h ′ i) ≥ c with probability at least Prob(Sim(fimg(xi),h ′ i) ≥ c) =1− ∫ (c−1)/ξ+c −1 Γ(d/2 + 1/2)√ πΓ(d/2) (1− x2)d/2−1dx where d is the dimension number of features, Γ(z) =∫∞ 0 xz−1e−xdx is the Gamma function. Proof. Without loss of generality, we omit the subscript for clearness. Sim(fimg(x),h ′) = ⟨fimg(x),h ′⟩ ∥fimg(x)∥2∥h′∥2 = ⟨fimg(x), fimg(x) + ξϵ∥fimg(x)∥2/∥ϵ∥2⟩ ∥fimg(x)∥2 ∥∥∥fimg(x) + ξϵ∥fimg(x)∥2/∥ϵ∥2 ∥∥∥ 2 = ∥fimg(x)∥2 + ξϵ⊺fimg(x)∥fimg(x)∥2/∥ϵ∥2 ∥fimg(x)∥2 ∥∥∥fimg(x) + ξϵ∥fimg(x)∥2/∥ϵ∥2 ∥∥∥ 2 Denote a = fimg(x)/∥fimg(x)∥2, b = ϵ/∥ϵ∥2, then we have Sim(fimg(x),h ′) = ∥fimg(x)∥2 + ξϵ⊺fimg(x)∥fimg(x)∥2/∥ϵ∥2 ∥fimg(x)∥2 ∥∥∥fimg(x) + ξϵ∥fimg(x)∥2/∥ϵ∥2 ∥∥∥ 2 = 1 + ξa⊺b ∥a+ ξb∥2 ≥ 1 + ξa⊺b ∥a∥2 + ξ∥b∥2 = 1 + ξa⊺b

Cited by

Sorted by citations

Authors
Zhou, Y., Zhang, R., Chen, C., Li, C., Tensmeyer, C., Yu, T., Gu, J., Xu, J., & Sun, T.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2021
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2111.13792
arXiv
2111.13792
OpenAlex
W3216237230
Cited by
38
References
2
Citations by year
2025: 3 · 2024: 10 · 2023: 22 · 2022: 3
Type
posted content
Language
en
Open access
Licence
creativecommons.org/licenses/by/4.0/legalcode
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords
text-to-image generation, CLIP, image-text alignment, image-text pairs, pre-trained model fine-tuning, MS COCO

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review