MirrorGAN: Learning Text-To-Image Generation by Redescription
Open accessAbstract
Generating an image from a given text description has two goals: visual realism and semantic consistency. Although significant progress has been made in generating high-quality and visually realistic images using generative adversarial networks, guaranteeing semantic consistency between the text description and visual content remains very challenging. In this paper, we address this problem by proposing a novel global-local attentive and semantic-preserving text-to-image-to-text framework called MirrorGAN. MirrorGAN exploits the idea of learning text-to-image generation by redescription and consists of three modules: a semantic text embedding module (STEM), a global-local collaborative attentive module for cascaded image generation (GLAM), and a semantic text regeneration and alignment module (STREAM). STEM generates word- and sentence-level embeddings. GLAM has a cascaded architecture for generating target images from coarse to fine scales, leveraging both local word attention and global sentence attention to progressively enhance the diversity and semantic consistency of the generated images. STREAM seeks to regenerate the text description from the generated image, which semantically aligns with the given text description. Thorough experiments on two public benchmark datasets demonstrate the superiority of MirrorGAN over other representative state-of-the-art methods.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 2× · see citing works
“The image encoder is a convolutional neural network (CNN) [11] pretrained on ImageNet [5], and the decoder is a RNN [12].”
MethodHochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 3× · see citing works
“The image encoder is a convolutional neural network (CNN) [11] pretrained on ImageNet [5], and the decoder is a RNN [12].”
MethodImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · cited 1× · see citing works
“The image encoder is a convolutional neural network (CNN) [11] pretrained on ImageNet [5], and the decoder is a RNN [12].”
MethodBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · cited 1× · see citing works
“Second, we only utilize a basic method for text embedding in STEM and image captioning in STREAM, which could be further improved, for example, by using the recently proposed BERT model [7] and state-of-the-art image captioning models [2, 3].”
MethodMicrosoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460 · cited 1× · see citing works
“We evaluated our model on two commonly used datasets, CUB bird dataset [30] and MS COCO dataset [17].”
MethodLearning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1179 · Cited by 25,545 · cited 4× · see citing works
“The image encoder is a convolutional neural network (CNN) [11] pretrained on ImageNet [5], and the decoder is a RNN [12].”
MethodInfluentialUnpaired image-to-image translation using cycle-consistent adversarial networks
Zhu, J.-Y., Park, T., Isola, P., & Efros, A. A. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.244 · Cited by 23,602 · cited 1× · see citing works
“Similar ideas to our own have recently been used in CycleGAN and DualGAN, which handle the bi-directional translations within two domains together [42, 36, 1], significantly advance image-to-image translation [14, 28, 15, 37, 23].”
Image-to-Image Translation with Conditional Adversarial Networks
Isola, P., Zhu, J.-Y., Zhou, T., & Efros, A. A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.632 · Cited by 22,930 · cited 1× · see citing works
“Similar ideas to our own have recently been used in CycleGAN and DualGAN, which handle the bi-directional translations within two domains together [43, 37, 1, 32], significantly advance image-to-image translation [14, 28, 15, 38, 23].”
Labaca-Castro, R. (2023) · Springer eBooks · doi:10.1007/978-3-658-40442-0_9 · Cited by 21,100 · see citing works
GAN（Generative Adversarial Nets）
淳司, 柴. (2017) · Journal of Japan Society for Fuzzy Theory and Intelligent Informatics · doi:10.3156/jsoft.29.5_177_2 · Cited by 19,268 · see citing works
Perceptual Losses for Real-Time Style Transfer and Super-Resolution
Johnson, J., Alahi, A., & Fei-Fei, L. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46475-6_43 · Cited by 11,283 · cited 1× · see citing works
“Similar ideas to our own have recently been used in CycleGAN and DualGAN, which handle the bi-directional translations within two domains together [43, 37, 1, 32], significantly advance image-to-image translation [14, 28, 15, 38, 23].”
Bidirectional recurrent neural networks
Schuster, M., & Paliwal, K. K. (1997) · IEEE Transactions on Signal Processing · doi:10.1109/78.650093 · Cited by 10,172 · cited 2× · see citing works
“Followed [35], a pre-trained bi-directional LSTM [27] was used to calculate the semantic embedding from text descriptions.”
MethodEffective Approaches to Attention-based Neural Machine Translation
Luong, T., Pham, H., & Manning, C. D. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1166 · Cited by 8,382 · cited 1× · see citing works
“Attention models have been extensively exploited in computer vision and natural language processing, for instance in object detection [21, 6, 18, 41], image/video captioning [34, 9, 31], visual question answering [2, 33, 36, 22], and neural machine translation [19, 8].”
BackgroundShow, Attend and Tell: Neural Image Caption Generation with Visual Attention
Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhutdinov, R., Zemel, R., & Bengio, Y. (2015) · arXiv · doi:10.48550/arxiv.1502.03044 · Cited by 8,248 · cited 7× · see citing works
“As shown in Figure 1 (a) and (c), if an image generated by T2I is semantically consistent with the given text description, its redescription by I2T should have exactly the same semantics with the given text description.”
BackgroundInfluentialImproved Techniques for Training GANs
Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., & Chen, X. (2016) · arXiv · doi:10.48550/arxiv.1606.03498 · Cited by 6,965 · cited 13× · see citing works
“The Inception Score and the R-precision were calculated accordingly as in [38, 34].”
MethodInfluentialShow and tell: A neural image caption generator
Vinyals, O., Toshev, A., Bengio, S., & Erhan, D. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298935 · Cited by 6,318 · cited 8× · see citing works
“As shown in Figure 1 (a) and (c), if an image generated by T2I is semantically consistent with the given text description, its redescription by I2T should have exactly the same semantics with the given text description.”
BackgroundMethodInfluentialBottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering
Anderson, P., He, X., Buehler, C., Teney, D., Johnson, M., Gould, S. J., & Zhang, L. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00636 · Cited by 5,117 · cited 2× · see citing works
“Attention models have been extensively exploited in computer vision and natural language processing, for instance in object detection [21, 6, 18, 41], image/video captioning [34, 9, 31], visual question answering [2, 33, 36, 22], and neural machine translation [19, 8].”
BackgroundMethodDeep visual-semantic alignments for generating image descriptions
Karpathy, A., & Fei-Fei, L. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298932 · Cited by 4,902 · see citing works
Caltech-UCSD Birds-200-2011 Dataset
Wah, C. (2024) · CaltechAUTHORS (California Institute of Technology) · doi:10.57702/i9oeewf6 · Cited by 3,875 · see citing works
StackGAN: Text to Photo-Realistic Image Synthesis with Stacked Generative Adversarial Networks
Zhang, H., Xu, T., Li, H., Zhang, S., Wang, X., Huang, X., & Metaxas, D. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.629 · Cited by 3,021 · cited 12× · see citing works
“proposed StackGAN to generate images in two separate stages [39].”
BackgroundMethodInfluentialGenerative Image Inpainting with Contextual Attention
Yu, J., Lin, Z., Yang, J., Shen, X., Lu, X., & Huang, T. S. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00577 · Cited by 2,633 · cited 1× · see citing works
“Similar ideas to our own have recently been used in CycleGAN and DualGAN, which handle the bi-directional translations within two domains together [43, 37, 1, 32], significantly advance image-to-image translation [14, 28, 15, 38, 23].”
Generative Adversarial Text to Image Synthesis
Reed, S., Akata, Z., Yan, X., Logeswaran, L., Schiele, B., & Lee, H. (2016) · arXiv · doi:10.48550/arxiv.1605.05396 · Cited by 2,494 · cited 4× · see citing works
“First, we compare MirrorGAN with state-of-the-art text-to-image methods [24, 25, 39, 40, 20, 35] using the Inception Score and R-precision score on both CUB and COCO datasets.”
BackgroundMethodInfluentialDualGAN: Unsupervised Dual Learning for Image-to-Image Translation
Yi, Z., Zhang, H., Tan, P., & Gong, M. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.310 · Cited by 2,226 · cited 1× · see citing works
“Similar ideas to our own have recently been used in CycleGAN and DualGAN, which handle the bi-directional translations within two domains together [42, 36, 1], significantly advance image-to-image translation [14, 28, 15, 37, 23].”
Stacked Attention Networks for Image Question Answering
Yang, Z., He, X., Gao, J., Deng, L., & Smola, A. J. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.10 · Cited by 2,036 · cited 1× · see citing works
“Attention models have been extensively exploited in computer vision and natural language processing, for instance in object detection [21, 6, 18, 41], image/video captioning [34, 9, 31], visual question answering [2, 33, 36, 22], and neural machine translation [19, 8].”
BackgroundSaccade target selection and object recognition: Evidence for a common attentional mechanism
Deubel, H., & Schneider, W. X. (1996) · Vision Research · doi:10.1016/0042-6989(95)00294-4 · Cited by 2,004 · see citing works
AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks
Xu, T., Zhang, P., Huang, Q., Zhang, H., Gan, Z., Huang, X., & He, X. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00143 · Cited by 1,923 · see citing works
StackGAN++: Realistic Image Synthesis with Stacked Generative Adversarial Networks
Zhang, H., Xu, T., Li, H., Zhang, S., Wang, X., Huang, X., & Metaxas, D. (2018) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2856256 · Cited by 1,222 · cited 6× · see citing works
“First, we compare MirrorGAN with state-of-the-art text-to-image methods [24, 25, 39, 40, 20, 35] using the Inception Score and R-precision score on both CUB and COCO datasets.”
MethodResultInfluentialUnsupervised Cross-Domain Image GenerationPreprint
Taigman, Y., Polyak, A., & Wolf, L. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.02200 · Cited by 901 · cited 1× · see citing works
“Similar ideas to our own have recently been used in CycleGAN and DualGAN, which handle the bi-directional translations within two domains together [42, 36, 1], sig-nificantly advance image-to-image translation [14, 28, 15, 37, 23].”
MethodAsk, Attend and Answer: Exploring Question-Guided Spatial Attention for Visual Question Answering
Xu, H., & Saenko, K. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46478-7_28 · Cited by 753 · cited 1× · see citing works
“Attention models have been extensively exploited in computer vision and natural language processing, for instance in object detection [21, 6, 18, 41], image/video captioning [34, 9, 31], visual question answering [2, 33, 36, 22], and neural machine translation [19, 8].”
BackgroundProgressive Attention Guided Recurrent Network for Salient Object Detection
Zhang, X., Wang, T., Qi, J., Lu, H., & Wang, G. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00081 · Cited by 685 · cited 1× · see citing works
“Attention models have been extensively exploited in computer vision and natural language processing, for instance in object detection [21, 6, 18, 41], image/video captioning [34, 9, 31], visual question answering [2, 33, 36, 22], and neural machine translation [19, 8].”
BackgroundVideo Captioning With Attention-Based LSTM and Semantic Consistency
Gao, L., Guo, Z. Z., Zhang, H., Xu, X., & Shen, H. T. (2017) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2017.2729019 · Cited by 637 · cited 1× · see citing works
“dresses visual similarities. Attention models have been extensively exploited in computer vision and natural language processing, for instance in object detection [21,6,18,40], image/video captioning [33,9,31], visual question answering [2,32,35, 22], and neural machine translation [19,8]. Attention can be modeled spa…”
BackgroundPlug & Play Generative Networks: Conditional Iterative Generation of Images in Latent Space
Nguyen, A., Clune, J., Bengio, Y., Dosovitskiy, A., & Yosinski, J. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.374 · Cited by 597 · see citing works
Multi-Way, Multilingual Neural Machine Translation with a Shared Attention Mechanism
Fırat, O., Cho, K., & Bengio, Y. (2016) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n16-1101 · Cited by 570 · cited 1× · see citing works
“Attention models have been extensively exploited in computer vision and natural language processing, for instance in object detection [21, 6, 18, 41], image/video captioning [34, 9, 31], visual question answering [2, 33, 36, 22], and neural machine translation [19, 8].”
BackgroundTop-down control of visual attention in object detection
Oliva, A., Torralba, A. B., Castelhano, M. S., & Henderson, J. M. (2004) · Proceedings - International Conference on Image Processing · doi:10.1109/icip.2003.1246946 · Cited by 474 · see citing works
DecideNet: Counting Varying Density Crowds Through Attention Guided Detection and Density Estimation
Liu, J., Gao, C., Meng, D., & Hauptmann, A. G. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00545 · Cited by 423 · cited 1× · see citing works
“Attention models have been extensively exploited in computer vision and natural language processing, for instance in object detection [21, 6, 18, 41], image/video captioning [34, 9, 31], visual question answering [2, 33, 36, 22], and neural machine translation [19, 8].”
BackgroundLearning What and Where to DrawPreprint
Reed, S., Akata, Z., Mohan, S., Tenka, S., Schiele, B., & Lee, H. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1610.02454 · Cited by 411 · cited 4× · see citing works
“2.88 ± 0.04 7.88 ± 0.07 GAWWN [25] 3.62 ± 0.07 - StackGAN [38] 3.70 ± 0.04 8.45 ± 0.03 StackGAN++ [39] 3.82 ± 0.06 - PPGN [20] - 9.58 ± 0.21 AttnGAN [34] 4 It can be seen that the image details generated by At-tnGAN are lost, colors are inconsistent with the text descriptions ( 3 rd and 4 th column), and the shape loo…”
MethodInfluentialAugmented CycleGAN: Learning Many-to-Many Mappings from Unpaired DataPreprint
Almahairi, A., Rajeswar, S., Sordoni, A., Bachman, P., & Courville, A. C. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.10151 · Cited by 356 · cited 1× · see citing works
“Similar ideas to our own have recently been used in CycleGAN and DualGAN, which handle the bi-directional translations within two domains together [42, 36, 1], significantly advance image-to-image translation [14, 28, 15, 37, 23].”
Inferring Semantic Layout for Hierarchical Text-to-Image Synthesis
Hong, S., Yang, D., Choi, J., & Lee, H. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00833 · Cited by 348 · cited 1× · see citing works
“Although significant progress has been made in generating visually realistic images using generative adversarial networks (GANs) such as in [39, 42, 35, 13], guaranteeing semantic alignment of the generated image with the input text remains challenging.”
BackgroundPhotographic Text-to-Image Synthesis with a Hierarchically-Nested Adversarial Network
Zhang, Z., Xie, Y., & Yang, L. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00649 · Cited by 331 · cited 2× · see citing works
“proposed training a T2I generator with hierarchically nested adversarial objectives [42].”
BackgroundBidirectional Attentive Fusion with Context Gating for Dense Video Captioning
Wang, J., Jiang, W., Ma, L., Liu, W., & Xu, Y. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00751 · Cited by 231 · cited 1× · see citing works
“Attention models have been extensively exploited in computer vision and natural language processing, for instance in object detection [21, 6, 18, 41], image/video captioning [34, 9, 31], visual question answering [2, 33, 36, 22], and neural machine translation [19, 8].”
BackgroundExploring Human-Like Attention Supervision in Visual Question Answering
Qiao, T., Dong, J., & Xu, D. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.12272 · Cited by 100 · cited 1× · see citing works
“Attention models have been extensively exploited in computer vision and natural language processing, for instance in object detection [21, 6, 18, 41], image/video captioning [34, 9, 31], visual question answering [2, 33, 36, 22], and neural machine translation [19, 8].”
BackgroundSelf-citationGroupCap: Group-Based Image Captioning with Structured Relevance and Diversity Constraints
Chen, F., Ji, R., Sun, X., Wu, Y., & Su, J. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00146 · Cited by 85 · cited 1× · see citing works
“Second, we only utilize a basic method for text embedding in STEM and image captioning in STREAM, which could be further improved, for example, by using the recently proposed BERT model [7] and state-of-the-art image captioning models [2, 3].”
MethodExploiting Local Feature Patterns for Unsupervised Domain Adaptation
Wen, J., Liu, R., Zheng, N., Zheng, Q., Gong, Z., & Yuan, J. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33015401 · Cited by 42 · cited 1× · see citing works
“Similar ideas to our own have recently been used in CycleGAN and DualGAN, which handle the bi-directional translations within two domains together [43, 37, 1, 32], significantly advance image-to-image translation [14, 28, 15, 38, 23].”
Exploring Human-like Attention Supervision in Visual Question AnsweringPreprint
Qiao, T., Dong, J., & Xu, D. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1709.06308 · Cited by 42 · cited 1× · see citing works
“Attention models have been extensively exploited in computer vision and natural language processing, for instance in object detection [21, 6, 18, 41], image/video captioning [34, 9, 31], visual question answering [2, 33, 36, 22], and neural machine translation [19, 8].”
BackgroundSelf-citationAncient Painting to Natural Image: A New Solution for Painting Processing
Qiao, T., Zhang, W., Zhang, M., Ma, Z., & Xu, D. (2019) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv.2019.00061 · Cited by 20 · cited 1× · see citing works
“Similar ideas to our own have recently been used in CycleGAN and DualGAN, which handle the bi-directional translations within two domains together [43, 37, 1, 32], significantly advance image-to-image translation [14, 28, 15, 38, 23].”
Self-citationBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805
Flagged first, then by citations
Research works printed but not linked: 2
- 10.1109/iccv.2017.629 · DOI · Paper page
- taigman, 2017, Unsupervised crossdomain image generation, International Conference on Learning Representations (ICLR)
Cited by
Sorted by citations
DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation
Ruiz, N., Li, Y., Jampani, V., Pritch, Y., Rubinstein, M., & Aberman, K. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.02155 · Cited by 2,295 · see citing works
“There also exists work on text-to-image synthesis [14, 16, 19, 24, 27, 35, 36, 50, 51, 55, 58, 67, 74].”
BackgroundA Review on Generative Adversarial Networks: Algorithms, Theory, and Applications
Gui, J., Sun, Z., Wen, Y., Tao, D., & Ye, J. (2023) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2021.3130191 · Cited by 1,231 · see citing works
“Field Subfield Method Super-resolution SRGAN [63], ESRGAN [64], Cycle-in-Cycle GANs [65], SRDGAN [66], TGAN [67] DR-GAN [68], TP-GAN [69], PG2 [70], PSGAN [71], Image synthesis and manipulation APDrawingGAN [72], IGAN [73], Image processing and computer vision introspective adversarial networks [74], GauGAN [75] Textu…”
BackgroundSelf-citationBlended Diffusion for Text-driven Editing of Natural Images
Avrahami, O., Lischinski, D., & Fried, O. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01767 · Cited by 799 · see citing works
“Recent works that leverage the tremendous progress in vision-language models and datadriven image generation have demonstrated that text-based interfaces for image creation and manipulation are now finally within reach [12, 23, 29, 30, 38, 39, 41, 47, 51, 57].”
BackgroundZhang, J., & Tao, D. (2021) · IEEE Internet of Things Journal · doi:10.1109/jiot.2020.3039359 · Cited by 774 · see citing works
“…and text-to-image generation are two generative tasks related to cross-modal matching, where captioning refers to generating a piece of text description for a given image or video [179] while text-to-image generation aims to generate a realistic image that matches the given text description [180].”
BackgroundSelf-citationVector Quantized Diffusion Model for Text-to-Image Synthesis
Gu, S., Chen, D., Bao, J., Wen, F., Zhang, B., Chen, D., Yuan, L., & Guo, B. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01043 · Cited by 720 · see citing works
Multimodal intelligence: Representation learning, information fusion, and applications
Zhang, C., Yang, Z., He, X., & Deng, L. (2020) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2020.2987728 · Cited by 511 · see citing works
“Analogous to cycle consistency [203], MirrorGAN is proposed to improve the semantic consistency between the two modalities using an extra image captioning module [204].”
MethodText2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators
Khachatryan, L., Movsisyan, A., Tadevosyan, V., Henschel, R. D., Wang, Z., Navasardyan, S., & Shi, H. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01462 · Cited by 426 · see citing works
“These include StackGAN [46], AttnGAN [43], and MirrorGAN [24], which further improve image quality and diversity by introducing novel architectures and attention mechanisms.”
MethodCross-Modal Contrastive Learning for Text-to-Image Generation
Zhang, H., Koh, J. Y., Baldridge, J., Lee, H., & Yang, Y. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00089 · Cited by 359 · see citing works
DF-GAN: A Simple and Effective Baseline for Text-to-Image Synthesis
Tao, M., Tang, H., Wu, F., Jing, X., Bao, B., & Xu, C. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01602 · Cited by 399 · see citing works
“Although impressive results have been presented by previous works [9,19,21,32,33,51,60], there still remain three problems.”
BackgroundMethodResultInfluentialAdversarial Representation Learning for Text-to-Image Matching
Sarafianos, N., Xu, X., & Kakadiaris, I. A. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00591 · Cited by 258 · see citing works
“Flowers and CUB datasets, which is in line with existing work [42, 62, 70], this was not the case for the CUHK-PEDES and Flickr30K datasets.”
A Review on Generative Adversarial Networks: Algorithms, Theory, and ApplicationsPreprint
Gui, J., Sun, Z., Wen, Y., Tao, D., & Ye, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2001.06937 · Cited by 252 · see citing works
“Field Subfield Method Super-resolution SRGAN [63], ESRGAN [64], Cycle-in-Cycle GANs [65], SRDGAN [66], TGAN [67] DR-GAN [68], TP-GAN [69], PG2 [70], PSGAN [71], Image synthesis and manipulation APDrawingGAN [72], IGAN [73], Image processing and computer vision introspective adversarial networks [74], GauGAN [75] Textu…”
BackgroundSelf-citationMore Control for Free! Image Synthesis with Semantic Diffusion Guidance
Liu, X., Park, D. H., Azadi, S., Zhang, G., Chopikyan, A., Hu, Y., Shi, H., Rohrbach, A., & Darrell, T. (2023) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv56688.2023.00037 · Cited by 239 · see citing works
“Various methods have been studied, proposing many different text-adaptive architectures and loss functions to enforce better semantic alignment between the input text and generated image, resulting in significant improvements in editing quality and correctness [55,60,51,27,11,61,33,35,17,28,7,23,40].”
MethodAdversarial text-to-image synthesis: A review
Frolov, S., Hinz, T., Raue, F., Hees, J., & Dengel, A. R. (2021) · Neural Networks · doi:10.1016/j.neunet.2021.07.019 · Cited by 214 · see citing works
“Figure 9: Simplified MirrorGAN [68] architecture.”
MethodInfluentialDeep Graph-neighbor Coherence Preserving Network for Unsupervised Cross-modal Hashing
Yu, J., Zhou, H., Zhan, Y., & Tao, D. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i5.16592 · Cited by 207 · see citing works
“Due to the explosion of multimedia data on the Internet (Yu, Rui, and Chen 2013; Yang et al. 2020c), cross-modal retrieval has become an essential technique in information retrieval (Yang et al. 2020c) and multi-modal data mining (Qiao et al. 2019b,a; Yang et al. 2020a,c; Li et al. 2020).”
BackgroundSelf-citationText to Image Generation with Semantic-Spatial Aware GAN
Liao, W., Hu, K., Yang, M. Y., & Rosenhahn, B. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01765 · Cited by 194 · see citing works
“Most of the recent T2I methods follow this framework [16, 21, 4, 24, 33].”
MethodGALIP: Generative Adversarial CLIPs for Text-to-Image Synthesis
Tao, M., Bao, B., Tang, H., & Xu, C. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01366 · Cited by 181 · see citing works
3DALL-E: Integrating Text-to-Image AI in 3D Design Workflows
Liu, V., Vermeulen, J., Fitzmaurice, G. W., & Matejka, J. (2023) · Designing Interactive Systems Conference · doi:10.1145/3563657.3596098 · Cited by 179 · see citing works
“However, many early models were class-conditional, meaning that they were only robust at generating images from the classes they were trained on [39, 40, 53, 54, 66, 68].”
BackgroundVision-Language Pre-Training: Basics, Recent Advances, and Future Trends
Gan, Z., Li, L., Li, C., Wang, L., Liu, Z., & Gao, J. (2022) · Foundations and Trends® in Computer Graphics and Vision · doi:10.1561/0600000105 · Cited by 155 · see citing works
Semantic Object Accuracy for Generative Text-to-Image Synthesis
Hinz, T., Heinrich, S., & Wermter, S. (2020) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2020.3021209 · Cited by 152 · see citing works
“Similarly, [16] learn text-to-image by redescription, i.”
TextFuseNet: Scene Text Detection with Richer Fused Features
Ye, J., Chen, Z., Liu, J., et al. (2020) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2020/72 · Cited by 149 · see citing works
Elasri, M., Elharrouss, O., Al-Maadeed, S. A., & Tairi, H. (2022) · Neural Processing Letters · doi:10.1007/s11063-022-10777-x · Cited by 146 · see citing works
“[54] proposed MirrorGAN method that consist of generating images from text description by combining cascaded image generation (GLAM) module, semantic text embedding module (STEM), and a semantic text regeneration and alignment module (STREAM).”
MethodInfluentialMultimodal Image Synthesis and Editing: The Generative AI Era
Zhan, F., Yu, Y., Wu, R., Zhang, J., Lu, S., Liu, L., Kortylewski, A., Theobalt, C., & Xing, E. P. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3305243 · Cited by 162 · see citing works
“Specifically, some research [55], [151], [152], [153], [154] explores to pass the generated images through an inverse network to yield the conditional input, which imposes a cycle-consistency of conditional input.”
BackgroundSelf-citationPan, Y., Liu, M., Xia, Y., & Shen, D. G. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3091214 · Cited by 138 · see citing works
“Generative adversarial networks (GANs) have been developed for image transformation from a source domain/ modality to a target domain/modality [25], [26], [27], [28], [29], [30], [31], [32].”
BackgroundDAE-GAN: Dynamic Aspect-aware GAN for Text-to-Image Synthesis
Ruan, S., Zhang, Y., Zhang, K., Fan, Y., Tang, F., Liu, Q., & Chen, E. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01370 · Cited by 133 · see citing works
“[18] proposed MirroGAN by semantically aligning the re-description of the generated image with the given text description.”
MethodDream3D: Zero-Shot Text-to-3D Synthesis Using 3D Shape Prior and Text-to-Image Diffusion Models
Xu, J., Wang, X., Cheng, W., Cao, Y., Shan, Y., Qie, X., & Gao, S. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.02003 · Cited by 135 · see citing works
DF-GAN: Deep Fusion Generative Adversarial Networks for Text-to-Image SynthesisPreprint
Tao, M., Tang, H., Wu, S., Sebe, N., Wu, F. Q., & Jing, X. (2020) · arXiv (Cornell University) · Cited by 121 · see citing works
RiFeGAN: Rich Feature Generation for Text-to-Image Synthesis From Prior Knowledge
Cheng, J. S., Wu, F., Tian, Y., Wang, L., & Tao, D. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01092 · Cited by 119 · see citing works
“[20] introduce a MirrorGAN to connect the dual tasks, text-to-image synthesis and text captioning, and constrain the regenerated caption aligning with the given caption.”
BackgroundSelf-citationDeep learning and the global workspace theory
VanRullen, R., & Kanai, R. (2021) · Trends in Neurosciences · doi:10.1016/j.tins.2021.04.005 · Cited by 128 · see citing works
“Similar methods have been applied to neural translation between varied domains, e.g. unpaired image-to-image translation [49–51], text-to-image translation [26, 28, 52] or touch-to-image translation [32].”
MethodA survey and taxonomy of adversarial neural networks for text‐to‐image synthesis
Agnese, J., Herrera, J., Tao, H., & Xingquan Zhu, A. (2020) · Wiley Interdisciplinary Reviews Data Mining and Knowledge Discovery · doi:10.1002/widm.1345 · Cited by 112 · see citing works
“(Reed et al., 2016b) StackGAN (Zhang et al., 2017b) StackGAN++ (Zhang et al., 2018b) AttnGAN (Xu et al., 2017) DC-GAN (Reed et al., 2016b) HDGAN (Zhang et al., 2018d) MirrorGAN (Qiao et al., 2019)”
BackgroundInfluentialA Review of AI Image Generator: Influences, Challenges, and Future Prospects for Architectural Field
Enjellina, Beyan, E. V. P., & Rossy, A. G. C. (2023) · Journal of Artificial Intelligence in Architecture · doi:10.24002/jarina.v2i1.6662 · Cited by 124 · see citing works
Multimodal research in vision and language: A review of current and emerging trends
Uppal, S., Bhagat, S., Hazarika, D., Majumder, N., Poria, S., Zimmermann, R., & Zadeh, A. (2021) · Information Fusion · doi:10.1016/j.inffus.2021.07.009 · Cited by 112 · see citing works
An Overview on Generative AI at Scale With Edge–Cloud Computing
Wang, Y.-C., Xue, J., Wei, C., & Kuo, C. ‐. J. (2023) · IEEE Open Journal of the Communications Society · doi:10.1109/ojcoms.2023.3320646 · Cited by 117 · see citing works
“It is a specific category of AI that aims to autonomously generate new content that imitates the content created by humans in different modalities, including images [42], [94], audio [100], [101], text [13], [32], and even 3D objects [81], [87].”
BackgroundOpal: Multimodal Image Generation for News Illustration
Liu, V., Qiao, H., & Chilton, L. (2022) · Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology · doi:10.1145/3526113.3545621 · Cited by 104 · see citing works
“As tools moved away from statistical-based techniques to GANs, a number of text-to-image architectures [39, 40, 52, 53] were proposed.”
MethodSemantics-Enhanced Adversarial Nets for Text-to-Image Synthesis
Tan, H., Liu, X., Li, X. S., Zhang, Y., & Yin, B. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.01060 · Cited by 93 · see citing works
“Then, we compare our SEGAN with other state-of-the-art GAN models [28, 30, 25, 42, 9, 13, 44, 43, 13, 40, 14, 18, 27].”
MethodData science for engineering design: State of the art and future directions
Chiarello, F., Belingheri, P., & Fantoni, G. (2021) · Computers in Industry · doi:10.1016/j.compind.2021.103447 · Cited by 93 · see citing works
A survey of multimodal learning: Methods, applications, and future
Yuan, Y., Li, Z., & Zhao, B. (2025) · ACM Computing Surveys · doi:10.1145/3713070 · Cited by 90 · see citing works
“To end this, Qiao et al. [92] proposed a text-to-image-to-text framework MirrorGAN that can keep the semantics of image text consistent.”
Dual Compositional Learning in Interactive Image Retrieval
Kim, J.-S., Yu, Y., Kim, H., & Kim, G. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i2.16271 · Cited by 87 · see citing works
“(Qiao et al. 2019; Joseph et al. 2019) jointly train an image generation model with a captioning model for better text-to-image generation.”
MethodLearn, Imagine and Create: Text-to-Image Generation from Prior Knowledge
Qiao, T., Zhang, J., Xu, D., & Tao, D. (2019) · neural information processing systems · Cited by 86 · see citing works
Self-citationLearning Canonical Representations for Scene Graph to Image Generation
Herzig, R., Bar, A., Xu, H., Chechik, G., Darrell, T., & Globerson, A. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58574-7_13 · Cited by 92 · see citing works
“Conditioning inputs may include class labels [7,32,36], source images [15,16,29,52,68,69], model interventions [2], and text [14,40,43,44,49,59,60,63].”
BackgroundKT-GAN: Knowledge-Transfer Generative Adversarial Network for Text-to-Image Synthesis
Tan, H., Liu, X., Liu, M., Yin, B., & Li, X. S. (2020) · IEEE Transactions on Image Processing · doi:10.1109/tip.2020.3026728 · Cited by 85 · see citing works
“However, such a discriminator alone is usually insufficient to model underlying semantic consistency between text and image [23], and consequently, results in semantic or structural errors in synthesized images (see Figure 1, the “Direct T2I” column).”
BackgroundMethodInfluentialResearch on image inpainting algorithm of improved total variation minimization method
Chen, Y., Zhang, H., Liu, L., Tao, J., Zhang, Q., Yang, K., Xia, R., & Xie, J. (2021) · Journal of Ambient Intelligence and Humanized Computing · doi:10.1007/s12652-020-02778-2 · Cited by 84 · see citing works
“It is obviously more visual than others (He et al. 2016; Zhang et al. 2019; Qiao et al. 2019; Chen et al. 2020a).”
BackgroundMultimedia Intelligence: When Multimedia Meets Artificial Intelligence
Zhu, W., Wang, X. Y., & Gao, W. (2020) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2020.2969791 · Cited by 80 · see citing works
“Recently, some progress has also been achieved in the inverse task — image/video generation from natural language [87], [88], [96],”
BackgroundMulti-Modal Machine Learning in Engineering Design: A Review and Future Directions
Song, B., Zhou, R., & Ahmed, F. (2023) · Journal of Computing and Information Science in Engineering · doi:10.1115/1.4063954 · Cited by 83 · see citing works
Khan, M. U. G., Jabeen, S., Khan, M. U. G., Saba, T., Rehmat, A., Rehman, A. U., & Tariq, U. (2020) · IEEE Access · doi:10.1109/access.2020.3015656 · Cited by 77 · see citing works
“DCGAN [21], CycleGAN [22], Pro-GAN [6], BigGAN [16], StyleGAN [10], StarGAN [9] are the examples of this problem.”
BackgroundMethodDeep image captioning: A review of methods, trends and future challenges
Xu, L., Tang, Q., Lv, J., Zheng, B., Zeng, X., & Li, W. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.126287 · Cited by 78 · see citing works
Gradient Normalization for Generative Adversarial Networks
Wu, Y.-L., Shuai, H.-H., Tam, Z. R., & Chiu, H.-Y. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00631 · Cited by 75 · see citing works
“Generative Adversarial Networks (GANs) [9] have recently achieved great success for synthesizing new data from a given prior distribution, which facilitates a variety of applications, e.g ., super resolution imaging [15], style transfer between domains [20, 32].”
BackgroundRich Human Feedback for Text-to-Image Generation
Liang, Y., He, J., Li, G., Li, P., Klimovskiy, A., Carolan, N., Sun, J., Pont-Tuset, J., Young, S., Yang, F., et al. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01835 · Cited by 78 · see citing works
“An early work is the Generative Adversarial Network (GAN) [3, 16, 26], which trains a generator for image generation and a discriminator to distinguish between real and generated images, in parallel (also see [32, 38, 47, 55, 60, 62] among others).”
BackgroundCPGAN: Content-Parsing Generative Adversarial Networks for Text-to-Image Synthesis
Liang, J., Pei, W., & Lu, F. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58548-8_29 · Cited by 77 · see citing works
“Most existing methods [4, 9–11, 17, 32, 34, 39, 43, 44, 49, 50, Fig.”
BackgroundMethodInfluentialText to Image Synthesis for Improved Image Captioning
Hossain, M. Z., Sohel, F. A., Shiratuddin, M. F., Laga, H., & Bennamoun, M. (2021) · IEEE Access · doi:10.1109/access.2021.3075579 · Cited by 67 · see citing works
“[27] proposed a text-to-image generation method in ‘‘MirrorGAN: Learning Text-to-Image Generation by Redescription’’.”
MethodStoryDALL-E: Adapting Pretrained Text-to-Image Transformers for Story Continuation
Maharana, A., Hannan, D., & Bansal, M. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19836-6_5 · Cited by 72 · see citing works
Conditional GAN for timeseries generationPreprint
Smith, K. E., & Smith, A. O. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.16477 · Cited by 90 · see citing works
“Compared to the image counterpart [6, 7, 8, 9, 10], the available literature that describes work with time series GANs is meager.”
BackgroundChen, X., & Wu, D. (2024) · IEEE Transactions on Learning Technologies · doi:10.1109/tlt.2024.3378279 · Cited by 76 · see citing works
RenAIssance: A Survey Into AI Text-to-Image Generation in the Era of Large Model
Bie, F., Yang, Y., Zhou, Z., Ghanem, A., Zhang, M., Yao, Z., Wu, X., Holmes, C., Golnari, P. A., Clifton, D. A., et al. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3522305 · Cited by 76 · see citing works
Self-citationNeural Architecture Search With a Lightweight Transformer for Text-to-Image Synthesis
Li, W., Wen, S., Shi, K., Yang, Y., & Huang, T. (2022) · IEEE Transactions on Network Science and Engineering · doi:10.1109/tnse.2022.3147787 · Cited by 65 · see citing works
Peng, D., Yang, W., Liu, C., & Lü, S. (2021) · Neural Networks · doi:10.1016/j.neunet.2021.01.023 · Cited by 63 · see citing works
Towards Implicit Text-Guided 3D Shape Generation
Liu, Z., Wang, Y., Qi, X., & Fu, C. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01737 · Cited by 62 · see citing works
Alhabeeb, S., & Al-Shargabi, A. A. (2024) · IEEE Access · doi:10.1109/access.2024.3365043 · Cited by 69 · see citing works
“Torebuild textual descriptions from the generated images, MirrorGAN [48] presents a text-to-image-to-text architecture with three models.”
MethodLiu, C., Chen, K., Zhao, R., Zou, Z., & Shi, Z. (2025) · IEEE Geoscience and Remote Sensing Magazine · doi:10.1109/mgrs.2025.3560455 · Cited by 63 · see citing works
“MirrorGAN [41] further emphasized bidirectional mapping between text and images to preserve textual coherence.”
Multi-Sentence Auxiliary Adversarial Networks for Fine-Grained Text-to-Image Synthesis
Yang, Y., Wang, L., Xie, D., Deng, C. D., & Tao, D. (2021) · IEEE Transactions on Image Processing · doi:10.1109/tip.2021.3055062 · Cited by 61 · see citing works
“In order to illustrate the effectiveness of our proposed MA-GAN, we conduct comparisons with ten previous textto-image synthesis methods, namely as StackGAN [2], StackGAN++ [3], HDGAN [5], AttnGAN [4], PPAN [46], MirrorGAN [11], LeicaGAN [47], SEGAN [7], SD-GAN [12] and DM-GAN [48].”
MethodInfluentialSelf-citationPrompt Tuning Inversion for Text-Driven Image Editing Using Diffusion Models
Dong, W., Xue, S., Duan, X., & Han, S. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00683 · Cited by 61 · see citing works
“Works based on generative adversarial networks (GANs) [37, 23, 33, 60] have been proposed for text-to-image synthesis.”
BackgroundInfluentialUnified Generative Adversarial Networks for Controllable Image-to-Image Translation
Tang, H., Liu, H., & Sebe, N. (2020) · IEEE Transactions on Image Processing · doi:10.1109/tip.2020.3021789 · Cited by 58 · see citing works
“Existing approaches employ the idea of GANs for conditional image generation, such as image-to-image translation [2], text-to-image translation [27], [28], audio-to-image [29], and sketch generation [30], [31].”
MethodLiu, Y., Nadai, M. D., Cai, D., Li, H., Alameda-Pineda, X., Sebe, N., & Lepri, B. (2020) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3394171.3413505 · Cited by 62 · see citing works
“ur of the desired object, while the second one further refines the image and synthesize 256x256 images. The same authors improved the model with multiple generators through StackGAN++ [54]. Qiao etal.[41] focused instead on semantic consistency, enforcing the generated images to have the same semantics with the input…”
MethodCommonGen: A constrained text generation challenge for generative commonsense reasoning
Lin, B. Y., Zhou, W., Shen, M., Zhou, P., Bhagavatula, C., Choi, Y., & Ren, X. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.165 · Cited by 239 · see citing works
“nguage generation models. We argue that such an ability of recovering natural scenes of daily life can benefit a wide range of natural language generation (NLG) tasks including image/video captioning (Qiao et al., 2019;Wang et al.,2019), scene-based visual reasoning and VQA (Hudson and Manning,2019), storytelling (Guan…”
BackgroundFareed, M. W., Nassif, A. B., & Nofal, E. (2024) · Heritage · doi:10.3390/heritage7030081 · Cited by 66 · see citing works
Deep Multiple-Attribute-Perceived Network for Real-World Texture Recognition
Zhai, W., Cao, Y. H., Zhang, J., & Zha, Z.-J. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00371 · Cited by 55 · see citing works
“Through an attribute transfer module, the cascaded global context corresponding to visual attribute are leveraged as guidance of spatial feature encoding at the next branch [26].”
MethodSelf-citationA Novel Visual Representation on Text Using Diverse Conditional GAN for Visual Recognition
Hu, T., Long, C., & Xiao, C. (2021) · IEEE Transactions on Image Processing · doi:10.1109/tip.2021.3061927 · Cited by 53 · see citing works
“Qiao et al. [10] also used the attention mechanism to guarantee semantic consistency based on word-level attention and sentence-level attention model.”
BackgroundDR-GAN: Distribution Regularization for Text-to-Image Generation
Tan, H., Liu, X., Yin, B., & Li, X. (2022) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3165573 · Cited by 53 · see citing works
“MirrorGAN [33] combined the T2I generation and Image-to-Text generation to improve the global semantic consistency between text description and the generated image.”
BackgroundMethodInfluentialLanguage-Driven Artistic Style Transfer
Fu, T.-J., Wang, X. E., & Wang, W. Y. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-20059-5_41 · Cited by 53 · see citing works
“To further enhance the alignment with inputs, inspired by cycle consistency [60,51,38,55], we consider the content matching loss Lcm and the style matching loss Lsm of transferred results Ô.”
MethodRecurrent Affine Transformation for Text-to-Image Synthesis
Ye, S., Wang, H., Tan, M., & Liu, F. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3266607 · Cited by 53 · see citing works
Pan, Y., Liu, M., Lian, C., Xia, Y., & Shen, D. G. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-32248-9_16 · Cited by 53 · see citing works
“To utilize all available subjects, a cycleconsistent generative adversarial network (CGAN) has recently been employed to impute missing PET images [2,3].”
MethodA survey on generative adversarial network-based text-to-image synthesis
Zhou, R. H., Jiang, C. F., & Xu, Q. (2021) · Neurocomputing · doi:10.1016/j.neucom.2021.04.069 · Cited by 52 · see citing works
Intelligent Home 3D: Automatic 3D-House Design From Linguistic Descriptions Only
Chen, Q., Wu, Q., Tang, R., Wang, Y., Wang, S., & Tan, M. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01264 · Cited by 52 · see citing works
“Thanks to the fast development of Deep Learning (DL) [6, 9, 11, 43, 44, 48, 49, 50, 52], especially Generative Adversarial Network (GAN) [3, 4, 8, 10] and visionlanguage research [13, 45], we can turn this problem into a text-to-image generation problem, which has been studied in [22, 29, 31, 41, 42].”
BackgroundMethodCycle-Consistent Inverse GAN for Text-to-Image Synthesis
Wang, H., Lin, G., Hoi, S. C. H., & Miao, C. (2021) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3474085.3475226 · Cited by 52 · see citing works
“StackGAN++ [37] 4.04 ± 0.06 - AttnGAN [35] 4.36 ± 0.03 23.98 MirrorGAN [20] 4.56 ± 0.05 - ControlGAN [15] 4.58 ± 0.09 - SD-GAN [36] 4.67 ± 0.09 - DM-GAN [42] 4.75 ± 0.07 16.09 DF-GAN [26] 4.86 ± 0.04 19.24 RiFeGAN [6] 5.23 ± 0.09 - DALL-E (zero-shot) [21] 2.85 ± 0.04 51.00 CI-GAN (Ours) 5.72 ± 0.12 9.78 can get the be…”
MethodInfluentialDTGAN: Dual Attention Generative Adversarial Networks for Text-to-Image Generation
Zhang, Z., & Schomaker, L. (2021) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn52387.2021.9533527 · Cited by 52 · see citing works
Ak, K. E., Lim, J., Tham, J. Y., & Kassim, A. A. (2020) · Pattern Recognition Letters · doi:10.1016/j.patrec.2020.02.030 · Cited by 48 · see citing works
Image synthesis: a review of methods, datasets, evaluation metrics, and future outlook
Baraheem, S. S., Le, T.-N., & Nguyen, T. (2023) · Artificial Intelligence Review · doi:10.1007/s10462-023-10434-2 · Cited by 48 · see citing works
“Text-to-image synthesis (Zhu et al. 2007; Dosovitskiy et al. 2015; Yan et al. 2015; Gregor et al. 2015; Mansimov et al. 2015; Cai et al. 2017; Mirza and Osindero 2014; Reed et al. 2016a; Odena et al. 2016; Zhang et al. 2016, 2017, 2021a; Sharma et al. 2018; Park et al. 2018; Xu et al. 2017; Qiao et al. 2019; Gou et al…”
BackgroundDeep Structure-Revealed Network for Texture Recognition
Zhai, W., Cao, Y. H., Zha, Z.-J., Xie, H., & Wu, F. C. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01102 · Cited by 48 · see citing works
Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods
Mogadala, A., Kalimuthu, M., & Klakow, D. (2021) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.11688 · Cited by 128 · see citing works
“47 64x64 MirrorGAN (Qiao et al., 2019) 128x128 256x256 26.”
BackgroundInfluentialExploring Global and Local Linguistic Representations for Text-to-Image Synthesis
Li, R., Wang, N., Feng, F., Zhang, G., & Wang, X. (2020) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2020.2972856 · Cited by 47 · see citing works
“7) MirrorGAN [7] emphasizes the semantic consistency using the idea of re-description.”
MethodInvestigating the Bilateral Connections in Generative Zero-Shot Learning
Li, J., Jing, M., Lü, K., Zhu, L., & Shen, H. T. (2021) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2021.3050803 · Cited by 47 · see citing works
Generative AI in the context of assistive technologies: Trends, limitations and future directions
Fu, B., Hadid, A., & Damer, N. (2024) · Image and Vision Computing · doi:10.1016/j.imavis.2024.105347 · Cited by 52 · see citing works
New Ideas and Trends in Deep Multimodal Content Understanding: A Review
Chen, W., Wang, W., Liu, L., & Lew, M. S. (2020) · Neurocomputing · doi:10.1016/j.neucom.2020.10.042 · Cited by 44 · see citing works
“Down [32], Attention-aware [78], BSSAN [62], CRAN[149], CBP [36], SOT [5], PAGNet [34], MirrorGAN [153], DAI [154], T2I2T [11], CCGAN [13], C4Synth [155], Cycle-Attn+ [156], Coupled CycleGAN [157], TCCM [158], CycleMatch [159], VQA-Rephrasings [160], iQAN [161], MLAN [162], MFB [163], DANs [164],…”
FICE: Text-conditioned fashion-image editing with guided GAN inversion
Pernuš, M., Fookes, C., Štruc, V., & Dobrišek, S. (2024) · Pattern Recognition · doi:10.1016/j.patcog.2024.111022 · Cited by 44 · see citing works
Generative Imagination Elevates Machine Translation
Long, Q., Wang, M., & Li, L. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.457 · Cited by 46 · see citing works
“MirrorGAN (Qiao et al. 2019) employs a mirror structure, which reversely learns from the inverse task of T2I to further validate whether generated images are consistent with the input texts.”
BackgroundMethodInfluentialText to Image Synthesis With Bidirectional Generative Adversarial Network
Wang, Z., Quan, Z., Wang, Z., Hu, X., & Chen, Y. (2020) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme46284.2020.9102904 · Cited by 43 · see citing works
Cross-Modal Semantic Matching Generative Adversarial Networks for Text-to-Image Synthesis
Tan, H., Liu, X., Yin, B., & Li, X. S. (2021) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2021.3060291 · Cited by 43 · see citing works
StyleT2I: Toward Compositional and High-Fidelity Text-to-Image Synthesis
Li, Z., Min, M. R., Li, K., & Xu, C. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01766 · Cited by 43 · see citing works
“Text-to-Image Synthesis Many previous works [8,15,25,29, 30,32,46,48,50,58,68,70–73 , 75] have studied text-to-image text-to-image synthesis.”
BackgroundIntegrating Visuospatial, Linguistic, and Commonsense Structure into Story Visualization
Maharana, A., & Bansal, M. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.543 · Cited by 40 · see citing works
“Dual learning has served as an effective method for promoting desirable characteristics in target output for both text-to-image generation (Qiao et al., 2019) and story visualization (Maharana et al.”
MethodDeep Vision Multimodal Learning: Methodology, Benchmark, and Trend
Chai, W., & Wang, G. (2022) · Applied Sciences · doi:10.3390/app12136588 · Cited by 43 · see citing works
TIME: Text and Image Mutual-Translation Adversarial Networks
Liu, B., Song, K., Zhu, Y., de Melo, G., & Elgammal, A. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i3.16305 · Cited by 40 · see citing works
“MirrorGAN [24] incorporates a pre-trained text re-description RNN to better align the images with the given texts.”
BackgroundMethodInfluentialCharacter-Preserving Coherent Story Visualization
Song, Y.-Z., Tam, Z. R., Chen, H.-J., Lu, H.-H., & Shuai, H.-H. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58520-4_2 · Cited by 40 · see citing works
“MirrorGAN [23] employs a mirror structure, which reversely learns from generated images to output texts for validating whether generated images are consistent with the input texts.”
BackgroundKnowledge-Driven Generative Adversarial Network for Text-to-Image Synthesis
Peng, J., Zhou, Y., Sun, X., Cao, L., Wu, Y., Huang, F., & Ji, R. (2021) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2021.3116416 · Cited by 41 · see citing works
Synthesizing Coherent Story with Auto-Regressive Latent Diffusion Models
Pan, X., Qin, P., Li, Y., Xue, H., & Chen, W. (2024) · 2024 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv57701.2024.00290 · Cited by 47 · see citing works
“GANs like Stackgan [41], Attngan [38], Mirrorgan [21], and MXC-GAN [40] perform adversarial training between generators and discriminators to learn to generate high-quality images.”
BackgroundDual Adversarial Inference for Text-to-Image Synthesis
Lao, Q., Havaei, M., Pesaranghader, A., Dutil, F., Jorio, L. D., & Fevens, T. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00766 · Cited by 38 · see citing works
“Several attempts have been made to leverage additional available information, such as object location [27], class label [5, 2], attention extracted from word features [30, 24] and text regeneration [24].”
BackgroundMethodInfluentialTransformer models for enhancing AttnGAN based text to image generation
Naveen, S., Kiran, M. S. S. R., Indupriya, M., Manikanta, T., & Sudeep, P. V. (2021) · Image and Vision Computing · doi:10.1016/j.imavis.2021.104284 · Cited by 40 · see citing works
RiFeGAN2: Rich Feature Generation for Text-to-Image Synthesis From Constrained Prior Knowledge
Cheng, J. S., Wu, F., Tian, Y., Wang, L., & Tao, D. (2021) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2021.3136857 · Cited by 38 · see citing works
Self-citationUnifying Multimodal Transformer for Bi-directional Image and Text Generation
Huang, Y., Xue, H., Liu, B., & Lu, Y. (2021) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3474085.3481540 · Cited by 37 · see citing works
“MirrorGAN [34] focus on the text-toimage generation task and uses an off-the-shelf captioning model to regularize the redescription of the generated images to be semantic consistent with the original text.”
MethodAIGC Image Quality Assessment via Image-Prompt Correspondence
Peng, F., Fu, H., Ming, A., Wang, C., Ma, H., He, S., Dou, Z., & Chen, S. (2024) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw63382.2024.00644 · Cited by 40 · see citing works
“In recent years, the field of generative models within deep learning has seen an extraordinary evolution, particularly * Corresponding author with the advent of Generative Adversarial Networks [6, 11, 13, 15, 32] (GANs) and diffusion models [7, 17, 37, 38, 58].”
BackgroundMultimodal fusion techniques: Review, data representation, information fusion, and application areas
Hangloo, S., & Arora, B. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.130827 · Cited by 43 · see citing works
Improving Generation and Evaluation of Visual Stories via Semantic Consistency
Maharana, A., Hannan, D., & Bansal, M. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.194 · Cited by 46 · see citing works
- Authors
- Qiao, T., Zhang, J., Xu, D., & Tao, D.
- Affiliations
- The University of Sydney (AU)
- University of Technology Sydney (AU)
- Venue
- 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2019-06
- Pages
- 1505-1514
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W2962845008
- Semantic Scholar
- 5af2d424bb38db7e6a72f7bec2cfd8a5bb8af7fb
- Cited by
- 644
- References
- 45
- Citations by year
- 2026: 17 · 2025: 71 · 2024: 99 · 2023: 134 · 2022: 102 · 2021: 113 · 2020: 92 · 2019: 16
- FWCI
- 26.32
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Natural Science Foundation of China (61806062)
- Cultural Heritage Administration
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- text-to-image generation, semantic consistency, redescription, global-local attention, image-text alignment, generative adversarial networks
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
