The DOI matches its Crossref record; 46 of the 46 references it cites were found in a registry or an index.
MirrorGAN: Learning Text-To-Image Generation by Redescription
Open accessAbstract
Generating an image from a given text description has two goals: visual realism and semantic consistency. Although significant progress has been made in generating high-quality and visually realistic images using generative adversarial networks, guaranteeing semantic consistency between the text description and visual content remains very challenging. In this paper, we address this problem by proposing a novel global-local attentive and semantic-preserving text-to-image-to-text framework called MirrorGAN. MirrorGAN exploits the idea of learning text-to-image generation by redescription and consists of three modules: a semantic text embedding module (STEM), a global-local collaborative attentive module for cascaded image generation (GLAM), and a semantic text regeneration and alignment module (STREAM). STEM generates word- and sentence-level embeddings. GLAM has a cascaded architecture for generating target images from coarse to fine scales, leveraging both local word attention and global sentence attention to progressively enhance the diversity and semantic consistency of the generated images. STREAM seeks to regenerate the text description from the generated image, which semantically aligns with the given text description. Thorough experiments on two public benchmark datasets demonstrate the superiority of MirrorGAN over other representative state-of-the-art methods.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 2×
“The image encoder is a convolutional neural network (CNN) [11] pretrained on ImageNet [5], and the decoder is a RNN [12].”
MethodHochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 3×
“The image encoder is a convolutional neural network (CNN) [11] pretrained on ImageNet [5], and the decoder is a RNN [12].”
MethodBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · cited 1×
“The image encoder is a convolutional neural network (CNN) [11] pretrained on ImageNet [5], and the decoder is a RNN [12].”
MethodBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 1×
“Second, we only utilize a basic method for text embedding in STEM and image captioning in STREAM, which could be further improved, for example, by using the recently proposed BERT model [7] and state-of-the-art image captioning models [2, 3].”
MethodMicrosoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460 · cited 1×
“We evaluated our model on two commonly used datasets, CUB bird dataset [30] and MS COCO dataset [17].”
MethodLearning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1179 · Cited by 25,545 · cited 4×
“The image encoder is a convolutional neural network (CNN) [11] pretrained on ImageNet [5], and the decoder is a RNN [12].”
MethodInfluentialUnpaired image-to-image translation using cycle-consistent adversarial networks
Zhu, J.-Y., Park, T., Isola, P., & Efros, A. A. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.244 · Cited by 23,602 · cited 1×
“Similar ideas to our own have recently been used in CycleGAN and DualGAN, which handle the bi-directional translations within two domains together [42, 36, 1], significantly advance image-to-image translation [14, 28, 15, 37, 23].”
Image-to-Image Translation with Conditional Adversarial Networks
Isola, P., Zhu, J.-Y., Zhou, T., & Efros, A. A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.632 · Cited by 22,930 · cited 1×
“Similar ideas to our own have recently been used in CycleGAN and DualGAN, which handle the bi-directional translations within two domains together [43, 37, 1, 32], significantly advance image-to-image translation [14, 28, 15, 38, 23].”
Labaca-Castro, R. (2023) · Springer eBooks · doi:10.1007/978-3-658-40442-0_9 · Cited by 21,119
Flagged first, then by citations
Research works printed but not linked: 2
- 10.1109/iccv.2017.629 · DOI · Paper page
- taigman, 2017, Unsupervised crossdomain image generation, International Conference on Learning Representations (ICLR)
Cited by
Sorted by citations
DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation
Ruiz, N., Li, Y., Jampani, V., Pritch, Y., Rubinstein, M., & Aberman, K. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.02155 · Cited by 2,295
“There also exists work on text-to-image synthesis [14, 16, 19, 24, 27, 35, 36, 50, 51, 55, 58, 67, 74].”
BackgroundA Review on Generative Adversarial Networks: Algorithms, Theory, and Applications
Gui, J., Sun, Z., Wen, Y., Tao, D., & Ye, J. (2023) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2021.3130191 · Cited by 1,231
“Field Subfield Method Super-resolution SRGAN [63], ESRGAN [64], Cycle-in-Cycle GANs [65], SRDGAN [66], TGAN [67] DR-GAN [68], TP-GAN [69], PG2 [70], PSGAN [71], Image synthesis and manipulation APDrawingGAN [72], IGAN [73], Image processing and computer vision introspective adversarial networks [74], GauGAN [75] Textu…”
BackgroundSelf-citationBlended Diffusion for Text-driven Editing of Natural Images
Avrahami, O., Lischinski, D., & Fried, O. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01767 · Cited by 889
“Recent works that leverage the tremendous progress in vision-language models and datadriven image generation have demonstrated that text-based interfaces for image creation and manipulation are now finally within reach [12, 23, 29, 30, 38, 39, 41, 47, 51, 57].”
BackgroundZhang, J., & Tao, D. (2021) · IEEE Internet of Things Journal · doi:10.1109/jiot.2020.3039359 · Cited by 774
“…and text-to-image generation are two generative tasks related to cross-modal matching, where captioning refers to generating a piece of text description for a given image or video [179] while text-to-image generation aims to generate a realistic image that matches the given text description [180].”
BackgroundSelf-citationVector Quantized Diffusion Model for Text-to-Image Synthesis
Gu, S., Chen, D., Bao, J., Wen, F., Zhang, B., Chen, D., Yuan, L., & Guo, B. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01043 · Cited by 720
Multimodal intelligence: Representation learning, information fusion, and applications
Zhang, C., Yang, Z., He, X., & Deng, L. (2020) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2020.2987728 · Cited by 511
“Analogous to cycle consistency [203], MirrorGAN is proposed to improve the semantic consistency between the two modalities using an extra image captioning module [204].”
MethodText2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators
Khachatryan, L., Movsisyan, A., Tadevosyan, V., Henschel, R. D., Wang, Z., Navasardyan, S., & Shi, H. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01462 · Cited by 620
“These include StackGAN [46], AttnGAN [43], and MirrorGAN [24], which further improve image quality and diversity by introducing novel architectures and attention mechanisms.”
MethodCross-Modal Contrastive Learning for Text-to-Image Generation
Zhang, H., Koh, J. Y., Baldridge, J., Lee, H., & Yang, Y. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00089 · Cited by 375
DF-GAN: A Simple and Effective Baseline for Text-to-Image Synthesis
Tao, M., Tang, H., Wu, F., Jing, X., Bao, B., & Xu, C. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01602 · Cited by 513
“Although impressive results have been presented by previous works [9,19,21,32,33,51,60], there still remain three problems.”
BackgroundMethodResultInfluentialAdversarial Representation Learning for Text-to-Image Matching
Sarafianos, N., Xu, X., & Kakadiaris, I. A. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00591 · Cited by 261
“Flowers and CUB datasets, which is in line with existing work [42, 62, 70], this was not the case for the CUHK-PEDES and Flickr30K datasets.”
A Review on Generative Adversarial Networks: Algorithms, Theory, and ApplicationsPreprint
Gui, J., Sun, Z., Wen, Y., Tao, D., & Ye, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2001.06937 · Cited by 252
“Field Subfield Method Super-resolution SRGAN [63], ESRGAN [64], Cycle-in-Cycle GANs [65], SRDGAN [66], TGAN [67] DR-GAN [68], TP-GAN [69], PG2 [70], PSGAN [71], Image synthesis and manipulation APDrawingGAN [72], IGAN [73], Image processing and computer vision introspective adversarial networks [74], GauGAN [75] Textu…”
BackgroundSelf-citationMore Control for Free! Image Synthesis with Semantic Diffusion Guidance
Liu, X., Park, D. H., Azadi, S., Zhang, G., Chopikyan, A., Hu, Y., Shi, H., Rohrbach, A., & Darrell, T. (2023) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv56688.2023.00037 · Cited by 274
“Various methods have been studied, proposing many different text-adaptive architectures and loss functions to enforce better semantic alignment between the input text and generated image, resulting in significant improvements in editing quality and correctness [55,60,51,27,11,61,33,35,17,28,7,23,40].”
MethodAdversarial text-to-image synthesis: A review
Frolov, S., Hinz, T., Raue, F., Hees, J., & Dengel, A. R. (2021) · Neural Networks · doi:10.1016/j.neunet.2021.07.019 · Cited by 228
“Figure 9: Simplified MirrorGAN [68] architecture.”
MethodInfluentialDeep Graph-neighbor Coherence Preserving Network for Unsupervised Cross-modal Hashing
Yu, J., Zhou, H., Zhan, Y., & Tao, D. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i5.16592 · Cited by 208
“Due to the explosion of multimedia data on the Internet (Yu, Rui, and Chen 2013; Yang et al. 2020c), cross-modal retrieval has become an essential technique in information retrieval (Yang et al. 2020c) and multi-modal data mining (Qiao et al. 2019b,a; Yang et al. 2020a,c; Li et al. 2020).”
BackgroundSelf-citationText to Image Generation with Semantic-Spatial Aware GAN
Liao, W., Hu, K., Yang, M. Y., & Rosenhahn, B. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01765 · Cited by 208
“Most of the recent T2I methods follow this framework [16, 21, 4, 24, 33].”
MethodGALIP: Generative Adversarial CLIPs for Text-to-Image Synthesis
Tao, M., Bao, B., Tang, H., & Xu, C. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01366 · Cited by 194
3DALL-E: Integrating Text-to-Image AI in 3D Design Workflows
Liu, V., Vermeulen, J., Fitzmaurice, G. W., & Matejka, J. (2023) · Designing Interactive Systems Conference · doi:10.1145/3563657.3596098 · Cited by 206
“However, many early models were class-conditional, meaning that they were only robust at generating images from the classes they were trained on [39, 40, 53, 54, 66, 68].”
BackgroundVision-Language Pre-Training: Basics, Recent Advances, and Future Trends
Gan, Z., Li, L., Li, C., Wang, L., Liu, Z., & Gao, J. (2022) · Foundations and Trends® in Computer Graphics and Vision · doi:10.1561/0600000105 · Cited by 155
Semantic Object Accuracy for Generative Text-to-Image Synthesis
Hinz, T., Heinrich, S., & Wermter, S. (2020) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2020.3021209 · Cited by 176
“Similarly, [16] learn text-to-image by redescription, i.”
TextFuseNet: Scene Text Detection with Richer Fused Features
Ye, J., Chen, Z., Liu, J., et al. (2020) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2020/72 · Cited by 149
Elasri, M., Elharrouss, O., Al-Maadeed, S. A., & Tairi, H. (2022) · Neural Processing Letters · doi:10.1007/s11063-022-10777-x · Cited by 146
“[54] proposed MirrorGAN method that consist of generating images from text description by combining cascaded image generation (GLAM) module, semantic text embedding module (STEM), and a semantic text regeneration and alignment module (STREAM).”
MethodInfluentialMultimodal Image Synthesis and Editing: The Generative AI Era
Zhan, F., Yu, Y., Wu, R., Zhang, J., Lu, S., Liu, L., Kortylewski, A., Theobalt, C., & Xing, E. P. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3305243 · Cited by 197
“Specifically, some research [55], [151], [152], [153], [154] explores to pass the generated images through an inverse network to yield the conditional input, which imposes a cycle-consistency of conditional input.”
BackgroundSelf-citationPan, Y., Liu, M., Xia, Y., & Shen, D. G. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3091214 · Cited by 138
“Generative adversarial networks (GANs) have been developed for image transformation from a source domain/ modality to a target domain/modality [25], [26], [27], [28], [29], [30], [31], [32].”
BackgroundDAE-GAN: Dynamic Aspect-aware GAN for Text-to-Image Synthesis
Ruan, S., Zhang, Y., Zhang, K., Fan, Y., Tang, F., Liu, Q., & Chen, E. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01370 · Cited by 133
“[18] proposed MirroGAN by semantically aligning the re-description of the generated image with the given text description.”
MethodDream3D: Zero-Shot Text-to-3D Synthesis Using 3D Shape Prior and Text-to-Image Diffusion Models
Xu, J., Wang, X., Cheng, W., Cao, Y., Shan, Y., Qie, X., & Gao, S. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.02003 · Cited by 168
DF-GAN: Deep Fusion Generative Adversarial Networks for Text-to-Image SynthesisPreprint
Tao, M., Tang, H., Wu, S., Sebe, N., Wu, F. Q., & Jing, X. (2020) · arXiv (Cornell University) · Cited by 121
RiFeGAN: Rich Feature Generation for Text-to-Image Synthesis From Prior Knowledge
Cheng, J. S., Wu, F., Tian, Y., Wang, L., & Tao, D. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01092 · Cited by 122
“[20] introduce a MirrorGAN to connect the dual tasks, text-to-image synthesis and text captioning, and constrain the regenerated caption aligning with the given caption.”
BackgroundSelf-citationDeep learning and the global workspace theory
VanRullen, R., & Kanai, R. (2021) · Trends in Neurosciences · doi:10.1016/j.tins.2021.04.005 · Cited by 128
“Similar methods have been applied to neural translation between varied domains, e.g. unpaired image-to-image translation [49–51], text-to-image translation [26, 28, 52] or touch-to-image translation [32].”
MethodA survey and taxonomy of adversarial neural networks for text‐to‐image synthesis
Agnese, J., Herrera, J., Tao, H., & Xingquan Zhu, A. (2020) · Wiley Interdisciplinary Reviews Data Mining and Knowledge Discovery · doi:10.1002/widm.1345 · Cited by 117
“(Reed et al., 2016b) StackGAN (Zhang et al., 2017b) StackGAN++ (Zhang et al., 2018b) AttnGAN (Xu et al., 2017) DC-GAN (Reed et al., 2016b) HDGAN (Zhang et al., 2018d) MirrorGAN (Qiao et al., 2019)”
BackgroundInfluentialA Review of AI Image Generator: Influences, Challenges, and Future Prospects for Architectural Field
Enjellina, Beyan, E. V. P., & Rossy, A. G. C. (2023) · Journal of Artificial Intelligence in Architecture · doi:10.24002/jarina.v2i1.6662 · Cited by 124
Multimodal research in vision and language: A review of current and emerging trends
Uppal, S., Bhagat, S., Hazarika, D., Majumder, N., Poria, S., Zimmermann, R., & Zadeh, A. (2021) · Information Fusion · doi:10.1016/j.inffus.2021.07.009 · Cited by 118
An Overview on Generative AI at Scale With Edge–Cloud Computing
Wang, Y.-C., Xue, J., Wei, C., & Kuo, C. ‐. J. (2023) · IEEE Open Journal of the Communications Society · doi:10.1109/ojcoms.2023.3320646 · Cited by 124
“It is a specific category of AI that aims to autonomously generate new content that imitates the content created by humans in different modalities, including images [42], [94], audio [100], [101], text [13], [32], and even 3D objects [81], [87].”
BackgroundOpal: Multimodal Image Generation for News Illustration
Liu, V., Qiao, H., & Chilton, L. (2022) · Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology · doi:10.1145/3526113.3545621 · Cited by 104
“As tools moved away from statistical-based techniques to GANs, a number of text-to-image architectures [39, 40, 52, 53] were proposed.”
MethodSemantics-Enhanced Adversarial Nets for Text-to-Image Synthesis
Tan, H., Liu, X., Li, X. S., Zhang, Y., & Yin, B. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.01060 · Cited by 93
“Then, we compare our SEGAN with other state-of-the-art GAN models [28, 30, 25, 42, 9, 13, 44, 43, 13, 40, 14, 18, 27].”
MethodData science for engineering design: State of the art and future directions
Chiarello, F., Belingheri, P., & Fantoni, G. (2021) · Computers in Industry · doi:10.1016/j.compind.2021.103447 · Cited by 99
A survey of multimodal learning: Methods, applications, and future
Yuan, Y., Li, Z., & Zhao, B. (2025) · ACM Computing Surveys · doi:10.1145/3713070 · Cited by 90
“To end this, Qiao et al. [92] proposed a text-to-image-to-text framework MirrorGAN that can keep the semantics of image text consistent.”
Dual Compositional Learning in Interactive Image Retrieval
Kim, J.-S., Yu, Y., Kim, H., & Kim, G. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i2.16271 · Cited by 90
“(Qiao et al. 2019; Joseph et al. 2019) jointly train an image generation model with a captioning model for better text-to-image generation.”
MethodLearn, Imagine and Create: Text-to-Image Generation from Prior Knowledge
Qiao, T., Zhang, J., Xu, D., & Tao, D. (2019) · neural information processing systems · Cited by 85
Self-citationLearning Canonical Representations for Scene Graph to Image Generation
Herzig, R., Bar, A., Xu, H., Chechik, G., Darrell, T., & Globerson, A. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58574-7_13 · Cited by 92
“Conditioning inputs may include class labels [7,32,36], source images [15,16,29,52,68,69], model interventions [2], and text [14,40,43,44,49,59,60,63].”
BackgroundKT-GAN: Knowledge-Transfer Generative Adversarial Network for Text-to-Image Synthesis
Tan, H., Liu, X., Liu, M., Yin, B., & Li, X. S. (2020) · IEEE Transactions on Image Processing · doi:10.1109/tip.2020.3026728 · Cited by 85
“However, such a discriminator alone is usually insufficient to model underlying semantic consistency between text and image [23], and consequently, results in semantic or structural errors in synthesized images (see Figure 1, the “Direct T2I” column).”
BackgroundMethodInfluentialResearch on image inpainting algorithm of improved total variation minimization method
Chen, Y., Zhang, H., Liu, L., Tao, J., Zhang, Q., Yang, K., Xia, R., & Xie, J. (2021) · Journal of Ambient Intelligence and Humanized Computing · doi:10.1007/s12652-020-02778-2 · Cited by 84
“It is obviously more visual than others (He et al. 2016; Zhang et al. 2019; Qiao et al. 2019; Chen et al. 2020a).”
BackgroundMultimedia Intelligence: When Multimedia Meets Artificial Intelligence
Zhu, W., Wang, X. Y., & Gao, W. (2020) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2020.2969791 · Cited by 87
“Recently, some progress has also been achieved in the inverse task — image/video generation from natural language [87], [88], [96],”
BackgroundMulti-Modal Machine Learning in Engineering Design: A Review and Future Directions
Song, B., Zhou, R., & Ahmed, F. (2023) · Journal of Computing and Information Science in Engineering · doi:10.1115/1.4063954 · Cited by 105
Khan, M. U. G., Jabeen, S., Khan, M. U. G., Saba, T., Rehmat, A., Rehman, A. U., & Tariq, U. (2020) · IEEE Access · doi:10.1109/access.2020.3015656 · Cited by 77
“DCGAN [21], CycleGAN [22], Pro-GAN [6], BigGAN [16], StyleGAN [10], StarGAN [9] are the examples of this problem.”
BackgroundMethodDeep image captioning: A review of methods, trends and future challenges
Xu, L., Tang, Q., Lv, J., Zheng, B., Zeng, X., & Li, W. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.126287 · Cited by 79
Gradient Normalization for Generative Adversarial Networks
Wu, Y.-L., Shuai, H.-H., Tam, Z. R., & Chiu, H.-Y. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00631 · Cited by 75
“Generative Adversarial Networks (GANs) [9] have recently achieved great success for synthesizing new data from a given prior distribution, which facilitates a variety of applications, e.g ., super resolution imaging [15], style transfer between domains [20, 32].”
BackgroundRich Human Feedback for Text-to-Image Generation
Liang, Y., He, J., Li, G., Li, P., Klimovskiy, A., Carolan, N., Sun, J., Pont-Tuset, J., Young, S., Yang, F., et al. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01835 · Cited by 122
“An early work is the Generative Adversarial Network (GAN) [3, 16, 26], which trains a generator for image generation and a discriminator to distinguish between real and generated images, in parallel (also see [32, 38, 47, 55, 60, 62] among others).”
BackgroundCPGAN: Content-Parsing Generative Adversarial Networks for Text-to-Image Synthesis
Liang, J., Pei, W., & Lu, F. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58548-8_29 · Cited by 77
“Most existing methods [4, 9–11, 17, 32, 34, 39, 43, 44, 49, 50, Fig.”
BackgroundMethodInfluentialText to Image Synthesis for Improved Image Captioning
Hossain, M. Z., Sohel, F. A., Shiratuddin, M. F., Laga, H., & Bennamoun, M. (2021) · IEEE Access · doi:10.1109/access.2021.3075579 · Cited by 78
“[27] proposed a text-to-image generation method in ‘‘MirrorGAN: Learning Text-to-Image Generation by Redescription’’.”
MethodStoryDALL-E: Adapting Pretrained Text-to-Image Transformers for Story Continuation
Maharana, A., Hannan, D., & Bansal, M. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19836-6_5 · Cited by 81
Conditional GAN for timeseries generationPreprint
Smith, K. E., & Smith, A. O. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.16477 · Cited by 118
“Compared to the image counterpart [6, 7, 8, 9, 10], the available literature that describes work with time series GANs is meager.”
BackgroundChen, X., & Wu, D. (2024) · IEEE Transactions on Learning Technologies · doi:10.1109/tlt.2024.3378279 · Cited by 85
RenAIssance: A Survey Into AI Text-to-Image Generation in the Era of Large Model
Bie, F., Yang, Y., Zhou, Z., Ghanem, A., Zhang, M., Yao, Z., Wu, X., Holmes, C., Golnari, P. A., Clifton, D. A., et al. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3522305 · Cited by 96
Self-citationNeural Architecture Search With a Lightweight Transformer for Text-to-Image Synthesis
Li, W., Wen, S., Shi, K., Yang, Y., & Huang, T. (2022) · IEEE Transactions on Network Science and Engineering · doi:10.1109/tnse.2022.3147787 · Cited by 67
Peng, D., Yang, W., Liu, C., & Lü, S. (2021) · Neural Networks · doi:10.1016/j.neunet.2021.01.023 · Cited by 64
Towards Implicit Text-Guided 3D Shape Generation
Liu, Z., Wang, Y., Qi, X., & Fu, C. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01737 · Cited by 69
Alhabeeb, S., & Al-Shargabi, A. A. (2024) · IEEE Access · doi:10.1109/access.2024.3365043 · Cited by 77
“Torebuild textual descriptions from the generated images, MirrorGAN [48] presents a text-to-image-to-text architecture with three models.”
MethodLiu, C., Chen, K., Zhao, R., Zou, Z., & Shi, Z. (2025) · IEEE Geoscience and Remote Sensing Magazine · doi:10.1109/mgrs.2025.3560455 · Cited by 63
“MirrorGAN [41] further emphasized bidirectional mapping between text and images to preserve textual coherence.”
Multi-Sentence Auxiliary Adversarial Networks for Fine-Grained Text-to-Image Synthesis
Yang, Y., Wang, L., Xie, D., Deng, C. D., & Tao, D. (2021) · IEEE Transactions on Image Processing · doi:10.1109/tip.2021.3055062 · Cited by 61
“In order to illustrate the effectiveness of our proposed MA-GAN, we conduct comparisons with ten previous textto-image synthesis methods, namely as StackGAN [2], StackGAN++ [3], HDGAN [5], AttnGAN [4], PPAN [46], MirrorGAN [11], LeicaGAN [47], SEGAN [7], SD-GAN [12] and DM-GAN [48].”
MethodInfluentialSelf-citationPrompt Tuning Inversion for Text-Driven Image Editing Using Diffusion Models
Dong, W., Xue, S., Duan, X., & Han, S. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00683 · Cited by 81
“Works based on generative adversarial networks (GANs) [37, 23, 33, 60] have been proposed for text-to-image synthesis.”
BackgroundInfluentialUnified Generative Adversarial Networks for Controllable Image-to-Image Translation
Tang, H., Liu, H., & Sebe, N. (2020) · IEEE Transactions on Image Processing · doi:10.1109/tip.2020.3021789 · Cited by 58
“Existing approaches employ the idea of GANs for conditional image generation, such as image-to-image translation [2], text-to-image translation [27], [28], audio-to-image [29], and sketch generation [30], [31].”
MethodLiu, Y., Nadai, M. D., Cai, D., Li, H., Alameda-Pineda, X., Sebe, N., & Lepri, B. (2020) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3394171.3413505 · Cited by 63
“ur of the desired object, while the second one further refines the image and synthesize 256x256 images. The same authors improved the model with multiple generators through StackGAN++ [54]. Qiao etal.[41] focused instead on semantic consistency, enforcing the generated images to have the same semantics with the input…”
MethodCommonGen: A constrained text generation challenge for generative commonsense reasoning
Lin, B. Y., Zhou, W., Shen, M., Zhou, P., Bhagavatula, C., Choi, Y., & Ren, X. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.165 · Cited by 239
“nguage generation models. We argue that such an ability of recovering natural scenes of daily life can benefit a wide range of natural language generation (NLG) tasks including image/video captioning (Qiao et al., 2019;Wang et al.,2019), scene-based visual reasoning and VQA (Hudson and Manning,2019), storytelling (Guan…”
BackgroundFareed, M. W., Nassif, A. B., & Nofal, E. (2024) · Heritage · doi:10.3390/heritage7030081 · Cited by 75
Deep Multiple-Attribute-Perceived Network for Real-World Texture Recognition
Zhai, W., Cao, Y. H., Zhang, J., & Zha, Z.-J. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00371 · Cited by 55
“Through an attribute transfer module, the cascaded global context corresponding to visual attribute are leveraged as guidance of spatial feature encoding at the next branch [26].”
MethodSelf-citationA Novel Visual Representation on Text Using Diverse Conditional GAN for Visual Recognition
Hu, T., Long, C., & Xiao, C. (2021) · IEEE Transactions on Image Processing · doi:10.1109/tip.2021.3061927 · Cited by 53
“Qiao et al. [10] also used the attention mechanism to guarantee semantic consistency based on word-level attention and sentence-level attention model.”
BackgroundDR-GAN: Distribution Regularization for Text-to-Image Generation
Tan, H., Liu, X., Yin, B., & Li, X. (2022) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3165573 · Cited by 53
“MirrorGAN [33] combined the T2I generation and Image-to-Text generation to improve the global semantic consistency between text description and the generated image.”
BackgroundMethodInfluentialLanguage-Driven Artistic Style Transfer
Fu, T.-J., Wang, X. E., & Wang, W. Y. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-20059-5_41 · Cited by 53
“To further enhance the alignment with inputs, inspired by cycle consistency [60,51,38,55], we consider the content matching loss Lcm and the style matching loss Lsm of transferred results Ô.”
MethodRecurrent Affine Transformation for Text-to-Image Synthesis
Ye, S., Wang, H., Tan, M., & Liu, F. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3266607 · Cited by 53
Pan, Y., Liu, M., Lian, C., Xia, Y., & Shen, D. G. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-32248-9_16 · Cited by 53
“To utilize all available subjects, a cycleconsistent generative adversarial network (CGAN) has recently been employed to impute missing PET images [2,3].”
MethodA survey on generative adversarial network-based text-to-image synthesis
Zhou, R. H., Jiang, C. F., & Xu, Q. (2021) · Neurocomputing · doi:10.1016/j.neucom.2021.04.069 · Cited by 56
Intelligent Home 3D: Automatic 3D-House Design From Linguistic Descriptions Only
Chen, Q., Wu, Q., Tang, R., Wang, Y., Wang, S., & Tan, M. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01264 · Cited by 52
“Thanks to the fast development of Deep Learning (DL) [6, 9, 11, 43, 44, 48, 49, 50, 52], especially Generative Adversarial Network (GAN) [3, 4, 8, 10] and visionlanguage research [13, 45], we can turn this problem into a text-to-image generation problem, which has been studied in [22, 29, 31, 41, 42].”
BackgroundMethodCycle-Consistent Inverse GAN for Text-to-Image Synthesis
Wang, H., Lin, G., Hoi, S. C. H., & Miao, C. (2021) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3474085.3475226 · Cited by 52
“StackGAN++ [37] 4.04 ± 0.06 - AttnGAN [35] 4.36 ± 0.03 23.98 MirrorGAN [20] 4.56 ± 0.05 - ControlGAN [15] 4.58 ± 0.09 - SD-GAN [36] 4.67 ± 0.09 - DM-GAN [42] 4.75 ± 0.07 16.09 DF-GAN [26] 4.86 ± 0.04 19.24 RiFeGAN [6] 5.23 ± 0.09 - DALL-E (zero-shot) [21] 2.85 ± 0.04 51.00 CI-GAN (Ours) 5.72 ± 0.12 9.78 can get the be…”
MethodInfluentialDTGAN: Dual Attention Generative Adversarial Networks for Text-to-Image Generation
Zhang, Z., & Schomaker, L. (2021) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn52387.2021.9533527 · Cited by 52
Ak, K. E., Lim, J., Tham, J. Y., & Kassim, A. A. (2020) · Pattern Recognition Letters · doi:10.1016/j.patrec.2020.02.030 · Cited by 49
Image synthesis: a review of methods, datasets, evaluation metrics, and future outlook
Baraheem, S. S., Le, T.-N., & Nguyen, T. (2023) · Artificial Intelligence Review · doi:10.1007/s10462-023-10434-2 · Cited by 52
“Text-to-image synthesis (Zhu et al. 2007; Dosovitskiy et al. 2015; Yan et al. 2015; Gregor et al. 2015; Mansimov et al. 2015; Cai et al. 2017; Mirza and Osindero 2014; Reed et al. 2016a; Odena et al. 2016; Zhang et al. 2016, 2017, 2021a; Sharma et al. 2018; Park et al. 2018; Xu et al. 2017; Qiao et al. 2019; Gou et al…”
BackgroundDeep Structure-Revealed Network for Texture Recognition
Zhai, W., Cao, Y. H., Zha, Z.-J., Xie, H., & Wu, F. C. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01102 · Cited by 48
Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods
Mogadala, A., Kalimuthu, M., & Klakow, D. (2021) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.11688 · Cited by 146
“47 64x64 MirrorGAN (Qiao et al., 2019) 128x128 256x256 26.”
BackgroundInfluentialExploring Global and Local Linguistic Representations for Text-to-Image Synthesis
Li, R., Wang, N., Feng, F., Zhang, G., & Wang, X. (2020) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2020.2972856 · Cited by 47
“7) MirrorGAN [7] emphasizes the semantic consistency using the idea of re-description.”
MethodInvestigating the Bilateral Connections in Generative Zero-Shot Learning
Li, J., Jing, M., Lü, K., Zhu, L., & Shen, H. T. (2021) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2021.3050803 · Cited by 46
Generative AI in the context of assistive technologies: Trends, limitations and future directions
Fu, B., Hadid, A., & Damer, N. (2024) · Image and Vision Computing · doi:10.1016/j.imavis.2024.105347 · Cited by 57
New Ideas and Trends in Deep Multimodal Content Understanding: A Review
Chen, W., Wang, W., Liu, L., & Lew, M. S. (2020) · Neurocomputing · doi:10.1016/j.neucom.2020.10.042 · Cited by 45
“Down [32], Attention-aware [78], BSSAN [62], CRAN[149], CBP [36], SOT [5], PAGNet [34], MirrorGAN [153], DAI [154], T2I2T [11], CCGAN [13], C4Synth [155], Cycle-Attn+ [156], Coupled CycleGAN [157], TCCM [158], CycleMatch [159], VQA-Rephrasings [160], iQAN [161], MLAN [162], MFB [163], DANs [164],…”
FICE: Text-conditioned fashion-image editing with guided GAN inversion
Pernuš, M., Fookes, C., Štruc, V., & Dobrišek, S. (2024) · Pattern Recognition · doi:10.1016/j.patcog.2024.111022 · Cited by 44
Generative Imagination Elevates Machine Translation
Long, Q., Wang, M., & Li, L. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.457 · Cited by 46
“MirrorGAN (Qiao et al. 2019) employs a mirror structure, which reversely learns from the inverse task of T2I to further validate whether generated images are consistent with the input texts.”
BackgroundMethodInfluentialText to Image Synthesis With Bidirectional Generative Adversarial Network
Wang, Z., Quan, Z., Wang, Z., Hu, X., & Chen, Y. (2020) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme46284.2020.9102904 · Cited by 43
Cross-Modal Semantic Matching Generative Adversarial Networks for Text-to-Image Synthesis
Tan, H., Liu, X., Yin, B., & Li, X. S. (2021) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2021.3060291 · Cited by 43
StyleT2I: Toward Compositional and High-Fidelity Text-to-Image Synthesis
Li, Z., Min, M. R., Li, K., & Xu, C. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01766 · Cited by 46
“Text-to-Image Synthesis Many previous works [8,15,25,29, 30,32,46,48,50,58,68,70–73 , 75] have studied text-to-image text-to-image synthesis.”
BackgroundIntegrating Visuospatial, Linguistic, and Commonsense Structure into Story Visualization
Maharana, A., & Bansal, M. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.543 · Cited by 40
“Dual learning has served as an effective method for promoting desirable characteristics in target output for both text-to-image generation (Qiao et al., 2019) and story visualization (Maharana et al.”
MethodDeep Vision Multimodal Learning: Methodology, Benchmark, and Trend
Chai, W., & Wang, G. (2022) · Applied Sciences · doi:10.3390/app12136588 · Cited by 48
TIME: Text and Image Mutual-Translation Adversarial Networks
Liu, B., Song, K., Zhu, Y., de Melo, G., & Elgammal, A. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i3.16305 · Cited by 40
“MirrorGAN [24] incorporates a pre-trained text re-description RNN to better align the images with the given texts.”
BackgroundMethodInfluentialCharacter-Preserving Coherent Story Visualization
Song, Y.-Z., Tam, Z. R., Chen, H.-J., Lu, H.-H., & Shuai, H.-H. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58520-4_2 · Cited by 41
“MirrorGAN [23] employs a mirror structure, which reversely learns from generated images to output texts for validating whether generated images are consistent with the input texts.”
BackgroundKnowledge-Driven Generative Adversarial Network for Text-to-Image Synthesis
Peng, J., Zhou, Y., Sun, X., Cao, L., Wu, Y., Huang, F., & Ji, R. (2021) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2021.3116416 · Cited by 41
Synthesizing Coherent Story with Auto-Regressive Latent Diffusion Models
Pan, X., Qin, P., Li, Y., Xue, H., & Chen, W. (2024) · 2024 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv57701.2024.00290 · Cited by 47
“GANs like Stackgan [41], Attngan [38], Mirrorgan [21], and MXC-GAN [40] perform adversarial training between generators and discriminators to learn to generate high-quality images.”
BackgroundDual Adversarial Inference for Text-to-Image Synthesis
Lao, Q., Havaei, M., Pesaranghader, A., Dutil, F., Jorio, L. D., & Fevens, T. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00766 · Cited by 38
“Several attempts have been made to leverage additional available information, such as object location [27], class label [5, 2], attention extracted from word features [30, 24] and text regeneration [24].”
BackgroundMethodInfluentialTransformer models for enhancing AttnGAN based text to image generation
Naveen, S., Kiran, M. S. S. R., Indupriya, M., Manikanta, T., & Sudeep, P. V. (2021) · Image and Vision Computing · doi:10.1016/j.imavis.2021.104284 · Cited by 41
RiFeGAN2: Rich Feature Generation for Text-to-Image Synthesis From Constrained Prior Knowledge
Cheng, J. S., Wu, F., Tian, Y., Wang, L., & Tao, D. (2021) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2021.3136857 · Cited by 39
Self-citationUnifying Multimodal Transformer for Bi-directional Image and Text Generation
Huang, Y., Xue, H., Liu, B., & Lu, Y. (2021) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3474085.3481540 · Cited by 41
“MirrorGAN [34] focus on the text-toimage generation task and uses an off-the-shelf captioning model to regularize the redescription of the generated images to be semantic consistent with the original text.”
MethodAIGC Image Quality Assessment via Image-Prompt Correspondence
Peng, F., Fu, H., Ming, A., Wang, C., Ma, H., He, S., Dou, Z., & Chen, S. (2024) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw63382.2024.00644 · Cited by 40
“In recent years, the field of generative models within deep learning has seen an extraordinary evolution, particularly * Corresponding author with the advent of Generative Adversarial Networks [6, 11, 13, 15, 32] (GANs) and diffusion models [7, 17, 37, 38, 58].”
BackgroundMultimodal fusion techniques: Review, data representation, information fusion, and application areas
Hangloo, S., & Arora, B. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.130827 · Cited by 46
Improving Generation and Evaluation of Visual Stories via Semantic Consistency
Maharana, A., Hannan, D., & Bansal, M. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.194 · Cited by 55
Metadata
- Authors
- Qiao, T., Zhang, J., Xu, D., & Tao, D.
- Affiliations
- The University of Sydney (AU)
- University of Technology Sydney (AU)
- Venue
- 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2019-06
- Pages
- 1505-1514
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W2962845008
- Semantic Scholar
- 5af2d424bb38db7e6a72f7bec2cfd8a5bb8af7fb
- Cited by
- 644
- References
- 45
- Citations by year
- 2026: 17 · 2025: 71 · 2024: 99 · 2023: 134 · 2022: 102 · 2021: 113 · 2020: 92 · 2019: 16
- FWCI
- 26.32
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Natural Science Foundation of China (61806062)
- Cultural Heritage Administration
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- text-to-image generation, semantic consistency, redescription, global-local attention, image-text alignment, generative adversarial networks
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
