CoChat
Sign Up
Cited by 73Generative Adversarial Networks and Image SynthesisMultimodal Machine Learning ApplicationsDigital Media Forensic Detection

Multimodal Conditional Image Synthesis with Product-of-Experts GANs

Open access
Huang, Xun, Mallya, Arun, Wang, Ting-Chun, Liu, Ming-Yu·2022·Lecture Notes in Computer Science
DOI 10.1007/978-3-031-19787-1_6
Open this paper in CoChat — free accountOpen PDF

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1007/978-3-031-19787-1_6

Cites

6 of 6 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Huang et al. introduced PoE-GAN, which used a multimodal generator based on PoE to fuse multimodal or unimodal inputs for converting to an image domain.

    “Huang et al. introduced PoE-GAN, which used a multimodal generator based on PoE to fuse multimodal or unimodal inputs for converting to an image domain [60].”

    From Challenges in AI-driven Biomedical Multimodal Data Fusion and Analysis (2025) · cites this paper as [60]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 77

  1. Alexander A Alemi, Ian Fischer, Joshua V Dillon, and Kevin Murphy. Deep variational information bottleneck. In ICLR, 2017. 5
  2. Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In ICLR, 2019. 2
  3. Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. COCO- Stuff: Thing and stuff classes in context. In CVPR, 2018. 6, 12
  4. Lucy Chai, David Bau, Ser-Nam Lim, and Phillip Isola. What makes fake images detectable? understanding properties that generalize. In ECCV, 2020. 8
  5. Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolu- tion, and fully connected crfs. IEEE transactions on pattern analysis and machine intelligence, 2017. 12
  6. Shu-Yu Chen, Wanchao Su, Lin Gao, Shihong Xia, and Hongbo Fu. DeepFaceDrawing: Deep generation of face im- ages from sketches. ACM Transactions on Graphics (TOG), 2020. 2, 12
  7. Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geof- frey Hinton. A simple framework for contrastive learning of visual representations. In ICML, 2020. 5
  8. Wengling Chen and James Hays. SketchyGAN: Towards diverse and realistic sketch to image synthesis. In CVPR, 2018. 2
  9. Rewon Child. Very deep VAEs generalize autoregressive models and can outperform them on images. In ICLR, 2021. 2, 3
  10. Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In CVPR, 2021. 2, 6, 7, 14, 16
  11. Arnab Ghosh, Richard Zhang, Puneet K Dokania, Oliver Wang, Alexei A Efros, Philip HS Torr, and Eli Shechtman. In- teractive sketch & fill: Multiclass sketch-to-image translation. In ICCV, 2019. 2
  12. Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. NeurIPS, 2014. 1, 2, 4, 5
  13. Junlin Han, Mehrdad Shoeiby, Lars Petersson, and Moham- mad Ali Armin. Dual contrastive learning for unsupervised image-to-image translation. In CVPR, 2021. 5
  14. Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual repre- sentation learning. In CVPR, 2020. 5
  15. Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bern- hard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In NeurIPS, 2017. 7
  16. Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normalization. In ICCV, 2017. 4
  17. Xun Huang, Ming-Yu Liu, Serge Belongie, and Jan Kautz. Multimodal unsupervised image-to-image translation. In ECCV, 2018. 8
  18. Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial net- works. In CVPR, 2017. 1, 2
  19. Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of GANs for improved quality, stability, and variation. In ICLR, 2018. 2, 12
  20. Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adver- sarial networks with limited data. In NeurIPS, 2020. 12
  21. Tero Karras, Miika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Alias-free generative adversarial networks. In NeurIPS, 2021. 2
  22. Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of StyleGAN. In CVPR, 2020. 2, 6, 7, 12, 14
  23. Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2014. 12
  24. Diederik P Kingma and Max Welling. Auto-encoding varia- tional bayes. In ICLR, 2014. 2
  25. Svetlana Kutuzova, Oswin Krause, Douglas McCloskey, Mads Nielsen, and Christian Igel. Multimodal variational autoencoders for semi-supervised learning: In defense of product-of-experts. arXiv preprint arXiv:2101.07240, 2021. 2 · arXiv
  26. Cheng-Han Lee, Ziwei Liu, Lingyun Wu, and Ping Luo. Maskgan: Towards diverse and interactive facial image ma- nipulation. In CVPR, 2020. 12
  27. Bowen Li, Xiaojuan Qi, Thomas Lukasiewicz, and Philip HS Torr. Controllable text-to-image generation. In NeurIPS, 2019. 15
  28. Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft COCO: Common objects in context. In ECCV, 2014. 2, 6, 12
  29. Rui Liu, Yixiao Ge, Ching Lam Choi, Xiaogang Wang, and Hongsheng Li. DivCo: Diverse conditional image synthesis via contrastive generative adversarial network. In CVPR, 2021. 5
  30. Xihui Liu, Guojun Yin, Jing Shao, Xiaogang Wang, and Hong- sheng Li. Learning to predict layout-to-image conditional convolutions for semantic image synthesis. In NeurIPS, 2019. 2, 5
  31. Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In ICCV, 2015. 12
  32. Lars Maaløe, Marco Fraccaro, Valentin Liévin, and Ole Winther. BIVA: A very deep hierarchy of latent variables for generative modeling. NeurIPS, 2019. 2, 3
  33. Andrew L Maas, Awni Y Hannun, and Andrew Y Ng. Recti- fier nonlinearities improve neural network acoustic models. In ICML, 2013. 12
  34. Lars Mescheder, Andreas Geiger, and Sebastian Nowozin. Which training methods for GANs do actually converge? In ICML, 2018. 5, 12
  35. Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adver- sarial networks. In ICLR, 2018. 2
  36. Takeru Miyato and Masanori Koyama. cGANs with projec- tion discriminator. In ICLR, 2018. 2, 4
  37. Augustus Odena, Christopher Olah, and Jonathon Shlens. Conditional image synthesis with auxiliary classifier GANs. In ICML, 2017. 1, 2
  38. Taesung Park, Alexei A Efros, Richard Zhang, and Jun-Yan Zhu. Contrastive learning for unpaired image-to-image trans- lation. In ECCV. Springer, 2020. 5
  39. Taesung Park, Ming-Yu Liu, Ting-Chun Wang, and Jun-Yan Zhu. Semantic image synthesis with spatially-adaptive nor- malization. In CVPR, 2019. 2, 4, 6, 7, 14, 16
  40. Gaurav Parmar, Richard Zhang, and Jun-Yan Zhu. On buggy resizing libraries and surprising subtleties in FID calculation. arXiv preprint arXiv:2104.11222, 2021. 7 · arXiv
  41. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. PyTorch: An imperative style, high-performance deep learning library. In NeurIPS, 2019. 12
  42. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021. 4, 5, 12, 14
  43. Alec Radford, Luke Metz, and Soumith Chintala. Unsuper- vised representation learning with deep convolutional genera- tive adversarial networks. In CVPR, 2016. 2
  44. Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In ICML, 2021. 2
  45. Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Lo- geswaran, Bernt Schiele, and Honglak Lee. Generative adver- sarial text to image synthesis. In ICML, 2016. 2
  46. Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wier- stra. Stochastic backpropagation and approximate inference in deep generative models. In ICML, 2014. 2
  47. Elad Richardson, Yuval Alaluf, Or Patashnik, Yotam Nitzan, Yaniv Azar, Stav Shapiro, and Daniel Cohen-Or. Encoding in style: a StyleGAN encoder for image-to-image translation. In CVPR, 2021. 6, 7, 12, 14
  48. Patsorn Sangkloy, Jingwan Lu, Chen Fang, Fisher Yu, and James Hays. Scribbler: Controlling deep image synthesis with sketch and color. In CVPR, 2017. 2
  49. Edgar Schönfeld, Vadim Sushko, Dan Zhang, Juergen Gall, Bernt Schiele, and Anna Khoreva. You only need adversarial supervision for semantic image synthesis. In ICLR, 2020. 2, 6, 7, 14, 16
  50. Yuge Shi, N Siddharth, Brooks Paige, and Philip Torr. Varia- tional mixture-of-experts autoencoders for multi-modal deep generative models. In NeurIPS, 2019. 2
  51. Edgar Simo-Serra, Satoshi Iizuka, Kazuma Sasaki, and Hi- roshi Ishikawa. Learning to simplify: fully convolutional networks for rough sketch cleanup. ACM Transactions on Graphics (TOG), 2016. 12
  52. Karen Simonyan and Andrew Zisserman. Very deep convolu- tional networks for large-scale image recognition. In ICLR, 2015. 5
  53. Casper Kaae Sønderby, Tapani Raiko, Lars Maaløe, Søren Kaae Sønderby, and Ole Winther. Ladder variational autoencoders. In NeurIPS, 2016. 2, 3
  54. Thomas M Sutter, Imant Daunhawer, and Julia E Vogt. Gen- eralized multimodal ELBO. In ICLR, 2020. 2
  55. Masahiro Suzuki, Kotaro Nakayama, and Yutaka Matsuo. Joint multimodal learning with deep generative models. In ICLR workshop, 2017. 2
  56. Ming Tao, Hao Tang, Songsong Wu, Nicu Sebe, Xiao-Yuan Jing, Fei Wu, and Bingkun Bao. DF-GAN: Deep fusion generative adversarial networks for text-to-image synthesis. arXiv preprint arXiv:2008.05865, 2020. 2, 6, 7, 14, 15, 16 · arXiv
  57. Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Im- proved texture networks: Maximizing quality and diversity in feed-forward stylization and texture synthesis. In CVPR, 2017. 13
  58. Arash Vahdat and Jan Kautz. NVAE: A deep hierarchical variational autoencoder. In NeurIPS, 2020. 2, 3, 12
  59. Ramakrishna Vedantam, Ian Fischer, Jonathan Huang, and Kevin Murphy. Generative models of visually grounded imag- ination. In ICLR, 2018. 2
  60. Sheng-Yu Wang, Oliver Wang, Richard Zhang, Andrew Owens, and Alexei A Efros. CNN-generated images are surprisingly easy to spot... for now. In CVPR, 2020. 8
  61. Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. High-resolution image synthesis and semantic manipulation with conditional GANs. In CVPR, 2018. 2
  62. Mika Westerlund. The emergence of deepfake technology: A review. Technology Innovation Management Review, 2019. 8
  63. Christopher KI Williams, Felix V Agakov, and Stephen N Felderhof. Products of gaussians. In NeurIPS, 2001. 3
  64. Mike Wu and Noah Goodman. Multimodal generative models for scalable weakly-supervised learning. In NeurIPS, 2018. 2
  65. Weihao Xia, Yujiu Yang, Jing-Hao Xue, and Baoyuan Wu. TediGAN: Text-guided diverse face image generation and manipulation. In CVPR, 2021. 2, 6, 12, 14, 15
  66. Saining Xie and Zhuowen Tu. Holistically-nested edge detec- tion. In ICCV, 2015. 12
  67. Dingdong Yang, Seunghoon Hong, Yunseok Jang, Tianchen Zhao, and Honglak Lee. Diversity-sensitive conditional gen- erative adversarial networks. In ICLR, 2019. 8
  68. Hui Ye, Xiulong Yang, Martin Takac, Rajshekhar Sunderra- man, and Shihao Ji. Improving text-to-image synthesis using contrastive learning. arXiv preprint arXiv:2107.02423, 2021. 2, 6, 7, 14, 16 · arXiv
  69. Ning Yu, Vladislav Skripniuk, Sahar Abdelnabi, and Mario Fritz. Artificial fingerprinting for generative models: Rooting deepfake attribution in training data. In ICCV, 2021. 8
  70. Han Zhang, Jing Yu Koh, Jason Baldridge, Honglak Lee, and Yinfei Yang. Cross-modal contrastive learning for text-to- image generation. In CVPR, 2021. 5
  71. Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiao- gang Wang, Xiaolei Huang, and Dimitris N Metaxas. Stack- GAN: Text to photo-realistic image synthesis with stacked generative adversarial networks. In ICCV, 2017. 2
  72. Richard Zhang. Making convolutional networks shift- invariant again. In ICML, 2019. 12
  73. Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In CVPR, 2018. 7
  74. Yuhao Zhang, Hang Jiang, Yasuhide Miura, Christopher D Manning, and Curtis P Langlotz. Contrastive learning of medical visual representations from paired images and text. arXiv preprint arXiv:2010.00747, 2020. 5 · arXiv
  75. Zhu Zhang, Jianxin Ma, Chang Zhou, Rui Men, Zhikang Li, Ming Ding, Jie Tang, Jingren Zhou, and Hongxia Yang. M6- UFC: Unifying multi-modal controls for conditional image synthesis. In NeurIPS, 2021. 2, 14, 15
  76. Jun-Yan Zhu, Richard Zhang, Deepak Pathak, Trevor Darrell, Alexei A Efros, Oliver Wang, and Eli Shechtman. Toward multimodal image-to-image translation. In NeurIPS, 2017. 8
  77. Minfeng Zhu, Pingbo Pan, Wei Chen, and Yi Yang. DM- GAN: Dynamic memory generative adversarial networks for text-to-image synthesis. In CVPR, 2019. 2, 15

Cited by

Sorted by citations

Authors
Huang, X., Mallya, A., Wang, T.-C., & Liu, M.-Y.
Affiliations
Nvidia (United States) (US)
Corresponding author
Huang, Xun
ORCID
Venue
Lecture Notes in Computer Science
Publisher
Springer Nature Switzerland
Published
2022
Pages
91-109
ISSN
  • 0302-9743 (ISSN-L)
  • 1611-3349
DOI
10.1007/978-3-031-19787-1_6
OpenAlex
W4225862877
Cited by
73
References
6
Citations by year
2026: 6 · 2025: 13 · 2024: 17 · 2023: 30 · 2022: 7
FWCI
5.92
Type
conference paper
Language
en
Open access
Licence
Locations
  • Lecture notes in computer science (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Digital Media Forensic Detection
Keywords
product of experts, generative adversarial networks, text-to-image generation, sketch-to-image generation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review