CoChat
Sign Up
PreprintCited by 224Generative Adversarial Networks and Image SynthesisDigital Media Forensic DetectionAdvanced Image Processing Techniques

Large Scale Image Completion via Co-Modulated Generative Adversarial Networks

Open access
Zhao, Shengyu, Cui, Jonathan, Sheng, Yilun, Dong, Yue, Liang, Xiao, Chang, Eric I, Xu, Yan·2021·arXiv
DOI 10.48550/arxiv.2103.10428arXiv 2103.10428
Open this paper in CoChat — free accountOpen PDF

Abstract

Numerous task-specific variants of conditional generative adversarial networks have been developed for image completion. Yet, a serious limitation remains that all existing algorithms tend to fail when handling large-scale missing regions. To overcome this challenge, we propose a generic new approach that bridges the gap between image-conditional and recent modulated unconditional generative architectures via co-modulation of both conditional and stochastic style representations. Also, due to the lack of good quantitative metrics for image completion, we propose the new Paired/Unpaired Inception Discriminative Score (P-IDS/U-IDS), which robustly measures the perceptual fidelity of inpainted images compared to real images via linear separability in a feature space. Experiments demonstrate superior performance in terms of both quality and diversity over state-of-the-art methods in free-form image completion and easy generalization to image-to-image translation. Code is available at https://github.com/zsyzzsoft/co-mod-gan.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2103.10428

Cites

8 of 8 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The improvement from image recovery is more apparent for Gib-E because a large portion of 3D meshes for reconstructing Gibson scenes is low-quality, leading to largely broken or distorted rendered views.

    “Such improvement is more apparent for Gib-E because a large portion of 3D meshes for reconstructing Gibson scenes is in lowquality [84], which leads to largely broken or distorted rendered views.”

    From Scaling Data Generation in Vision-and-Language Navigation (2023) · cites this paper as [84]

    The Image Completion section (p. 8) discusses improvements on FFHQ, and Appendix F (p. 15) discusses artifacts on Places2. Searches for Gibson, Gib-E, 3D meshes, and distorted rendered views found no passage addressing the claimed reason for the improvement. (LLM, conf=0.76)

    Experimental — accuracy not yet measured

  • Recent large datasets contain hundreds of interactive scenes and vast human demonstrations, leading to significant improvement in agent performance.

    “recent large datasets which contain hundreds of interactive scenes have been created [20, 66, 84], as well as a vast amount of human demonstrations have been collected [45, 67] for learning visual navigation, leading to significant improvement in agent's performance.”

    From Scaling Data Generation in Vision-and-Language Navigation (2023) · cites this paper as [84]

    Appendix F (p. 15) discusses the Places2 image dataset and an interactive demo of image completion, not interactive scenes for visual navigation. Searches for human demonstrations and agent performance found no passage addressing those claims; the paper’s user study (Appendix C, p. 14) concerns judgments of generated images. (LLM, conf=0.76)

    Experimental — accuracy not yet measured

  • The proposed paradigm applies 1200+ photo-realistic environments from the HM3D and Gibson datasets.

    “which applies 1200+ photo-realistic environments from HM3D and Gibson datasets”

    From Scaling Data Generation in Vision-and-Language Navigation (2023) · cites this paper as [84]

    Section 5.1 (p. 7) describes image-completion experiments on FFHQ and Places2. Searches for HM3D, Gibson, and 1200 photo-realistic environments found no passage in our copy addressing the claimed use of those datasets. (LLM, conf=0.75)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 72

  1. Fazil Altinel, Mete Ozay, and Takayuki Okatani. Deep structured energy-based image inpainting. In 2018 24th International Conference on Pattern Recognition (ICPR) , pp. 423-428. IEEE, 2018.
  2. Coloma Ballester, Marcelo Bertalmio, Vicent Caselles, Guillermo Sapiro, and Joan Verdera. Filling-in by joint interpolation of vector fields and gray levels. IEEE transactions on image processing , 10(8):1200-1211, 2001.
  3. Connelly Barnes, Eli Shechtman, Adam Finkelstein, and Dan B Goldman. Patchmatch: A randomized correspondence algorithm for structural image editing. In ACM Transactions on Graphics (ToG) , pp. 24. ACM, 2009.
  4. Mikołaj Bi´ nkowski, Dougal J Sutherland, Michael Arbel, and Arthur Gretton. Demystifying mmd gans. arXiv preprint arXiv:1801.01401 , 2018. · arXiv
  5. Yochai Blau and Tomer Michaeli. The perception-distortion tradeoff. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pp. 6228-6237, 2018.
  6. Léon Bottou and Chih-Jen Lin. Support vector machine solvers. Large scale kernel machines , 3(1):301-320, 2007.
  7. Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. Coco-stuff: Thing and stuff classes in context. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pp. 1209-1218, 2018.
  8. Weiwei Cai and Zhanguo Wei. Diversity-generated image inpainting with style extraction. arXiv preprint arXiv:1912.01834 , 2019. · arXiv
  9. Ting Chen, Mario Lucic, Neil Houlsby, and Sylvain Gelly. On self modulation for generative adversarial networks. In International Conference on Learning Representations , 2019. URL https://openreview. net/forum?id=Hkl5aoR5tm .
  10. Soheil Darabi, Eli Shechtman, Connelly Barnes, Dan B Goldman, and Pradeep Sen. Image melding: Combining inconsistent images using patch-based synthesis. ACM Trans. Graph. , 31(4):82-1, 2012.
  11. Terrance DeVries, Adriana Romero, Luis Pineda, Graham W Taylor, and Michal Drozdzal. On the evaluation of conditional gans. arXiv preprint arXiv:1907.08175 , 2019. · arXiv
  12. Ding Ding, Sundaresh Ram, and Jeffrey J Rodríguez. Image inpainting using nonlocal texture matching and nonlinear filtering. IEEE Transactions on Image Processing , 28(4):1705-1719, 2018.
  13. Vincent Dumoulin, Jonathon Shlens, and Manjunath Kudlur. A learned representation for artistic style. arXiv preprint arXiv:1610.07629 , 2016. · arXiv
  14. Alexei A Efros and William T Freeman. Image quilting for texture synthesis and transfer. In Proceedings of the 28th annual conference on Computer graphics and interactive techniques , pp. 341-346. ACM, 2001.
  15. Alexei A Efros and Thomas K Leung. Texture synthesis by non-parametric sampling. In Proceedings of the seventh IEEE international conference on computer vision , volume 2, pp. 1033-1038. IEEE, 1999.
  16. Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural information processing systems , pp. 2672-2680, 2014.
  17. Swaminathan Gurumurthy, Ravi Kiran Sarvadevabhatla, and R Venkatesh Babu. Deligan: Generative adversarial networks for diverse and limited data. In Proceedings of the IEEE conference on computer vision and pattern recognition , pp. 166-174, 2017.
  18. Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normalization. In Proceedings of the IEEE International Conference on Computer Vision , pp. 1501-1510, 2017.
  19. Xun Huang, Ming-Yu Liu, Serge Belongie, and Jan Kautz. Multimodal unsupervised image-to-image translation. In Proceedings of the European Conference on Computer Vision (ECCV) , pp. 172-189, 2018.
  20. Satoshi Iizuka, Edgar Simo-Serra, and Hiroshi Ishikawa. Globally and locally consistent image completion. ACM Transactions on Graphics (ToG) , 36(4):107, 2017.
  21. Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition , pp. 1125-1134, 2017.
  22. Jianmin Jiang, Hossam M Kasem, and Kwok-Wai Hung. Robust image completion via deep feature transformations. IEEE Access , 7:113916-113930, 2019.
  23. Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. arXiv preprint arXiv:1912.04958 , 2019b. · arXiv
  24. Junho Kim, Minjae Kim, Hyeonwoo Kang, and Kwanghee Lee. U-gat-it: unsupervised generative attentional networks with adaptive layer-instance normalization for image-to-image translation. arXiv preprint arXiv:1907.10830 , 2019. · arXiv
  25. Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014. · arXiv
  26. Rolf Köhler, Christian Schuler, Bernhard Schölkopf, and Stefan Harmeling. Mask-specific inpainting with deep neural networks. In German Conference on Pattern Recognition , pp. 523-534. Springer, 2014.
  27. Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. In Advances in Neural Information Processing Systems , pp. 3929-3938, 2019.
  28. Avisek Lahiri, Arnav Jain, Prabir Kumar Biswas, and Pabitra Mitra. Improving consistency and correctness of sequence inpainting using semantically guided generative adversarial network. arXiv preprint arXiv:1711.06106 , 2017. · arXiv
  29. Avisek Lahiri, Arnav Kumar Jain, Sanskar Agrawal, Pabitra Mitra, and Prabir Kumar Biswas. Prior guided gan based semantic inpainting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 13696-13705, 2020.
  30. Jingyuan Li, Ning Wang, Lefei Zhang, Bo Du, and Dacheng Tao. Recurrent feature reasoning for image inpainting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 7760-7768, 2020.
  31. Hongyu Liu, Bin Jiang, Yi Xiao, and Chao Yang. Coherent semantic attention for image inpainting. arXiv preprint arXiv:1905.12384 , 2019a. · arXiv
  32. Hongyu Liu, Bin Jiang, Yibing Song, Wei Huang, and Chao Yang. Rethinking image inpainting via a mutual encoder-decoder with feature equalizations. arXiv preprint arXiv:2007.06929 , 2020. · arXiv
  33. Ming-Yu Liu, Thomas Breuel, and Jan Kautz. Unsupervised image-to-image translation networks. In Advances in neural information processing systems , pp. 700-708, 2017.
  34. Ming-Yu Liu, Xun Huang, Arun Mallya, Tero Karras, Timo Aila, Jaakko Lehtinen, and Jan Kautz. Few-shot unsupervised image-to-image translation. In Proceedings of the IEEE International Conference on Computer Vision , pp. 10551-10560, 2019b.
  35. David Lopez-Paz and Maxime Oquab. Revisiting classifier two-sample tests. arXiv preprint arXiv:1610.06545 , 2016. · arXiv
  36. Yuqing Ma, Xianglong Liu, Shihao Bai, Lei Wang, Aishan Liu, Dacheng Tao, and Edwin Hancock. Region-wise generative adversarial imageinpainting for large missing areas. arXiv preprint arXiv:1909.12507 , 2019. · arXiv
  37. Qi Mao, Hsin-Ying Lee, Hung-Yu Tseng, Siwei Ma, and Ming-Hsuan Yang. Mode seeking generative adversarial networks for diverse image synthesis. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pp. 1429-1437, 2019.
  38. Lars Mescheder, Andreas Geiger, and Sebastian Nowozin. Which training methods for gans do actually converge? arXiv preprint arXiv:1801.04406 , 2018. · arXiv
  39. Kamyar Nazeri, Eric Ng, Tony Joseph, Faisal Qureshi, and Mehran Ebrahimi. Edgeconnect: Generative image inpainting with adversarial edge learning. arXiv preprint arXiv:1901.00212 , 2019. · arXiv
  40. Augustus Odena, Jacob Buckman, Catherine Olsson, Tom B Brown, Christopher Olah, Colin Raffel, and Ian Goodfellow. Is generator conditioning causally related to gan performance? arXiv preprint arXiv:1802.08768 , 2018. · arXiv
  41. Taesung Park, Ming-Yu Liu, Ting-Chun Wang, and Jun-Yan Zhu. Semantic image synthesis with spatiallyadaptive normalization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pp. 2337-2346, 2019.
  42. Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. Context encoders: Feature learning by inpainting. In Proceedings of the IEEE conference on computer vision and pattern recognition , pp. 2536-2544, 2016.
  43. Pengda Qin, Weiran Xu, and William Yang Wang. Dsgan: Generative adversarial training for distant supervision relation extraction. arXiv preprint arXiv:1805.09929 , 2018. · arXiv
  44. Jimmy SJ Ren, Li Xu, Qiong Yan, and Wenxiu Sun. Shepard convolutional neural networks. In Advances in Neural Information Processing Systems , pp. 901-909, 2015.
  45. Yurui Ren, Xiaoming Yu, Ruonan Zhang, Thomas H Li, Shan Liu, and Ge Li. Structureflow: Image inpainting via structure-aware appearance flow. In Proceedings of the IEEE International Conference on Computer Vision , pp. 181-190, 2019.
  46. Mark Sabini and Gili Rusak. Painting outside the box: Image outpainting with gans. arXiv preprint arXiv:1808.08483 , 2018. · arXiv
  47. Min-cheol Sagong, Yong-goo Shin, Seung-wook Kim, Seung Park, and Sung-jea Ko. Pepsi: Fast image inpainting with parallel decoding network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pp. 11360-11368, 2019.
  48. Mehdi SM Sajjadi, Olivier Bachem, Mario Lucic, Olivier Bousquet, and Sylvain Gelly. Assessing generative models via precision and recall. In Advances in Neural Information Processing Systems , pp. 5228-5237, 2018.
  49. Tim Salimans and Durk P Kingma. Weight normalization: A simple reparameterization to accelerate training of deep neural networks. In Advances in neural information processing systems , pp. 901-909, 2016.
  50. Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In Advances in neural information processing systems , pp. 2234-2242, 2016.
  51. Yi Wang, Xin Tao, Xiaojuan Qi, Xiaoyong Shen, and Jiaya Jia. Image inpainting via generative multi-column convolutional neural networks. In Advances in Neural Information Processing Systems , pp. 331-340, 2018.
  52. Yi Wang, Xin Tao, Xiaoyong Shen, and Jiaya Jia. Wide-context semantic image extrapolation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pp. 1399-1408, 2019a.
  53. Ze Wang, Xiuyuan Cheng, Guillermo Sapiro, and Qiang Qiu. Stochastic conditional generative networks with basis decomposition. arXiv preprint arXiv:1909.11286 , 2019b. · arXiv
  54. Sitao Xiang and Hao Li. On the effects of batch and weight normalization in generative adversarial networks. arXiv preprint arXiv:1704.03971 , 2017. · arXiv
  55. Chaohao Xie, Shaohui Liu, Chao Li, Ming-Ming Cheng, Wangmeng Zuo, Xiao Liu, Shilei Wen, and Errui Ding. Image inpainting with learnable bidirectional attention maps. In Proceedings of the IEEE International Conference on Computer Vision , pp. 8858-8867, 2019.
  56. Junyuan Xie, Linli Xu, and Enhong Chen. Image denoising and inpainting with deep neural networks. In Advances in neural information processing systems , pp. 341-349, 2012.
  57. Wei Xiong, Jiahui Yu, Zhe Lin, Jimei Yang, Xin Lu, Connelly Barnes, and Jiebo Luo. Foreground-aware image inpainting. In Proceedings of the IEEE conference on computer vision and pattern recognition , pp. 5840-5848, 2019.
  58. Zhaoyi Yan, Xiaoming Li, Mu Li, Wangmeng Zuo, and Shiguang Shan. Shift-net: Image inpainting via deep feature rearrangement. In Proceedings of the European Conference on Computer Vision (ECCV) , pp. 1-17, 2018.
  59. Chao Yang, Xin Lu, Zhe Lin, Eli Shechtman, Oliver Wang, and Hao Li. High-resolution image inpainting using multi-scale neural patch synthesis. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pp. 6721-6729, 2017.
  60. Yang Yang, Xiaojie Guo, Jiayi Ma, Lin Ma, and Haibin Ling. Lafin: Generative landmark guided face inpainting. arXiv preprint arXiv:1911.11394 , 2019a. · arXiv
  61. Zongxin Yang, Jian Dong, Ping Liu, Yi Yang, and Shuicheng Yan. Very long natural scenery image prediction by outpainting. In Proceedings of the IEEE International Conference on Computer Vision , pp. 10561-10570, 2019b.
  62. Zili Yi, Hao Zhang, Ping Tan, and Minglun Gong. Dualgan: Unsupervised dual learning for image-to-image translation. In Proceedings of the IEEE international conference on computer vision , pp. 2849-2857, 2017.
  63. Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S Huang. Generative image inpainting with contextual attention. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pp. 5505-5514, 2018.
  64. Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S Huang. Free-form image inpainting with gated convolution. In Proceedings of the IEEE International Conference on Computer Vision , pp. 4471-4480, 2019.
  65. Tao Yu, Zongyu Guo, Xin Jin, Shilin Wu, Zhibo Chen, Weiping Li, Zhizheng Zhang, and Sen Liu. Region Normalization for Image Inpainting. arXiv e-prints , art. arXiv:1911.10375, Nov 2019. · arXiv
  66. Yanhong Zeng, Jianlong Fu, Hongyang Chao, and Baining Guo. Learning pyramid-context encoder network for high-quality image inpainting. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pp. 1486-1494, 2019.
  67. Lei Zhao, Qihang Mo, Sihuan Lin, Zhizhong Wang, Zhiwen Zuo, Haibo Chen, Wei Xing, and Dongming Lu. Uctgan: Diverse image inpainting based on unsupervised cross-space translation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 5741-5750, 2020a.
  68. Shengyu Zhao, Zhijian Liu, Ji Lin, Jun-Yan Zhu, and Song Han. Differentiable augmentation for data-efficient gan training. Advances in Neural Information Processing Systems , 33, 2020b.
  69. Chuanxia Zheng, Tat-Jen Cham, and Jianfei Cai. Pluralistic image completion. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pp. 1438-1447, 2019.
  70. Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE transactions on pattern analysis and machine intelligence , 40(6): 1452-1464, 2017.
  71. Tong Zhou, Changxing Ding, Shaowen Lin, Xinchao Wang, and Dacheng Tao. Learning oracle attention for high-fidelity face completion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 7680-7689, 2020.
  72. Jun-Yan Zhu, Richard Zhang, Deepak Pathak, Trevor Darrell, Alexei A Efros, Oliver Wang, and Eli Shechtman. Toward multimodal image-to-image translation. In Advances in neural information processing systems , pp. 465-476, 2017b.

Cited by

Sorted by citations

Authors
Zhao, S., Cui, J., Sheng, Y., Dong, Y., Liang, X., Chang, E. I., & Xu, Y.
Affiliations
  • Tsinghua University (CN)
  • Microsoft (United States) (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2021
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2103.10428
arXiv
2103.10428
OpenAlex
W3130440474
Cited by
224
References
8
Citations by year
2026: 8 · 2025: 24 · 2024: 72 · 2023: 62 · 2022: 50 · 2021: 8
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • Microsoft Research
  • National Science and Technology Major Project
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Digital Media Forensic Detection
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Image Processing Techniques
Keywords
image inpainting, conditional generative adversarial networks, image-to-image translation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review