The DOI matches its Crossref record; 54 of the 54 references it cites were found in a registry or an index.
Free-Form Image Inpainting With Gated Convolution
Open accessAbstract
We present a generative image inpainting system to complete images with free-form mask and guidance. The system is based on gated convolutions learned from millions of images without additional labelling efforts. The proposed gated convolution solves the issue of vanilla convolution that treats all input pixels as valid ones, generalizes partial convolution by providing a learnable dynamic feature selection mechanism for each channel at each spatial location across all layers. Moreover, as free-form masks may appear anywhere in images with any shape, global and local GANs designed for a single rectangular mask are not applicable. Thus, we also present a patch-based GAN loss, named SN-PatchGAN, by applying spectral-normalized discriminator on dense image patches. SN-PatchGAN is simple in formulation, fast and stable in training. Results on automatic image inpainting and user-guided extension demonstrate that our system generates higher-quality and more flexible results than previous methods. Our system helps user quickly remove distracting objects, modify image layouts, clear watermarks and edit faces. Code, demo and models are available at: \url{https://github.com/JiahuiYu/generative_inpainting}.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
U-Net: Convolutional Networks for Biomedical Image Segmentation
Ronneberger, O., Fischer, P., & Brox, T. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24574-4_28 · Cited by 99,404
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380
Squeeze-and-Excitation Networks
Hu, J., Shen, L., & Sun, G. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00745 · Cited by 35,667
Image-to-Image Translation with Conditional Adversarial Networks
Isola, P., Zhu, J.-Y., Zhou, T., & Efros, A. A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.632 · Cited by 22,930
Squeeze-and-Excitation Networks
Hu, J., Shen, L., Albanie, S., Sun, G., & Wu, E. (2020) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2019.2913372 · Cited by 12,157
TensorFlow: A system for large-scale machine learning
Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S., Irving, G., Isard, M., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.08695 · Cited by 11,822
Perceptual Losses for Real-Time Style Transfer and Super-Resolution
Johnson, J., Alahi, A., & Fei-Fei, L. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46475-6_43 · Cited by 11,283
Context Encoders: Feature Learning by Inpainting
Pathak, D., Krahenbuhl, P., Donahue, J., Darrell, T., & Efros, A. A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.278 · Cited by 5,691
WaveNet: a generative model for raw audio
Oord, A. V. D., Dieleman, S., Zen, H., Simonyan, K., Vinyals, O., Graves, A., Kalchbrenner, N., Senior, A., & Kavukcuoglu, K. (2016) · arXiv · doi:10.48550/arxiv.1609.03499 · Cited by 4,911
Progressive Growing of GANs for Improved Quality, Stability, and Variation
Karras, T., Aila, T., Laine, S., & Lehtinen, J. (2017) · arXiv · doi:10.48550/arxiv.1710.10196 · Cited by 4,727
High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs
Wang, T.-C., Liu, M.-Y., Zhu, J.-Y., Tao, A., Kautz, J., & Catanzaro, B. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00917 · Cited by 4,332
Places: A 10 Million Image Database for Scene Recognition
Zhou, B., Lapedriza, A., Khosla, A., Oliva, A., & Torralba, A. (2018) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2017.2723009 · Cited by 4,042
Bertalmio, M., Sapiro, G., Caselles, V., & Ballester, C. (2000) · Proceedings of the 27th annual conference on Computer graphics and interactive techniques - SIGGRAPH '00 · doi:10.1145/344779.344972 · Cited by 3,580
Spectral Normalization for Generative Adversarial Networks
Miyato, T., Kataoka, T., Koyama, M., & Yoshida, Y. (2018) · arXiv · doi:10.48550/arxiv.1802.05957 · Cited by 3,438
Region Filling and Object Removal by Exemplar-Based Image Inpainting
Criminisi, A., Perez, P., & Toyama, K. (2004) · IEEE Transactions on Image Processing · doi:10.1109/tip.2004.833105 · Cited by 3,258 · cited 2×
“…2001; Barnes et al. 2009; Drori et al. 2003; Sun et al. 2005], structures [Huang et al. 2014], transformation or distortion information [Huang et al. 2013; Pavi´c et al. 2006] and image exemplars [Criminisi et al. 2004; Hays and Efros 2007; Kwatra et al. 2005; Whyte et al. 2009; Zhao et al. 2018].”
BackgroundMethodHolistically-Nested Edge Detection
Xie, S., & Tu, Z. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.164 · Cited by 3,096
Texture synthesis by non-parametric sampling
Efros, A. A., & Leung, T. K. (1999) · Proceedings of the Seventh IEEE International Conference on Computer Vision · doi:10.1109/iccv.1999.790383 · Cited by 3,023
PatchMatch: a randomized correspondence algorithm for structural image editing
Barnes, C., Shechtman, E., Finkelstein, A., & Goldman, D. B. (2009) · ACM Transactions on Graphics · doi:10.1145/1531326.1531330 · Cited by 2,880
Generative Image Inpainting with Contextual Attention
Yu, J., Lin, Z., Yang, J., Shen, X., Lu, X., & Huang, T. S. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00577 · Cited by 2,527
Self-citationImage Inpainting for Irregular Holes Using Partial Convolutions
Liu, G., Reda, F. A., Shih, K. J., Wang, T.-C., Tao, A., & Catanzaro, B. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01252-6_6 · Cited by 2,419 · cited 24×
“Next, we compare our model with previous state-of-theart methods [15, 23, 49].”
BackgroundMethodResultInfluentialImage quilting for texture synthesis and transfer
Efros, A. A., & Freeman, W. T. (2001) · Proceedings of the 28th annual conference on Computer graphics and interactive techniques · doi:10.1145/383259.383296 · Cited by 2,347
Globally and locally consistent image completion
Iizuka, S., Simo-Serra, E., & Ishikawa, H. (2017) · ACM Transactions on Graphics · doi:10.1145/3072959.3073659 · Cited by 2,203 · cited 19×
“For previous inpainting networks which try to fill a single rectangular hole, an additional local GAN is used on the masked rectangular region to improve results [15, 49].”
BackgroundMethodResultInfluentialConditional Image Generation with PixelCNN Decoders
van den Oord, A., Kalchbrenner, N., Vinyals, O., Espeholt, L., Graves, A., & Kavukcuoglu, K. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.05328 · Cited by 1,833
Precomputed Real-Time Texture Synthesis with Markovian Generative Adversarial Networks
Li, C., & Wand, M. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46487-9_43 · Cited by 1,421
Semantic Image Inpainting with Deep Generative Models
Yeh, R. A., Chen, C., Lim, T. Y., Schwing, A. G., Hasegawa-Johnson, M., & Do, M. N. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.728 · Cited by 1,206
Filling-in by joint interpolation of vector fields and gray levels
Ballester, C., Bertalmio, M., Caselles, V., Sapiro, G., & Verdera, J. (2001) · IEEE Transactions on Image Processing · doi:10.1109/83.935036 · Cited by 962
Srivastava, R. K., Greff, K., & Schmidhuber, J. (2015) · arXiv · doi:10.48550/arxiv.1505.00387 · Cited by 912
(2017) · International Conference on Machine Learning · Cited by 758
Ashikhmin, M. (2001) · Proceedings of the 2001 symposium on Interactive 3D graphics · doi:10.1145/364338.364405 · Cited by 737
Scene completion using millions of photographs
Hays, J., & Efros, A. A. (2007) · ACM SIGGRAPH 2007 papers · doi:10.1145/1275808.1276382 · Cited by 687
Li, Y., Liu, S., Yang, J., & Yang, M.-H. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.624 · Cited by 645
Self-citationSummarizing visual data using bidirectional similarity
Simakov, D., Caspi, Y., Shechtman, E., & Irani, M. (2008) · 2008 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2008.4587842 · Cited by 613
Texture optimization for example-based synthesis
Kwatra, V., Essa, I., Bobick, A., & Kwatra, N. (2005) · ACM Transactions on Graphics · doi:10.1145/1073204.1073263 · Cited by 608
Real-time user-guided image colorization with learned deep priors
Zhang, R., Zhu, J.-Y., Isola, P., Geng, X., Lin, A. S., Yu, T., & Efros, A. A. (2017) · ACM Transactions on Graphics · doi:10.1145/3072959.3073703 · Cited by 569
Image melding: combining inconsistent images using patch-based synthesis
Darabi, S., Shechtman, E., Barnes, C., Goldman, D. B., & Sen, P. (2012) · ACM Transactions on Graphics · doi:10.1145/2185520.2185578 · Cited by 550
EdgeConnect: Generative Image Inpainting with Adversarial Edge Learning
Nazeri, K., Ng, E., Joseph, T., Qureshi, F. Z., & Ebrahimi, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.00212 · Cited by 538
Image completion with structure propagation
Sun, J., Yuan, L., Jia, J., & Shum, H.-Y. (2005) · ACM Transactions on Graphics · doi:10.1145/1073204.1073274 · Cited by 526
Zheng, C., Cham, T.-J., & Cai, J. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00153 · Cited by 522
Scribbler: Controlling Deep Image Synthesis with Sketch and Color
Sangkloy, P., Lu, J., Fang, C., Yu, F., & Hays, J. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.723 · Cited by 497
Fragment-based image completion
Drori, I., Cohen-Or, D., & Yeshurun, H. (2003) · ACM SIGGRAPH 2003 Papers · doi:10.1145/1201775.882267 · Cited by 492
Foreground-Aware Image Inpainting
Xiong, W., Yu, J., Lin, Z., Yang, J., Lu, X., Barnes, C., & Luo, J. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00599 · Cited by 402
Self-citationImage completion using planar structure guidance
Huang, J.-B., Kang, S. B., Ahuja, N., & Kopf, J. (2014) · ACM Transactions on Graphics · doi:10.1145/2601097.2601205 · Cited by 322
Wide Activation for Efficient and Accurate Image Super-Resolution
Yu, J., Fan, Y., Yang, J., Xu, N., Wang, Z., Wang, X., & Huang, T. S. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1808.08718 · Cited by 301
Self-citationContextual-Based Image Inpainting: Infer, Match, and Translate
Song, Y., Yang, C., Lin, Z., Liu, X., Huang, Q., Li, H., & Kuo, C.-C. J. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01216-8_1 · Cited by 234
Self-citationKim, D., Woo, S., Lee, J.-Y., & Kweon, I. S. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00594 · Cited by 211
SPG-Net: Segmentation Prediction and Guidance Network for Image InpaintingPreprint
Song, Y., Yang, C., Shen, Y., Wang, P., Huang, Q., & Kuo, C. ‐. J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.03356 · Cited by 203
Gated Convolutional Neural Network for Semantic Segmentation in High-Resolution Images
Wang, H., Wang, Y., Zhang, Q., Xiang, S., & Pan, C. (2017) · Remote Sensing · doi:10.3390/rs9050446 · Cited by 201
Image Completion Approaches Using the Statistics of Similar Patches
He, K., & Sun, J. (2014) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2014.2330611 · Cited by 172
PEPSI : Fast Image Inpainting With Parallel Decoding Network
Sagong, M.-C., Shin, Y.-G., Kim, S., Park, S., & Ko, S.-J. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.01162 · Cited by 162
PEPSI++: Fast and Lightweight Network for Image Inpainting
Shin, Y.-G., Sagong, M.-C., Yeo, Y.-J., Kim, S., & Ko, S.-J. (2020) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2020.2978501 · Cited by 135
Get Out of my Picture! Internet-based Inpainting
Whyte, O., Šivic, J., & Zisserman, A. (2009) · British Machine Vision Conference (BMVC) · doi:10.5244/c.23.116 · Cited by 114
Interactive image completion with perspective correction
Pavić, D., Schönefeld, V., & Kobbelt, L. P. (2006) · The Visual Computer · doi:10.1007/s00371-006-0050-2 · Cited by 68
Guided Image Inpainting: Replacing an Image Region by Pulling Content From Another Image
Zhao, Y., Price, B. L., Cohen, S. D., & Gurari, D. (2019) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv.2019.00166 · Cited by 50
Transformation guided image completion
Huang, J., Kopf, J., Ahuja, N., & Kang, S. B. (2013) · IEEE International Conference on Computational Photography (ICCP) · doi:10.1109/iccphot.2013.6528313 · Cited by 30
Research works printed but not linked: 1
- mount, 1998, ANN Library for Approximate Nearest Neighbour Searching
Cited by
High-Resolution Image Synthesis with Latent Diffusion Models
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01042 · Cited by 20,302
Palette: Image-to-Image Diffusion Models
Saharia, C., Chan, W., Chang, H., Lee, C., Ho, J., Salimans, T., Fleet, D., & Norouzi, M. (2022) · Special Interest Group on Computer Graphics and Interactive Techniques Conference Proceedings · doi:10.1145/3528233.3530757 · Cited by 1,717
RePaint: Inpainting using Denoising Diffusion Probabilistic Models
Lugmayr, A., Danelljan, M., Romero, A., Yu, F., Timofte, R., & Van Gool, L. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01117 · Cited by 1,778
SimMIM: a Simple Framework for Masked Image Modeling
Xie, Z., Zhang, Z., Cao, Y., Lin, Y., Bao, J., Yao, Z., Dai, Q., & Hu, H. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00943 · Cited by 1,255
Resolution-robust Large Mask Inpainting with Fourier Convolutions
Suvorov, R., Logacheva, E., Mashikhin, A., Remizova, A., Ashukha, A., Silvestrov, A., Kong, N., Goka, H., Park, K., & Lempitsky, V. (2022) · 2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv51458.2022.00323 · Cited by 1,123
Metadata
- Authors
- Yu, J., Lin, Z., Yang, J., Shen, X., Lu, X., & Huang, T.
- Venue
- 2019 IEEE/CVF International Conference on Computer Vision (ICCV)
- Publisher
- IEEE
- Published
- 2019-10
- Pages
- 4470-4479
- OpenAlex
- W2807633959
- Cited by
- 1,928
- References
- 54
- FWCI
- 79.11
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.