Image Inpainting for Irregular Holes Using Partial Convolutions
The DOI matches its Crossref record; 42 of the 45 references it cites were found in a registry or an index.
Abstract
Existing deep learning based image inpainting methods use a standard convolutional network over the corrupted image, using convolutional filter responses conditioned on both valid pixels as well as the substitute values in the masked holes (typically the mean value). This often leads to artifacts such as color discrepancy and blurriness. Post-processing is usually used to reduce such artifacts, but are expensive and may fail. We propose the use of partial convolutions, where the convolution is masked and renormalized to be conditioned on only valid pixels. We further include a mechanism to automatically generate an updated mask for the next layer as part of the forward pass. Our model outperforms other methods for irregular masks. We show qualitative and quantitative comparisons with other methods to validate our approach.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“For real videos, we rely on approximate geometry, materials and forces inferred from vision models and heuristics [38,39,40,51,53,54,56].This can introduce discrepancies between the true and modeled dynamics.”
From PhyStruct-Video: Structured Physics State Spaces for Physically Consistent Video Generation (2025) · cites this paper as [38]
Citations
Cites
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148 · cited 2×
“The perceptual loss computes the L 1 distances between both I out and I comp and the ground truth, but after projecting these images into higher level feature spaces using an ImageNet-pretrained VGG-16 [29].”
MethodU-Net: Convolutional Networks for Biomedical Image Segmentation
Ronneberger, O., Fischer, P., & Brox, T. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24574-4_28 · Cited by 99,404 · cited 1×
“We design a UNet-like architecture [23] similar to the one used in [11], replacing all convolutional layers with partial convolutional layers and using nearest neighbor up-sampling in the decoding stage.”
MethodAdam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Image quality assessment: from error visibility to structural similarity
Wang, Z., Bovik, A. C., Sheikh, H. R., & Simoncelli, E. P. (2004) · IEEE Transactions on Image Processing · doi:10.1109/tip.2003.819861 · Cited by 57,892 · cited 3×
“Nevertheless we follow the previous image inpainting works [36,38] by reporting (cid:96) 1 error, PSNR, SSIM [35], and the inception score [27].”
MethodImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380 · cited 9×
“ImageNet and Places2 models train for 10 days, whereas CelebA-HQ trains in 3 days.”
BackgroundMethodInfluentialImage-to-Image Translation with Conditional Adversarial Networks
Isola, P., Zhu, J.-Y., Zhou, T., & Efros, A. A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.632 · Cited by 22,930 · cited 1×
“We design a UNet-like architecture [23] similar to the one used in [11], replacing all convolutional layers with partial convolutional layers and using nearest neighbor up-sampling in the decoding stage.”
MethodDelving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification
He, K., Zhang, X., Ren, S., & Sun, J. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.123 · Cited by 19,741 · cited 1×
“We initialize the weights using the initialization method described in [9] and use Adam [14] for optimization.”
Method(2017) · Cited by 11,625
Perceptual Losses for Real-Time Style Transfer and Super-Resolution
Johnson, J., Alahi, A., & Fei-Fei, L. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46475-6_43 · Cited by 11,283 · cited 3×
“Perceptual loss [12] is known to generate checkerboard artifacts.”
BackgroundInfluentialDeep Learning Face Attributes in the Wild
Liu, Z., Luo, P., Wang, X., & Tang, X. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.425 · Cited by 7,747 · cited 6×
“ImageNet and Places2 models train for 10 days, whereas CelebA-HQ trains in 3 days.”
MethodInfluentialDeformable Convolutional Networks
Dai, J., Qi, H., Xiong, Y., Li, Y., Zhang, G., Hu, H., & Wei, Y. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.89 · Cited by 7,513
Improved Techniques for Training GANs
Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., & Chen, X. (2016) · arXiv · doi:10.48550/arxiv.1606.03498 · Cited by 6,965 · cited 1×
“Nevertheless we follow the previous image inpainting works [34, 36] by reporting l1 error, PSNR, SSIM [33], and the inception score [25].”
MethodContext Encoders: Feature Learning by Inpainting
Pathak, D., Krahenbuhl, P., Donahue, J., Darrell, T., & Efros, A. A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.278 · Cited by 5,691 · cited 4×
“Content Encoders [22] first embed the 128 × 128 image with 64 × 64 center hole into low dimensional feature space and then decode the feature to a 64x64 image.”
InfluentialProgressive Growing of GANs for Improved Quality, Stability, and Variation
Karras, T., Aila, T., Laine, S., & Lehtinen, J. (2017) · arXiv · doi:10.48550/arxiv.1710.10196 · Cited by 4,727
Places: A 10 Million Image Database for Scene Recognition
Zhou, B., Lapedriza, A., Khosla, A., Oliva, A., & Torralba, A. (2018) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2017.2723009 · Cited by 4,042 · cited 1×
“Training Data We use 3 separate image datasets for training and testing: ImageNet dataset [24], Places2 dataset [37] and CelebA-HQ [17, 13].”
MethodBertalmio, M., Sapiro, G., Caselles, V., & Ballester, C. (2000) · Proceedings of the 27th annual conference on Computer graphics and interactive techniques - SIGGRAPH '00 · doi:10.1145/344779.344972 · Cited by 3,580 · cited 1×
“Non-learning approaches to image inpainting rely on propagating appearance information from neighboring pixels to the target region using some mechanisms like distance field[3, 1, 30].”
BackgroundPatchMatch: a randomized correspondence algorithm for structural image editing
Barnes, C., Shechtman, E., Finkelstein, A., & Goldman, D. B. (2009) · ACM Transactions on Graphics · doi:10.1145/1531326.1531330 · Cited by 2,880
A Neural Algorithm of Artistic Style
Gatys, L., Ecker, A., & Bethge, M. (2016) · Journal of Vision · doi:10.1167/16.12.326 · Cited by 2,612 · cited 1×
“We further include the style-loss term, which is similar to the perceptual loss [6], but we first perform an autocorrelation (Gram matrix) on each feature map before applying the L(1).”
MethodGenerative Image Inpainting with Contextual Attention
Yu, J., Lin, Z., Yang, J., Shen, X., Lu, X., & Huang, T. S. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00577 · Cited by 2,527 · cited 11×
“[36] remove the centered hole assumption and can handle irregular shaped holes, but do not perform an extensive quantitative analysis on a large number of images with irregular masks (51 test images in [8]).”
BackgroundMethodResultInfluentialImage quilting for texture synthesis and transfer
Efros, A. A., & Freeman, W. T. (2001) · Proceedings of the 28th annual conference on Computer graphics and interactive techniques · doi:10.1145/383259.383296 · Cited by 2,347 · cited 1×
“Patch-based methods such as [5, 15] operate by searching for relevant patches from the image’s non-hole regions or other source images in an iterative fashion.”
MethodGlobally and locally consistent image completion
Iizuka, S., Simo-Serra, E., & Ishikawa, H. (2017) · ACM Transactions on Graphics · doi:10.1145/3072959.3073659 · Cited by 2,203 · cited 11×
“2(c): inpainted result of Iizuka et al.[10].”
BackgroundMethodInfluentialPérez, P., Gangnet, M., & Blake, A. (2003) · ACM SIGGRAPH 2003 Papers · doi:10.1145/1201775.882269 · Cited by 2,020 · cited 1×
“[10] uses fast marching [30] and Poisson image blending [21], while Yu et al.”
MethodConditional Image Generation with PixelCNN Decoders
van den Oord, A., Kalchbrenner, N., Vinyals, O., Espeholt, L., Graves, A., & Kavukcuoglu, K. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.05328 · Cited by 1,833 · cited 2×
“It has also been used for full-image generation in PixelCNN [20], to condition the next pixel only on previously synthesized pixels.”
MethodPérez, P., Gangnet, M., & Blake, A. (2003) · ACM Transactions on Graphics · doi:10.1145/882262.882269 · Cited by 1,773
Conditional Image Generation with PixelCNN DecodersPreprint
van den Oord, A., Kalchbrenner, N., Vinyals, O., et al. (2016) · arXiv (Cornell University) · Cited by 1,586
An Image Inpainting Technique Based on the Fast Marching Method
Telea, A. (2004) · Journal of Graphics Tools · doi:10.1080/10867651.2004.10487596 · Cited by 1,421 · cited 3×
“Non-learning approaches to image inpainting rely on propagating appearance information from neighboring pixels to the target region using some mechanisms like distance field[3, 1, 30].”
BackgroundMethodSemantic Image Inpainting with Deep Generative Models
Yeh, R. A., Chen, C., Lim, T. Y., Schwing, A. G., Hasegawa-Johnson, M., & Do, M. N. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.728 · Cited by 1,206
Filling-in by joint interpolation of vector fields and gray levels
Ballester, C., Bertalmio, M., Caselles, V., Sapiro, G., & Verdera, J. (2001) · IEEE Transactions on Image Processing · doi:10.1109/83.935036 · Cited by 962 · cited 1×
“Non-learning approaches to image inpainting rely on propagating appearance information from neighboring pixels to the target region using some mechanisms like distance field[3, 1, 30].”
BackgroundUhrig, J., Schneider, N., Schneider, L., Franke, U., Brox, T., & Geiger, A. (2017) · 2017 International Conference on 3D Vision (3DV) · doi:10.1109/3dv.2017.00012 · Cited by 924 · cited 1×
“[31] proposed sparsity invariant CNNs with reweighted convolution and max pooling based mask updating mechanism for depth completion.”
BackgroundHigh-Resolution Image Inpainting Using Multi-scale Neural Patch Synthesis
Yang, C., Lu, X., Lin, Z., Shechtman, E., Wang, O., & Li, H. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.434 · Cited by 877 · cited 3×
“[34] takes the result from Content Encoders as input and then propagates the texture information from non-hole regions to fill the hole regions as postprocessing.”
BackgroundMethodScene completion using millions of photographs
Hays, J., & Efros, A. A. (2007) · ACM SIGGRAPH 2007 papers · doi:10.1145/1275808.1276382 · Cited by 687
Li, Y., Liu, S., Yang, J., & Yang, M.-H. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.624 · Cited by 645 · cited 1×
“Li et al. [17] and Iizuka et al. [10] extended Content Encoders by defining both global and local discriminators; then Iizuka et al. [10] apply Poisson blending as a post-process.”
BackgroundSummarizing visual data using bidirectional similarity
Simakov, D., Caspi, Y., Shechtman, E., & Irani, M. (2008) · 2008 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2008.4587842 · Cited by 613 · cited 1×
“However, these steps often come at a large computation cost such as in [26].”
BackgroundTexture optimization for example-based synthesis
Kwatra, V., Essa, I., Bobick, A., & Kwatra, N. (2005) · ACM Transactions on Graphics · doi:10.1145/1073204.1073263 · Cited by 608
Scene completion using millions of photographs
Hays, J., & Efros, A. A. (2007) · ACM Transactions on Graphics · doi:10.1145/1276377.1276382 · Cited by 534
Deformable Convolutional NetworksPreprint
Dai, J., Qi, H., Xiong, Y., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.06211 · Cited by 428 · cited 1×
“Discussions of other CNN variants like [4] are beyond the scope of this work.”
BackgroundDense Point Trajectories by GPU-Accelerated Large Displacement Optical Flow
Sundaram, N., Brox, T., & Keutzer, K. (2010) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-15549-9_32 · Cited by 422 · cited 1×
“We found the results of occlusion/dis-occlusion mask estimation method between two consecutive frames for videos described in [29] to be a good source of such patterns.”
Semantic Image Inpainting with Perceptual and Contextual Losses.Preprint
Yeh, R. A., Chen, C., Lim, T.-Y., Hasegawa‐Johnson, M., & Do, M. N. (2016) · arXiv (Cornell University) · Cited by 319
Shepard convolutional neural networks
Ren, J., Xu, L., Yan, Q., & Sun, W. (2015) · neural information processing systems · Cited by 201
Segmentation-Aware Convolutional Networks Using Local Attention Masks
Harley, A. W., Derpanis, K. G., & Kokkinos, I. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.539 · Cited by 142 · cited 3×
“The partial convolution at every location, similarly defined in [7], is expressed as:”
BackgroundMethodInfluentialContextual-based Image Inpainting: Infer, Match, and TranslatePreprint
Song, Y., Yang, C., Lin, Z., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.08590 · Cited by 8
Ulyanov, D., Vedaldi, A., & Lempitsky, V. (2017) · arXiv · doi:10.48550/arxiv.1711.10925
Cited by
Free-Form Image Inpainting With Gated Convolution
Yu, J., Lin, Z., Yang, J., Shen, X., Lu, X., & Huang, T. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00457 · Cited by 1,928
“Next, we compare our model with previous state-of-theart methods [15, 23, 49].”
BackgroundMethodResultInfluentialPalette: Image-to-Image Diffusion Models
Saharia, C., Chan, W., Chang, H., Lee, C., Ho, J., Salimans, T., Fleet, D., & Norouzi, M. (2022) · Special Interest Group on Computer Graphics and Interactive Techniques Conference Proceedings · doi:10.1145/3528233.3530757 · Cited by 1,717
“While there are other important prior works on image inpainting, such as [Liu et al. 2018a, 2020; Zheng et al. 2019], we were not able to compare with all of them.”
BackgroundMethodResultInfluentialRePaint: Inpainting using Denoising Diffusion Probabilistic Models
Lugmayr, A., Danelljan, M., Romero, A., Yu, F., Timofte, R., & Van Gool, L. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01117 · Cited by 1,778
“As image inpainting requires a high-level semantic context, and to explicitly include it in the generation pipeline, there exist hand-crafted architectural designs such as Dilated Convolutions [16, 44] to increase the receptive field, Partial Convolutions [19] and Gated Convolutions [47] to guide the convolution kerne…”
MethodMaskGAN: Towards Diverse and Interactive Facial Image Manipulation
Lee, C.-H., Liu, Z., Wu, L., & Luo, P. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00559 · Cited by 1,130
Resolution-robust Large Mask Inpainting with Fourier Convolutions
Suvorov, R., Logacheva, E., Mashikhin, A., Remizova, A., Ashukha, A., Silvestrov, A., Kong, N., Goka, H., Park, K., & Lempitsky, V. (2022) · 2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv51458.2022.00323 · Cited by 1,123
“The subject has been studied in pre-deep learning era [1, 5, 13], and the progress accelerated in recent years through the use of deep and wide neural networks [26, 30, 25] and adversarial learning [34, 18, 56, 44, 57, 32, 54, 61].”
BackgroundMethodInfluential
Metadata
- Authors
- Liu, G., Reda, F. A., Shih, K. J., Wang, T.-C., Tao, A., & Catanzaro, B.
- Affiliations
- Nvidia (United States) (US)
- Corresponding author
- Liu, Guilin
- Venue
- Lecture Notes in Computer Science
- Publisher
- Springer International Publishing
- Published
- 2018
- Pages
- 89-105
- ISSN
- 0302-9743 (ISSN-L)
- 1611-3349
- OpenAlex
- W2798365772
- Cited by
- 2,419
- References
- 45
- Citations by year
- 2026: 178 · 2025: 279 · 2024: 322 · 2023: 408 · 2022: 353 · 2021: 379 · 2020: 304 · 2019: 172 · 2018: 24
- FWCI
- 66.24
- Type
- conference paper
- Language
- en
- Open access
- closed
- Licence
- www.springer.com/tdm (tdm)
- www.springer.com/tdm (vor)
- Locations
- Lecture notes in computer science (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Image Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- image inpainting, partial convolution
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.