Image Inpainting With Local and Global Refinement
The DOI matches its Crossref record; 55 of the 55 references it cites were found in a registry or an index.
Abstract
Image inpainting has made remarkable progress with recent advances in deep learning. Popular networks mainly follow an encoder-decoder architecture (sometimes with skip connections) and possess sufficiently large receptive field, i.e., larger than the image resolution. The receptive field refers to the set of input pixels that are path-connected to a neuron. For image inpainting task, however, the size of surrounding areas needed to repair different kinds of missing regions are different, and the very large receptive field is not always optimal, especially for the local structures and textures. In addition, a large receptive field tends to involve more undesired completion results, which will disturb the inpainting process. Based on these insights, we rethink the process of image inpainting from a different perspective of receptive field, and propose a novel three-stage inpainting framework with local and global refinement. Specifically, we first utilize an encoder-decoder network with skip connection to achieve coarse initial results. Then, we introduce a shallow deep model with small receptive field to conduct the local refinement, which can also weaken the influence of distant undesired completion results. Finally, we propose an attention-based encoder-decoder network with large receptive field to conduct the global refinement. Experimental results demonstrate that our method outperforms the state of the arts on three popular publicly available datasets for image inpainting. Our local and global refinement network can be directly inserted into the end of any existing networks to further improve their inpainting performance. Code is available at https://github.com/weizequan/LGNet.git.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 1×
“Zhang et al. [25] divided the image inpainting process into four different phases, and used an LSTM (long short-term memory) architecture [46] to control the information flow of the progressive process.”
MethodU-Net: Convolutional Networks for Biomedical Image Segmentation
Ronneberger, O., Fischer, P., & Brox, T. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24574-4_28 · Cited by 99,404
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · cited 1×
“Similar to many previous works [14], [16], [17], [28], the perceptual loss [50] and style loss [51] defined on the VGG-16 [52] (pre-trained on ImageNet [53]) are also applied to better recover the structural and textual information.”
Image quality assessment: from error visibility to structural similarity
Wang, Z., Bovik, A. C., Sheikh, H. R., & Simoncelli, E. P. (2004) · IEEE Transactions on Image Processing · doi:10.1109/tip.2003.819861 · Cited by 57,892 · cited 1×
“For the evaluation metrics, we adopt several common metrics in the image inpainting task: (cid:96) 1 error, PSNR (peak signal-to-noise ratio), SSIM (the structural similarity index) [58], FID (Fr´echet inception distance) [59], and LPIPS (learned perceptual image patch similarity) [60].”
MethodThe unreasonable effectiveness of deep features as a perceptual metric
Zhang, R., Isola, P., Efros, A. A., Shechtman, E., & Wang, O. (2018) · arXiv · doi:10.48550/arxiv.1801.03924 · Cited by 24,812
Self-citationReducing the dimensionality of data with neural networks
Hinton, G. E., & Salakhutdinov, R. R. (2006) · Science · doi:10.1126/science.1127647 · Cited by 20,928
GAN(Generative Adversarial Nets)
(2017) · Journal of Japan Society for Fuzzy Theory and Intelligent Informatics · doi:10.3156/jsoft.29.5_177_2 · Cited by 19,270
The Unreasonable Effectiveness of Deep Features as a Perceptual Metric
Zhang, R., Isola, P., Efros, A. A., Shechtman, E., & Wang, O. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00068 · Cited by 14,623
Self-citationPerceptual Losses for Real-Time Style Transfer and Super-Resolution
Johnson, J., Alahi, A., & Fei-Fei, L. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46475-6_43 · Cited by 11,283 · cited 1×
“Similar to many previous works [14], [16], [17], [28], the perceptual loss [50] and style loss [51] defined on the VGG-16 [52] (pre-trained on ImageNet [53]) are also applied to better recover the structural and textual information.”
GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., & Hochreiter, S. (2017) · arXiv · doi:10.48550/arxiv.1706.08500 · Cited by 8,365
Deep Learning Face Attributes in the Wild
Liu, Z., Luo, P., Wang, X., & Tang, X. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.425 · Cited by 7,747
Image Style Transfer Using Convolutional Neural Networks
Gatys, L. A., Ecker, A. S., & Bethge, M. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.265 · Cited by 6,025 · cited 1×
“Similar to many previous works [14], [16], [17], [28], the perceptual loss [50] and style loss [51] defined on the VGG-16 [52] (pre-trained on ImageNet [53]) are also applied to better recover the structural and textual information.”
Context Encoders: Feature Learning by Inpainting
Pathak, D., Krahenbuhl, P., Donahue, J., Darrell, T., & Efros, A. A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.278 · Cited by 5,691 · cited 2×
“From the perspective of network design, these inpainting methods can be roughly categorized into one-stage networks (one generator) [12]–[18], two-stage networks (two generators) [19]–[24], and progressive networks (one or multiple generators applied in an iterative manner) [25]–[29].”
Background“Least squares generative adversarial networks,”
Mao, X., Li, Q., Xie, H., Lau, R. Y. K., Wang, Z., & Smolley, S. P. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.304 · Cited by 5,437 · cited 1×
“For GAN loss, we use the least square loss [49], the corresponding loss functions for the coarse inpainting network and the discriminator are defined as: where I is the merged image.”
MethodProgressive Growing of GANs for Improved Quality, Stability, and Variation
Karras, T., Aila, T., Laine, S., & Lehtinen, J. (2017) · arXiv · doi:10.48550/arxiv.1710.10196 · Cited by 4,727
Learning Deconvolution Network for Semantic Segmentation
Noh, H., Hong, S., & Han, B. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.178 · Cited by 4,422 · cited 1×
“(3) The receptive field is an important aspect of deep neural networks, which has gained more attention in image classification and semantic segmentation [32], [33].”
BackgroundPlaces: A 10 Million Image Database for Scene Recognition
Zhou, B., Lapedriza, A., Khosla, A., Oliva, A., & Torralba, A. (2018) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2017.2723009 · Cited by 4,042 · cited 3×
“• Places2 dataset [35]: A large-scale scene recognition dataset.”
BackgroundMethodBertalmio, M., Sapiro, G., Caselles, V., & Ballester, C. (2000) · Proceedings of the 27th annual conference on Computer graphics and interactive techniques - SIGGRAPH '00 · doi:10.1145/344779.344972 · Cited by 3,580 · cited 3×
“The use of diffusion for image inpainting was pioneered by Bertalmio et al. [1], specifically, the anisotropic diffusion was iteratively applied without losing sharpness in the reconstruction process.”
BackgroundMethodPatchMatch: a randomized correspondence algorithm for structural image editing
Barnes, C., Shechtman, E., Finkelstein, A., & Goldman, D. B. (2009) · ACM Transactions on Graphics · doi:10.1145/1531326.1531330 · Cited by 2,880
Generative Image Inpainting with Contextual Attention
Yu, J., Lin, Z., Yang, J., Shen, X., Lu, X., & Huang, T. S. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00577 · Cited by 2,527
Image Inpainting for Irregular Holes Using Partial Convolutions
Liu, G., Reda, F. A., Shih, K. J., Wang, T.-C., Tao, A., & Catanzaro, B. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01252-6_6 · Cited by 2,419 · cited 9×
“Liu et al. [14] designed a partial convolution operation followed by an automatic mask-update step to better fill irregular holes.”
BackgroundMethodInfluentialGlobally and locally consistent image completion
Iizuka, S., Simo-Serra, E., & Ishikawa, H. (2017) · ACM Transactions on Graphics · doi:10.1145/3072959.3073659 · Cited by 2,203 · cited 1×
“To improve the consistency of image completion, Iizuka et al. [13] introduced the global and local context discriminators to train the fully-convolutional completion network.”
BackgroundFree-Form Image Inpainting With Gated Convolution
Yu, J., Lin, Z., Yang, J., Shen, X., Lu, X., & Huang, T. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00457 · Cited by 1,928 · cited 5×
“Inspired by [14], Yu et al. [20] subsequently upgraded the previous work [19] by introducing gated convolution and a patch-based GAN discriminator.”
MethodInfluentialUnderstanding the Effective Receptive Field in Deep Convolutional Neural Networks
Luo, W., Li, Y., Urtasun, R., & Zemel, R. (2017) · arXiv · doi:10.48550/arxiv.1701.04128 · Cited by 1,575
Filling-in by joint interpolation of vector fields and gray levels
Ballester, C., Bertalmio, M., Caselles, V., Sapiro, G., & Verdera, J. (2001) · IEEE Transactions on Image Processing · doi:10.1109/83.935036 · Cited by 962 · cited 1×
“Based on the joint interpolation of the image gray levels and the gradient directions, Ballester et al. [2] formulated the image inpainting as a variational problem.”
MethodVector-valued image regularization with PDEs: a common framework for different applications
Tschumperle, D., & Deriche, R. (2005) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2005.87 · Cited by 671 · cited 3×
“Tschumperl´e and Deriche [3] proposed a single generic anisotropic diffusion equation and obtained the better inpainting results.”
BackgroundMethodWhat makes Paris look like Paris?
Doersch, C., Singh, S., Gupta, A., Sivic, J., & Efros, A. A. (2012) · ACM Transactions on Graphics · doi:10.1145/2185520.2185597 · Cited by 663 · cited 3×
“• Paris StreetView dataset [36]: This dataset consists of street-level imagery.”
BackgroundMethodImage Inpainting : Overview and Recent Advances
Guillemot, C., & Le Meur, O. (2014) · IEEE Signal Processing Magazine · doi:10.1109/msp.2013.2273004 · Cited by 608 · cited 1×
“We refer readers to the surveys [37], [38] for more details of the traditional approaches.”
MethodEdgeConnect: Structure Guided Image Inpainting using Edge Prediction
Nazeri, K., Ng, E., Joseph, T., Qureshi, F., & Ebrahimi, M. (2019) · 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW) · doi:10.1109/iccvw.2019.00408 · Cited by 562 · cited 1×
“Nazeri et al. [21] proposed an edge-guided two-stage image inpainting method.”
Image melding: combining inconsistent images using patch-based synthesis
Darabi, S., Shechtman, E., Barnes, C., Goldman, D. B., & Sen, P. (2012) · ACM Transactions on Graphics · doi:10.1145/2185520.2185578 · Cited by 550
Shift-Net: Image Inpainting via Deep Feature Rearrangement
Yan, Z., Li, X., Li, M., Zuo, W., & Shan, S. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01264-9_1 · Cited by 511
Learning Pyramid-Context Encoder Network for High-Quality Image Inpainting
Zeng, Y., Fu, J., Chao, H., & Guo, B. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00158 · Cited by 500 · cited 5×
“The training strategy also adopts their original setting [15], [17].”
BackgroundMethodInfluentialRecurrent Feature Reasoning for Image Inpainting
Li, J., Wang, N., Zhang, L., Du, B., & Tao, D. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00778 · Cited by 484 · cited 9×
“In this work, we use the simple self-attention method [19], [28], and more advanced attention method can also be used in our framework.”
BackgroundMethodInfluentialDense Point Trajectories by GPU-Accelerated Large Displacement Optical Flow
Sundaram, N., Brox, T., & Keutzer, K. (2010) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-15549-9_32 · Cited by 422 · cited 1×
“In [14], these masks are generated via random dilation, rotation, and cropping from the raw masks obtained by occlusion/dis-occlusion mask estimation method [56] between two consecutive frames of videos.”
MethodStructureFlow: Image Inpainting via Structure-Aware Appearance Flow
Ren, Y., Yu, X., Zhang, R., Li, T. H., Liu, S., & Li, G. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00027 · Cited by 396
Self-citationCoherent Semantic Attention for Image Inpainting
Liu, H., Jiang, B., Xiao, Y., & Yang, C. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00427 · Cited by 383 · cited 1×
“In literature, several aspects have gained more attentions, e.g. , simultaneous or step-wise structure and texture inpainting [22], different attention strategies for context information collection [19], [30], progressive filling from border to center [28], and so on.”
BackgroundContextual Residual Aggregation for Ultra High-Resolution Image Inpainting
Yi, Z., Tang, Q., Azizi, S., Jang, D., & Xu, Z. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00753 · Cited by 374
Elharrouss, O., Almaadeed, N., Al-Maadeed, S., & Akbari, Y. (2020) · Neural Processing Letters · doi:10.1007/s11063-019-10163-0 · Cited by 341 · cited 1×
“We refer readers to the surveys [37], [38] for more details of the traditional approaches.”
MethodImage completion using planar structure guidance
Huang, J.-B., Kang, S. B., Ahuja, N., & Kopf, J. (2014) · ACM Transactions on Graphics · doi:10.1145/2601097.2601205 · Cited by 322 · cited 1×
“Through using mid-level structural cues, Huang et al. [6] proposed an automatic completion algorithm based on augmented patch-based matching.”
MethodRethinking Image Inpainting via a Mutual Encoder-Decoder with Feature Equalizations
Liu, H., Jiang, B., Song, Y., Huang, W., & Yang, C. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58536-5_43 · Cited by 290
Image Inpainting With Learnable Bidirectional Attention Maps
Xie, C., Liu, S., Li, C., Cheng, M.-M., Zuo, W., Liu, X., Wen, S., & Ding, E. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00895 · Cited by 282 · cited 2×
“Similar to many previous works [14], [16], [17], [28], the perceptual loss [50] and style loss [51] defined on the VGG-16 [52] (pre-trained on ImageNet [53]) are also applied to better recover the structural and textual information.”
BackgroundContextual-Based Image Inpainting: Infer, Match, and Translate
Song, Y., Yang, C., Lin, Z., Liu, X., Huang, Q., Li, H., & Kuo, C.-C. J. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01216-8_1 · Cited by 234 · cited 1×
“Inspired by several existing methods [19], [39], [40] that use attention mechanism to fill in the missing regions, Zeng et al. [15] proposed a pyramid-context encoder network for image completion via attention transfer.”
MethodProgressive Reconstruction of Visual Structure for Image Inpainting
Li, J., He, F., Zhang, L., Du, B., & Tao, D. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00606 · Cited by 182 · cited 1×
“Li et al. [27] progressively reconstructed the visual structure to entangle the visual feature reconstruction for image inpainting.”
BackgroundImage Inpainting Using Nonlocal Texture Matching and Nonlinear Filtering
Ding, D., Ram, S., & Rodriguez, J. J. (2019) · IEEE Transactions on Image Processing · doi:10.1109/tip.2018.2880681 · Cited by 169 · cited 3×
“Early works can be classified into two categories: diffusion-based approaches [1]–[3] and patch-based approaches [4]– [8].”
BackgroundMethodImage Inpainting by End-to-End Cascaded Refinement With Mask Awareness
Zhu, M., He, D., Li, X., Li, C., Li, F., Liu, X., Ding, E., & Zhang, Z. (2021) · IEEE Transactions on Image Processing · doi:10.1109/tip.2021.3076310 · Cited by 168 · cited 5×
“Similarly, Zhu et al. [44] proposed a mask-aware convolution and point-wise normalization for image inpainting, standing from the dynamic concept as well.”
BackgroundMethodInfluentialDynamic Selection Network for Image Inpainting
Wang, N., Zhang, Y., & Zhang, L. (2021) · IEEE Transactions on Image Processing · doi:10.1109/tip.2020.3048629 · Cited by 165 · cited 1×
“Wang et al. [43] proposed the validness migratable convolution and regional composite normalization modules to better utilize the valid pixels during the inpainting process.”
Self-citationGuidance and Evaluation: Semantic-Aware Image Inpainting for Mixed Scenes
Liao, L., Xiao, J., Wang, Z., Lin, C., & Satoh, S. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58583-9_41 · Cited by 164
High-Resolution Image Inpainting with Iterative Confidence Feedback and Guided Upsampling
Zeng, Y., Lin, Z., Yang, J., Zhang, J., Shechtman, E., & Lu, H. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58529-7_1 · Cited by 160
Semantic Image Inpainting with Progressive Generative Networks
Zhang, H., Hu, Z., Luo, C., Zuo, W., & Wang, M. (2018) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3240508.3240625 · Cited by 156 · cited 2×
“From the perspective of network design, these inpainting methods can be roughly categorized into one-stage networks (one generator) [12]–[18], two-stage networks (two generators) [19]–[24], and progressive networks (one or multiple generators applied in an iterative manner) [25]–[29].”
BackgroundMethodProgressive Image Inpainting with Full-Resolution Residual Network
Guo, Z., Chen, Z., Yu, T., Chen, J., & Liu, S. (2019) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3343031.3351022 · Cited by 139 · cited 1×
“To address this limitation, Guo et al. [26] proposed full-resolution residual networks with several dilation modules.”
BackgroundExemplar-based Inpainting: Technical Review and new Heuristics for better Geometric Reconstructions
Buyssens, P., Daisy, M., Tschumperlé, D., & Lézoray, O. (2015) · IEEE Transactions on Image Processing · doi:10.1109/tip.2015.2411437 · Cited by 110 · cited 1×
“Moreover, patch-based method is also often used in exemplar-based inpainting [5], [7].”
MethodVCNet: A Robust Approach to Blind Image Inpainting
Wang, Y., Chen, Y.-C., Tao, X., & Jia, J. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58595-2_45 · Cited by 110
Scene text removal via cascaded text stroke detection and erasing
Bian, X., Wang, C., Quan, W., Ye, J., Zhang, X., & Yan, D.-M. (2021) · Computational Visual Media · doi:10.1007/s41095-021-0242-8 · Cited by 40 · cited 1×
“We conduct the experiments on the recent real-world text erasing datasets [62], including 11,040 training samples and 1,080 testing samples.”
MethodSelf-citationPixel‐wise Dense Detector for Image Inpainting
Zhang, R., Quan, W., Wu, B., Li, Z., & Yan, D.-M. (2020) · Computer Graphics Forum · doi:10.1111/cgf.14160 · Cited by 24 · cited 3×
“From the perspective of network design, these inpainting methods can be roughly categorized into one-stage networks (one generator) [12]–[18], two-stage networks (two generators) [19]–[24], and progressive networks (one or multiple generators applied in an iterative manner) [25]–[29].”
BackgroundMethodSelf-citation
Cited by
Self-supervised Learning of Adversarial Example: Towards Good Generalizations for Deepfake Detection
Chen, L., Zhang, Y., Song, Y., Liu, L., & Wang, J. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01815 · Cited by 307
“The realistic image generation brought by the generative adversarial network (GAN) raises a security issue that human portraits can be easily substituted to provide malicious bioinformatics [11, 45, 15, 44, 51, 56, 40, 54, 48].”
BackgroundSelf-citationDual degradation image inpainting method via adaptive feature fusion and U-net network
Chen, Y., Xia, R., Yang, K., & Zou, K. (2025) · Applied Soft Computing · doi:10.1016/j.asoc.2025.113010 · Cited by 85
Metadata
- Authors
- Quan, W., Zhang, R., Zhang, Y., Li, Z., Wang, J., & Yan, D.-M.
- Venue
- IEEE Transactions on Image Processing
- Published
- 2022
- Volume
- 31
- Pages
- 2405-2420
- ISSN
- 1057-7149 (ISSN-L)
- 1941-0042
- OpenAlex
- W4225592316
- Cited by
- 197
- References
- 55
- FWCI
- 13.52
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.