Globally and locally consistent image completion
Abstract
We present a novel approach for image completion that results in images that are both locally and globally consistent. With a fully-convolutional neural network, we can complete images of arbitrary resolutions by filling-in missing regions of any shape. To train this image completion network to be consistent, we use global and local context discriminators that are trained to distinguish real images from completed ones. The global discriminator looks at the entire image to assess if it is coherent as a whole, while the local discriminator looks only at a small area centered at the completed region to ensure the local consistency of the generated patches. The image completion network is then trained to fool the both context discriminator networks, which requires it to generate images that are indistinguishable from real ones with regard to overall consistency as well as in details. We show that our approach can be used to complete a wide variety of scenes. Furthermore, in contrast with the patch-based approaches such as PatchMatch, our approach can generate fragments that do not appear elsewhere in the image, which allows us to naturally complete the images of objects with familiar and highly specific structures, such as faces.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Cites
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · see citing works
Fully convolutional networks for semantic segmentation
Long, J., Shelhamer, E., & Darrell, T. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298965 · Cited by 38,834 · see citing works
Learning representations by back-propagating errors
Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986) · Nature · doi:10.1038/323533a0 · Cited by 31,892 · see citing works
Image-to-Image Translation with Conditional Adversarial Networks
Isola, P., Zhu, J.-Y., Zhou, T., & Efros, A. A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.632 · Cited by 22,930 · see citing works
GAN（Generative Adversarial Nets）
淳司, 柴. (2017) · Journal of Japan Society for Fuzzy Theory and Intelligent Informatics · doi:10.3156/jsoft.29.5_177_2 · Cited by 19,268 · see citing works
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate ShiftPreprint
Ioffe, S. (2024) · arXiv (Cornell University) · doi:10.57702/o9raffed · Cited by 15,273 · see citing works
(2010) · International Conference on Machine Learning · Cited by 13,995 · see citing works
Backpropagation Applied to Handwritten Zip Code Recognition
LeCun, Y., Boser, B., Denker, J. S., Henderson, D., Howard, R. E., Hubbard, W., & Jackel, L. D. (1989) · Neural Computation · doi:10.1162/neco.1989.1.4.541 · Cited by 11,891 · see citing works
Deep Learning Face Attributes in the Wild
Liu, Z., Luo, P., Wang, X., & Tang, X. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.425 · Cited by 7,747 · see citing works
Improved Techniques for Training GANs
Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., & Chen, X. (2016) · arXiv · doi:10.48550/arxiv.1606.03498 · Cited by 6,965 · see citing works
Context Encoders: Feature Learning by Inpainting
Pathak, D., Krahenbuhl, P., Donahue, J., Darrell, T., & Efros, A. A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.278 · Cited by 5,691 · see citing works
ADADELTA: An Adaptive Learning Rate Method
Zeiler, M. D. (2012) · arXiv · doi:10.48550/arxiv.1212.5701 · Cited by 5,322 · see citing works
Bertalmı́o, M., Sapiro, G., Caselles, V., & Ballester, C. (2000) · ACM SIGGRAPH Conference (SIGGRAPH) · doi:10.1145/344779.344972 · Cited by 3,747 · see citing works
Region Filling and Object Removal by Exemplar-Based Image Inpainting
Criminisi, A., Perez, P., & Toyama, K. (2004) · IEEE Transactions on Image Processing · doi:10.1109/tip.2004.833105 · Cited by 3,258 · cited 3× · see citing works
“Approaches for automatic estimation of the scene structure have also been proposed: utilizing the tensor-voting algorithm to smoothly connect curves across holes [Jia and Tang 2003]; exploiting structure-based priority for patch ordering [Criminisi et al. 2004], tile-based search space constraints [Kopf et al.”
MethodTexture synthesis by non-parametric sampling
Efros, A. A., & Leung, T. K. (1999) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.1999.790383 · Cited by 3,058 · see citing works
Barnes, C., Shechtman, E., Finkelstein, A., & Goldman, D. B. (2009) · ACM Transactions on Graphics · doi:10.1145/1531326.1531330 · Cited by 2,888 · see citing works
Image quilting for texture synthesis and transfer
Efros, A. A., & Freeman, W. T. (2001) · ACM SIGGRAPH Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH) · doi:10.1145/383259.383296 · Cited by 2,426 · see citing works
Pérez, P., Gangnet, M., & Blake, A. (2003) · ACM SIGGRAPH Conference (SIGGRAPH) · doi:10.1145/1201775.882269 · Cited by 2,041 · see citing works
Pérez, P., Gangnet, M., & Blake, A. (2003) · ACM Transactions on Graphics · doi:10.1145/882262.882269 · Cited by 1,769 · see citing works
An Image Inpainting Technique Based on the Fast Marching Method
Telea, A. (2004) · Journal of Graphics Tools · doi:10.1080/10867651.2004.10487596 · Cited by 1,421 · see citing works
Kwatra, V., Schödl, A., Essa, I. A., Turk, G., & Bobick, A. (2003) · ACM SIGGRAPH Conference (SIGGRAPH) · doi:10.1145/1201775.882264 · Cited by 1,216 · see citing works
Neocognitron: A hierarchical neural network capable of visual pattern recognition
Fukushima, K. (1988) · Neural Networks · doi:10.1016/0893-6080(88)90014-7 · Cited by 1,097 · see citing works
Simultaneous structure and texture image inpainting
Bertalmı́o, M., Vese, L. A., Sapiro, G., & Osher, S. (2003) · IEEE Transactions on Image Processing · doi:10.1109/tip.2003.815261 · Cited by 1,083 · see citing works
Filling-in by joint interpolation of vector fields and gray levels
Ballester, C., Bertalmı́o, M., Caselles, V., Sapiro, G., & Verdera, J. (2001) · IEEE Transactions on Image Processing · doi:10.1109/83.935036 · Cited by 962 · see citing works
High-Resolution Image Inpainting Using Multi-scale Neural Patch Synthesis
Yang, C., Lu, X., Lin, Z., Shechtman, E., Wang, O., & Li, H. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.434 · Cited by 894 · see citing works
Kwatra, V., Schödl, A., Essa, I. A., et al. (2003) · ACM Transactions on Graphics · doi:10.1145/882262.882264 · Cited by 788 · see citing works
Scene completion using millions of photographs
Hays, J. H., & Efros, A. A. (2007) · ACM SIGGRAPH Conference (SIGGRAPH) · doi:10.1145/1275808.1276382 · Cited by 714 · see citing works
Space-Time Completion of Video
Wexler, Y., Shechtman, E., & Irani, M. (2007) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2007.60 · Cited by 681 · see citing works
Summarizing visual data using bidirectional similarity
Simakov, D., Caspi, Y., Shechtman, E., & Irani, M. (2008) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2008.4587842 · Cited by 615 · see citing works
Texture optimization for example-based synthesis
Kwatra, V., Essa, I. A., Bobick, A., & Kwatra, N. (2005) · ACM Transactions on Graphics · doi:10.1145/1073204.1073263 · Cited by 614 · see citing works
Conference on Neural Information Processing Systems
Liu, L., Zhou, T., Long, G., Jiang, J., & Zhang, C. (2019) · UTS ePRESS (University of Technology Sydney) · Cited by 593 · see citing works
Darabi, S., Shechtman, E., Barnes, C., et al. (2012) · ACM Transactions on Graphics · doi:10.1145/2185520.2185578 · Cited by 577 · see citing works
Image completion with structure propagation
Sun, J. Z., Yuan, L., Jia, J., & Shum, H. (2005) · ACM Transactions on Graphics · doi:10.1145/1073204.1073274 · Cited by 541 · see citing works
The Generalized PatchMatch Correspondence Algorithm
Barnes, C., Shechtman, E., Goldman, D. B., & Finkelstein, A. (2010) · Lecture notes in computer science · doi:10.1007/978-3-642-15558-1_3 · Cited by 538 · see citing works
Scene completion using millions of photographs
Hays, J. H., & Efros, A. A. (2007) · ACM Transactions on Graphics · doi:10.1145/1276377.1276382 · Cited by 537 · see citing works
Deep Learning Based Imaging Data Completion for Improved Brain Disease Diagnosis
Li, R., Zhang, W., Suk, H., Wang, L., Jiang, L., Shen, D. G., & Ji, S. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-10443-0_39 · Cited by 480 · see citing works
Spatial Pattern Templates for Recognition of Objects with Regular Structure
Tyleček, R., & Šára, R. (2013) · Lecture notes in computer science · doi:10.1007/978-3-642-40602-7_39 · Cited by 362 · see citing works
Image completion using planar structure guidance
Huang, J., Kang, S. B., Ahuja, N., et al. (2014) · ACM Transactions on Graphics · doi:10.1145/2601097.2601205 · Cited by 343 · see citing works
Fragment-based image completion
Drori, I., Cohen‐Or, D., & Yeshurun, H. (2003) · ACM Transactions on Graphics · doi:10.1145/882262.882267 · Cited by 343 · see citing works
Learning how to inpaint from global image statistics
Левин, Zomet, & Weiß (2003) · IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2003.1238360 · Cited by 337 · see citing works
PatchMatch: A Randomized Correspondence Algorithm for Structural Image Editing
Barnes, C., Shechtman, E., Finkelstein, A., & Goldman, D. B. (2023) · ACM eBooks · doi:10.1145/3596711.3596777 · Cited by 276 · see citing works
Image repairing: robust image synthesis by adaptive ND tensor voting
Jia, J., & Tang, C. (2003) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2003.1211414 · Cited by 203 · see citing works
Statistics of Patch Offsets for Image Completion
He, K., & Sun, J. Z. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-33709-3_2 · Cited by 192 · see citing works
Places: An Image Database for Deep Scene UnderstandingPreprint
Zhou, B., Khosla, A., Lapedriza, À., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1610.02055 · Cited by 173 · see citing works
Shepard convolutional neural networks
Ren, J., Xu, L., Yan, Q., & Sun, W. (2015) · neural information processing systems · Cited by 161 · see citing works
Mask-Specific Inpainting with Deep Neural Networks
Köhler, R., Schuler, C. J., Schölkopf, B., & Stefan Harmeling, B. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-11752-2_43 · Cited by 131 · see citing works
Multitask Twin Support Vector Machines
Xie, X., & Sun, S. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-34481-7_42 · Cited by 123 · see citing works
Get Out of my Picture! Internet-based Inpainting
Whyte, O., Šivic, J., & Zisserman, A. (2009) · British Machine Vision Conference (BMVC) · doi:10.5244/c.23.116 · Cited by 102 · see citing works
Mohammed, U., Prince, S. J. D., & Kautz, J. (2009) · ACM Transactions on Graphics · doi:10.1145/1531326.1531363 · Cited by 89 · see citing works
Quality prediction for image completion
Kopf, J., Kienzle, W., Drucker, S. M., et al. (2012) · ACM Transactions on Graphics · doi:10.1145/2366145.2366150 · Cited by 84 · see citing works
Graph Laplace for Occluded Face Completion and Recognition
Deng, Y. C., Dai, Q., & Zhang, Z. (2011) · IEEE Transactions on Image Processing · doi:10.1109/tip.2011.2109729 · Cited by 84 · see citing works
Interactive image completion with perspective correction
Pavić, D., Schönefeld, V., & Kobbelt, L. P. (2006) · The Visual Computer · doi:10.1007/s00371-006-0050-2 · Cited by 69 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Image Inpainting for Irregular Holes Using Partial Convolutions
Liu, G., Reda, F. A., Shih, K. J., Wang, T.-C., Tao, A., & Catanzaro, B. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01252-6_6 · Cited by 2,419 · see citing works
“2(c): inpainted result of Iizuka et al.[10].”
BackgroundMethodInfluentialRePaint: Inpainting using Denoising Diffusion Probabilistic Models
Lugmayr, A., Danelljan, M., Romero, A., Yu, F., Timofte, R., & Van Gool, L. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01117 · Cited by 1,778 · see citing works
Palette: Image-to-Image Diffusion Models
Saharia, C., Chan, W., Chang, H., Lee, C., Ho, J., Salimans, T., Fleet, D., & Norouzi, M. (2022) · Special Interest Group on Computer Graphics and Interactive Techniques Conference Proceedings · doi:10.1145/3528233.3530757 · Cited by 1,717 · see citing works
A Review on Generative Adversarial Networks: Algorithms, Theory, and Applications
Gui, J., Sun, Z., Wen, Y., Tao, D., & Ye, J. (2023) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2021.3130191 · Cited by 1,231 · see citing works
Attentive Generative Adversarial Network for Raindrop Removal from A Single Image
Qian, R., Tan, R. T., Yang, W., Su, J., & Liu, J. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00263 · Cited by 821 · see citing works
FakeCatcher: Detection of Synthetic Portrait Videos using Biological Signals
Ciftci, U. A., Demir, I., & Yin, L. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2020.3009287 · Cited by 499 · see citing works
Durall, R., Keuper, M., & Keuper, J. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00791 · Cited by 26 · see citing works
DeepFake Detection by Analyzing Convolutional Traces
Guarnera, L., Giudice, O., & Battiato, S. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw50498.2020.00341 · Cited by 374 · see citing works
All in One Bad Weather Removal Using Architectural Search
Li, R., Tan, R. T., & Cheong, L.-F. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00324 · Cited by 377 · see citing works
Self-Supervised Learning in Remote Sensing: A review
Wang, Y., Albrecht, C. M., Braham, N. A. A., Mou, L., & Zhu, X. X. (2022) · IEEE Geoscience and Remote Sensing Magazine · doi:10.1109/mgrs.2022.3198244 · Cited by 364 · see citing works
SC-FEGAN: Face Editing Generative Adversarial Network With User’s Sketch and Color
Jo, Y., & Park, J. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00183 · Cited by 346 · see citing works
Detecting GAN generated Fake Images using Co-occurrence Matrices
Nataraj, L., Mohammed, T. M., Manjunath, B. S., Chandrasekaran, S., Flenner, A., Bappy, J. H., & Roy-Chowdhury, A. K. (2019) · Electronic Imaging · doi:10.2352/issn.2470-1173.2019.5.mwsf-532 · Cited by 331 · see citing works
Februus: Input Purification Defense Against Trojan Attacks on Deep Neural Network Systems
Doan, B. G., Abbasnejad, E., & Ranasinghe, D. C. (2020) · Annual Computer Security Applications Conference · doi:10.1145/3427228.3427264 · Cited by 263 · see citing works
Generative Adversarial Networks for Image and Video Synthesis: Algorithms and Applications
Liu, M.-Y., Huang, X., Yu, J., Wang, T.-C., & Mallya, A. (2021) · Proceedings of the IEEE · doi:10.1109/jproc.2021.3049196 · Cited by 236 · see citing works
Semantic photo manipulation with a generative image prior
Bau, D., Strobelt, H., Peebles, W., Wulff, J., Zhou, B., Zhu, J.-Y., & Torralba, A. (2019) · ACM Transactions on Graphics · doi:10.1145/3306346.3323023 · Cited by 207 · see citing works
Countering Malicious DeepFakes: Survey, Battleground, and Horizon
Juefei-Xu, F., Wang, R., Huang, Y., Guo, Q., Ma, L., & Liu, Y. (2022) · International Journal of Computer Vision · doi:10.1007/s11263-022-01606-8 · Cited by 191 · see citing works
Fighting Deepfake by Exposing the Convolutional Traces on Images
Guarnera, L., Giudice, O., & Battiato, S. (2020) · IEEE Access · doi:10.1109/access.2020.3023037 · Cited by 149 · see citing works
RSGAN: face swapping and editing using face and hair representation in latent spaces
Natsume, R., Yatagawa, T., & Morishima, S. (2018) · ACM SIGGRAPH 2018 Posters · doi:10.1145/3230744.3230818 · Cited by 140 · see citing works
Towards Generalizable Deepfake Detection with Locality-aware AutoEncoder
Du, M., Pentyala, S., Li, Y., & Hu, X. (2020) · Proceedings of the 29th ACM International Conference on Information & Knowledge Management · doi:10.1145/3340531.3411892 · Cited by 114 · see citing works
A detailed analysis of image and video forgery detection techniques
Tyagi, S., & Yadav, D. (2023) · The Visual Computer · doi:10.1007/s00371-021-02347-4 · Cited by 109 · see citing works
Neural point cloud rendering via multi-plane projection
Dai, P., Zhang, Y., Li, Z., Liu, S., & Zeng, B. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00785 · Cited by 80 · see citing works
Refacing: Reconstructing Anonymized Facial Features Using GANS
Abramian, D., & Eklund, A. (2019) · 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019) · doi:10.1109/isbi.2019.8759515 · Cited by 63 · see citing works
A Layer-Based Sequential Framework for Scene Generation with GANs
Turkoglu, M. O., Thong, W., Spreeuwers, L., & Kicanaoglu, B. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33018901 · Cited by 41 · see citing works
Improving the Segmentation Accuracy of Ovarian-Tumor Ultrasound Images Using Image Inpainting
Chen, L., Qiao, C., Wu, M., Cai, L., Yin, C., Yang, M., Sang, X., & Bai, W. (2023) · Bioengineering · doi:10.3390/bioengineering10020184 · Cited by 27 · see citing works
- Authors
- Iizuka, S., Simo-Serra, E., & Ishikawa, H.
- Affiliations
- Waseda University (JP)
- Waseda University
- ORCID
- Iizuka, Satoshi
- Venue
- ACM Transactions on Graphics
- Published
- 2017-07-20
- Volume
- 36
- Issue
- 4
- Pages
- 1-14
- ISSN
- 0730-0301 (ISSN-L)
- 1557-7368
- OpenAlex
- W2738588019
- Cited by
- 2,203
- References
- 52
- Citations by year
- 2026: 87 · 2025: 171 · 2024: 233 · 2023: 312 · 2022: 302 · 2021: 341 · 2020: 359 · 2019: 262 · 2018: 125 · 2017: 10 · 2016: 1
- FWCI
- 85.78
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- ACM Transactions on Graphics (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
- Physical Sciences › Engineering › Computational Mechanics › 3D Shape Modeling and Analysis
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Image Processing Techniques
- Keywords
- image inpainting, global consistency, local consistency, fully convolutional network, patch generation, face completion
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
