PatchMatch: a randomized correspondence algorithm for structural image editing
The DOI matches its Crossref record; 30 of the 30 references it cites were found in a registry or an index.
Abstract
This paper presents interactive image editing tools using a new randomized algorithm for quickly finding approximate nearest-neighbor matches between image patches. Previous research in graphics and vision has leveraged such nearest-neighbor searches to provide a variety of high-level digital image editing tools. However, the cost of computing a field of such matches for an entire image has eluded previous efforts to provide interactive performance. Our algorithm offers substantial performance improvements over the previous state of the art (20-100x), enabling its use in interactive editing tools. The key insights driving the algorithm are that some good patch matches can be found via random sampling, and that natural coherence in the imagery allows us to propagate such matches quickly to surrounding areas. We offer theoretical analysis of the convergence properties of the algorithm, as well as empirical and practical evidence for its high quality and performance. This one simple algorithm forms the basis for a variety of tools -- image retargeting, completion and reshuffling -- that can be used together in the context of a high-level image editing application. Finally, we propose additional intuitive constraints on the synthesis process that offer the user a level of control unavailable in previous methods.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Fischler, M. A., & Bolles, R. C. (1981) · Communications of the ACM · doi:10.1145/358669.358692 · Cited by 25,893
A Non-Local Algorithm for Image Denoising
Buades, A., Coll, B., & Morel, J.-M. (2005) · 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) · doi:10.1109/cvpr.2005.38 · Cited by 7,616 · cited 2×
“Dense nearest neighbor patch search was also shown to be useful for image denoising [Buades et al. 2005] and learning based superresolution [Freeman et al.”
Texture synthesis by non-parametric sampling
Efros, A. A., & Leung, T. K. (1999) · Proceedings of the Seventh IEEE International Conference on Computer Vision · doi:10.1109/iccv.1999.790383 · Cited by 3,023
Example-based super-resolution
Freeman, W. T., Jones, T. R., & Pasztor, E. C. (2002) · IEEE Computer Graphics and Applications · doi:10.1109/38.988747 · Cited by 2,498
Image quilting for texture synthesis and transfer
Efros, A. A., & Freeman, W. T. (2001) · Proceedings of the 28th annual conference on Computer graphics and interactive techniques · doi:10.1145/383259.383296 · Cited by 2,347
Hertzmann, A., Jacobs, C. E., Oliver, N., Curless, B., & Salesin, D. H. (2001) · Proceedings of the 28th annual conference on Computer graphics and interactive techniques · doi:10.1145/383259.383295 · Cited by 1,518
Fast texture synthesis using tree-structured vector quantization
Wei, L.-Y., & Levoy, M. (2000) · Proceedings of the 27th annual conference on Computer graphics and interactive techniques - SIGGRAPH '00 · doi:10.1145/344779.345009 · Cited by 1,356
Seam carving for content-aware image resizing
Avidan, S., & Shamir, A. (2007) · ACM Transactions on Graphics · doi:10.1145/1276377.1276390 · Cited by 1,092
Szeliski, R., Zabih, R., Scharstein, D., Veksler, O., Kolmogorov, V., Agarwala, A., Tappen, M., & Rother, C. (2008) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2007.70844 · Cited by 966
Object removal by exemplar-based inpainting
Criminisi, A., Perez, P., & Toyama, K. (2003) · 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings. · doi:10.1109/cvpr.2003.1211538 · Cited by 847
Research works printed but not linked: 6
- 10.1145/1276377.1276392 · doi:10.1145/1276377.1276392
- 10.1145/1073204.1073261 · DOI
- Mount , D. M. , and Arya , S. , 1997 . ANN: A library for approximate nearest neighbor searching, Oct. 28. Mount, D. M., and Arya, S., 1997. ANN: A library for approximate nearest neighbor searching, Oct. 28.
- 10.1145/1111411.1111431 · doi:10.1145/1111411.1111431
- 10.1145/1186822.1073274 · doi:10.1145/1186822.1073274
- 10.1145/1457515.1409071 · DOI
Cited by
Wang, X., Girshick, R., Gupta, A., & He, K. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00813 · Cited by 11,596
“Non-local matching is also the essence of successful texture synthesis [12], super-resolution [16], and inpainting [1] algorithms.”
BackgroundGenerative Image Inpainting with Contextual Attention
Yu, J., Lin, Z., Yang, J., Shen, X., Lu, X., & Huang, T. S. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00577 · Cited by 2,527
Semantic Image Inpainting with Deep Generative Models
Yeh, R. A., Chen, C., Lim, T. Y., Schwing, A. G., Hasegawa-Johnson, M., & Do, M. N. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.728 · Cited by 1,206
LiteFlowNet: A Lightweight Convolutional Neural Network for Optical Flow Estimation
Hui, T.-W., Tang, X., & Loy, C. C. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00936 · Cited by 817
Relation-Aware Global Attention for Person Re-Identification
Zhang, Z., Lan, C., Zeng, W., Jin, X., & Chen, Z. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00325 · Cited by 652
Massively Parallel Multiview Stereopsis by Surface Normal Diffusion
Galliani, S., Lasinger, K., & Schindler, K. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.106 · Cited by 599
MaskGIT: Masked Generative Image Transformer
Chang, H., Zhang, H., Jiang, L., Liu, C., & Freeman, W. T. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01103 · Cited by 446
“Traditional patch-based methods [3] work well on texture regions, while deep learning based methods [14,38,48,50,54] have been demonstrated to synthesize images requiring better semantic coherence.”
BackgroundState of the Art on Neural Rendering
Tewari, A., Fried, O., Thies, J., Sitzmann, V., Lombardi, S., Sunkavalli, K., Martin‐Brualla, R., Simon, T., Saragih, J., Nießner, M., et al. (2020) · Computer Graphics Forum · doi:10.1111/cgf.14022 · Cited by 430
Self-citationInstance-Sensitive Fully Convolutional Networks
Dai, J., He, K., Li, Y., Ren, S., & Sun, J. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46466-4_32 · Cited by 413
Learning Blind Video Temporal Consistency
Lai, W.-S., Huang, J.-B., Wang, O., Shechtman, E., Yumer, E., & Yang, M.-H. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01267-0_11 · Cited by 405
Self-citationForeground-Aware Image Inpainting
Xiong, W., Yu, J., Lin, Z., Yang, J., Lu, X., Barnes, C., & Luo, J. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00599 · Cited by 402
Self-citationStructureFlow: Image Inpainting via Structure-Aware Appearance Flow
Ren, Y., Yu, X., Zhang, R., Li, T. H., Liu, S., & Li, G. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00027 · Cited by 396
“Double-DIP”: Unsupervised Image Decomposition via Coupled Deep-Image-Priors
Gandelsman, Y., Shocher, A., & Irani, M. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.01128 · Cited by 355
SC-FEGAN: Face Editing Generative Adversarial Network With User’s Sketch and Color
Jo, Y., & Park, J. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00183 · Cited by 346
A global sampling method for alpha matting
He, K., Rhemann, C., Rother, C., Tang, X., & Sun, J. (2011) · CVPR 2011 · doi:10.1109/cvpr.2011.5995495 · Cited by 310
Han, X., Li, Z., Huang, H., Kalogerakis, E., & Yu, Y. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.19 · Cited by 304
TextureGAN: Controlling Deep Image Synthesis with Texture Patches
Xian, W., Sangkloy, P., Agrawal, V., Raj, A., Lu, J., Fang, C., Yu, F., & Hays, J. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00882 · Cited by 297
Rethinking Image Inpainting via a Mutual Encoder-Decoder with Feature Equalizations
Liu, H., Jiang, B., Song, Y., Huang, W., & Yang, C. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58536-5_43 · Cited by 290
Single-Image HDR Reconstruction by Learning to Reverse the Camera Pipeline
Liu, Y.-L., Lai, W.-S., Chen, Y.-S., Kao, Y.-L., Yang, M.-H., Chuang, Y.-Y., & Huang, J.-B. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00172 · Cited by 287
Image Inpainting With Learnable Bidirectional Attention Maps
Xie, C., Liu, S., Li, C., Cheng, M.-M., Zuo, W., Liu, X., Wen, S., & Ding, E. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00895 · Cited by 282
Multi-Scale Geometric Consistency Guided Multi-View Stereo
Xu, Q., & Tao, W. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00563 · Cited by 257
Metadata
- Authors
- Barnes, C., Shechtman, E., Finkelstein, A., & Goldman, D. B.
- Venue
- ACM SIGGRAPH 2009 papers
- Publisher
- ACM
- Published
- 2009-07-27
- Pages
- 1-11
- OpenAlex
- W4249710618
- Cited by
- 364
- References
- 30
- FWCI
- 9.00
- Type
- conference paper
- Language
- en
- Open access
- closed
- Funders
- Division of Information and Intelligent Systems (IIS-0511965)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.