Real-Time High-Resolution Background Matting
Open accessThe DOI matches its Crossref record; 37 of the 37 references it cites were found in a registry or an index.
Abstract
We introduce a real-time, high-resolution background replacement technique which operates at 30fps in 4K resolution, and 60fps for HD on a modern GPU. Our technique is based on background matting, where an additional frame of the background is captured and used in recovering the alpha matte and the foreground layer. The main challenge is to compute a high-quality alpha matte, preserving strand-level hair details, while processing high-resolution images in real-time. To achieve this goal, we employ two neural networks; a base network computes a low-resolution result which is refined by a second network operating at high-resolution on selective patches. We introduce two large-scale video and image matting datasets: VideoMatte240K and PhotoMatte13K/85. Our approach yields higher quality results compared to the previous state-of-the-art in background matting, while simultaneously yielding a dramatic boost in both speed and resolution.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 5×
“We adopt ResNet-50 [12] for our encoder backbone, which can be replaced by ResNet-101 and MobileNetV2 [26] to trade-off between speed and quality.”
MethodInfluentialPyTorch: An Imperative Style, High-Performance Deep Learning Library
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. (2019) · arXiv · doi:10.48550/arxiv.1912.01703 · Cited by 31,473
He, K., Gkioxari, G., Dollar, P., & Girshick, R. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.322 · Cited by 29,137
MobileNetV2: Inverted Residuals and Linear Bottlenecks
Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., & Chen, L.-C. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00474 · Cited by 28,755 · cited 4×
“Role of the base network We experiment with replacing ResNet-50 with ResNet-101 and MobileNetV2 as our encoder backbone in the base network.”
MethodInfluentialBatch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S., & Szegedy, C. (2015) · arXiv · doi:10.48550/arxiv.1502.03167 · Cited by 27,818 · cited 3×
“Our decoder network applies bilinear upsampling at each step, concatenated with the skip connection from the backbone, and followed by a 3×3 convolution, Batch Normalization [14], and ReLU activation [22] (except the last layer).”
MethodEncoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation
Chen, L.-C., Zhu, Y., Papandreou, G., Schroff, F., & Adam, H. (2018) · arXiv · doi:10.48550/arxiv.1802.02611 · Cited by 20,063 · cited 4×
“For fair comparison in the real-time scenario, where manual trimaps cannot be crafted, we construct trimaps by morphing segmentation result from DeepLabV3+, as suggested in [27].”
BackgroundMethodInfluentialEncoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation
Chen, L.-C., Zhu, Y., Papandreou, G., Schroff, F., & Adam, H. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01234-2_49 · Cited by 20,054
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S. (2024) · arXiv (Cornell University) · doi:10.57702/o9raffed · Cited by 14,214
(2010) · International Conference on Machine Learning · Cited by 13,995
Rethinking Atrous Convolution for Semantic Image Segmentation
Chen, L.-C., Papandreou, G., Schroff, F., & Adam, H. (2017) · arXiv · doi:10.48550/arxiv.1706.05587 · Cited by 7,517
He, K., Gkioxari, G., Dollar, P., & Girshick, R. (2020) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2844175 · Cited by 3,457
A Closed-Form Solution to Natural Image Matting
Levin, A., Lischinski, D., & Weiss, Y. (2008) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2007.1177 · Cited by 1,616
PointRend: Image Segmentation As Rendering
Kirillov, A., Wu, Y., He, K., & Girshick, R. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00982 · Cited by 1,135
A Bayesian approach to digital matting
Chuang, Y.-Y., Curless, B., Salesin, D. H., & Szeliski, R. (2005) · Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001 · doi:10.1109/cvpr.2001.990970 · Cited by 796 · cited 3×
“Matting with known natural background had been previously explored in [23], Bayesian matting [7] and Poisson matting [29, 10] which also requires a trimap.”
BackgroundSelf-citationSun, J., Jia, J., Tang, C.-K., & Shum, H.-Y. (2004) · ACM Transactions on Graphics · doi:10.1145/1015706.1015721 · Cited by 583
Xu, N., Price, B., Cohen, S., & Huang, T. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.41 · Cited by 447 · cited 6×
“It also requires large volumes of images with high-quality alpha mattes to generalize; the publicly available datasets [32, 24] are too limited.”
BackgroundMethodInfluentialHe, K., Gkioxari, G., Dollár, P., & Girshick, R. (2017) · arXiv · doi:10.48550/arxiv.1703.06870 · Cited by 406
A perceptually motivated online benchmark for image matting
Rhemann, C., Rother, C., Wang, J., Gelautz, M., Kohli, P., & Rott, P. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206503 · Cited by 367
Look into Person: Joint Body Parsing & Pose Estimation Network and a New Benchmark
Liang, X., Gong, K., Shen, X., & Lin, L. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2820063 · Cited by 365
Shared Sampling for Real‐Time Alpha Matting
Gastal, E. S. L., & Oliveira, M. M. (2010) · Computer Graphics Forum · doi:10.1111/j.1467-8659.2009.01627.x · Cited by 347 · cited 2×
“Traditional (non-learning based) matting algorithms [6, 16, 17, 29, 9, 2, 7] require manual annotation (a trimap) and solve for the alpha matte in the ‘unknown’ region of the trimap.”
BackgroundImage and Video Matting: A Survey
Wang, J., & Cohen, M. F. (2007) · Foundations and Trends® in Computer Graphics and Vision · doi:10.1561/0600000019 · Cited by 329 · cited 1×
“Different matting techniques are reviewed in the survey by Wang and Cohen [31].”
MethodLevin, A., Rav Acha, A., & Lischinski, D. (2008) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2008.168 · Cited by 297
Pose2Seg: Detection Free Human Instance Segmentation
Zhang, S.-H., Li, R., Dong, X., Rosin, P., Cai, Z., Han, X., Yang, D., Huang, H., & Hu, S.-M. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00098 · Cited by 231 · cited 1×
“Specific applications of human segmentation and parsing have also received considerable attention in recent works [33, 19].”
BackgroundDeep Automatic Portrait Matting
Shen, X., Tao, X., Gao, H., Zhou, C., & Jia, J. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46448-0_6 · Cited by 223 · cited 1×
“Portrait matting without a trimap [35, 28] is one of the more successful applications due to less variability among portrait images compared to full body humans.”
BackgroundIndices Matter: Learning to Index for Deep Image Matting
Lu, H., Dai, Y., Shen, C., & Xu, S. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00336 · Cited by 196 · cited 2×
“, Context-Aware Matting [13], Index Matting [21], sampling-based matting [30] and opacity propagationbased matting [18].”
BackgroundBackground Matting: The World Is Your Green Screen
Sengupta, S., Jayaram, V., Curless, B., Seitz, S. M., & Kemelmacher-Shlizerman, I. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00236 · Cited by 194
Self-citationA Late Fusion CNN for Digital Matting
Zhang, Y., Gong, L., Fan, L., Ren, P., Huang, Q., Bao, H., & Xu, W. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00765 · Cited by 193 · cited 1×
“Recent approaches like Late Fusion Matting [34] and HAttMatting [24] aim to solve for the alpha matte directly from the image, but these approaches can often fail to generalize as shown in [27].”
BackgroundSun, J., Jia, J., Tang, C.-K., & Shum, H.-Y. (2004) · ACM SIGGRAPH 2004 Papers · doi:10.1145/1186562.1015721 · Cited by 189 · cited 3×
“Matting with known natural background had been previously explored in [23], Bayesian matting [7] and Poisson matting [29, 10] which also requires a trimap.”
Attention-Guided Hierarchical Structure Aggregation for Image Matting
Qiao, Y., Liu, Y., Yang, X., Zhou, D., Xu, M., Zhang, Q., & Wei, X. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01369 · Cited by 175 · cited 4×
“We then train on publicly available datasets [32, 24] that are manually curated to learn fine-grained details.”
BackgroundMethodInfluentialContext-Aware Image Matting for Simultaneous Foreground and Alpha Estimation
Hou, Q., & Liu, F. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00423 · Cited by 162 · cited 2×
“, Context-Aware Matting [13], Index Matting [21], sampling-based matting [30] and opacity propagationbased matting [18].”
BackgroundLearning-Based Sampling for Natural Image Matting
Tang, J., Aksoy, Y., Öztireli, C., Groß, M. H., & Aydın, T. O. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00317 · Cited by 156 · cited 1×
“, Context-Aware Matting [13], Index Matting [21], sampling-based matting [30] and opacity propagationbased matting [18].”
BackgroundCai, S., Zhang, X., Fan, H., Huang, H., Liu, J., Liu, J., Liu, J., Wang, J., & Sun, J. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00891 · Cited by 144 · cited 1×
“Although the performance of these methods depends on the quality of the annotations, some recent methods consider coarse [20] or faulty human annotations [3] to predict the alpha matte.”
Boosting Semantic Human Matting With Coarse Annotations
Liu, J., Yao, Y., Hou, W., Cui, M., Xie, X., Zhang, C., & Hua, X. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00859 · Cited by 120
Aksoy, Y., Oh, T.-H., Paris, S., Pollefeys, M., & Matusik, W. (2018) · ACM Transactions on Graphics · doi:10.1145/3197517.3201275 · Cited by 112 · cited 1×
“Soft segmentation for natural images had also been explored in [1].”
Background$F$, $B$, Alpha MattingPreprint
Forte, M., & Pitié, F. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.07711 · Cited by 103
Fast Deep Matting for Portrait Animation on Mobile Phone
Zhu, B., Chen, Y., Wang, J., Liu, S., Zhang, B., & Tang, M. (2017) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3123266.3123286 · Cited by 76 · cited 1×
“Portrait matting without a trimap [35, 28] is one of the more successful applications due to less variability among portrait images compared to full body humans.”
BackgroundHierarchical Opacity Propagation for Image MattingPreprint
Li, Y., Xu, Q., & Lu, H. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.03249 · Cited by 14 · cited 1×
“, Context-Aware Matting [13], Index Matting [21], sampling-based matting [30] and opacity propagationbased matting [18].”
Background
Metadata
- Authors
- Lin, S., Ryabtsev, A., Sengupta, S., Curless, B., Seitz, S., & Kemelmacher-Shlizerman, I.
- Venue
- 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2021-06
- Pages
- 8758-8767
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W3176288137
- Cited by
- 233
- References
- 37
- FWCI
- 15.51
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.