The DOI matches its Crossref record; 47 of the 47 references it cites were found in a registry or an index.
Deep bilateral learning for real-time image enhancement
Open accessAbstract
Performance is a critical challenge in mobile image processing. Given a reference imaging pipeline, or even human-adjusted pairs of images, we seek to reproduce the enhancements and enable real-time evaluation. For this, we introduce a new neural network architecture inspired by bilateral grid processing and local affine color transforms. Using pairs of input/output images, we train a convolutional neural network to predict the coefficients of a locally-affine model in bilateral space. Our architecture learns to make local, global, and content-dependent decisions to approximate the desired image transformation. At runtime, the neural network consumes a low-resolution version of the input image, produces a set of affine transformations in bilateral space, upsamples those transformations in an edge-preserving fashion using a new slicing node, and then applies those upsampled transformations to the full-resolution image. Our algorithm processes high-resolution images on a smartphone in milliseconds, provides a real-time viewfinder at 1080p resolution, and matches the quality of state-of-the-art approximation techniques on a large class of image operators. Unlike previous work, our model is trained off-line from data and therefore does not require access to the original operator at runtime. This allows our model to learn complex, scene-dependent transformations for which no reference implementation is available, such as the photographic edits of a human retoucher.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
U-Net: Convolutional Networks for Biomedical Image Segmentation
Ronneberger, O., Fischer, P., & Brox, T. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24574-4_28 · Cited by 99,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Fully convolutional networks for semantic segmentation
Long, J., Shelhamer, E., & Darrell, T. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298965 · Cited by 38,834
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S., & Szegedy, C. (2015) · arXiv · doi:10.48550/arxiv.1502.03167 · Cited by 27,818
Image-to-Image Translation with Conditional Adversarial Networks
Isola, P., Zhu, J.-Y., Zhou, T., & Efros, A. A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.632 · Cited by 22,930
Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification
He, K., Zhang, X., Ren, S., & Sun, J. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.123 · Cited by 19,741
Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising
Zhang, K., Zuo, W., Chen, Y., Meng, D., & Zhang, L. (2017) · IEEE Transactions on Image Processing · doi:10.1109/tip.2017.2662206 · Cited by 8,983
Bilateral Filtering for Gray and Color Images
Tomasi, C., & Manduchi, R. (2002) · Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271) · doi:10.1109/iccv.1998.710815 · Cited by 8,177 · cited 1×
“Similarly, many approaches have been proposed to accelerate bilateral filtering, due to the ubiquity of edge-aware image processing [Adams et al. 2010; Chen et al. 2007; Paris and Durand 2006; Tomasi and Manduchi 1998].”
BackgroundMulti-scale context aggregation by dilated convolutions
Yu, F., & Koltun, V. (2015) · arXiv · doi:10.48550/arxiv.1511.07122 · Cited by 7,738 · cited 2×
“far from real-time. In Figure 10, we compare our technique against a U-Net architecture [Ronneberger et al. 2015] adapted from Isola et al. [2016], and a linear network based on dilated convolutions [Yu and Koltun 2015]. We explore several settings for the depth (number of layers, 3 to 11) and the width (number of fil…”
MethodHe, K., Sun, J., & Tang, X. (2013) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2012.213 · Cited by 5,546
Learning a Deep Convolutional Network for Image Super-Resolution
Dong, C., Loy, C. C., He, K., & Tang, X. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10593-2_13 · Cited by 5,494
Depth Map Prediction from a Single Image using a Multi-Scale Deep Network
Eigen, D., Puhrsch, C., & Fergus, R. (2014) · arXiv · doi:10.48550/arxiv.1406.2283 · Cited by 3,442
A Closed-Form Solution to Natural Image Matting
Levin, A., Lischinski, D., & Weiss, Y. (2008) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2007.1177 · Cited by 1,616
FDDB: A benchmark for face detection in unconstrained settings
Jain, V., & Miller, E. G. (2010) · ScholarWorks@UMassAmherst (University of Massachusetts Amherst) · Cited by 1,072
Kopf, J., Cohen, M. F., Lischinski, D., & Uyttendaele, M. (2007) · ACM SIGGRAPH 2007 papers · doi:10.1145/1275808.1276497 · Cited by 872
Kopf, J., Cohen, M. F., Lischinski, D., & Uyttendaele, M. (2007) · ACM Transactions on Graphics · doi:10.1145/1276377.1276497 · Cited by 845
Iizuka, S., Simo-Serra, E., & Ishikawa, H. (2016) · ACM Transactions on Graphics · doi:10.1145/2897824.2925974 · Cited by 814
Learning photographic global tonal adjustment with a database of input/output image pairs
Bychkovsky, V., Paris, S., Chan, E., & Durand, F. (2011) · CVPR 2011 · doi:10.1109/cvpr.2011.5995413 · Cited by 536
Self-citationLearning photographic global tonal adjustment with a database of input / output image pairs
Bychkovsky, V., Paris, S., Chan, E., & Durand, F. (2011) · CVPR 2011 · doi:10.1109/cvpr.2011.5995332 · Cited by 529
Self-citationBurst photography for high dynamic range and low-light imaging on mobile cameras
Hasinoff, S. W., Sharlet, D., Geiss, R., Adams, A., Barron, J. T., Kainz, F., Chen, J., & Levoy, M. (2016) · ACM Transactions on Graphics · doi:10.1145/2980179.2980254 · Cited by 483
Self-citationA Fast Approximation of the Bilateral Filter Using a Signal Processing Approach
Paris, S., & Durand, F. (2006) · Lecture Notes in Computer Science · doi:10.1007/11744085_44 · Cited by 481
Self-citationDeep joint demosaicking and denoising
Gharbi, M., Chaurasia, G., Paris, S., & Durand, F. (2016) · ACM Transactions on Graphics · doi:10.1145/2980179.2982399 · Cited by 473
Self-citationReal-time edge-aware image processing with the bilateral grid
Chen, J., Paris, S., & Durand, F. (2007) · ACM Transactions on Graphics · doi:10.1145/1276377.1276506 · Cited by 395
Self-citationFast High‐Dimensional Filtering Using the Permutohedral Lattice
Adams, A., Baek, J., & Davis, M. A. (2010) · Computer Graphics Forum · doi:10.1111/j.1467-8659.2009.01645.x · Cited by 394
Real-time edge-aware image processing with the bilateral grid
Chen, J., Paris, S., & Durand, F. (2007) · ACM SIGGRAPH 2007 papers · doi:10.1145/1275808.1276506 · Cited by 394
Self-citationLocal Laplacian filters: edge-aware image processing with a Laplacian pyramid
Paris, S., Hasinoff, S. W., & Kautz, J. (2011) · ACM Transactions on Graphics · doi:10.1145/2010324.1964963 · Cited by 384
Self-citationLocal Laplacian filters: edge-aware image processing with a Laplacian pyramid
Paris, S., Hasinoff, S. W., & Kautz, J. (2011) · ACM SIGGRAPH 2011 papers · doi:10.1145/1964921.1964963 · Cited by 322
Self-citationBarron, J. T., & Poole, B. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46487-9_38 · Cited by 309
Self-citationAutomatic Photo Adjustment Using Deep Neural Networks
Yan, Z., Zhang, H., Wang, B., Paris, S., & Yu, Y. (2016) · ACM Transactions on Graphics · doi:10.1145/2790296 · Cited by 271
Decoupling algorithms from schedules for easy optimization of image processing pipelines
Ragan-Kelley, J., Adams, A., Paris, S., Levoy, M., Amarasinghe, S., & Durand, F. (2012) · ACM Transactions on Graphics · doi:10.1145/2185520.2185528 · Cited by 251
Self-citationLi, X., Ren, J., Yan, Q., Liao, R., & Jia, J. (2015) · International Conference on Machine Learning · Cited by 230
Deep Automatic Portrait Matting
Shen, X., Tao, X., Gao, H., Zhou, C., & Jia, J. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46448-0_6 · Cited by 223
Fast Local Laplacian Filters: Theory and Applications
Aubry, M., Paris, S., Hasinoff, S. W., Kautz, J., & Durand, F. (2014) · ACM Transactions on Graphics · doi:10.1145/2629645 · Cited by 197
Self-citationLearning Sparse High Dimensional Filters: Image Filtering, Dense CRFs and Bilateral Neural Networks
Jampani, V., Kiefel, M., & Gehler, P. V. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.482 · Cited by 188
Data-driven hallucination of different times of day from a single outdoor photo
Shih, Y., Paris, S., Durand, F., & Freeman, W. T. (2013) · ACM Transactions on Graphics · doi:10.1145/2508363.2508419 · Cited by 187
Self-citationLearning Recursive Filters for Low-Level Vision via a Hybrid Neural Network
Liu, S., Pan, J., & Yang, M. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46493-0_34 · Cited by 178
Darkroom: compiling high-level image processing code into hardware pipelines
Hegarty, J., Brunhaver, J., DeVito, Z., Ragan-Kelley, J., Cohen, N., Bell, S., Vasilyev, A., Horowitz, M., & Hanrahan, P. (2014) · ACM Transactions on Graphics · doi:10.1145/2601097.2601174 · Cited by 177
Automatically scheduling halide image processing pipelines
Mullapudi, R. T., Adams, A., Sharlet, D., Ragan‐Kelley, J., & Fatahalian, K. (2016) · ACM Transactions on Graphics · doi:10.1145/2897824.2925952 · Cited by 173
Fast bilateral-space stereo for synthetic defocus
Barron, J. T., Adams, A., Shih, Y., & Hernández, C. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7299076 · Cited by 169
Self-citationFlowNet 2.0: Evolution of Optical Flow Estimation with Deep NetworksPreprint
Ilg, E., Mayer, N., Saikia, T., Keuper, M., Dosovitskiy, A., & Brox, T. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1612.01925 · Cited by 162
Chen, J., Adams, A., Wadhwa, N., & Hasinoff, S. W. (2016) · ACM Transactions on Graphics · doi:10.1145/2980179.2982423 · Cited by 150
Self-citationFast Guided FilterPreprint
He, K., & Sun, J. D. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1505.00996 · Cited by 106
Farbman, Z., Fattal, R., & Lischinski, D. (2011) · ACM Transactions on Graphics · doi:10.1145/2070781.2024209 · Cited by 57
Content‐Aware Automatic Photo Enhancement
Kaufman, L., Lischinski, D., & Werman, M. (2012) · Computer Graphics Forum · doi:10.1111/j.1467-8659.2012.03225.x · Cited by 48
Transform recipes for efficient cloud photo enhancement
Gharbi, M., Shih, Y., Chaurasia, G., Ragan‐Kelley, J., Paris, S., & Durand, F. (2015) · ACM Transactions on Graphics · doi:10.1145/2816795.2818127 · Cited by 30
Self-citationHigh quality image reconstruction from RAW and JPEG image pair
Yuan, L., & Sun, J. Z. (2011) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2011.6126492 · Cited by 20
Cited by
EnlightenGAN: Deep Light Enhancement Without Paired Supervision
Jiang, Y., Gong, X., Liu, D., Cheng, Y., Fang, C., Shen, X., Yang, J., Zhou, P., & Wang, Z. (2021) · IEEE Transactions on Image Processing · doi:10.1109/tip.2021.3051462 · Cited by 2,715
Learning a Deep Single Image Contrast Enhancer from Multi-Exposure Images
Cai, J., Gu, S., & Zhang, L. (2018) · IEEE Transactions on Image Processing · doi:10.1109/tip.2018.2794218 · Cited by 1,234
A Deep Learning System to Screen Novel Coronavirus Disease 2019 Pneumonia
Xu, X., Jiang, X., Ma, C., Du, P., Li, X., Lv, S., Yu, L., Ni, Q., Chen, Y., Su, J., et al. (2020) · Engineering · doi:10.1016/j.eng.2020.04.010 · Cited by 1,209
Underexposed Photo Enhancement Using Deep Illumination Estimation
Wang, R., Zhang, Q., Fu, C.-W., Shen, X., Zheng, W.-S., & Jia, J. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00701 · Cited by 983
Learning Enriched Features for Real Image Restoration and Enhancement
Zamir, S. W., Arora, A., Khan, S., Hayat, M., Khan, F. S., Yang, M.-H., & Shao, L. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58595-2_30 · Cited by 946
Metadata
- Authors
- Gharbi, M., Chen, J., Barron, J. T., Hasinoff, S. W., & Durand, F.
- Venue
- ACM Transactions on Graphics
- Published
- 2017-07-20
- Volume
- 36
- Issue
- 4
- Pages
- 1-12
- ISSN
- 0730-0301 (ISSN-L)
- 1557-7368
- OpenAlex
- W2735974062
- Cited by
- 800
- References
- 47
- FWCI
- 21.90
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.