The DOI matches its Crossref record; 31 of the 31 references it cites were found in a registry or an index.
Zoom to Learn, Learn to Zoom
Abstract
This paper shows that when applying machine learning to digital zoom, it is beneficial to operate on real, RAW sensor data. Existing learning-based super-resolution methods do not use real sensor data, instead operating on processed RGB images. We show that these approaches forfeit detail and accuracy that can be gained by operating on raw data, particularly when zooming in on distant objects. The key barrier to using real sensor data for training is that ground-truth high-resolution imagery is missing. We show how to obtain such ground-truth data via optical zoom and contribute a dataset, SR-RAW, for real-world computational zoom. We use SR-RAW to train a deep network with a novel contextual bilateral loss that is robust to mild misalignment between input and outputs images. The trained network achieves state-of-the-art performance in 4X and 8X computational zoom. We also show that synthesizing sensor data by resampling high-resolution RGB images is an oversimplified approximation of real sensor data and noise, resulting in worse image quality.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Self-citationVery Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
The Unreasonable Effectiveness of Deep Features as a Perceptual Metric
Zhang, R., Isola, P., Efros, A. A., Shechtman, E., & Wang, O. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00068 · Cited by 14,623
Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network
Ledig, C., Theis, L., Huszar, F., Caballero, J., Cunningham, A., Acosta, A., Aitken, A., Tejani, A., Totz, J., Wang, Z., et al. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.19 · Cited by 12,696
Perceptual Losses for Real-Time Style Transfer and Super-Resolution
Johnson, J., Alahi, A., & Fei-Fei, L. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46475-6_43 · Cited by 11,283
Image Super-Resolution Using Deep Convolutional Networks
Dong, C., Loy, C. C., He, K., & Tang, X. (2016) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2015.2439281 · Cited by 10,105
Bilateral Filtering for Gray and Color Images
Tomasi, C., & Manduchi, R. (2002) · Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271) · doi:10.1109/iccv.1998.710815 · Cited by 8,177 · cited 1×
“Inspired by the edge-preserving bilateral filter [28], we integrate the spatial pixel coordinates and pixel-level RGB information into the image features.”
MethodAccurate Image Super-Resolution Using Very Deep Convolutional Networks
Kim, J., Lee, J. K., & Lee, K. M. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.182 · Cited by 7,870
Enhanced Deep Residual Networks for Single Image Super-Resolution
Lim, B., Son, S., Kim, H., Nah, S., & Lee, K. M. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2017.151 · Cited by 7,736
ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks
Wang, X., Yu, K., Wu, S., Gu, J., Liu, Y., Dong, C., Qiao, Y., & Loy, C. C. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-11021-5_5 · Cited by 4,731
Residual Dense Network for Image Super-Resolution
Zhang, Y., Tian, Y., Kong, Y., Zhong, B., & Fu, Y. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00262 · Cited by 4,168
Single image super-resolution from transformed self-exemplars
Huang, J.-B., Singh, A., & Ahuja, N. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7299156 · Cited by 3,472
Deeply-Recursive Convolutional Network for Image Super-Resolution
Kim, J., Lee, J. K., & Lee, K. M. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.181 · Cited by 3,025
Deep Laplacian Pyramid Networks for Fast and Accurate Super-Resolution
Lai, W.-S., Huang, J.-B., Ahuja, N., & Yang, M.-H. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.618 · Cited by 2,976
Example-based super-resolution
Freeman, W. T., Jones, T. R., & Pasztor, E. C. (2002) · IEEE Computer Graphics and Applications · doi:10.1109/38.988747 · Cited by 2,498
New edge-directed interpolation
Li, X., & Orchard, M. T. (2001) · IEEE Transactions on Image Processing · doi:10.1109/83.951537 · Cited by 2,023
Super-resolution from a single image
Glasner, D., Bagon, S., & Irani, M. (2009) · 2009 IEEE 12th International Conference on Computer Vision · doi:10.1109/iccv.2009.5459271 · Cited by 1,889
Chen, C., Chen, Q., Xu, J., & Koltun, V. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00347 · Cited by 1,376
Self-citationRecovering Realistic Texture in Image Super-Resolution by Deep Spatial Feature Transform
Wang, X., Yu, K., Dong, C., & Change Loy, C. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00070 · Cited by 1,170
EnhanceNet: Single Image Super-Resolution Through Automated Texture Synthesis
Sajjadi, M. S. M., Scholkopf, B., & Hirsch, M. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.481 · Cited by 1,058
Photographic Image Synthesis with Cascaded Refinement Networks
Chen, Q., & Koltun, V. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.168 · Cited by 887
Self-citationParametric Image Alignment Using Enhanced Correlation Coefficient Maximization
Evangelidis, G. D., & Psarakis, E. Z. (2008) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2008.113 · Cited by 731
The 2018 PIRM Challenge on Perceptual Image Super-Resolution
Blau, Y., Mechrez, R., Timofte, R., Michaeli, T., & Zelnik-Manor, L. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-11021-5_21 · Cited by 593
The relativistic discriminator: a key element missing from standard GAN
Jolicoeur‐Martineau, A. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1807.00734 · Cited by 532
Deep joint demosaicking and denoising
Gharbi, M., Chaurasia, G., Paris, S., & Durand, F. (2016) · ACM Transactions on Graphics · doi:10.1145/2980179.2982399 · Cited by 473
The Contextual Loss for Image Transformation with Non-aligned Data
Mechrez, R., Talmi, I., & Zelnik-Manor, L. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01264-9_47 · Cited by 411
Burst Denoising with Kernel Prediction Networks
Mildenhall, B., Barron, J. T., Chen, J., Sharlet, D., Ng, R., & Carroll, R. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00265 · Cited by 404
Self-citationSingle Image Reflection Separation with Perceptual Losses
Zhang, X., Ng, R., & Chen, Q. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00503 · Cited by 385
Self-citationMultiframe demosaicing and super-resolution of color images
Farsiu, S., Elad, M., & Milanfar, P. (2006) · IEEE Transactions on Image Processing · doi:10.1109/tip.2005.860336 · Cited by 319
DeepISP: Toward Learning an End-to-End Image Processing Pipeline
Schwartz, E., Giryes, R., & Bronstein, A. M. (2019) · IEEE Transactions on Image Processing · doi:10.1109/tip.2018.2872858 · Cited by 265
Deep Residual Network for Joint Demosaicing and Super-Resolution
Zhou, R., Achanta, R., & Susstrunk, S. E. (2018) · Color and Imaging Conference · doi:10.2352/issn.2169-2629.2018.26.75 · Cited by 35
Cited by
Deep Learning for Image Super-Resolution: A Survey
Wang, Z., Chen, J., & Hoi, S. C. H. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2020.2982166 · Cited by 1,969
Learning to Upsample by Learning to Sample
Liu, W., Lu, H., Fu, H., & Cao, Z. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00554 · Cited by 792
Toward Real-World Single Image Super-Resolution: A New Benchmark and a New Model
Cai, J., Zeng, H., Yong, H., Cao, Z., & Zhang, L. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00318 · Cited by 583
Real-world single image super-resolution: A brief review
Chen, H., He, X., Qing, L., Wu, Y., Ren, C., Sheriff, R. E., & Zhu, C. (2022) · Information Fusion · doi:10.1016/j.inffus.2021.09.005 · Cited by 405
Real-World Super-Resolution via Kernel Estimation and Noise Injection
Ji, X., Cao, Y., Tai, Y., Wang, C., Li, J., & Huang, F. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw50498.2020.00241 · Cited by 332
Metadata
- Authors
- Zhang, X., Chen, Q., Ng, R., & Koltun, V.
- Venue
- 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2019-06
- Pages
- 3757-3765
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W2943960148
- Cited by
- 260
- References
- 31
- FWCI
- 11.47
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.