End-to-End Image Super-Resolution via Deep and Shallow Convolutional Networks
Open accessAbstract
In this paper, we propose a new image super-resolution (SR) approach based on a convolutional neural network (CNN), which jointly learns the feature extraction, upsampling, and high-resolution (HR) reconstruction modules, yielding a completely end-to-end trainable deep CNN. However, directly training such a deep network in an end-to-end fashion is challenging, which takes a longer time to converge and may lead to sub-optimal results. To address this issue, we propose to jointly train an ensemble of deep and shallow networks. The shallow network with weaker learning capability restores the main structure of the image content, while the deep network with stronger representation power captures the high-frequency details. Since the shallow network is much easier to optimize, it significantly lowers the difficulty of deep network optimization during joint training. To further ensure more accurate restoration of HR images, the high-frequency details are reconstructed in a multi-scale manner to simultaneously incorporate both short- and long-range contextual information. The proposed method is extensively evaluated on widely adopted data sets and compares favorably against state-of-the-art methods. In-depth ablation studies are conducted to verify the contributions of different network designs to image SR, providing additional insights for future research.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Zero-padded identity mapping requires fewer parameters than projection-based shortcut connections, which in turn leads to enhanced generalization.
“In comparison to the projection-based shortcut connections, zero-padded identity mapping requires fewer parameters, which in turn leads to enhanced generalization [124].”
From Review of deep learning: concepts, CNN architectures, challenges, applications, future directions (2021) · cites this paper as [124]
The Feature Extraction section describes an identity-mapping shortcut and says zero padding preserves feature-map size, but it does not describe a zero-padded identity mapping. The shortcut-connection analysis compares a network with and without shortcuts, not identity mappings with projection-based shortcuts. Searches for that comparison and for improved generalization from fewer parameters found no supporting passage in our copy. (LLM, conf=0.72)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · see citing works
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · see citing works
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · see citing works
Image quality assessment: from error visibility to structural similarity
Wang, Z., Bovik, A. C., Sheikh, H. R., & Simoncelli, E. P. (2004) · IEEE Transactions on Image Processing · doi:10.1109/tip.2003.819861 · Cited by 57,892 · see citing works
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562 · see citing works
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S., & Szegedy, C. (2015) · arXiv · doi:10.48550/arxiv.1502.03167 · Cited by 27,818 · see citing works
Learning Deep Features for Discriminative Localization
Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., & Torralba, A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.319 · Cited by 11,428 · see citing works
Caffe: Convolutional Architecture for Fast Feature Embedding
Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., & Darrell, T. (2014) · Proceedings of the 22nd ACM international conference on Multimedia · doi:10.1145/2647868.2654889 · Cited by 11,077 · see citing works
Image Super-Resolution Using Deep Convolutional Networks
Dong, C., Loy, C. C., He, K., & Tang, X. (2016) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2015.2439281 · Cited by 10,105 · see citing works
Martin, D., Fowlkes, C., Tal, D., & Malik, J. (2002) · Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001 · doi:10.1109/iccv.2001.937655 · Cited by 8,216 · see citing works
Beyond Bags of Features: Spatial Pyramid Matching for Recognizing Natural Scene Categories
Lazebnik, S., Schmid, C., & Ponce, J. (2006) · 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Volume 2 (CVPR'06) · doi:10.1109/cvpr.2006.68 · Cited by 7,916 · see citing works
Accurate Image Super-Resolution Using Very Deep Convolutional Networks
Kim, J., Lee, J. K., & Lee, K. M. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.182 · Cited by 7,870 · see citing works
Shi, W., Caballero, J., Huszar, F., Totz, J., Aitken, A. P., Bishop, R., Rueckert, D., & Wang, Z. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.207 · Cited by 7,570 · see citing works
Learning a Deep Convolutional Network for Image Super-Resolution
Dong, C., Loy, C. C., He, K., & Tang, X. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-10593-2_13 · Cited by 5,827 · see citing works
Image Super-Resolution Via Sparse Representation
Yang, J., Wright, J. N., Huang, T. S., & Ma, Y. (2010) · IEEE Transactions on Image Processing · doi:10.1109/tip.2010.2050625 · Cited by 5,241 · see citing works
Residual Dense Network for Image Super-Resolution
Zhang, Y., Tian, Y., Kong, Y., Zhong, B., & Fu, Y. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00262 · Cited by 4,371 · see citing works
Accelerating the Super-Resolution Convolutional Neural Network
Dong, C., Loy, C. C., & Tang, X. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46475-6_25 · Cited by 3,786 · see citing works
Cubic convolution interpolation for digital image processing
Keys, R. G. (1981) · IEEE Transactions on Acoustics Speech and Signal Processing · doi:10.1109/tassp.1981.1163711 · Cited by 3,770 · see citing works
On Single Image Scale-Up Using Sparse-Representations
Zeyde, R., Elad, M., & Protter, M. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-27413-8_47 · Cited by 3,371 · see citing works
Low-Complexity Single-Image Super-Resolution based on Nonnegative Neighbor Embedding
Bevilacqua, M., Roumy, A., Guillemot, C. M., & Morel, M.-L. A. (2012) · British Machine Vision Conference (BMVC) · doi:10.5244/c.26.135 · Cited by 3,162 · see citing works
Deeply-Recursive Convolutional Network for Image Super-Resolution
Kim, J., Lee, J. K., & Lee, K. M. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.181 · Cited by 3,122 · see citing works
Deep Laplacian Pyramid Networks for Fast and Accurate Super-Resolution
Lai, W., Huang, J., Ahuja, N., & Yang, M. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.618 · Cited by 3,111 · see citing works
Example-based super-resolution
Freeman, W. T., Jones, T. R., & Pasztor, E. C. (2002) · IEEE Computer Graphics and Applications · doi:10.1109/38.988747 · Cited by 2,497 · see citing works
Image Super-Resolution via Deep Recursive Residual Network
Tai, Y. Y., Yang, J. J., & Liu, X. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.298 · Cited by 2,494 · see citing works
Super-resolution through neighbor embedding
Chang, H., Yeung, D., & Xiong, Y. (2004) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2004.1315043 · Cited by 1,995 · see citing works
MemNet: A Persistent Memory Network for Image Restoration
Tai, Y. Y., Yang, J. J., Liu, X., & Xu, C. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.486 · Cited by 1,940 · see citing works
Super-resolution from a single image
Glasner, D., Bagon, S., & Irani, M. (2009) · 2009 IEEE 12th International Conference on Computer Vision · doi:10.1109/iccv.2009.5459271 · Cited by 1,889 · see citing works
Freeman, W. T., Pasztor, E. C., & Carmichael, O. T. (2000) · International Journal of Computer Vision · doi:10.1023/a:1026501619075 · Cited by 1,504 · see citing works
A+: Adjusted Anchored Neighborhood Regression for Fast Super-Resolution
Timofte, R., Smet, V. D., & Gool, L. V. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-16817-3_8 · Cited by 1,470 · see citing works
Training Very Deep NetworksPreprint
Srivastava, R. K., Greff, K., & Schmidhuber, J. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1507.06228 · Cited by 1,453 · see citing works
Anchored Neighborhood Regression for Fast Example-Based Super-Resolution
Timofte, R., De, V., & Gool, L. V. (2013) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2013.241 · Cited by 1,373 · see citing works
Dong, W., Lei Zhang, A., Shi, G., & Wu, X. (2011) · IEEE Transactions on Image Processing · doi:10.1109/tip.2011.2108306 · Cited by 1,162 · see citing works
Dong, W., Lei Zhang, A., Shi, G., Member, S., Xiaolin Wu, C., & Member, S. (2011) · Cited by 1,023 · see citing works
Image super-resolution using gradient profile prior
Sun, J., Xu, Z., & Shum, H. (2008) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2008.4587659 · Cited by 948 · see citing works
Deep Networks for Image Super-Resolution with Sparse Prior
Wang, Z., Liu, D., Yang, J., Han, W., & Huang, T. S. (2015) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.50 · Cited by 807 · see citing works
Coupled Dictionary Training for Image Super-Resolution
Yang, J., Wang, Z., Lin, Z., Cohen, S., & Huang, T. S. (2012) · IEEE Transactions on Image Processing · doi:10.1109/tip.2012.2192127 · Cited by 786 · see citing works
Generalizing the Nonlocal-Means to Super-Resolution Reconstruction
Protter, M., Elad, M., Takeda, H., & Milanfar, P. (2008) · IEEE Transactions on Image Processing · doi:10.1109/tip.2008.2008067 · Cited by 709 · see citing works
Image and video upscaling from local self-examples
Freedman, G., & Fattal, R. (2011) · ACM Transactions on Graphics · doi:10.1145/1944846.1944852 · Cited by 682 · see citing works
Fast and accurate image upscaling with super-resolution forests
Schulter, S., Leistner, C., & Bischof, H. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7299003 · Cited by 633 · see citing works
Single Image Super-Resolution With Non-Local Means and Steering Kernel Regression
Zhang, K., Gao, X., Tao, D., & Li, X. (2012) · IEEE Transactions on Image Processing · doi:10.1109/tip.2012.2208977 · Cited by 578 · see citing works
Convolutional Sparse Coding for Image Super-Resolution
Gu, S., Zuo, W., Xie, Q., Meng, D., Feng, X., & Lei Zhang, A. (2015) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.212 · Cited by 352 · see citing works
Fast Direct Super-Resolution by Simple Functions
Yang, C.-Y., & Yang, M. (2013) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2013.75 · Cited by 336 · see citing works
Super-Resolution with Deep Convolutional Sufficient StatisticsPreprint
Bruna, J., Sprechmann, P., & LeCun, Y. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.05666 · Cited by 264 · see citing works
A Comparison of Interpolation Methods for Sparse Data: Application to Wind and Concentration Fields
Goodin, W. R., McRae, G. J., & Seinfeld, J. H. (1979) · Journal of applied meteorology · doi:10.1175/1520-0450(1979)018 · Cited by 208 · see citing works
Fast Image Interpolation via Random Forests
Huang, J., Siu, W.-C., & Liu, T.-R. (2015) · IEEE Transactions on Image Processing · doi:10.1109/tip.2015.2440751 · Cited by 107 · see citing works
Learning Super-Resolution Jointly From External and Internal Examples
Wang, Z., Yang, Y., Wang, Z., Chang, S., Yang, J., & Huang, T. S. (2015) · IEEE Transactions on Image Processing · doi:10.1109/tip.2015.2462113 · Cited by 90 · see citing works
Learning Super-Resolution Jointly From External and Internal Examples
Wang, Z., Member, S., Yang, Y., Wang, Z., Chang, S., Yang, J., Huang, T. S., & Fellow, L. (2016) · Cited by 89 · see citing works
Image Super-Resolution with Fast Approximate Convolutional Sparse Coding
Osendorfer, C., Soyer, H., & van der Smagt, P. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-12643-2_31 · Cited by 71 · see citing works
Ensemble based deep networks for image super-resolution
Wang, L., Huang, Z., Gong, Y., & Pan, C. (2017) · Pattern Recognition · doi:10.1016/j.patcog.2017.02.027 · Cited by 65 · see citing works
Self-citationGUN: Gradual Upsampling Network for Single Image Super-Resolution
Zhao, Y., Li, G., Xie, W., Jia, W., Min, H., & Liu, X. (2018) · IEEE Access · doi:10.1109/access.2018.2855127 · Cited by 64 · see citing works
Flagged first, then by citations
Research works printed but not linked: 4
- J. Salvador and E. Perez-Pellitero, ''Naïve Bayes super-resulution forest,'' in Proc. IEEE Int. Conf. Comput. Vis., Dec. 2015, pp. 325-333.
- C. E. Duchon, ''Lanczos filtering in one and two dimensions,'' J. Appl. Meteorol., vol. 18, no. 8, pp. 1016-1022, 1979.
- D. R. Martin, C. C. Fowlkes, D. Tal, and J. Malik, ''A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics,'' in Proc. IEEE Int. Conf. Comput. Vis., Jul. 2001, pp. 416-425.
- PEIHUA LI received the Ph.D. degree from the Harbin Institute of Technology, in 2002. He was a recipient of the honorary nomination of National Excellent Doctoral Dissertation, in 2005. He was supported by the Program for New Century Excel- lent Talents in University of the Ministry of Educa- tion of China, in 2011. He is currently a Professor with the School of Information and Communica- tion Engineering, Dalian University of Technol- ogy. He has published over 50 papers in referred conferences and journals. His current research interests include image clas- sification and search using theoretical and computational methods of infor- mation geometry.
Cited by
Sorted by citations
Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network
Ledig, C., Theis, L., Huszar, F., Caballero, J., Cunningham, A., Acosta, A., Aitken, A., Tejani, A., Totz, J., Wang, Z., et al. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.19 · Cited by 12,696 · see citing works
Review of deep learning: concepts, CNN architectures, challenges, applications, future directions
Alzubaidi, L., Zhang, J., Humaidi, A. J., Al-Dujaili, A., Duan, Y., Al-Shamma, O., Santamaría, J., Fadhel, M. A., Al-Amidie, M., & Farhan, L. (2021) · Journal of Big Data · doi:10.1186/s40537-021-00444-8 · Cited by 7,906 · see citing works
A multiscale and multidepth convolutional neural network for remote sensing imagery pan-sharpening
Yuan, Q., Wei, Y., Meng, X., Shen, H., & Zhang, L. (2018) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2018.2794888 · Cited by 649 · see citing works
- Authors
- Wang, Y., Wang, L., Wang, H., & Li, P.
- Affiliations
- Dalian University of Technology (CN)
- Venue
- IEEE Access
- Published
- 2019
- Volume
- 7
- Pages
- 31959-31970
- ISSN
- 2169-3536 (ISSN-L)
- OpenAlex
- W2963103155
- Cited by
- 143
- References
- 50
- Citations by year
- 2026: 1 · 2025: 9 · 2024: 8 · 2023: 13 · 2022: 10 · 2021: 22 · 2020: 30 · 2019: 22 · 2018: 22 · 2017: 5 · 2016: 1
- FWCI
- 9.39
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- IEEE Access (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- arXiv (Cornell University)
- Funders
- National Natural Science Foundation of China (61471082, 61671103)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Image Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image Enhancement Techniques
- Keywords
- image super-resolution, end-to-end learning, network ensembles, upsampling, high-frequency detail reconstruction, multi-scale reconstruction, feature extraction
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
