The DOI matches its Crossref record; 36 of the 36 references it cites were found in a registry or an index.
Deep Convolutional Framelets: A General Deep Learning Framework for Inverse Problems
Open accessAbstract
Recently, deep learning approaches with various network architectures have achieved significant performance improvement over existing iterative reconstruction methods in various imaging problems. However, it is still unclear why these deep learning architectures work for specific inverse problems. Moreover, in contrast to the usual evolution of signal processing theory around the classical theories, the link between deep learning and the classical signal processing approaches, such as wavelets, nonlocal processing, and compressed sensing, are not yet well understood. To address these issues, here we show that the long-sought missing link is the convolution framelets for representing a signal by convolving local and nonlocal bases. The convolution framelets were originally developed to generalize the theory of low-rank Hankel matrix approaches for inverse problems, and this paper further extends this idea so as to obtain a deep neural network using multilayer convolution framelets with perfect reconstruction (PR) under rectified linear unit (ReLU) nonlinearity. Our analysis also shows that the popular deep network components such as residual blocks, redundant filter channels, and concatenated ReLU (CReLU) do indeed help to achieve PR, while the pooling and unpooling layers should be augmented with high-pass branches to meet the PR condition. Moreover, by changing the number of filter channels and bias, we can control the shrinkage behaviors of the neural network. This discovery reveals the limitations of many existing deep learning architectures for inverse problems, and leads us to propose a novel theory for a deep convolutional framelet neural network. Using numerical experiments with various inverse problems, we demonstrate that our deep convolutional framelets network shows consistent improvement over existing deep architectures. This discovery suggests that the success of deep learning stems not from a magical black box, but rather from the power of a novel signal representation using a nonlocal basis combined with a data-driven local basis, which is indeed a natural extension of classical signal processing theory.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
Image quality assessment: from error visibility to structural similarity
Wang, Z., Bovik, A. C., Sheikh, H. R., & Simoncelli, E. P. (2004) · IEEE Transactions on Image Processing · doi:10.1109/tip.2003.819861 · Cited by 57,892
Donoho, D. L. (2006) · IEEE Transactions on Information Theory · doi:10.1109/tit.2006.871582 · Cited by 23,994
Daubechies, I. (1992) · doi:10.1137/1.9781611970104 · Cited by 15,979
Candes, E. J., Romberg, J., & Tao, T. (2006) · IEEE Transactions on Information Theory · doi:10.1109/tit.2005.862083 · Cited by 15,639
Image denoising by sparse 3-D transform-domain collaborative filtering
Dabov, K., Foi, A., Katkovnik, V., & Egiazarian, K. (2007) · IEEE Transactions on Image Processing · doi:10.1109/tip.2007.901238 · Cited by 9,283
Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising
Zhang, K., Zuo, W., Chen, Y., Meng, D., & Zhang, L. (2017) · IEEE Transactions on Image Processing · doi:10.1109/tip.2017.2662206 · Cited by 8,983
A Non-Local Algorithm for Image Denoising
Buades, A., Coll, B., & Morel, J.-M. (2005) · 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) · doi:10.1109/cvpr.2005.38 · Cited by 7,616 · cited 1×
“Non-local image processing techniques such as non-local means [8], BM3D [16], etc have also demonstrated impressive performance for many image processing applications.”
BackgroundContext Encoders: Feature Learning by Inpainting
Pathak, D., Krahenbuhl, P., Donahue, J., Darrell, T., & Efros, A. A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.278 · Cited by 5,691
Ahmed, N., Natarajan, T., & Rao, K. R. (1974) · IEEE Transactions on Computers · doi:10.1109/t-c.1974.223784 · Cited by 5,156
Understanding deep learning requires rethinking generalization
Zhang, C., Bengio, S., Hardt, M., Recht, B., & Vinyals, O. (2016) · arXiv · doi:10.48550/arxiv.1611.03530 · Cited by 2,838
Deep Convolutional Neural Network for Inverse Problems in Imaging
Jin, K. H., McCann, M. T., Froustey, E., & Unser, M. (2017) · IEEE Transactions on Image Processing · doi:10.1109/tip.2017.2713099 · Cited by 2,529 · cited 7×
“For example, in image denoising [40, 76] or streaking artifact problems [28, 35], a residual network works better, since the artifacts are random and have complicated distribution.”
BackgroundMethodInfluentialSidky, E. Y., & Pan, X. (2008) · Physics in Medicine and Biology · doi:10.1088/0031-9155/53/17/021 · Cited by 1,995
Image reconstruction by domain-transform manifold learning
Zhu, B., Liu, J. Z., Cauley, S. F., Rosen, B. R., & Rosen, M. S. (2018) · Nature · doi:10.1038/nature25988 · Cited by 1,694
Invariant Scattering Convolution Networks
Bruna, J., & Mallat, S. (2013) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2012.230 · Cited by 1,633
Matrix pencil method for estimating parameters of exponentially damped/undamped sinusoids in noise
Hua, Y., & Sarkar, T. K. (1990) · IEEE Transactions on Acoustics, Speech, and Signal Processing · doi:10.1109/29.56027 · Cited by 1,537
Sampling signals with finite rate of innovation
Vetterli, M., Marziliano, P., & Blu, T. (2002) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2002.1003065 · Cited by 1,079
Blind identification and equalization based on second-order statistics: a time domain approach
Tong, L., Xu, G., & Kailath, T. (1994) · IEEE Transactions on Information Theory · doi:10.1109/18.312157 · Cited by 1,078
Mallat, S. (2012) · Communications on Pure and Applied Mathematics · doi:10.1002/cpa.21413 · Cited by 1,038
Framelets: MRA-based constructions of wavelet frames
Daubechies, I., Han, B., Ron, A., & Shen, Z. (2003) · Applied and Computational Harmonic Analysis · doi:10.1016/s1063-5203(02)00511-0 · Cited by 752
Deep Pyramidal Residual Networks
Han, D., Kim, J., & Kim, J. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.668 · Cited by 715
A deep convolutional neural network using directional wavelets for low‐dose X‐ray CT reconstruction
Kang, E., Min, J., & Ye, J. C. (2017) · Medical Physics · doi:10.1002/mp.12344 · Cited by 709
Self-citationSplit Bregman Methods and Frame Based Image Restoration
Cai, J.-F., Osher, S., & Shen, Z. (2010) · Multiscale Modeling & Simulation · doi:10.1137/090753504 · Cited by 670
Hankel Matrix Rank Minimization with Applications to System Identification and Realization
Fazel, M., Pong, T. K., Sun, D., & Tseng, P. (2013) · SIAM Journal on Matrix Analysis and Applications · doi:10.1137/110853996 · Cited by 487
A Mathematical Theory of Deep Convolutional Neural Networks for Feature Extraction
Wiatowski, T., & Bolcskei, H. (2018) · IEEE Transactions on Information Theory · doi:10.1109/tit.2017.2776228 · Cited by 399
Benefits of Depth in Neural Networks
Telgarsky, M. (2016) · arXiv · doi:10.48550/arxiv.1602.04485 · Cited by 341
Image restoration: Total variation, wavelet frames, and beyond
Cai, J.-F., Dong, B., Osher, S., & Shen, Z. (2012) · Journal of the American Mathematical Society · doi:10.1090/s0894-0347-2012-00740-1 · Cited by 339
A framelet-based image inpainting algorithm
Cai, J.-F., Chan, R. H., & Shen, Z. (2008) · Applied and Computational Harmonic Analysis · doi:10.1016/j.acha.2007.10.002 · Cited by 338
Jin, K. H., Lee, D., & Ye, J. C. (2016) · IEEE Transactions on Computational Imaging · doi:10.1109/tci.2016.2601296 · Cited by 245
Self-citationAnnihilating Filter-Based Low-Rank Hankel Matrix Approach for Image Inpainting
Jin, K. H., & Ye, J. C. (2015) · IEEE Transactions on Image Processing · doi:10.1109/tip.2015.2446943 · Cited by 177
Self-citationA parallel MR imaging method using multilayer perceptron
Kwon, K., Kim, D., & Park, H. (2017) · Medical Physics · doi:10.1002/mp.12600 · Cited by 164
Off-the-Grid Recovery of Piecewise Constant Images from Few Fourier Samples
Ongie, G., & Jacob, M. (2016) · SIAM Journal on Imaging Sciences · doi:10.1137/15m1042280 · Cited by 134
Acceleration of MR parameter mapping using annihilating filter‐based low rank hankel matrix (ALOHA)
Lee, D., Jin, K. H., Kim, E. Y., Park, S., & Ye, J. C. (2016) · Magnetic Resonance in Medicine · doi:10.1002/mrm.26081 · Cited by 110
Self-citationCompressive Sampling Using Annihilating Filter-Based Low-Rank Interpolation
Ye, J. C., Kim, J., Jin, K. H., & Lee, K. (2016) · IEEE Transactions on Information Theory · doi:10.1109/tit.2016.2629078 · Cited by 77
Self-citationLee, J., Jin, K. H., & Ye, J. C. (2016) · Magnetic Resonance in Medicine · doi:10.1002/mrm.26077 · Cited by 68
Self-citationA Tale of Two Bases: Local-Nonlocal Regularization on Image Patches with Convolution Framelets
Yin, R., Gao, T., Lu, Y., & Daubechies, I. C. (2017) · SIAM Journal on Imaging Sciences · doi:10.1137/16m1091447 · Cited by 53
Cited by
Deep Learning on Image Denoising: An overview
Tian, C., Fei, L., Zheng, W., Xu, Y., Zuo, W., & Lin, C.-W. (2020) · Neural Networks · doi:10.1016/j.neunet.2020.07.025 · Cited by 1,091
Multi-level Wavelet-CNN for Image Restoration
Liu, P., Zhang, H., Zhang, K., Lin, L., & Zuo, W. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2018.00121 · Cited by 933
Solving inverse problems using data-driven models
Arridge, S., Maass, P., Öktem, O., & Schönlieb, C.-B. (2019) · Acta Numerica · doi:10.1017/s0962492919000059 · Cited by 686
Framing U-Net via Deep Convolutional Framelets: Application to Sparse-View CT
Han, Y., & Ye, J. C. (2018) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2018.2823768 · Cited by 588
Self-citationDeep Learning Techniques for Inverse Problems in Imaging
Ongie, G., Jalal, A., Metzler, C. A., Baraniuk, R. G., Dimakis, A. G., & Willett, R. (2020) · IEEE Journal on Selected Areas in Information Theory · doi:10.1109/jsait.2020.2991563 · Cited by 577
Metadata
- Authors
- Ye, J. C., Han, Y., & Cha, E.
- Venue
- SIAM Journal on Imaging Sciences
- Published
- 2018-01
- Volume
- 11
- Issue
- 2
- Pages
- 991-1048
- ISSN
- 1936-4954 (ISSN-L)
- OpenAlex
- W2777802649
- Cited by
- 330
- References
- 36
- FWCI
- 25.42
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.