FlexISP: a flexible camera image processing framework
The DOI matches its Crossref record; 57 of the 57 references it cites were found in a registry or an index.
Abstract
Conventional pipelines for capturing, displaying, and storing images are usually defined as a series of cascaded modules, each responsible for addressing a particular problem. While this divide-and-conquer approach offers many benefits, it also introduces a cumulative error, as each step in the pipeline only considers the output of the previous step, not the original sensor data. We propose an end-to-end system that is aware of the camera and image model, enforces natural-image priors, while jointly accounting for common image processing steps like demosaicking, denoising, deconvolution, and so forth, all directly in a given output representation (e.g., YUV, DCT). Our system is flexible and we demonstrate it on regular Bayer images as well as images from custom sensors. In all cases, we achieve large improvements in image quality and signal reconstruction compared to state-of-the-art techniques. Finally, we show that our approach is capable of very efficiently handling high-resolution images, making even mobile implementations feasible.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“particularly, the regularization term only corresponds to a denoising subproblem [18,31,61].”
From Learning Deep CNN Denoiser Prior for Image Restoration (2017) · cites this paper as [31]
Citations
Cites
Nonlinear total variation based noise removal algorithms
Rudin, L. I., Osher, S., & Fatemi, E. (1992) · Physica D: Nonlinear Phenomena · doi:10.1016/0167-2789(92)90242-f · Cited by 15,704
Image denoising by sparse 3-D transform-domain collaborative filtering
Dabov, K., Foi, A., Katkovnik, V., & Egiazarian, K. (2007) · IEEE Transactions on Image Processing · doi:10.1109/tip.2007.901238 · Cited by 9,283
Self-citationA Non-Local Algorithm for Image Denoising
Buades, A., Coll, B., & Morel, J.-M. (2005) · 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) · doi:10.1109/cvpr.2005.38 · Cited by 7,616 · cited 14×
“Non-local means (NLM) [Buades et al. 2005] filters a single image using a weighted average of similar patches for each pixel.”
BackgroundMethodInfluentialA First-Order Primal-Dual Algorithm for Convex Problems with Applications to Imaging
Chambolle, A., & Pock, T. (2011) · Journal of Mathematical Imaging and Vision · doi:10.1007/s10851-010-0251-1 · Cited by 4,336
Parikh, N., & Boyd, S. (2014) · Foundations and Trends® in Optimization · doi:10.1561/2400000003 · Cited by 4,102
The JPEG still picture compression standard
Wallace, G. K. (1991) · Communications of the ACM · doi:10.1145/103085.103089 · Cited by 3,784
Improving resolution by image registration
Irani, M., & Peleg, S. (1991) · CVGIP: Graphical Models and Image Processing · doi:10.1016/1049-9652(91)90045-l · Cited by 1,929
From learning models of natural image patches to whole image restoration
Zoran, D., & Weiss, Y. (2011) · 2011 International Conference on Computer Vision · doi:10.1109/iccv.2011.6126278 · Cited by 1,522
Fast Image Deconvolution using Hyper-Laplacian Priors
Krishnan, D., & Fergus, R. (2009) · neural information processing systems · Cited by 1,470
High Dynamic Range Imaging: Acquisition, Display, and Image-Based Lighting
Reinhard, E., Ward, G., Pattanaik, S., Debevec, P. E., Heidrich, W., & Myszkowski, K. (2010) · Cited by 1,359
Self-citationCurvelets: A Surprisingly Effective Nonadaptive Representation for Objects with Edges
Candès, E. J., & Donoho, D. L. (2000) · Cited by 1,330
Fast Image Recovery Using Variable Splitting and Constrained Optimization
Afonso, M. V., Bioucas-Dias, J. M., & Figueiredo, M. A. T. (2010) · IEEE Transactions on Image Processing · doi:10.1109/tip.2010.2047910 · Cited by 1,223 · cited 3×
“…and the related Alternating Direction Method of Multipliers (ADMM), have been used in many imaging applications, such as deconvolution, inpainting [Afonso et al. 2010], superresolution, optical flow [Chambolle and Pock 2011], denoising [Venkatakrishnan et al. 2013], and deconvolution for simple…”
InfluentialPlug-and-Play priors for model based reconstruction
Venkatakrishnan, S. V., Bouman, C. A., & Wohlberg, B. (2013) · 2013 IEEE Global Conference on Signal and Information Processing · doi:10.1109/globalsip.2013.6737048 · Cited by 1,178 · cited 3×
“2013] PD DC TV + cross-channel [Venkatakrishnan et al. 2013] ADMM Tomographic DN KSVD | BM3D | TV | .”
Buades, A., Coll, B., & Morel, J.-M. (2011) · Image Processing On Line · doi:10.5201/ipol.2011.bcm_nlm · Cited by 1,090
High performance imaging using large camera arrays
Wilburn, B., Joshi, N., Vaish, V., Talvala, E.-V., Antunez, E., Barth, A., Adams, A., Horowitz, M., & Levoy, M. (2005) · ACM Transactions on Graphics · doi:10.1145/1073204.1073259 · Cited by 1,042
Two-Phase Kernel Estimation for Robust Motion Deblurring
Xu, L., & Jia, J. (2010) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-15549-9_12 · Cited by 1,027
Image and depth from a conventional camera with a coded aperture
Levin, A., Fergus, R., Durand, F., & Freeman, W. T. (2007) · ACM Transactions on Graphics · doi:10.1145/1276377.1276464 · Cited by 980
Roth, S., & Black, M. J. (2009) · International Journal of Computer Vision · doi:10.1007/s11263-008-0197-6 · Cited by 847
Beyond pixels: exploring new representations and applications for motion analysis
Freeman, W. T., Adelson, E. H., & Liu, C. (2009) · Cited by 805
Pointwise Shape-Adaptive DCT for High-Quality Denoising and Deblocking of Grayscale and Color Images
Foi, A., Katkovnik, V., & Egiazarian, K. (2007) · IEEE Transactions on Image Processing · doi:10.1109/tip.2007.891788 · Cited by 725
Self-citationBM3D Frames and Variational Image Deblurring
Danielyan, A., Katkovnik, V., & Egiazarian, K. (2012) · IEEE Transactions on Image Processing · doi:10.1109/tip.2011.2176954 · Cited by 611
Self-citationColor demosaicking by local directional interpolation and nonlocal adaptive thresholding
Wu, X. (2011) · Journal of Electronic Imaging · doi:10.1117/1.3600632 · Cited by 544
SIFT Flow: Dense Correspondence across Different Scenes
Liu, C., Yuen, J., Torralba, A., Sivic, J., & Freeman, W. T. (2008) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-88690-7_3 · Cited by 540
Image Processing and Data Analysis: The Multiscale Approach
Starck, J.-L., Murtagh, F. D., & Bijaoui, A. (1998) · Cambridge University Press eBooks · doi:10.1017/cbo9780511564352 · Cited by 538
Image Processing and Data Analysis, The Multiscale Approach
Seymour, L., Starck, J. L., Murtagh, F., & Bijaoui, A. (1999) · Journal of the American Statistical Association · doi:10.2307/2669962 · Cited by 467
Image restoration by sparse 3D transform-domain collaborative filtering
Dabov, K., Foi, A., Katkovnik, V., & Egiazarian, K. (2008) · SPIE Proceedings · doi:10.1117/12.766355 · Cited by 461
Self-citation1 BM3D frames and variational image deblurring
Danielyan, A., Katkovnik, V., Egiazarian, K., & Member, S. (2012) · Cited by 414
Self-citationShao, L., Yan, R., Li, X., & Liu, Y. (2014) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2013.2278548 · Cited by 389
Video Denoising By Sparse 3D Transform-Domain Collaborative Filtering
Dabov, K., Foi, A., & Egiazarian, K. (2007) · doi:10.5281/zenodo.40233 · Cited by 373
Self-citationColor image processing pipeline
Ramanath, R., Snyder, W. E., Yoo, Y., & Drew, M. S. (2005) · IEEE Signal Processing Magazine · doi:10.1109/msp.2005.1407713 · Cited by 369 · cited 2×
“The complexity of this process is traditionally tackled by splitting the image processing into several independent pipeline stages [Ramanath et al. 2005].”
MethodA Machine Learning Approach for Non-blind Image Deconvolution
Schuler, C. J., Burger, H. C., Harmeling, S., & Scholkopf, B. (2013) · 2013 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2013.142 · Cited by 319 · cited 1×
“…priors: EPLL (Gaussian mixture models) [Zoran and Weiss 2011], heavy-tailed gradient distributions [Levin et al. 2007; Krishnan and Fergus 2009], self-similarity [Danielyan et al. 2012; Dabov et al. 2008], Field-of-Experts [Roth and Black 2009], and neural networks [Schuler et al. 2013].”
BackgroundMultiframe demosaicing and super-resolution of color images
Farsiu, S., Elad, M., & Milanfar, P. (2006) · IEEE Transactions on Image Processing · doi:10.1109/tip.2005.860336 · Cited by 319
PiCam: an ultra-thin high performance monolithic camera array
Venkataraman, K., Lelescu, D., Duparré, J., McMahon, A., Molina, G., Chatterjee, P., Mullis, R., & Nayar, S. (2013) · ACM Transactions on Graphics · doi:10.1145/2508363.2508390 · Cited by 226
High-quality computational imaging through simple lenses
Heide, F., Rouf, M., Hullin, M. B., Labitzke, B., Heidrich, W., & Kolb, A. (2013) · ACM Transactions on Graphics · doi:10.1145/2516971.2516974 · Cited by 216
Self-citationAdaptive dynamic range imaging: optical control of pixel exposures over space and time
Nayar, & Branzoi (2003) · Proceedings Ninth IEEE International Conference on Computer Vision · doi:10.1109/iccv.2003.1238624 · Cited by 172
Coded rolling shutter photography: Flexible space-time sampling
Gu, J., Hitomi, Y., Mitsunaga, T., & Nayar, S. K. (2010) · IEEE International Conference on Computational Photography (ICCP) · doi:10.1109/iccphot.2010.5585094 · Cited by 153
European Signal Processing Conference
Raitio, T., Lu, H. Z., Kane, J., Suni, A., Vainio, M., King, S., & Alku, P. (2014) · Cited by 135
The Frankencamera: an experimental platform for computational photography
Adams, A., Talvala, E.-V., Park, S. H., Jacobs, D. E., Ajdin, B., Gelfand, N., Dolson, J., Vaquero, D., Baek, J., Tico, M., et al. (2010) · ACM Transactions on Graphics · doi:10.1145/1778765.1778766 · Cited by 134
Self-citationNon-stationary correction of optical aberrations
Schuler, C. J., Hirsch, M., Harmeling, S., & Schölkopf, B. (2011) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2011.6126301 · Cited by 124
Combining the power of Internal and External denoising
Mosseri, I., Zontak, M., & Irani, M. (2013) · IEEE International Conference on Computational Photography (ICCP) · doi:10.1109/iccphot.2013.6528298 · Cited by 106
A note on multi-image denoising
Buades, T., Lou, Y., Morel, J., & Tang, Z. (2009) · International Workshop on Local and Non-Local Approximation in Image Processing (LNLA) · doi:10.1109/lnla.2009.5278408 · Cited by 95
Sharp MSE Bounds for Proximal Denoising
Oymak, S., & Hassibi, B. (2015) · Foundations of Computational Mathematics · doi:10.1007/s10208-015-9278-4 · Cited by 92
Enhancing resolution along multiple imaging dimensions using assorted pixels
Narasimhan, S. G., & Nayar, S. K. (2005) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2005.76 · Cited by 78
Noise suppression in low-light images through joint denoising and demosaicing
Chatterjee, P., Joshi, N., Kang, S. B., & Matsushita, Y. (2011) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2011.5995371 · Cited by 69
Direct PSF estimation using a random noise target
Brauers, J., Seiler, C., & Aach, T. (2009) · Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE · doi:10.1117/12.837591 · Cited by 51
Jeon, G., & Dubois, É. (2012) · IEEE Transactions on Image Processing · doi:10.1109/tip.2012.2214041 · Cited by 50
Image enhancement method via blur and noisy image fusion
Tico, M., & Pulli, K. (2009) · Proceedings - International Conference on Image Processing · doi:10.1109/icip.2009.5413626 · Cited by 47
Self-citationMitra, K., Cossairt, O. S., & Veeraraghavan, A. (2014) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2014.2313118 · Cited by 40
Adaptive denoising and lossy compression of images in transform domain
Egiazarian, K., Astola, J. T., Helsingius, M. P., & Kuosmanen, P. (1999) · Journal of Electronic Imaging · doi:10.1117/1.482673 · Cited by 35
Self-citationMulti-Frame Image Denoising And Stabilization
Tico, M. (2008) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.40794 · Cited by 29
Video and Image Bayesian Demosaicing with a Two Color Image Prior
Bennett, E. P., Uyttendaele, M., Zitnick, C. L., Szeliski, R., & Kang, S. B. (2006) · Lecture notes in computer science · doi:10.1007/11744023_40 · Cited by 27
HDR Reconstruction for Alternating Gain (ISO) Sensor Readout
Hajisharif, S., Kronander, J., & Unger, J. (2014) · Eurographics · doi:10.2312/egsh.20141006 · Cited by 23
Improvement of Color Video Demosaicking in Temporal Domain
Wu, X., & Zhang, L. (2006) · IEEE Transactions on Image Processing · doi:10.1109/tip.2006.877504 · Cited by 20
Recovering Stereo Pairs from Anaglyphs
Joulin, A., & Kang, S. B. (2013) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2013.44 · Cited by 15
Fast ANN for High‐Quality Collaborative Filtering
Tsai, Y., Steinberger, M., Pająk, D., & Pulli, K. (2015) · Computer Graphics Forum · doi:10.1111/cgf.12715 · Cited by 10
Self-citationDemosaicing by smoothing along 1D features
Ajdin, B., Hullin, M. B., Fuchs, C., Seidel, H., & Lensch, H. P. A. (2008) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2008.4587653 · Cited by 8
Fast ANN for High-Quality Collaborative Filtering
Tsai, Y., Steinberger, M., Pająk, D., et al. (2014) · Eurographics · doi:10.2312/hpg.20141094 · Cited by 5
Self-citation
Research works printed but not linked: 1
- Coifman , R. R. , and Donoho , D. L . 1995 . Translation-invariant de noising. Tech. Rep. 475 , Stanford University . Coifman, R. R., and Donoho, D. L. 1995. Translation-invariant denoising. Tech. Rep. 475, Stanford University.
Cited by
FFDNet: Toward a Fast and Flexible Solution for CNN-Based Image Denoising
Zhang, K., Zuo, W., & Zhang, L. (2018) · IEEE Transactions on Image Processing · doi:10.1109/tip.2018.2839891 · Cited by 2,678
Loss Functions for Image Restoration With Neural Networks
Zhao, H., Gallo, O., Frosio, I., & Kautz, J. (2017) · IEEE Transactions on Computational Imaging · doi:10.1109/tci.2016.2644865 · Cited by 2,641
Self-citationLearning Deep CNN Denoiser Prior for Image Restoration
Zhang, K., Zuo, W., Gu, S., & Zhang, L. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.300 · Cited by 2,116
“In [31], the authors used an alternative to ADMM and HQS, i.e ., the primal-dual algorithm [11], to decouple fidelity term and regularization term.”
MethodPlug-and-Play Image Restoration With Deep Denoiser Prior
Zhang, K., Li, Y., Zuo, W., Zhang, L., Van Gool, L., & Timofte, R. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3088914 · Cited by 1,113
Real Image Denoising With Feature Attention
Anwar, S., & Barnes, N. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00325 · Cited by 784
Metadata
- Authors
- Heide, F., Steinberger, M., Tsai, Y.-T., Rouf, M., Pająk, D., Reddy, D., Gallo, O., Liu, J., Heidrich, W., Egiazarian, K., Kautz, J., & Pulli, K.
- Affiliations
- University of British Columbia (CA)
- Graz University of Technology (AT)
- Nvidia (United Kingdom) (GB)
- University of California, Santa Cruz (US)
- NVIDIA and TUT
- Venue
- ACM Transactions on Graphics
- Published
- 2014-11-19
- Volume
- 33
- Issue
- 6
- Pages
- 1-13
- ISSN
- 0730-0301 (ISSN-L)
- 1557-7368
- OpenAlex
- W2021347102
- Cited by
- 298
- References
- 56
- Citations by year
- 2026: 10 · 2025: 18 · 2024: 25 · 2023: 35 · 2022: 26 · 2021: 48 · 2020: 31 · 2019: 39 · 2018: 26 · 2017: 16 · 2016: 12 · 2015: 12
- FWCI
- 11.87
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- ACM Transactions on Graphics (publishedVersion)
- King Abdullah University of Science and Technology Repository (King Abdullah University of Science and Technology) (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Image Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image Enhancement Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image and Signal Denoising Methods
- Keywords
- natural image priors, camera model, demosaicing, image denoising, deconvolution, YUV, discrete cosine transform, high-resolution imagery, mobile image processing, signal reconstruction
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.