The DOI matches its Crossref record; 49 of the 49 references it cites were found in a registry or an index.
End-to-end optimization of optics and image processing for achromatic extended depth of field and super-resolution imaging
Abstract
In typical cameras the optical system is designed first; once it is fixed, the parameters in the image processing algorithm are tuned to get good image reproduction. In contrast to this sequential design approach, we consider joint optimization of an optical system (for example, the physical shape of the lens) together with the parameters of the reconstruction algorithm. We build a fully-differentiable simulation model that maps the true source image to the reconstructed one. The model includes diffractive light propagation, depth and wavelength-dependent effects, noise and nonlinearities, and the image post-processing. We jointly optimize the optical parameters and the image processing algorithm parameters so as to minimize the deviation between the true and reconstructed image, over a large set of images. We implement our joint optimization method using autodifferentiation to efficiently compute parameter gradients in a stochastic optimization algorithm. We demonstrate the efficacy of this approach by applying it to achromatic extended depth of field and snapshot super-resolution imaging.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Principles of Optics: Electromagnetic Theory of Propagation, Interference and Diffraction of Light
Born, M., Wolf, E., Bhatia, A. B., Clemmow, P. C., Gabor, D., Stokes, A. R., Taylor, A. M., Wayman, P. A., & Wilcock, W. L. (1999) · doi:10.1017/cbo9781139644181 · Cited by 26,290
Image Super-Resolution Using Deep Convolutional Networks
Dong, C., Loy, C. C., He, K., & Tang, X. (2016) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2015.2439281 · Cited by 10,105
Martin, D., Fowlkes, C., Tal, D., & Malik, J. (2002) · Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001 · doi:10.1109/iccv.2001.937655 · Cited by 8,216
Accurate Image Super-Resolution Using Very Deep Convolutional Networks
Kim, J., Lee, J. K., & Lee, K. M. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.182 · Cited by 7,870
Shi, W., Caballero, J., Huszar, F., Totz, J., Aitken, A. P., Bishop, R., Rueckert, D., & Wang, Z. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.207 · Cited by 7,570
ADADELTA: An Adaptive Learning Rate Method
Zeiler, M. D. (2012) · arXiv · doi:10.48550/arxiv.1212.5701 · Cited by 5,322
Accelerating the Super-Resolution Convolutional Neural Network
Dong, C., Loy, C. C., & Tang, X. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46475-6_25 · Cited by 3,786
Single image super-resolution from transformed self-exemplars
Huang, J.-B., Singh, A., & Ahuja, N. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7299156 · Cited by 3,472
Deeply-Recursive Convolutional Network for Image Super-Resolution
Kim, J., Lee, J. K., & Lee, K. M. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.181 · Cited by 3,025
Deep Laplacian Pyramid Networks for Fast and Accurate Super-Resolution
Lai, W.-S., Huang, J.-B., Ahuja, N., & Yang, M.-H. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.618 · Cited by 2,976
A method for solving the convex programming problem with convergence rate O(1/k^2)
Nesterov, Y. E. (1983) · Доклады Академии наук · Cited by 2,662
Recovering high dynamic range radiance maps from photographs
Debevec, P. E., & Malik, J. (2008) · ACM SIGGRAPH 2008 classes · doi:10.1145/1401132.1401174 · Cited by 2,399
Hamming Embedding and Weak Geometric Consistency for Large Scale Image Search
Jegou, H., Douze, M., & Schmid, C. (2008) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-88682-2_24 · Cited by 1,901 · cited 1×
“We then sample images from a dataset of 1,244 highresolution images [Jegou et al. 2008] and assign one of the five depth settings randomly while considering all three wavelengths simultaneously.”
MethodA broadband achromatic metalens for focusing and imaging in the visible
Chen, W. T., Zhu, A. Y., Sanjeev, V., Khorasaninejad, M., Shi, Z., Lee, E., & Capasso, F. (2018) · Nature Nanotechnology · doi:10.1038/s41565-017-0034-6 · Cited by 1,866
Recovering high dynamic range radiance maps from photographs
Debevec, P. E., & Malik, J. (1997) · Proceedings of the 24th annual conference on Computer graphics and interactive techniques - SIGGRAPH '97 · doi:10.1145/258734.258884 · Cited by 1,775
Light Field Photography with a Hand-held Plenoptic Camera
Ng, R., Levoy, M., Brédif, M., Duval, G., Horowitz, M., & Hanrahan, P. (2005) · HAL · Cited by 1,749
A+: Adjusted Anchored Neighborhood Regression for Fast Super-Resolution
Timofte, R., De Smet, V., & Van Gool, L. (2015) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-16817-3_8 · Cited by 1,408
High Dynamic Range Imaging: Acquisition, Display, and Image-Based Lighting
Reinhard, E., Ward, G., Pattanaik, S., Debevec, P. E., Heidrich, W., & Myszkowski, K. (2010) · Cited by 1,359
Self-citationExtended depth of field through wave-front coding
Dowski, E. R., & Cathey, W. T. (1995) · Applied Optics · doi:10.1364/ao.34.001859 · Cited by 1,259
Single disperser design for coded aperture snapshot spectral imaging
Wagadarikar, A., John, R., Willett, R., & Brady, D. (2008) · Applied Optics · doi:10.1364/ao.47.000b44 · Cited by 1,047
Deep Convolutional Neural Network for Image Deconvolution
Li, X., Ren, J., Liu, C., & Jia, J. (2014) · neural information processing systems · Cited by 1,044
Pavani, S. R. P., Thompson, M. A., Biteen, J. S., Lord, S. J., Liu, N., Twieg, R. J., Piestun, R., & Moerner, W. E. (2009) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0900245106 · Cited by 1,040
Image and depth from a conventional camera with a coded aperture
Levin, A., Fergus, R., Durand, F., & Freeman, W. T. (2007) · ACM SIGGRAPH 2007 papers · doi:10.1145/1275808.1276464 · Cited by 1,023
Image and depth from a conventional camera with a coded aperture
Levin, A., Fergus, R., Durand, F., & Freeman, W. T. (2007) · ACM Transactions on Graphics · doi:10.1145/1276377.1276464 · Cited by 980
Deep Networks for Image Super-Resolution with Sparse Prior
Wang, Z., Liu, D., Yang, J., Han, W., & Huang, T. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.50 · Cited by 784
Fast and accurate image upscaling with super-resolution forests
Schulter, S., Leistner, C., & Bischof, H. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7299003 · Cited by 618
HDR image reconstruction from a single exposure using deep CNNs
Eilertsen, G., Kronander, J., Denes, G., Mantiuk, R. K., & Unger, J. (2017) · ACM Transactions on Graphics · doi:10.1145/3130800.3130816 · Cited by 598
Deep high dynamic range imaging of dynamic scenes
Kalantari, N. K., & Ramamoorthi, R. (2017) · ACM Transactions on Graphics · doi:10.1145/3072959.3073609 · Cited by 596
Coded exposure photography: motion deblurring using fluttered shutter
Raskar, R., Agrawal, A., & Tumblin, J. (2006) · ACM Transactions on Graphics · doi:10.1145/1141911.1141957 · Cited by 485
Burst photography for high dynamic range and low-light imaging on mobile cameras
Hasinoff, S. W., Sharlet, D., Geiss, R., Adams, A., Barron, J. T., Kainz, F., Chen, J., & Levoy, M. (2016) · ACM Transactions on Graphics · doi:10.1145/2980179.2980254 · Cited by 483
Veeraraghavan, A., Raskar, R., Agrawal, A., Mohan, A., & Tumblin, J. (2007) · ACM SIGGRAPH 2007 papers · doi:10.1145/1275808.1276463 · Cited by 469
Veeraraghavan, A., Raskar, R., Agrawal, A., Mohan, A., & Tumblin, J. (2007) · ACM Transactions on Graphics · doi:10.1145/1276377.1276463 · Cited by 444
Compressive light field photography using overcomplete dictionaries and optimized projections
Marwah, K., Wetzstein, G., Bando, Y., & Raskar, R. (2013) · ACM Transactions on Graphics · doi:10.1145/2461912.2461914 · Cited by 398
Self-citationOptimal Point Spread Function Design for 3D Imaging
Shechtman, Y., Sahl, S. J., Backer, A. S., & Moerner, W. E. (2014) · Physical Review Letters · doi:10.1103/physrevlett.113.133902 · Cited by 384
Reinhard, E., Ward, G., Pattanaik, S. N., & Debevec, P. E. (2005) · Cited by 372
Multiscale gigapixel photography
Brady, D. J., Gehm, M. E., Stack, R. A., Marks, D. L., Kittle, D. S., Golish, D. R., Vera, E. M., & Feller, S. D. (2012) · Nature · doi:10.1038/nature11150 · Cited by 331
Learning to Synthesize a 4D RGBD Light Field from a Single Image
Srinivasan, P. P., Wang, T., Sreelal, A., Ramamoorthi, R., & Ng, R. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.246 · Cited by 244
Morgan, B. C., Waits, C. M., Krizmanic, J. F., et al. (2004) · Journal of Microelectromechanical Systems · doi:10.1109/jmems.2003.823220 · Cited by 135
Learning Sensor Multiplexing Design through Back-propagationPreprint
Chakrabarti, A. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.07078 · Cited by 129
Light field video capture using a learning-based hybrid imaging system
Wang, T.-C., Zhu, J.-Y., Kalantari, N. K., Efros, A. A., & Ramamoorthi, R. (2017) · ACM Transactions on Graphics · doi:10.1145/3072959.3073614 · Cited by 120
Learning High Dynamic Range from Outdoor Panoramas
Zhang, J., & Lalonde, J. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.484 · Cited by 114
Achromatic hybrid refractive-diffractive lens with extended depth of focus
Flores, A. S., Wang, M. R., & Yang, J. J. (2004) · Applied Optics · doi:10.1364/ao.43.005618 · Cited by 106
The diffractive achromat full spectrum computational imaging with diffractive optics
Peng, Y., Fu, Q., Heide, F., & Heidrich, W. (2016) · ACM Transactions on Graphics · doi:10.1145/2897824.2925941 · Cited by 98
Self-citationDeepBinaryMask: Learning a binary mask for video compressive sensing
Iliadis, M., Spinoulas, L., & Katsaggelos, A. K. (2019) · Digital Signal Processing · doi:10.1016/j.dsp.2019.102591 · Cited by 98
Computational Plenoptic Imaging
Wetzstein, G., Ihrke, I., Lanman, D. R., & Heidrich, W. (2011) · Computer Graphics Forum · doi:10.1111/j.1467-8659.2011.02073.x · Cited by 86
Self-citationComputational Cameras: Redefining the Image
Nayar, S. K. (2006) · Computer · doi:10.1109/mc.2006.258 · Cited by 70
Single-shot diffuser-encoded light field imaging
Antipa, N., Sylvia, N., Ng, R., et al. (2016) · IEEE International Conference on Computational Photography (ICCP) · doi:10.1109/iccphot.2016.7492880 · Cited by 58
Glare encoding of high dynamic range images
Rouf, M., Mantiuk, R., Heidrich, W., et al. (2011) · doi:10.1109/cvpr.2011.5995335 · Cited by 41
Self-citationAchromatic flat focusing lens based on dispersion engineering of spoof surface plasmon polaritons
Yang, J., Wang, J., Feng, M., Li, Y., Wang, X.-H., Zhou, X., Cui, T., & Qu, S. (2017) · Applied Physics Letters · doi:10.1063/1.4983831 · Cited by 39
Cited by
Inference in artificial intelligence with deep optics and photonics
Wetzstein, G., Ozcan, A., Gigan, S., Fan, S., Englund, D., Soljačić, M., Denz, C., Miller, D. A. B., & Psaltis, D. (2020) · Nature · doi:10.1038/s41586-020-2973-6 · Cited by 1,031
Self-citationTowards real-time photorealistic 3D holography with deep neural networks
Shi, L., Li, B., Kim, C., Kellnhofer, P., & Matusik, W. (2021) · Nature · doi:10.1038/s41586-020-03152-0 · Cited by 637
Deep Learning Techniques for Inverse Problems in Imaging
Ongie, G., Jalal, A., Metzler, C. A., Baraniuk, R. G., Dimakis, A. G., & Willett, R. (2020) · IEEE Journal on Selected Areas in Information Theory · doi:10.1109/jsait.2020.2991563 · Cited by 577
Intelligent metasurfaces: control, communication and computing
Li, L., Zhao, H., Liu, C., Li, L., & Cui, T. J. (2022) · eLight · doi:10.1186/s43593-022-00013-3 · Cited by 347
DeepSTORM3D: dense 3D localization microscopy and PSF design by deep learning
Nehme, E., Freedman, D., Gordon, R., Ferdman, B., Weiss, L. E., Alalouf, O., Naor, T., Orange, R., Michaeli, T., & Shechtman, Y. (2020) · Nature Methods · doi:10.1038/s41592-020-0853-5 · Cited by 311
Metadata
- Authors
- Sitzmann, V., Diamond, S., Peng, Y., Dun, X., Boyd, S., Heidrich, W., Heide, F., & Wetzstein, G.
- Venue
- ACM Transactions on Graphics
- Published
- 2018-07-30
- Volume
- 37
- Issue
- 4
- Pages
- 1-13
- ISSN
- 0730-0301 (ISSN-L)
- 1557-7368
- OpenAlex
- W2809900749
- Cited by
- 478
- References
- 49
- FWCI
- 29.33
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.