The DOI matches its Crossref record; 49 of the 49 references it cites were found in a registry or an index.
Interactive reconstruction of Monte Carlo image sequences using a recurrent denoising autoencoder
Abstract
We describe a machine learning technique for reconstructing image sequences rendered using Monte Carlo methods. Our primary focus is on reconstruction of global illumination with extremely low sampling budgets at interactive rates. Motivated by recent advances in image restoration with deep convolutional networks, we propose a variant of these networks better suited to the class of noise present in Monte Carlo rendering. We allow for much larger pixel neighborhoods to be taken into account, while also improving execution speed by an order of magnitude. Our primary contribution is the addition of recurrent connections to the network in order to drastically improve temporal stability for sequences of sparsely sampled input images. Our method also has the desirable property of automatically modeling relationships based on auxiliary per-pixel input channels, such as depth and normals. We show significantly higher quality results compared to existing methods that run at comparable speeds, and furthermore argue a clear path for making our method run at realtime rates in the near future.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
U-Net: Convolutional Networks for Biomedical Image Segmentation
Ronneberger, O., Fischer, P., & Brox, T. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24574-4_28 · Cited by 99,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805
Image quality assessment: from error visibility to structural similarity
Wang, Z., Bovik, A. C., Sheikh, H. R., & Simoncelli, E. P. (2004) · IEEE Transactions on Image Processing · doi:10.1109/tip.2003.819861 · Cited by 57,892
Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation
Girshick, R., Donahue, J., Darrell, T., & Malik, J. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.81 · Cited by 32,684
Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification
He, K., Zhang, X., Ren, S., & Sun, J. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.123 · Cited by 19,741
Deep learning in neural networks: an overview
Schmidhuber, J. (2015) · Neural Networks · doi:10.1016/j.neunet.2014.09.003 · Cited by 18,570
Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network
Ledig, C., Theis, L., Huszar, F., Caballero, J., Cunningham, A., Acosta, A., Aitken, A., Tejani, A., Totz, J., Wang, Z., et al. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.19 · Cited by 12,696
Shi, W., Caballero, J., Huszar, F., Totz, J., Aitken, A. P., Bishop, R., Rueckert, D., & Wang, Z. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.207 · Cited by 7,570
Extracting and composing robust features with denoising autoencoders
Vincent, P., Larochelle, H., Bengio, Y., & Manzagol, P.-A. (2008) · Proceedings of the 25th international conference on Machine learning - ICML '08 · doi:10.1145/1390156.1390294 · Cited by 7,436
Context Encoders: Feature Learning by Inpainting
Pathak, D., Krahenbuhl, P., Donahue, J., Darrell, T., & Efros, A. A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.278 · Cited by 5,691
Flownet: Learning optical flow with convolutional networks
Dosovitskiy, A., Fischer, P., Ilg, E., Hausser, P., Hazirbas, C., Golkov, V., Smagt, P. V. D., Cremers, D., & Brox, T. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.316 · Cited by 4,313
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2015) · arXiv · doi:10.48550/arxiv.1512.03385 · Cited by 4,196
Estimation of the Mean of a Multivariate Normal Distribution
Stein, C. M. (1981) · The Annals of Statistics · doi:10.1214/aos/1176345632 · Cited by 2,729
Loss Functions for Image Restoration With Neural Networks
Zhao, H., Gallo, O., Frosio, I., & Kautz, J. (2017) · IEEE Transactions on Computational Imaging · doi:10.1109/tci.2016.2644865 · Cited by 2,641
Kajiya, J. T. (1986) · ACM SIGGRAPH Computer Graphics · doi:10.1145/15886.15902 · Cited by 2,238
EIE: efficient inference engine on compressed deep neural network
Han, S., Liu, X., Mao, H., Pu, J., Pedram, A., Horowitz, M. A., & Dally, W. J. (2016) · ACM SIGARCH Computer Architecture News · doi:10.1145/3007787.3001163 · Cited by 1,937
Static Analysis of Shape in TensorFlow Programs
The Theano Development Team, Al‐Rfou, R., Alain, G., Almahairi, A., Angermueller, C., Bahdanau, D., Ballas, N., Bastien, F., Bayer, J., Belikov, A., et al. (2016) · arXiv (Cornell University) · doi:10.4230/lipics.ecoop.2020.15 · Cited by 1,589
MR Image Reconstruction From Highly Undersampled k-Space Data by Dictionary Learning
Ravishankar, S., & Bresler, Y. (2011) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2010.2090538 · Cited by 1,059
Learning Representations for Automatic Colorization
Larsson, G., Maire, M., & Shakhnarovich, G. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46493-0_35 · Cited by 946
Generalization of backpropagation with application to a recurrent gas market model
Werbos, P. J. (1988) · Neural Networks · doi:10.1016/0893-6080(88)90007-x · Cited by 944
Kajiya, J. T. (1986) · Proceedings of the 13th annual conference on Computer graphics and interactive techniques · doi:10.1145/15922.15902 · Cited by 848
Iizuka, S., Simo-Serra, E., & Ishikawa, H. (2016) · ACM Transactions on Graphics · doi:10.1145/2897824.2925974 · Cited by 814
OptiX: a general purpose ray tracing engine
Parker, S. G., Bigler, J., Dietrich, A., Friedrich, H., Hoberock, J., Luebke, D., McAllister, D., McGuire, M., Morley, K., Robison, A., et al. (2010) · ACM Transactions on Graphics · doi:10.1145/1778765.1778803 · Cited by 684
Robust Monte Carlo methods for light transport simulation
Guibas, L., & Veach, E. (1997) · Cited by 660
Learning to generate chairs with convolutional neural networks
Dosovitskiy, A., Springenberg, J. T., & Brox, T. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298761 · Cited by 640
Algorithm 247: Radical-inverse quasi-random point sequence
Halton, J. H. (1964) · Communications of the ACM · doi:10.1145/355588.365104 · Cited by 601
Deep Stereo: Learning to Predict New Views from the World's Imagery
Flynn, J., Neulander, I., Philbin, J., & Snavely, N. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.595 · Cited by 574
FlowNet: Learning Optical Flow with Convolutional Networks
Fischer, P., Dosovitskiy, A., Ilg, E., Haeusser, P., Hazırbaş, C., Golkov, V., van der Smagt, P., Cremers, D., & Brox, T. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1504.06852 · Cited by 562
Image Restoration Using Convolutional Auto-encoders with Symmetric Skip Connections
Mao, X.-J., Shen, C., & Yang, Y.-B. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.08921 · Cited by 323
Kernel-predicting convolutional networks for denoising Monte Carlo renderings
Bako, S., Vogels, T., Mcwilliams, B., Meyer, M., NováK, J., Harvill, A., Sen, P., Derose, T., & Rousselle, F. (2017) · ACM Transactions on Graphics · doi:10.1145/3072959.3073708 · Cited by 308
Dieleman, S., Schlüter, J., Raffel, C., Olson, E. M., Sønderby, S. K., Nouri, D., Maturana, D., Thoma, M., Battenberg, E., Kelly, J., et al. (2015) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.27878 · Cited by 279
Adaptive manifolds for real-time high-dimensional filtering
Gastal, E. S. L., & Oliveira, M. M. (2012) · ACM Transactions on Graphics · doi:10.1145/2185520.2185529 · Cited by 241
Bidirectional recurrent convolutional networks for multi-frame super-resolution
Huang, Y., Wang, W., & Wang, L. (2015) · neural information processing systems · Cited by 222
Spatio-temporal video autoencoder with differentiable memory
Pătrăucean, V., Handa, A., & Cipolla, R. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.06309 · Cited by 211
Recent Advances in Adaptive Sampling and Reconstruction for Monte Carlo Rendering
Zwicker, M., Jarosz, W., Lehtinen, J., Moon, B., Ramamoorthi, R., Rousselle, F., Sen, P., Soler, C., & Yoon, S. E. (2015) · Computer Graphics Forum · doi:10.1111/cgf.12592 · Cited by 180
Deep Learning Made Easier by Linear Transformations in Perceptrons
Raiko, T., Harri Valpola, B., & LeCun, Y. (2012) · Cited by 171
A machine learning approach for filtering Monte Carlo noise
Kalantari, N. K., Bako, S., & Sen, P. (2015) · ACM Transactions on Graphics · doi:10.1145/2766977 · Cited by 159
SURE-based optimization for adaptive sampling and reconstruction
Li, T., Wu, Y., & Chuang, Y. (2012) · ACM Transactions on Graphics · doi:10.1145/2366145.2366213 · Cited by 140
On filtering the noise from the random parameters in Monte Carlo rendering
Sen, P., & Darabi, S. (2012) · ACM Transactions on Graphics · doi:10.1145/2167076.2167083 · Cited by 111
Robust Denoising using Feature and Color Information
Rousselle, F., Manzi, M., & Zwicker, M. (2013) · Computer Graphics Forum · doi:10.1111/cgf.12219 · Cited by 94
Anisotropic diffusion for Monte Carlo noise reduction
McCool, M. D. (1999) · ACM Transactions on Graphics · doi:10.1145/318009.318015 · Cited by 91
Nonlinearly Weighted First-order Regression for Denoising Monte Carlo Renderings
Bitterli, B., Rousselle, F., Moon, B., et al. (2016) · Computer Graphics Forum · doi:10.1111/cgf.12954 · Cited by 83
Edge-avoiding À-Trous wavelet transform for fast global illumination filtering
Dammertz, H., Sewtz, D., Hanika, J., et al. (2010) · High Performance Graphics · doi:10.5555/1921479.1921491 · Cited by 76
Guided Image Filtering for Interactive High‐quality Global Illumination
Bauszat, P., Eisemann, M., & Magnor, M. A. (2011) · Computer Graphics Forum · doi:10.1111/j.1467-8659.2011.01996.x · Cited by 76
Axis-aligned filtering for interactive physically-based diffuse indirect lighting
Mehta, S. U., Wang, B., Ramamoorthi, R., & Durand, F. (2013) · ACM Transactions on Graphics · doi:10.1145/2461912.2461947 · Cited by 54
Path‐space Motion Estimation and Decomposition for Robust Animation Filtering
Zimmer, H., Rousselle, F., Jakob, W., et al. (2015) · Computer Graphics Forum · doi:10.1111/cgf.12685 · Cited by 46
Adaptive rendering with linear predictions
Moon, B., Iglesias-Guitián, J. A., Yoon, S., & Mitchell, K. (2015) · ACM Transactions on Graphics · doi:10.1145/2766992 · Cited by 38
Cited by
State of the Art on Neural Rendering
Tewari, A., Fried, O., Thies, J., Sitzmann, V., Lombardi, S., Sunkavalli, K., Martin‐Brualla, R., Simon, T., Saragih, J., Nießner, M., et al. (2020) · Computer Graphics Forum · doi:10.1111/cgf.14022 · Cited by 430
Tewari, A., Thies, J., Mildenhall, B., Srinivasan, P., Tretschk, E., Yifan, W., Lassner, C., Sitzmann, V., Martin‐Brualla, R., Lombardi, S., et al. (2022) · Computer Graphics Forum · doi:10.1111/cgf.14507 · Cited by 339
Kernel-predicting convolutional networks for denoising Monte Carlo renderings
Bako, S., Vogels, T., Mcwilliams, B., Meyer, M., NováK, J., Harvill, A., Sen, P., Derose, T., & Rousselle, F. (2017) · ACM Transactions on Graphics · doi:10.1145/3072959.3073708 · Cited by 308
A point-cloud deep learning framework for prediction of fluid flow fields on irregular geometries
Kashefi, A., Rempe, D., & Guibas, L. J. (2021) · Physics of Fluids · doi:10.1063/5.0033376 · Cited by 295
tempoGAN: a temporally coherent, volumetric GAN for super-resolution fluid flow
Xie, Y., Franz, A., Chu, M., & Thuerey, N. (2018) · ACM Transactions on Graphics · doi:10.1145/3197517.3201304 · Cited by 275
Metadata
- Authors
- Chaitanya, C. R. A., Kaplanyan, A. S., Schied, C., Salvi, M., Lefohn, A., Nowrouzezahrai, D., & Aila, T.
- Venue
- ACM Transactions on Graphics
- Published
- 2017-07-20
- Volume
- 36
- Issue
- 4
- Pages
- 1-12
- ISSN
- 0730-0301 (ISSN-L)
- 1557-7368
- OpenAlex
- W2737368828
- Cited by
- 306
- References
- 49
- FWCI
- 70.26
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.