Splatter Image: Ultra-Fast Single-View 3D Reconstruction
Open accessThe DOI matches its Crossref record; 50 of the 50 references it cites were found in a registry or an index.
Abstract
We introduce the Splatter Image, an ultra-efficient approach for monocular 3D object reconstruction. Splatter Image is based on Gaussian Splatting, which allows fast and high-quality reconstruction of 3D scenes from multiple images. We apply Gaussian Splatting to monocular reconstruction by learning a neural network that, at test time, performs reconstruction in a feed-forward manner, at 38 FPS. Our main innovation is the surprisingly straightforward design of this network, which, using 2D operators, maps the input image to one 3D Gaussian per pixel. The resulting set of Gaussians thus has the form an image, the Splatter Image. We further extend the method take several images as input via cross-view attention. Owning to the speed of the renderer (588 FPS), we use a single GPU for training while generating entire images at each iteration to optimize perceptual metrics like LPIPS. On several synthetic, real, multi-category and large-scale benchmark datasets, we achieve better results in terms of PSNR, LPIPS, and other metries while training and evaluating much faster than prior works. Code, models and more results are available at https://szymanowiczs.github.io/splatter-image.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
U-Net: Convolutional Networks for Biomedical Image Segmentation
Ronneberger, O., Fischer, P., & Brox, T. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24574-4_28 · Cited by 99,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
The unreasonable effectiveness of deep features as a perceptual metric
Zhang, R., Isola, P., Efros, A. A., Shechtman, E., & Wang, O. (2018) · arXiv · doi:10.48550/arxiv.1801.03924 · Cited by 24,812
High-Resolution Image Synthesis with Latent Diffusion Models
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01042 · Cited by 20,302 · cited 1×
“We use in particular a U-Net [32] as those have demonstrated excellent performance in image generation [31].”
MethodThe Unreasonable Effectiveness of Deep Features as a Perceptual Metric
Zhang, R., Isola, P., Efros, A. A., Shechtman, E., & Wang, O. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00068 · Cited by 14,623
Denoising Diffusion Probabilistic Models
Ho, J., Jain, A., & Abbeel, P. (2020) · arXiv · doi:10.48550/arxiv.2006.11239 · Cited by 8,712 · cited 1×
“PVD [46] and PC 2 [23] extended this approach using Diffusion Models [10] by conditioning the de-noising process on partial point clouds and RGB images, respectively.”
Method3D Gaussian Splatting for Real-Time Radiance Field Rendering
Kerbl, B., Kopanas, G., Leimkuehler, T., & Drettakis, G. (2023) · ACM Transactions on Graphics · doi:10.1145/3592433 · Cited by 6,025 · cited 5×
“Gaussian Splat-ting [14] used non-isotropic 3D Gaussians with variable scale, thus removing the need for 2D inpainting networks.”
BackgroundMethodInfluentialNeRF: Representing Scenes as Neural Radiance Fields for View Synthesis
Mildenhall, B., Srinivasan, P. P., Tancik, M., Barron, J. T., Ramamoorthi, R., & Ng, R. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58452-8_24 · Cited by 5,351
Denoising Diffusion Implicit Models
Song, J., Meng, C., & Ermon, S. (2020) · arXiv · doi:10.48550/arxiv.2010.02502 · Cited by 2,507
Film: Visual reasoning with a general conditioning layer
Perez, E., Strub, F., De Vries, H., Dumoulin, V., & Courville, A. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11671 · Cited by 2,490
A Point Set Generation Network for 3D Object Reconstruction from a Single Image
Fan, H., Su, H., & Guibas, L. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.264 · Cited by 2,402 · cited 1×
“PointOutNet [9] adapted PointNet [27] to take image encoding as input and trained point cloud prediction networks using 3D point cloud supervision.”
MethodShapeNet: An Information-Rich 3D Model Repository
Chang, A. X., Funkhouser, T., Guibas, L., Hanrahan, P., Huang, Q., Li, Z., Savarese, S., Savva, M., Song, S., Su, H., et al. (2015) · arXiv · doi:10.48550/arxiv.1512.03012 · Cited by 2,330 · cited 1×
“We further validate Splatter Image by comparing it to alternative, slower recon-structors on standard benchmark datasets like ShapeNet [4] and CO3D [28].”
MethodpixelNeRF: Neural Radiance Fields from One or Few Images
Yu, A., Ye, V., Tancik, M., & Kanazawa, A. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00455 · Cited by 1,560 · cited 6×
“We measure the quality of our reconstructions by evaluating the quality of Novel View Synthesis, in line with related works [16, 44].”
BackgroundMethodInfluentialTensoRF: Tensorial Radiance Fields
Chen, A., Xu, Z., Geiger, A., Yu, J., & Su, H. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-19824-3_20 · Cited by 1,184
Efficient Geometry-aware 3D Generative Adversarial Networks
Chan, E. R., Lin, C. Z., Chan, M. A., Nagano, K., Pan, B., de Mello, S., Gallo, O., Guibas, L., Tremblay, J., Khamis, S., et al. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01565 · Cited by 1,093 · cited 1×
“A hybrid implicit-explicit triplane representation [2] has been proposed as a compromise between rendering speed and memory consumption.”
BackgroundMVDream: Multi-view Diffusion for 3D Generation
Shi, Y., Wang, P., Ye, J., Long, M., Li, K., & Yang, X. (2023) · International Conference on Learning Representations · doi:10.48550/arxiv.2308.16512 · Cited by 958 · cited 3×
“We do so in a manner similar to [34], but only at the lowest UNet resolution, which maintains the computational cost very low.”
BackgroundKato, H., Ushiku, Y., & Harada, T. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00411 · Cited by 954
Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstruction
Sun, C., Sun, M., & Chen, H.-T. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00538 · Cited by 939 · cited 1×
“Some follow-up works [8, 38] have used faster implicit representations based on voxel grids that encode opacities and colours directly [8, 38] — similar to DVGO [37], they can thus achieve significant speed-ups.”
BackgroundObjaverse-XL: A Universe of 10M+ 3D Objects
Deitke, M., Liu, R., Wallingford, M., Ngo, H., Michel, O., Kusupati, A., Fan, A., Laforte, C., Voleti, V., Gadre, S. Y., et al. (2023) · neural information processing systems · doi:10.52202/075280-1554 · Cited by 848
Zero-1-to-3: Zero-shot One Image to 3D Object
Liu, R., Wu, R., Van Hoorick, B., Tokmakov, P., Zakharov, S., & Vondrick, C. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00853 · Cited by 841 · cited 2×
“Diffusion Models have been employed for conditional Novel View Synthesis [3, 18, 19, 42].”
BackgroundMethodMVSNeRF: Fast Generalizable Radiance Field Reconstruction from Multi-View Stereo
Chen, A., Xu, Z., Zhao, F., Zhang, X., Xiang, F., Yu, J., & Su, H. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01386 · Cited by 797 · cited 1×
“When more than one view is available at the input, one can learn to interpolate between available views in the 3D space to estimate the scene geometry [5, 21], learn a view interpolation function [41] or optimize a 3D representation of a scene using semantic priors [11].”
BackgroundHigh-Resolution Image Synthesis with Latent Diffusion Models
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2021) · arXiv · doi:10.48550/arxiv.2112.10752 · Cited by 737
Objaverse: A Universe of Annotated 3D Objects
Deitke, M., Schwenk, D., Salvador, J., Weihs, L., Michel, O., VanderBilt, E., Schmidt, L., Ehsanit, K., Kembhavi, A., & Farhadi, A. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01263 · Cited by 725 · cited 4×
“Compared to these slower baselines, Splat-ter Image is not only competitive in terms of quality, but in fact state-of-the-art in several cases, improving both reconstruction PSNR and LPIPS.”
BackgroundInfluentialIBRNet: Learning Multi-View Image-Based Rendering
Wang, Q., Wang, Z., Genova, K., Srinivasan, P., Zhou, H., Barron, J. T., Martin-Brualla, R., Snavely, N., & Funkhouser, T. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00466 · Cited by 725 · cited 1×
“When more than one view is available at the input, one can learn to interpolate between available views in the 3D space to estimate the scene geometry [5, 21], learn a view interpolation function [41] or optimize a 3D representation of a scene using semantic priors [11].”
BackgroundElucidating the Design Space of Diffusion-Based Generative Models
Karras, T., Aittala, M., Aila, T., & Laine, S. (2022) · arXiv · doi:10.48550/arxiv.2206.00364 · Cited by 702
Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesis
Jain, A., Tancik, M., & Abbeel, P. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00583 · Cited by 463 · cited 1×
“When more than one view is available at the input, one can learn to interpolate between available views in the 3D space to estimate the scene geometry [5, 21], learn a view interpolation function [41] or optimize a 3D representation of a scene using semantic priors [11].”
BackgroundTripoSR: Fast 3D Object Reconstruction from a Single ImagePreprint
Tochilkin, D., Pankratz, D., Liu, Z., Huang, Z., Letts, A., Li, Y., Ding, L., Laforte, C., Jampani, V., & Cao, Y. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2403.02151 · Cited by 456 · cited 1×
“While they are not as fast to render as explicit representations, they are fast enough to be effectively used for single-view reconstruction [1, 7].”
Background3D Shape Generation and Completion through Point-Voxel Diffusion
Zhou, L., Du, Y., & Wu, J. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00577 · Cited by 421
SynSin: End-to-End View Synthesis From a Single Image
Wiles, O., Gkioxari, G., Szeliski, R., & Johnson, J. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00749 · Cited by 418 · cited 1×
“Other works [17, 30, 43] train networks for Novel Views Synthesis purely from videos and use point clouds as intermediate 3D representations for conditioning 2D inpainting or generation networks.”
BackgroundGoogle Scanned Objects: A High-Quality Dataset of 3D Scanned Household Items
Downs, L., Francis, A., Koenig, N., Kinman, B., Hickman, R., Reymann, K., McHugh, T. B., & Vanhoucke, V. (2022) · 2022 International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra46639.2022.9811809 · Cited by 377
Common Objects in 3D: Large-Scale Learning and Evaluation of Real-life 3D Category Reconstruction
Reizenstein, J., Shapovalov, R., Henzler, P., Sbordone, L., Labatut, P., & Novotny, D. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01072 · Cited by 352 · cited 2×
“However, in real datasets like CO3D [28], this ambiguity is present.”
BackgroundMethodADOP: approximate differentiable one-pixel point rendering
Rückert, D., Franke, L., & Stamminger, M. (2022) · ACM Transactions on Graphics · doi:10.1145/3528223.3530122 · Cited by 274
SparseNeuS: Fast Generalizable Neural Surface Reconstruction from Sparse Views
Long, X., Lin, C. C., Wang, P., Komura, T., & Wang, W. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19824-3_13 · Cited by 263
DiffRF: Rendering-Guided 3D Radiance Field Diffusion
Muller, N. E., Siddiqui, Y., Porzi, L., Bulò, S. R., Kontschieder, P., & Nießner, M. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00421 · Cited by 263 · cited 1×
“Other works build and use a 3D [6, 25] or 2D [7, 22] prior which can be used in an image-conditioned auto-decoding framework.”
RenderDiffusion: Image Diffusion for 3D Reconstruction, Inpainting and Generation
Anciukevicius, T., Xu, Z., Fisher, M., Henderson, P., Bilen, H., Mitra, N. J., & Guerrero, P. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01213 · Cited by 261 · cited 1×
“While they are not as fast to render as explicit representations, they are fast enough to be effectively used for single-view reconstruction [1, 7].”
BackgroundSingle-Stage Diffusion NeRF: A Unified Approach to 3D Generation and Reconstruction
Chen, H., Gu, J., Chen, A., Tian, W., Tu, Z., Liu, L., & Su, H. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00229 · Cited by 242 · cited 1×
“Other works build and use a 3D [6, 25] or 2D [7, 22] prior which can be used in an image-conditioned auto-decoding framework.”
NerfDiff: Single-image View Synthesis with NeRF-guided Distillation from 3D-aware DiffusionPreprint
Gu, J., Trevithick, A., Lin, K.-E., Susskind, J., Theobalt, C., Liu, L., & Ramamoorthi, R. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2302.10109 · Cited by 225 · cited 6×
“Triplanes can be predicted by networks in the camera view space, instead of a global reference frame, thus allowing reconstruction in the view-space [7].”
BackgroundMethodInfluentialZwicker, M., Pfister, H., van Baar, J., & Gross, M. (2005) · Proceedings Visualization, 2001. VIS '01. · doi:10.1109/visual.2001.964490 · Cited by 220
RealFusion 360° Reconstruction of Any Object from a Single Image
Melas-Kyriazi, L., Laina, I., Rupprecht, C., & Vedaldi, A. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.00816 · Cited by 200 · cited 1×
“Other works build and use a 3D [6, 25] or 2D [7, 22] prior which can be used in an image-conditioned auto-decoding framework.”
Self-citationSparseFusion: Distilling View-Conditioned Diffusion for 3D Reconstruction
Zhou, Z., & Tulsiani, S. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01211 · Cited by 171 · cited 2×
“Since we are proposing a deterministic reconstruction method, we do not compare to methods that employ Score Distil-lation [47] or feed-forward diffusion models [3, 42].”
MethodPC2: Projection-Conditioned Point Cloud Diffusion for Single-Image 3D Reconstruction
Melas-Kyriazi, L., Rupprecht, C., & Vedaldi, A. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01242 · Cited by 168 · cited 1×
“PVD [46] and PC 2 [23] extended this approach using Diffusion Models [10] by conditioning the de-noising process on partial point clouds and RGB images, respectively.”
MethodSelf-citationVision Transformer for NeRF-Based View Synthesis from a Single Input Image
Lin, K.-E., Lin, Y.-C., Lai, W., Lin, T.-Y., Shih, Y., & Ramamoorthi, R. (2023) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv56688.2023.00087 · Cited by 164 · cited 4×
“We measure the quality of our reconstructions by evaluating the quality of Novel View Synthesis, in line with related works [16, 44].”
BackgroundMethodInfluentialGenerative Novel View Synthesis with 3D-Aware Diffusion Models
Chan, E. R., Nagano, K., Chan, M. A., Bergman, A. W., Park, J. J., Levy, A., Aittala, M., De Mello, S., Karras, T., & Wetzstein, G. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00389 · Cited by 162 · cited 4×
“Since we are proposing a deterministic reconstruction method, we do not compare to methods that employ Score Distil-lation [47] or feed-forward diffusion models [3, 42].”
BackgroundMethodInfluentialCodeNeRF: Disentangled Neural Radiance Fields for Object Categories
Jang, W., & Agapito, L. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01271 · Cited by 160 · cited 2×
“For ShapeNet, we compare against implicit [12, 16, 35, 44], hydrid implicit-explicit [7] and explicit meth-ods [8, 38] methods.”
BackgroundMethodViewset Diffusion: (0-)Image-Conditioned 3D Generative Models from 2D Data
Szymanowicz, S., Rupprecht, C., & Vedaldi, A. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00814 · Cited by 152 · cited 8×
“We apply pre-processing following the protocol of [38], thus approximately fixing the scale of all objects.”
BackgroundMethodInfluentialSelf-citationInfinite Nature: Perpetual View Generation of Natural Scenes from a Single Image
Liu, A., Makadia, A., Tucker, R., Snavely, N., Jampani, V., & Kanazawa, A. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01419 · Cited by 141 · cited 1×
“Other works [17, 30, 43] train networks for Novel Views Synthesis purely from videos and use point clouds as intermediate 3D representations for conditioning 2D inpainting or generation networks.”
BackgroundViewFormer: NeRF-Free Neural Rendering from Few Images Using Transformers
Kulhánek, J., Derner, E., Sattler, T., & Babuška, R. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19784-0_12 · Cited by 102
Free3D: Consistent Novel View Synthesis Without 3D Representation
Zheng, C., & Vedaldi, A. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00928 · Cited by 96 · cited 2×
“4 shows that our method produces sharper and more accurate reconstructions than prior works in challenging situations with limited visibility and thin structures.”
MethodSelf-citationFWD: Real-time Novel View Synthesis with Forward Warping and Depth
Cao, A., Rockwell, C., & Johnson, J. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01526 · Cited by 67 · cited 1×
“While Viewset Diffusion [38] and NeRFDiff [7] were introduced as diffusion-based generative methods, one can use their deterministic variants by using their reconstruction network in a single forward pass.”
MethodFast and Explicit Neural View Synthesis
Guo, P., Bautista, M. Á., Colburn, A., Yang, L., Ulbricht, D., Susskind, J. M., & Shan, Q. (2022) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv51458.2022.00009 · Cited by 41
Cited by
MVSplat: Efficient 3D Gaussian Splatting from Sparse Multi-view Images
Chen, Y., Xu, H., Zheng, C., Zhuang, B., Pollefeys, M., Geiger, A., Cham, T.-J., & Cai, J. (2025) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-72664-4_21 · Cited by 231
Metadata
- Authors
- Szymanowicz, S., Rupprecht, C., & Vedaldi, A.
- Venue
- 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2024-06-16
- Pages
- 10208-10217
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W4402727821
- Cited by
- 180
- References
- 50
- FWCI
- 51.69
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.