CoChat
Sign Up
Cited by 69

Geometry-Free View Synthesis: Transformers and no 3D Priors

Rombach, Robin, Esser, Patrick, Ommer, Bjorn·2021·2021 IEEE/CVF International Conference on Computer Vision (ICCV)
DOI 10.1109/iccv48922.2021.01409
Open this paper in CoChat — free account

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/iccv48922.2021.01409

Cites

4 of 4 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Learning-based view synthesis has been explored through a purely data-driven approach using scalable Transformer architectures, bypassing the need for explicit 3D prior knowledge.

    “This bypasses the need for explicit 3D prior knowledge (Rombach et al., 2021; Sajjadi et al., 2022): instead of relying on neural rendering applied to 3D representations, novel views are synthesized directly by neural networks conditioned on camera inputs (Tatarchenko et al., 2016).”

    From Cosmos World Foundation Model Platform for Physical AI (2025) · cites this paper as Rombach et al., 2021

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 74

  1. Sameer Agarwal, Noah Snavely, Ian Simon, Steven M. Seitz, and Richard Szeliski. Building rome in a day. In ICCV, pages 72-79. IEEE Computer Society, 2009. 2
  2. Kara-Ali Aliev, Artem Sevastopolsky, Maria Kolos, Dmitry Ulyanov, and Victor S. Lempitsky. Neural point-based graphics. In ECCV (22), volume 12367 of Lecture Notes in Computer Science, pages 696-712. Springer, 2020. 2, 5
  3. Lei Jimmy Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization. CoRR, abs/1607.06450, 2016. · arXiv
  4. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In ICLR, 2015. 3
  5. Chris Buehler, Michael Bosse, Leonard McMillan, Steven J. Gortler, and Michael F. Cohen. Unstructured lumigraph rendering. In SIGGRAPH, pages 425-432. ACM, 2001. 2
  6. Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In ICML, volume 119 of Proceedings of Machine Learning Research, pages 1691-1703. PMLR, 2020. 5
  7. Xi Chen, Diederik P. Kingma, Tim Salimans, Yan Duan, Prafulla Dhariwal, John Schulman, Ilya Sutskever, and Pieter Abbeel. Variational lossy autoencoder. In ICLR (Poster). OpenReview.net, 2017. 3
  8. Inchang Choi, Orazio Gallo, Alejandro J. Troccoli, Min H. Kim, and Jan Kautz. Extreme view synthesis. In ICCV, pages 7780-7789. IEEE, 2019. 2
  9. Bin Dai and David P. Wipf. Diagnosing and enhancing VAE models. In ICLR (Poster). OpenReview.net, 2019. 3
  10. Paul E. Debevec, Camillo J. Taylor, and Jitendra Malik. Modeling and rendering architecture from photographs: A hybrid geometry- and image-based approach. In SIGGRAPH, pages 11-20. ACM, 1996. 1, 2
  11. Prafulla Dhariwal, Heewoo Jun, Christine Payne, Jong Wook Kim, Alec Radford, and Ilya Sutskever. Jukebox: A generative model for music. CoRR, abs/2005.00341, 2020. 3 · arXiv
  12. Alexey Dosovitskiy, Jost Tobias Springenberg, and Thomas Brox. Learning to generate chairs with convolutional neural networks. In CVPR, pages 1538-1546. IEEE Computer Society, 2015. 2
  13. Alexey Dosovitskiy, Jost Tobias Springenberg, Maxim Tatarchenko, and Thomas Brox. Learning to generate chairs, tables and cars with convolutional networks. IEEE Trans. Pattern Anal. Mach. Intell., 39(4):692-705, 2017. 2
  14. Patrick Esser, Robin Rombach, and Björn Ommer. A disentangling invertible interpretation network for explaining latent representa- tions. In CVPR, pages 9220-9229. IEEE, 2020. 3
  15. Patrick Esser, Robin Rombach, and Björn Ommer. Taming transformers for high-resolution image synthesis. CoRR, abs/2012.09841, 2020. 3, 4, 11 · arXiv
  16. John Flynn, Michael Broxton, Paul E. Debevec, Matthew DuVall, Graham Fyffe, Ryan S. Overbeck, Noah Snavely, and Richard Tucker. Deepview: View synthesis with learned gradient descent. In CVPR, pages 2367-2376. Computer Vision Foundation / IEEE, 2019. 2
  17. John Flynn, Ivan Neulander, James Philbin, and Noah Snavely. Deepstereo: Learning to predict new views from the world's imagery. CoRR, abs/1506.06825, 2015. 2 · arXiv
  18. Yasutaka Furukawa and Jean Ponce. Accurate, dense, and robust multiview stereopsis. IEEE Trans. Pattern Anal. Mach. Intell., 32(8):1362-1376, 2010. 2
  19. Clément Godard, Oisin Mac Aodha, and Gabriel J. Brostow. Unsupervised monocular depth estimation with left-right consistency. In CVPR, pages 6602-6611. IEEE Computer Society, 2017. 2
  20. Clément Godard, Oisin Mac Aodha, Michael Firman, and Gabriel J. Brostow. Digging into self-supervised monocular depth estima- tion. In ICCV, pages 3827-3837. IEEE, 2019. 2
  21. Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. Generative adversarial nets. In NIPS, pages 2672-2680, 2014. 2
  22. Steven J. Gortler, Radek Grzeszczuk, Richard Szeliski, and Michael F. Cohen. The lumigraph. In SIGGRAPH, pages 43-54. ACM, 1996. 1, 2
  23. Peter Hedman, Suhib Alsisan, Richard Szeliski, and Johannes Kopf. Casual 3d photography. ACM Trans. Graph., 36(6):234:1- 234:15, 2017. 2
  24. Peter Hedman and Johannes Kopf. Instant 3d photography. ACM Trans. Graph., 37(4):101:1-101:12, 2018.
  25. Peter Hedman, Julien Philip, True Price, Jan-Michael Frahm, George Drettakis, and Gabriel J. Brostow. Deep blending for free- viewpoint image-based rendering. ACM Trans. Graph., 37(6):257:1-257:15, 2018. 2
  26. Peter Hedman, Tobias Ritschel, George Drettakis, and Gabriel J. Brostow. Scalable inside-out image-based rendering. ACM Trans. Graph., 35(6):231:1-231:11, 2016. 2
  27. Youichi Horry, Ken-ichi Anjyo, and Kiyoshi Arai. Tour into the picture: using a spidery mesh interface to make animation from a single image. In SIGGRAPH, pages 225-232. ACM, 1997. 2
  28. Ronghang Hu, Nikhila Ravi, Alexander C. Berg, and Deepak Pathak. Worldsheet: Wrapping the world in a 3d sheet for view synthesis from a single image. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2021. 2
  29. Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. In ICLR (Poster). OpenReview.net, 2017. 3
  30. Nima Khademi Kalantari, Ting-Chun Wang, and Ravi Ramamoorthi. Learning-based view synthesis for light field cameras. ACM Trans. Graph., 35(6):193:1-193:10, 2016. 2
  31. Johannes Kopf, Kevin Matzen, Suhib Alsisan, Ocean Quigley, Francis Ge, Yangming Chong, Josh Patterson, Jan-Michael Frahm, Shu Wu, Matthew Yu, Peizhao Zhang, Zijian He, Peter Vajda, Ayush Saraf, and Michael F. Cohen. One shot 3d photography. ACM Trans. Graph., 39(4):76, 2020. 2
  32. Tejas D. Kulkarni, William F. Whitney, Pushmeet Kohli, and Joshua B. Tenenbaum. Deep convolutional inverse graphics network. In NIPS, pages 2539-2547, 2015. 2
  33. Marc Levoy and Pat Hanrahan. Light field rendering. In SIGGRAPH, pages 31-42. ACM, 1996. 1, 2
  34. Zhengqi Li, Wenqi Xian, Abe Davis, and Noah Snavely. Crowdsampling the plenoptic function. In ECCV (1), volume 12346 of Lecture Notes in Computer Science, pages 178-196. Springer, 2020. 2
  35. Tsung-Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: common objects in context. In ECCV (5), volume 8693 of Lecture Notes in Computer Science, pages 740-755. Springer, 2014. 7
  36. Andrew Liu, Richard Tucker, Varun Jampani, Ameesh Makadia, Noah Snavely, and Angjoo Kanazawa. Infinite nature: Perpetual view generation of natural scenes from a single image. CoRR, abs/2012.09855, 2020. 2, 4, 6, 7, 8, 12, 17 · arXiv
  37. Miaomiao Liu, Xuming He, and Mathieu Salzmann. Geometry-aware deep network for single-image novel view synthesis. In CVPR, pages 4616-4624. IEEE Computer Society, 2018. 2
  38. Ilya Loshchilov and Frank Hutter. SGDR: stochastic gradient descent with warm restarts. In ICLR (Poster). OpenReview.net, 2017. 11
  39. Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR (Poster). OpenReview.net, 2019. 11
  40. Chris J. Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A continuous relaxation of discrete random variables. In ICLR (Poster). OpenReview.net, 2017. 3
  41. Moustafa Meshry, Dan B. Goldman, Sameh Khamis, Hugues Hoppe, Rohit Pandey, Noah Snavely, and Ricardo Martin-Brualla. Neural rerendering in the wild. In CVPR, pages 6878-6887. Computer Vision Foundation / IEEE, 2019. 2
  42. Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV (1), volume 12346 of Lecture Notes in Computer Science, pages 405-421. Springer, 2020. 1, 2
  43. Simon Niklaus and Feng Liu. Softmax splatting for video frame interpolation. In CVPR, pages 5436-5445. IEEE, 2020. 4
  44. Simon Niklaus, Long Mai, Jimei Yang, and Feng Liu. 3d ken burns effect from a single image. ACM Trans. Graph., 38(6):184:1- 184:15, 2019. 2
  45. Ankur P. Parikh, Oscar Täckström, Dipanjan Das, and Jakob Uszkoreit. A decomposable attention model for natural language inference. In EMNLP, pages 2249-2255. The Association for Computational Linguistics, 2016. 3
  46. Eunbyung Park, Jimei Yang, Ersin Yumer, Duygu Ceylan, and Alexander C. Berg. Transformation-grounded image generation network for novel 3d view synthesis. In CVPR, pages 702-711. IEEE Computer Society, 2017. 2
  47. Eric Penner and Li Zhang. Soft 3d reconstruction for view synthesis. ACM Trans. Graph., 36(6):235:1-235:11, 2017.
  48. Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. CoRR, abs/2102.12092, 2021. 3 · arXiv
  49. René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. IEEE Trans. Pattern Anal. Mach. Intell., 2020. 2, 4, 7, 9, 12
  50. Ali Razavi, Aäron van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with VQ-VAE-2. In NeurIPS, pages 14837-14847, 2019. 3
  51. Gernot Riegler and Vladlen Koltun. Free view synthesis. In ECCV (19), volume 12364 of Lecture Notes in Computer Science, pages 623-640. Springer, 2020. 1, 2, 5
  52. Gernot Riegler and Vladlen Koltun. Stable view synthesis. CoRR, abs/2011.07233, 2020. 1, 2 · arXiv
  53. Tim Salimans, Andrej Karpathy, Xi Chen, and Diederik P. Kingma. Pixelcnn++: Improving the pixelcnn with discretized logistic mixture likelihood and other modifications. In ICLR (Poster). OpenReview.net, 2017. 3
  54. Johannes L. Schönberger, Enliang Zheng, Jan-Michael Frahm, and Marc Pollefeys. Pixelwise view selection for unstructured multi- view stereo. In ECCV (3), volume 9907 of Lecture Notes in Computer Science, pages 501-518. Springer, 2016. 2
  55. Jonathan Shade, Steven J. Gortler, Li-wei He, and Richard Szeliski. Layered depth images. In SIGGRAPH, pages 231-242. ACM, 1998.
  56. Meng-Li Shih, Shih-Yang Su, Johannes Kopf, and Jia-Bin Huang. 3d photography using context-aware layered depth inpainting. In CVPR, pages 8025-8035. IEEE, 2020. 2, 4, 7, 8, 12, 15, 17
  57. Pratul P. Srinivasan, Richard Tucker, Jonathan T. Barron, Ravi Ramamoorthi, Ren Ng, and Noah Snavely. Pushing the boundaries of view extrapolation with multiplane images. In CVPR, pages 175-184. Computer Vision Foundation / IEEE, 2019. 2
  58. Pratul P. Srinivasan, Tongzhou Wang, Ashwin Sreelal, Ravi Ramamoorthi, and Ren Ng. Learning to synthesize a 4d RGBD light field from a single image. In ICCV, pages 2262-2270. IEEE Computer Society, 2017. 2
  59. Shao-Hua Sun, Minyoung Huh, Yuan-Hong Liao, Ning Zhang, and Joseph J. Lim. Multi-view to novel view: Synthesizing novel views with self-learned confidence. In ECCV (3), volume 11207 of Lecture Notes in Computer Science, pages 162-178. Springer, 2018. 2
  60. Richard Szeliski and Polina Golland. Stereo matching with transparency and matting. Int. J. Comput. Vis., 32(1):45-61, 1999.
  61. Maxim Tatarchenko, Alexey Dosovitskiy, and Thomas Brox. Multi-view 3d models from single images with a convolutional network. In ECCV (7), volume 9911 of Lecture Notes in Computer Science, pages 322-337. Springer, 2016. 2
  62. Lucas Theis, Aäron van den Oord, and Matthias Bethge. A note on the evaluation of generative models. In ICLR, 2016. 6
  63. Justus Thies, Michael Zollhöfer, and Matthias Nießner. Deferred neural rendering: image synthesis using neural textures. ACM Trans. Graph., 38(4):66:1-66:12, 2019. 2
  64. Richard Tucker and Noah Snavely. Single-view view synthesis with multiplane images. In CVPR, pages 548-557. IEEE, 2020. 2
  65. Aäron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learning. In NIPS, pages 6306-6315, 2017. 3, 4
  66. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, pages 5998-6008, 2017. 3, 4, 11
  67. Olivia Wiles, Georgia Gkioxari, Richard Szeliski, and Justin Johnson. Synsin: End-to-end view synthesis from a single image. In CVPR, pages 7465-7475. IEEE, 2020. 1, 2, 7, 12, 15
  68. Junyuan Xie, Ross B. Girshick, and Ali Farhadi. Deep3d: Fully automatic 2d-to-3d video conversion with deep convolutional neural networks. In ECCV (4), volume 9908 of Lecture Notes in Computer Science, pages 842-857. Springer, 2016. 2
  69. Zexiang Xu, Sai Bi, Kalyan Sunkavalli, Sunil Hadap, Hao Su, and Ravi Ramamoorthi. Deep view synthesis from sparse photometric images. ACM Trans. Graph., 38(4):76:1-76:13, 2019. 2
  70. Jimei Yang, Scott E. Reed, Ming-Hsuan Yang, and Honglak Lee. Weakly-supervised disentangling with recurrent transformations for 3d view synthesis. In NIPS, pages 1099-1107, 2015. 2
  71. Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In CVPR, pages 586-595. IEEE Computer Society, 2018. 5, 6
  72. Tinghui Zhou, Matthew Brown, Noah Snavely, and David G. Lowe. Unsupervised learning of depth and ego-motion from video. In CVPR, pages 6612-6619. IEEE Computer Society, 2017. 2
  73. Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. Stereo magnification: learning view synthesis using multiplane images. ACM Trans. Graph., 37(4):65:1-65:12, 2018. 2, 6, 12
  74. Tinghui Zhou, Shubham Tulsiani, Weilun Sun, Jitendra Malik, and Alexei A. Efros. View synthesis by appearance flow. In ECCV (4), volume 9908 of Lecture Notes in Computer Science, pages 286-301. Springer, 2016. 2

Cited by

Sorted by citations

Authors
Rombach, R., Esser, P., & Ommer, B.
Affiliations
Ludwig Maximilian University of Munich & Heidelberg University,Germany
Venue
2021 IEEE/CVF International Conference on Computer Vision (ICCV)
Publisher
IEEE
Published
2021-10
Pages
14336-14346
DOI
10.1109/iccv48922.2021.01409
Cited by
69
References
4
Licence

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review