Language-driven synthesis of 3D scenes from scene databases
Open accessThe DOI matches its Crossref record; 44 of the 44 references it cites were found in a registry or an index.
Abstract
We introduce a novel framework for using natural language to generate and edit 3D indoor scenes, harnessing scene semantics and text-scene grounding knowledge learned from large annotated 3D scene databases. The advantage of natural language editing interfaces is strongest when performing semantic operations at the sub-scene level, acting on groups of objects. We learn how to manipulate these sub-scenes by analyzing existing 3D scenes. We perform edits by first parsing a natural language command from the user and transforming it into a semantic scene graph that is used to retrieve corresponding sub-scenes from the databases that match the command. We then augment this retrieved sub-scene by incorporating other objects that may be implied by the scene context. Finally, a new 3D scene is synthesized by aligning the augmented sub-scene with the user's current scene, where new objects are spliced into the environment, possibly triggering appropriate adjustments to the existing scene arrangement. A suggestive modeling interface with multiple interpretations of user commands is used to alleviate ambiguities in natural language. We conduct studies comparing our approach against both prior text-to-scene work and artist-made scenes and find that our method significantly outperforms prior work and is comparable to handmade scenes even when complex and varied natural sentences are used.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Existing Embodied AI environments are typically crafted through manual design [6, 13, 26, 27], 3D scanning [8, 43, 45], or procedurally generated with hard-coded rules [7].”
From Holodeck: Language Guided Generation of 3D Embodied AI Environments (2024) · cites this paper as [27]
Citations
Cites
The Stanford CoreNLP Natural Language Processing Toolkit
Manning, C., Surdeanu, M., Bauer, J., Finkel, J., Bethard, S., & McClosky, D. (2014) · Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations · doi:10.3115/v1/p14-5010 · Cited by 7,037
Indoor segmentation and support inference from rgbd images
Silberman, N., Hoiem, D., Kohli, P., & Fergus, R. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-33715-4_54 · Cited by 5,760
Approximate is Better than “Exact” for Interval Estimation of Binomial Proportions
Agresti, A., & Coull, B. A. (1998) · The American Statistician · doi:10.1080/00031305.1998.10480550 · Cited by 3,223
A Comparison and Evaluation of Multi-View Stereo Reconstruction Algorithms
Seitz, S. M., Curless, B., Diebel, J., Scharstein, D., & Szeliski, R. (2006) · 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Volume 1 (CVPR'06) · doi:10.1109/cvpr.2006.19 · Cited by 2,457
ShapeNet: An Information-Rich 3D Model Repository
Chang, A. X., Funkhouser, T., Guibas, L., Hanrahan, P., Huang, Q., Li, Z., Savarese, S., Savva, M., Song, S., Su, H., et al. (2015) · arXiv · doi:10.48550/arxiv.1512.03012 · Cited by 2,330
Self-citationSUN RGB-D: A RGB-D scene understanding benchmark suite
Song, S., Lichtenberg, S. P., & Xiao, J. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298655 · Cited by 1,943 · cited 1×
“Recent advances in computational design, VR/AR, and robotics are placing an increasing demand for rich 3D content, especially those in indoor environments [Hua et al. 2016; Silberman et al. 2012; Song et al. 2015, 2017].”
BackgroundShapeNet: An Information-Rich 3D Model Repository
Chang, A. X., Funkhouser, T. A., Guibas, L. J., Hanrahan, P., Huang, Q., Li, Z., Savarese, S., Savva, M., Song, S., Su, H., et al. (2023) · doi:10.5281/zenodo.10089705 · Cited by 1,623
Self-citationSemantic Scene Completion from a Single Depth Image
Song, S., Yu, F., Zeng, A., Chang, A. X., Savva, M., & Funkhouser, T. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.28 · Cited by 1,209
Approximate Is Better than "Exact" for Interval Estimation of Binomial Proportions
Agresti, A., & Coull, B. A. (1998) · The American Statistician · doi:10.2307/2685469 · Cited by 972
KnowRob: A knowledge processing infrastructure for cognition-enabled robots
Tenorth, M., & Beetz, M. (2013) · The International Journal of Robotics Research · doi:10.1177/0278364913481635 · Cited by 388
WordsEye: an automatic text-to-scene conversion system
Coyne, B., & Sproat, R. (2001) · Proceedings of the 28th annual conference on Computer graphics and interactive techniques · doi:10.1145/383259.383316 · Cited by 377
Example-based synthesis of 3D object arrangements
Fisher, M., Ritchie, D., Savva, M., Funkhouser, T., & Hanrahan, P. (2012) · ACM Transactions on Graphics · doi:10.1145/2366145.2366154 · Cited by 341
Self-citationSceneNN: A Scene Meshes Dataset with aNNotations
Hua, B.-S., Pham, Q.-H., Nguyen, D. T., Tran, M.-K., Yu, L.-F., & Yeung, S.-K. (2016) · 2016 Fourth International Conference on 3D Vision (3DV) · doi:10.1109/3dv.2016.18 · Cited by 333
Self-citationMake it home: automatic optimization of furniture arrangement
Yu, L.-F., Yeung, S.-K., Tang, C.-K., Terzopoulos, D., Chan, T. F., & Osher, S. J. (2011) · ACM Transactions on Graphics · doi:10.1145/2010324.1964981 · Cited by 276
Self-citationInteractive furniture layout using interior design guidelines
Merrell, P., Schkufza, E., Li, Z., Agrawala, M., & Koltun, V. (2011) · ACM Transactions on Graphics · doi:10.1145/2010324.1964982 · Cited by 260
Make it home: automatic optimization of furniture arrangement
Yu, L.-F., Yeung, S.-K., Tang, C.-K., Terzopoulos, D., Chan, T. F., & Osher, S. J. (2011) · ACM SIGGRAPH 2011 papers · doi:10.1145/1964921.1964981 · Cited by 254
Self-citationInteractive furniture layout using interior design guidelines
Merrell, P., Schkufza, E., Li, Z., Agrawala, M., & Koltun, V. (2011) · ACM SIGGRAPH 2011 papers · doi:10.1145/1964921.1964982 · Cited by 234
An interactive approach to semantic modeling of indoor scenes with an RGBD camera
Shao, T., Xu, W., Zhou, K., Wang, J., Li, D., & Guo, B. (2012) · ACM Transactions on Graphics · doi:10.1145/2366145.2366155 · Cited by 218
Deep convolutional priors for indoor scene synthesis
Wang, K., Savva, M., Chang, A. X., & Ritchie, D. (2018) · ACM Transactions on Graphics · doi:10.1145/3197517.3201362 · Cited by 204
A Survey of Methods for Volumetric Scene Reconstruction from Photographs
Slabaugh, G., Schafer, R., Malzbender, T., & Culbertson, B. (2001) · Eurographics · doi:10.1007/978-3-7091-6756-4_6 · Cited by 197
Acquiring 3D indoor environments with variability and repetition
Kim, Y. M., Mitra, N. J., Yan, D.-M., & Guibas, L. (2012) · ACM Transactions on Graphics · doi:10.1145/2366145.2366157 · Cited by 174
Self-citationA Survey of Methods for Volumetric Scene Reconstruction from Photographs
Slabaugh, G., Culbertson, B., Malzbender, T., et al. (2001) · City Research Online (City University London) · doi:10.2312/vg/vg01/081-101 · Cited by 161
Learning the Visual Interpretation of Sentences
Zitnick, C. L., Parikh, D., & Vanderwende, L. (2013) · 2013 IEEE International Conference on Computer Vision · doi:10.1109/iccv.2013.211 · Cited by 153
Learning Spatial Knowledge for Text to 3D Scene Generation
Chang, A., Savva, M., & Manning, C. D. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1217 · Cited by 153
Characterizing structural relationships in scenes using graph kernels
Fisher, M., Savva, M., & Hanrahan, P. (2011) · ACM SIGGRAPH 2011 papers · doi:10.1145/1964921.1964929 · Cited by 152
Self-citationGrounding spatial relations for human-robot interaction
Guadarrama, S., Riano, L., Golland, D., Gouhring, D., Jia, Y., Klein, D., Abbeel, P., & Darrell, T. (2013) · 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems · doi:10.1109/iros.2013.6696569 · Cited by 140
PiGraphs: learning interaction snapshots from observations
Savva, M., Chang, A. X., Hanrahan, P., Fisher, M., & Nießner, M. (2016) · ACM Transactions on Graphics · doi:10.1145/2897824.2925867 · Cited by 131
Self-citationCharacterizing structural relationships in scenes using graph kernels
Fisher, M., Savva, M., & Hanrahan, P. (2011) · ACM Transactions on Graphics · doi:10.1145/2010324.1964929 · Cited by 131
Self-citationActivity-centric scene synthesis for functional 3D scene modeling
Fisher, M., Savva, M., Li, Y., Hanrahan, P., & Nießner, M. (2015) · ACM Transactions on Graphics · doi:10.1145/2816795.2818057 · Cited by 122
Self-citationSemantically-enriched 3D models for common-sense knowledge
Savva, M., Chang, A. X., & Hanrahan, P. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2015.7301289 · Cited by 101
Style compatibility for 3D furniture models
Liu, T., Hertzmann, A., Li, W., & Funkhouser, T. A. (2015) · ACM Transactions on Graphics · doi:10.1145/2766898 · Cited by 89
Text to 3D Scene Generation with Rich Lexical GroundingPreprint
Chang, A., Monroe, W., Savva, M., Potts, C., & Manning, C. D. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1505.06289 · Cited by 66
Interaction context (ICON): towards a geometric functionality descriptor
Hu, R., Zhu, C., van Kaick, O. M., Liu, L., Shamir, A., & Zhang, H. (2015) · ACM Transactions on Graphics · doi:10.1145/2766914 · Cited by 65
Self-citationThe Clutterpalette: An Interactive Tool for Detailing Indoor Scenes
Yu, L.-F., Yeung, S.-K., & Terzopoulos, D. (2015) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2015.2417575 · Cited by 59
Self-citationTell Me Dave: Context-Sensitive Grounding of Natural Language to Manipulation Instructions
Kumar Misra, D., Sung, J., Lee, K., & Saxena, A. (2014) · Robotics: Science and Systems X · doi:10.15607/rss.2014.x.005 · Cited by 59
Action-driven 3D indoor scene evolution
Ma, R., Li, H., Zou, C., Liao, Z., Tong, X., & Zhang, H. (2016) · ACM Transactions on Graphics · doi:10.1145/2980179.2980223 · Cited by 57
Self-citationLearning Object Arrangements in 3D Scenes using Human Context
Yun, J., Lim, M., & Saxena, A. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1206.6462 · Cited by 56
Real-time automatic 3D scene generation from natural language voice and text descriptions
Seversky, L. M., & Yin, L. (2006) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/1180639.1180660 · Cited by 54
3D attention-driven depth acquisition for object identification
Xu, K., Shi, Y., Zheng, L., Zhang, J., Liu, M., Huang, H. F., Su, H., Cohen‐Or, D., & Chen, B. (2016) · ACM Transactions on Graphics · doi:10.1145/2980179.2980224 · Cited by 50
Organizing heterogeneous scene collections through contextual focal points
Xu, K., Ma, R., Zhang, H., Zhu, C., Shamir, A., Cohen‐Or, D., & Huang, H. F. (2014) · ACM Transactions on Graphics · doi:10.1145/2601097.2601109 · Cited by 49
Self-citationLearning 3D Scene Synthesis from Annotated RGB‐D Images
Kermani, Z. S., Liao, Z., Tan, P., & Zhang, H. (2016) · Computer Graphics Forum · doi:10.1111/cgf.12976 · Cited by 48
Self-citationInteractive Learning of Spatial Knowledge for Text to 3D Scene Generation
Chang, A., Savva, M., & Manning, C. D. (2014) · doi:10.3115/v1/w14-3102 · Cited by 39
Filling Your Shelves: Synthesizing Diverse Style-Preserving Artifact Arrangements
Majerowicz, L., Shamir, A., Sheffer, A., & Hoos, H. H. (2013) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2013.245 · Cited by 32
Annotation Tools and Knowledge Representation for a Text-To-Scene System
Coyne, R. E., Klapheke, A., Rouhizadeh, M., Sproat, R., & Bauer, D. (2012) · Columbia Academic Commons (Columbia University) · doi:10.7916/d84f205r · Cited by 19
Research works printed but not linked: 2
- Angel X. Chang , Mihail Eric , Manolis Savva , and Christopher D . Manning . 2017 . SceneSeer: 3D Scene Design with Natural Language. CoRR abs/1703.00050 (2017). http://arxiv.org/abs/1703.00050 Angel X. Chang, Mihail Eric, Manolis Savva, and Christopher D. Manning. 2017. SceneSeer: 3D Scene Design with Natural Language. CoRR abs/1703.00050 (2017). http://arxiv.org/abs/1703.00050 · arXiv
- 10.5555/2386438.2386446 · doi:10.5555/2386438.2386446
Cited by
Synthetic Data for Deep Learning
Nikolenko, S. I. (2021) · Springer Optimization and Its Applications · doi:10.1007/978-3-030-75178-4 · Cited by 423
Learning 3D Semantic Scene Graphs From 3D Indoor Reconstructions
Wald, J., Dhamo, H., Navab, N., & Tombari, F. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00402 · Cited by 244
GRAINS: Generative Recursive Autoencoders for INdoor Scenes
Li, M., Patil, A. G., Xu, K., Chaudhuri, S., Khan, O., Shamir, A., Tu, C., Chen, B., Cohen-Or, D., & Zhang, H. (2019) · ACM Transactions on Graphics · doi:10.1145/3303766 · Cited by 184
Self-citationPlanIT: planning and instantiating indoor scenes with relation graph and spatial prior networks
Wang, K., Lin, Y.-A., Weissmann, B., Savva, M., Chang, A. X., & Ritchie, D. (2019) · ACM Transactions on Graphics · doi:10.1145/3306346.3322941 · Cited by 195
SceneFormer: Indoor Scene Generation with Transformers
Wang, X., Yeshwanth, C., & Niesner, M. (2021) · 2021 International Conference on 3D Vision (3DV) · doi:10.1109/3dv53792.2021.00021 · Cited by 114
Metadata
- Authors
- Ma, R., Patil, A. G., Fisher, M., Li, M., Pirk, S., Hua, B.-S., Yeung, S.-K., Tong, X., Guibas, L., & Zhang, H.
- Affiliations
- Simon Fraser University (CA)
- Adobe Systems (United States) (US)
- Shandong University (CN)
- Stanford University (US)
- The University of Tokyo (JP)
- Hong Kong University of Science and Technology (HK)
- Microsoft Research Asia (China) (CN)
- ORCID
- Venue
- ACM Transactions on Graphics
- Published
- 2018-12-04
- Volume
- 37
- Issue
- 6
- Pages
- 1-16
- ISSN
- 0730-0301 (ISSN-L)
- 1557-7368
- OpenAlex
- W2902539442
- Cited by
- 100
- References
- 42
- Citations by year
- 2026: 7 · 2025: 15 · 2024: 19 · 2023: 17 · 2022: 6 · 2021: 16 · 2020: 12 · 2019: 7 · 2018: 1
- FWCI
- 6.63
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- ACM Transactions on Graphics (publishedVersion)
- Rare & Special e-Zone (The Hong Kong University of Science and Technology) (submittedVersion)
- Funders
- Natural Sciences and Engineering Research Council of Canada (611370)
- National Science Foundation (IIS-1528025)
- Ministry of Education - Singapore (MOE2016-T2-2-154)
- Amazon Web Services
- Adobe Systems
- Hong Kong University of Science and Technology (R9429)
- Stanford AI Lab-Toyota Center for Artificial Intelligence Research
- Topics
- Physical Sciences › Computer Science › Computer Graphics and Computer-Aided Design › 3D Shape Modeling and Analysis
- Physical Sciences › Computer Science › Computer Graphics and Computer-Aided Design › Computer Graphics and Visualization Techniques
- Physical Sciences › Computer Science › Human-Computer Interaction › Interactive and Immersive Displays
- Keywords
- 3D scene synthesis, indoor scenes, scene semantics, scene graph, natural language ambiguity
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.