Language-driven synthesis of 3D scenes from scene databases
Open accessThe DOI matches its Crossref record; 45 of the 45 references it cites were found in a registry or an index.
Abstract
We introduce a novel framework for using natural language to generate and edit 3D indoor scenes, harnessing scene semantics and text-scene grounding knowledge learned from large annotated 3D scene databases. The advantage of natural language editing interfaces is strongest when performing semantic operations at the sub-scene level, acting on groups of objects. We learn how to manipulate these sub-scenes by analyzing existing 3D scenes. We perform edits by first parsing a natural language command from the user and transforming it into a semantic scene graph that is used to retrieve corresponding sub-scenes from the databases that match the command. We then augment this retrieved sub-scene by incorporating other objects that may be implied by the scene context. Finally, a new 3D scene is synthesized by aligning the augmented sub-scene with the user's current scene, where new objects are spliced into the environment, possibly triggering appropriate adjustments to the existing scene arrangement. A suggestive modeling interface with multiple interpretations of user commands is used to alleviate ambiguities in natural language. We conduct studies comparing our approach against both prior text-to-scene work and artist-made scenes and find that our method significantly outperforms prior work and is comparable to handmade scenes even when complex and varied natural sentences are used.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Existing Embodied AI environments are typically crafted through manual design [6, 13, 26, 27], 3D scanning [8, 43, 45], or procedurally generated with hard-coded rules [7].”
From Holodeck: Language Guided Generation of 3D Embodied AI Environments (2024) · cites this paper as [27]
Citations
Cites
The Stanford CoreNLP Natural Language Processing Toolkit
Manning, C., Surdeanu, M., Bauer, J., Finkel, J., Bethard, S., & McClosky, D. (2014) · Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations · doi:10.3115/v1/p14-5010 · Cited by 7,037 · cited 1×
“We use the Stanford CoreNLP framework [Manning et al. 2014] to perform part-of-speech tagging and convert input statements into a dependency tree.”
MethodIndoor segmentation and support inference from rgbd images
Silberman, N., Hoiem, D., Kohli, P., & Fergus, R. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-33715-4_54 · Cited by 5,760
Approximate is Better than “Exact” for Interval Estimation of Binomial Proportions
Agresti, A., & Coull, B. A. (1998) · The American Statistician · doi:10.1080/00031305.1998.10480550 · Cited by 3,223 · cited 1×
“Based on Wilson score confidence interval for Bernoulli trials [Agresti and Coull 1998], there is a 95% chance that human users would prefer our method at least 43.3% of the time.”
MethodInfluentialA Comparison and Evaluation of Multi-View Stereo Reconstruction Algorithms
Seitz, S. M., Curless, B., Diebel, J., Scharstein, D., & Szeliski, R. (2006) · 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Volume 1 (CVPR'06) · doi:10.1109/cvpr.2006.19 · Cited by 2,457 · cited 2×
“, [Seitz et al. 2006; Slabaugh et al. 2001; Xiao 2012], to name just a few, and most recent works are taking a data-driven approach, e.”
BackgroundShapeNet: An Information-Rich 3D Model Repository
Chang, A. X., Funkhouser, T., Guibas, L., Hanrahan, P., Huang, Q., Li, Z., Savarese, S., Savva, M., Song, S., Su, H., et al. (2015) · arXiv · doi:10.48550/arxiv.1512.03012 · Cited by 2,330 · cited 3×
“We also encode enriched object annotations from ShapeNetSem as node attributes to describe the refined properties of an object.”
MethodSelf-citationSUN RGB-D: A RGB-D scene understanding benchmark suite
Song, S., Lichtenberg, S. P., & Xiao, J. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298655 · Cited by 1,943 · cited 1×
“Recent advances in computational design, VR/AR, and robotics are placing an increasing demand for rich 3D content, especially those in indoor environments [Hua et al. 2016; Silberman et al. 2012; Song et al. 2015, 2017].”
BackgroundShapeNet: An Information-Rich 3D Model Repository
Chang, A. X., Funkhouser, T. A., Guibas, L. J., Hanrahan, P., Huang, Q., Li, Z., Savarese, S., Savva, M., Song, S., Su, H., et al. (2023) · doi:10.5281/zenodo.10089705 · Cited by 1,623
Self-citationSemantic Scene Completion from a Single Depth Image
Song, S., Yu, F., Zeng, A., Chang, A. X., Savva, M., & Funkhouser, T. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.28 · Cited by 1,209 · cited 3×
“Our scene database includes 2984 3D scenes from SUNCG dataset [Song et al. 2017], 133 scenes from SceneSynth [Fisher et al. 2012] and 80 scenes modeled from the real-world SceneNN dataset [Hua et al. 2016].”
MethodApproximate Is Better than "Exact" for Interval Estimation of Binomial Proportions
Agresti, A., & Coull, B. A. (1998) · The American Statistician · doi:10.2307/2685469 · Cited by 972
KnowRob: A knowledge processing infrastructure for cognition-enabled robots
Tenorth, M., & Beetz, M. (2013) · The International Journal of Robotics Research · doi:10.1177/0278364913481635 · Cited by 388 · cited 2×
“However, both problems and their solutions share commonalities, including the need to utilize both rule-based models and data-driven learning of spatial and semantic relations, as well as the exploitation of common sense knowledge [Tenorth and Beetz 2013] to incorporate unstated facts.”
Background
Research works printed but not linked: 1
- 10.5555/2386438.2386446 · doi:10.5555/2386438.2386446
Cited by
Synthetic Data for Deep Learning
Nikolenko, S. I. (2021) · Springer Optimization and Its Applications · doi:10.1007/978-3-030-75178-4 · Cited by 423
“The original paper [557] presented state of the art results in semantic scene completion, but, naturally, SUNCG has been used for many different tasks related to depth estimation, indoor navigation, SLAM, and others [2,108,231,385,465], and it often serves as the basis for scene understanding competitions [566].”
MethodLearning 3D Semantic Scene Graphs From 3D Indoor Reconstructions
Wald, J., Dhamo, H., Navab, N., & Tombari, F. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00402 · Cited by 244
GRAINS: Generative Recursive Autoencoders for INdoor Scenes
Li, M., Patil, A. G., Xu, K., Chaudhuri, S., Khan, O., Shamir, A., Tu, C., Chen, B., Cohen-Or, D., & Zhang, H. (2019) · ACM Transactions on Graphics · doi:10.1145/3303766 · Cited by 184
“A recent work of Ma et al. [2018] uses language as a prior to guide the scene synthesis process.”
MethodSelf-citationPlanIT: planning and instantiating indoor scenes with relation graph and spatial prior networks
Wang, K., Lin, Y.-A., Weissmann, B., Savva, M., Chang, A. X., & Ritchie, D. (2019) · ACM Transactions on Graphics · doi:10.1145/3306346.3322941 · Cited by 195
“There exists prior work in language-based scene creation [Chang et al. 2015, 2014], including recent work that uses a graph-based intermediate representation [Ma et al. 2018a].”
MethodSceneFormer: Indoor Scene Generation with Transformers
Wang, X., Yeshwanth, C., & Niesner, M. (2021) · 2021 International Conference on 3D Vision (3DV) · doi:10.1109/3dv53792.2021.00021 · Cited by 114
“Text inputs have been used to create detailed partial scenes [21, 6], such as a table with several small objects on it, which are then inserted into the larger scene.”
MethodHolodeck: Language Guided Generation of 3D Embodied AI Environments
Yang, Y., Sun, F.-Y., Weihs, L., Vanderbilt, E., Herrasti, A., Han, W., Wu, J., Haber, N., Krishna, R., Liu, L., et al. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.01536 · Cited by 101
“Many works on scene design either learn spatial knowledge priors from existing 3D scene databases [4, 31, 49–51, 54, 59] or leverage user input and refine the 3D scene iteratively [3, 5].”
Ning, X., Yu, Z., Li, L., Li, W., & Tiwari, P. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.102033 · Cited by 101
VRCopilot: Authoring 3D Layouts with Generative AI Models in VR
Zhang, L., Pan, J., Gettig, J., Oney, S., & Guo, A. (2024) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3654777.3676451 · Cited by 111
“Prior work has demonstrated promising results in generating realistic 3D layouts [34, 45, 57] and text-to-layout generation [20, 39].”
Ma, P., Du, T., Zhang, J. Z., Wu, K., Spielberg, A., Katzschmann, R. K., & Matusik, W. (2021) · ACM Transactions on Graphics · doi:10.1145/3450626.3459832 · Cited by 95
Graph-to-3D: End-to-End Generation and Manipulation of 3D Scenes Using Scene Graphs
Dhamo, H., Manhardt, F., Navab, N., & Tombari, F. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01604 · Cited by 126
“[21] convert text to a scene graph with pairwise and group relationships, to progressively retrieve subscenes for 3D synthesis.”
MethodSceneHGN: Hierarchical Graph Networks for 3D Indoor Scene Generation With Fine-Grained Geometry
Gao, L., Sun, J.-M., Mo, K., Lai, Y., Guibas, L. J., & Yang, J. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3237577 · Cited by 94
“There are also other works learning to produce furniture layout under language [47], activity [48], human [49] and action [50] constraints.”
BackgroundSelf-citationLearning Generative Models of 3D Structures
Chaudhuri, S., Ritchie, D., Wu, J., Xu, K., & Zhang, H. (2020) · Computer Graphics Forum · doi:10.1111/cgf.14020 · Cited by 61
Self-citationA Survey of 3D Indoor Scene Synthesis
Zhang, S.-H., Zhang, S.-K., Liang, Y., & Hall, P. M. (2019) · Journal of Computer Science and Technology · doi:10.1007/s11390-019-1929-5 · Cited by 73
“…et al. [93] (2018) Initial scene or scene types Graph Context User interaction & MAP Ma et al. [43] (2016) Initial scene - Action graphs Action sequence (Markov chain) Ma et al. [94] User interaction (using text) Merrell et al. [3] (2011) Room shape & furniture set - Hard-coded layout guidelines”
MethodATISS: Autoregressive Transformers for Indoor Scene Synthesis
Paschalidou, D., Kar, A., Shugrina, M., Kreis, K., Geiger, A., & Fidler, S. (2021) · arXiv · doi:10.48550/arxiv.2110.03675 · Cited by 62
Fast 3D Indoor Scene Synthesis by Learning Spatial Relation Priors of Objects
Zhang, S., Zhang, S.-K., Xie, W., Luo, C., Yang, Y., & Fu, H. (2021) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2021.3050143 · Cited by 56
“A few works (e.g., [9], [10], [11], [12]) attempt to simplify layouts into independent cliques or subsets e.g., [10], [11].”
BackgroundLanguage-based Photo Color Adjustment for Graphic Designs
Wang, Z., Zhao, N., Hancke, G. P., & Lau, R. W. H. (2023) · ACM Transactions on Graphics · doi:10.1145/3592111 · Cited by 54
“However, we take advantage of the recent success of language-based interactions used in various computer vision tasks [Bahng et al. 2018; Chen et al. 2018; Jiang et al. 2021; Lüddecke and Ecker 2022; Ma et al. 2018; Weng et al. 2022; Zou et al. 2019], and present a language-based system for photo color adjustment in t…”
MethodHow People Prompt Generative AI to Create Interactive VR Scenes
Manesh, S. A., Zhang, T., Onishi, Y., Hara, K., Bateman, S., Li, J., & Tang, A. (2024) · Designing Interactive Systems Conference · doi:10.1145/3643834.3661547 · Cited by 59
“While early work already proposed language-based 3D model retrieval and scene composition [37, 49], recent breakthroughs in large multi-modal generative models have significantly expanded the range of vocabularies and intents such methods can understand and execute (e.g., [16]).”
BackgroundXie, H., Wen, B., Chen, Z., Hong, F., & Liu, Z. (2026) · International Journal of Computer Vision · doi:10.1007/s11263-026-02999-6 · Cited by 51
“Early data-driven approaches (Chang et al., 2014; Ker-mani et al., 2016; Fu et al., 2017; Ma et al., 2018) represent spatial relationships between objects using scene graphs, which serve as a blueprint for 3D scene generation through object retrieval and placement.”
MageAdd: Real-Time Interaction Simulation for Scene Synthesis
Zhang, S.-K., Li, Y., He, Y. H., Yang, Y., & Zhang, S. (2021) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3474085.3475194 · Cited by 37
“Human languages are also decoded as graphs for scene generations [2, 20].”
BackgroundLearning 3D Semantic Scene Graphs with Instance Embeddings
Wald, J., Navab, N., & Tombari, F. (2022) · International Journal of Computer Vision · doi:10.1007/s11263-021-01546-9 · Cited by 42
“Similarly, Ma et al. (2018) parse natural language into graphs and retrieve 3D scenes that fulfill requested compositions.”
BackgroundIndoor Scene Generation from a Collection of Semantic-Segmented Depth Images
Yang, M., Guo, Y., Zhou, B., & Tong, X. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01492 · Cited by 32
Self-citationAdvances in Data‐Driven Analysis and Synthesis of 3D Indoor Scenes
Patil, A. G., Patil, S. G., Li, M., Fisher, M., Savva, M., & Zhang, H. (2023) · Computer Graphics Forum · doi:10.1111/cgf.14927 · Cited by 41
Self-citationChat2Layout: Interactive 3D Furniture Layout With a Multimodal LLM
Wang, C., Zhong, H., Chai, M., He, M., Chen, D., & Liao, J. (2025) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2025.3626731 · Cited by 36
“Although rules for scene graph can be automatically derived from the scenes [1], [4], [6], [7], [8], the rule extraction process itself is complex and often depends on large datasets, making these approaches less flexible in dynamic environments.”
Learning meaningful controls for fluids
Chu, M., Thuerey, N., Seidel, H., Theobalt, C., & Zayer, R. (2021) · ACM Transactions on Graphics · doi:10.1145/3450626.3459845 · Cited by 30
Text2VRScene: Exploring the Framework of Automated Text-driven Generation System for VR Experience
Yin, Z., Wang, Y., Papatheodorou, T., & Hui, P. (2024) · IEEE Conference on Virtual Reality and 3D User Interfaces (VR) · doi:10.1109/vr58804.2024.00090 · Cited by 40
“To explore the automated generation of VR experience, several recent studies have focused on generating or reconstructing 3D scenes [10,14,28,44,52], but few of these studies have been able to automate the massive production of satisfying VR experiences.”
BackgroundSceneCraft: An LLM Agent for Synthesizing 3D Scene as Blender CodePreprint
Hu, Z., İşcen, A., Jain, A., Kipf, T., Yue, Y., Ross, D. A., Schmid, C., & Fathi, A. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2403.01248 · Cited by 109
“This system, and its follow-up works (Sever-sky & Yin, 2006; Chang et al., 2014; Ma et al., 2018), can generate 3D scenes from natural language.”
BackgroundSceneDirector: Interactive Scene Synthesis by Simultaneously Editing Multiple Objects in Real-Time
Zhang, S.-K., Tam, H. I. I., Li, Y., Ren, K.-X., Fu, H., & Zhang, S. (2023) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2023.3268115 · Cited by 30
“The techniques in [34] and [35] interpret texts into scene graphs for generating scenes.”
MethodGeometry-Based Layout Generation with Hyper-Relations AMONG Objects
Zhang, S.-K., Xie, W., & Zhang, S. (2021) · Graphical Models · doi:10.1016/j.gmod.2021.101104 · Cited by 27
ShapeTalk: A Language Dataset and Framework for 3D Shape Edits and Deformations
Achlioptas, P., Huang, I., Sung, M., Tulyakov, S., & Guibas, L. J. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01220 · Cited by 32
Self-citationMotion Planning for Convertible Indoor Scene Layout Design
Xiong, G., Fu, Q., Fu, H., Zhou, B., Luo, G., & Deng, Z. (2020) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2020.3005680 · Cited by 25
“, [11], [12], [13]), human-object relationships (e.”
SceneGen: Generative Contextual Scene Augmentation using Scene Graph PriorsPreprint
Keshavarzi, M., Parikh, A., Zhai, X., Mao, M., Caldas, L., & Yang, A. Y. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.12395 · Cited by 25
“More specically, (Ma et al. 2018) bears a minor resemblance to our approach, in 1) training on object relations, and 2) the ability to augment an initial input scene, but unlike our work, it augments scenes by merging in subscenes retrieved from a database.”
BackgroundInfluentialLUMINOUS: Indoor Scene Generation for Embodied AI ChallengesPreprint
Zhao, Y., Lin, K., Jia, Z., Gao, Q., Thattai, G., Thomason, J., & Sukhatme, G. S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.05527 · Cited by 26
“Other lines of research condition on the image [24, 27] or text [28, 29] representation of indoor scenes.”
BackgroundHSM: Hierarchical Scene Motifs for Multi-Scale Indoor Scene Generation
Pun, H. I. D., Tam, H. I. I., Wang, A. T., Huo, X., Chang, A. X., & Savva, M. (2026) · International Conference on 3D Vision (3DV) · doi:10.1109/3dv69130.2026.00131 · Cited by 33
“Of particular interest are methods that generate scenes from text [40, 69, 84, 85], with recent works increasingly incorporating large language models (LLMs) into the process [3, 5, 6, 11, 13, 15, 21, 28, 42– 44, 46, 47, 63, 64, 66, 70, 75, 77, 79, 82, 83].”
Haisor : Human-aware Indoor Scene Optimization via Deep Reinforcement Learning
Sun, J.-M., Yang, J., Mo, K., Lai, Y., Guibas, L. J., & Gao, L. (2023) · ACM Transactions on Graphics · doi:10.1145/3632947 · Cited by 27
“…solutions have been proposed by considering different settings, such as example-driven methods [Fisher et al. 2012], language-driven synthesis [Ma et al. 2018], learning-based methods [Kermani et al. 2016; Merrell et al. 2011; Zhang et al. 2022], and optimization-based methods [Henderson and…”
MethodSelf-citationSystematic Literature Review of Using Virtual Reality as a Social Platform in HCI Community
Wei, X., Jin, X., Kan, G. L., Yan, Y., & Fan, M. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3711078 · Cited by 21
“Previous studies have explored generating 3D objects or scenes from 2D sketches or text using artificial intelligence-generated content (AIGC) [82, 114, 119].”
Learning to manipulate amorphous materials
Zhang, Y., Yu, W., Liu, C. K., Kemp, C., & Turk, G. (2020) · ACM Transactions on Graphics · doi:10.1145/3414685.3417868 · Cited by 23
Write‐An‐Animation: High‐level Text‐based Animation Editing with Character‐Scene Interaction
Zhang, J., Xu, X., Shen, Z., Huang, Z., Zhao, Y., Cao, Y., Wan, P., & Wang, M. (2021) · Computer Graphics Forum · doi:10.1111/cgf.14415 · Cited by 22
Kim, S., Kim, D., Shin, J.-E., & Woo, W. (2024) · IEEE Conference on Virtual Reality and 3D User Interfaces (VR) · doi:10.1109/vr58804.2024.00099 · Cited by 17
“They have largely been employed to automatically arrange or move objects in a room based on their relationships for scene synthesis [28, 43–45].”
MethodSceneEval: Evaluating Semantic Coherence in Text-Conditioned 3D Indoor Scene Synthesis
Tam, H. I. I., Pun, H. I. D., Wang, A. T., Chang, A. X., & Savva, M. (2026) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv61042.2026.00710 · Cited by 40
“Subsequent work has focused on automating generation with rule-based [10–12, 43, 61], data-driven [9, 16, 29, 38, 45, 62] and deep learning methods [24, 33, 34, 41, 42, 44, 49– 51, 54, 55, 58, 63, 64].”
Scene-Context-Aware Indoor Object Selection and Movement in VR
Wang, M., Ye, Z.-M., Shi, J.-C., & Yang, Y.-L. (2021) · IEEE Conference on Virtual Reality and 3D User Interfaces (VR) · doi:10.1109/vr50410.2021.00045 · Cited by 16
“Note that scene contexts have also been employed to improve furniture arrangement computation as in [32,34,66].”
MethodPatch-Based 3D Natural Scene Generation from a Single Example
Li, W., Chen, X., Wang, J., & Chen, B. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01608 · Cited by 20
“Research has taken several routes, among which a prevalent one is learning to extract common patterns of the geometry or appearance from homogeneous scene samples, such as indoor scenes [14, 25, 34, 37, 59, 63, 71, 72, 75], terrains [15,19,21,26], urban scenes [12,30,44], etc.”
BackgroundActive Arrangement of Small Objects in 3D Indoor Scenes
Zhang, S., Han, Z., Lai, Y., Zwicker, M., & Zhang, H. (2019) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2019.2949295 · Cited by 12
“by” [37], which cannot be simply interpreted asmultiple pairwise relations.”
BackgroundSelf-citationProcedural Generation of Narrative Worlds
Balint, J. T., & Bidarra, R. (2023) · IEEE Transactions on Games · doi:10.1109/tg.2022.3216582 · Cited by 11
“SceneSeer [13] and Ma et al. [28] place objects in a room based on natural language text.”
BackgroundExternal Knowledge Enhanced 3D Scene Generation from Sketch
Wu, Z., Feng, M., Wang, Y., Xie, H., Dong, W., Miao, B., & Mian, A. S. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72658-3_17 · Cited by 15
A generalized semantic representation for procedural generation of rooms
Balint, J. T., & Bidarra, R. (2019) · International Conference on the Foundations of Digital Games (FDG) · doi:10.1145/3337722.3341848 · Cited by 12
“Their later work [18] adapts a sub-scene model, incorporating heterogeneous sets with specified relationships as sub-motifs with a given probability.”
BackgroundXie, J., Cheng, L., Wang, G., Hu, M., Yu, Z., Du, M., & Ning, X. (2024) · Displays · doi:10.1016/j.displa.2024.102773 · Cited by 10
FreeScene: Mixed Graph Diffusion for 3D Scene Synthesis from Free Prompts
Bai, T., Bai, W., Chen, D., Wu, T., Li, M., & Ma, R. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.00553 · Cited by 18
“Many related works have explored scene synthesis conditioned on text descriptions [3–6, 32, 44], sketches [25, 45, 57], images [19, 21, 35], etc.”
Self-citationMood-Driven Colorization of Virtual Indoor Scenes
Solah, M., Huang, H., Sheng, J., Feng, T., Pomplun, M., & Yu, L.-F. (2022) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2022.3150513 · Cited by 10
Illumination‐Guided Furniture Layout Optimization
Vitsas, N., Παπαϊωάννου, Γ., Gkaravelis, A., & Vasilakis, A. A. (2020) · Computer Graphics Forum · doi:10.1111/cgf.13930 · Cited by 10
Generation of small groups with rich behaviors from natural language interface
Chen, C., Wong, S., & Liu, W. (2020) · Computer Animation and Virtual Worlds · doi:10.1002/cav.1960 · Cited by 11
Placeit3d: Language-Guided Object Placement in Real 3D Scenes
Abdelreheem, A., Aleotti, F., Watson, J., Qureshi, Z., Eldesokey, A., Wonka, P., Brostow, G., Vicente, S., & Garcia-Hernando, G. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.00626 · Cited by 16
“Multiple datasets for language-guided 3D tasks [1–3, 6, 7, 43] have been built on top of ScanNet [10], taking advantage of the readily available annotations.”
A semantic‐driven generation of 3D Chinese opera performance scenes
Liang, H., Dong, X., Liu, X., Pan, J., Zhang, J., & Wang, R. (2022) · Computer Animation and Virtual Worlds · doi:10.1002/cav.2077 · Cited by 9
“7,8 Presently, the 3D performance scene of Chinese Opera still requires substantial manpower to import the model and generate the scene, which entails a heavy workload and lack of intelligence.”
BackgroundConditional room layout generation based on graph neural networks
Yao, Z., Chen, Y., Cui, J., Zhang, S., Li, S., & Hao, A. (2024) · Computers & Graphics · doi:10.1016/j.cag.2024.103971 · Cited by 13
RelScene: A Benchmark and baseline for Spatial Relations in text-driven 3D Scene Generation
Ye, Z., Zheng, X. T., Liu, Y., & Peng, Y. (2024) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3664647.3681653 · Cited by 12
“For example, Ma et al.[12] introduce a natural language framework to edit 3D indoor scenes with an annotated large 3D scene database.”
BackgroundLearning layout generation for virtual worlds
Cheng, W., & Shan, Y. (2024) · Computational Visual Media · doi:10.1007/s41095-023-0365-1 · Cited by 10
“A novel framework [24] considers scene synthesis at a sub-scene level which regularizes the outputs.”
Synthesizing Play-Ready VR Scenes with Natural Language Prompts Through GPT API
Kouzelis, L. R., & Spantidi, O. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-47966-3_2 · Cited by 8
Li, Y., Wang, X., Wu, Z., Li, G., Liu, S., & Zhou, M. (2020) · Computers & Graphics · doi:10.1016/j.cag.2020.08.002 · Cited by 10
Commonsense Knowledge-Driven Joint Reasoning Approach for Object Retrieval in Virtual Reality
Jiang, H., Weng, D., Dongye, X., Luo, L., & Zhang, Z. (2023) · ACM Transactions on Graphics · doi:10.1145/3618320 · Cited by 8
“Graphs have been used for novel scene synthesis and scene enhancement [Fu et al. 2017; Jiang et al. 2018; Kermani et al. 2016; Li et al. 2019; Ma et al. 2018; Xu et al. 2013].”
MethodA system of configurable 3D indoor scene synthesis via semantic relation learning
Yang, X., Hu, F., Ye, L., Chang, Z., & Li, J. (2022) · Displays · doi:10.1016/j.displa.2022.102168 · Cited by 6
Accelerating 3D Scene Development for the Metaverse: Lessons from Photogrammetry and Manual Modeling
Pentangelo, V., Dario, D. D., Martino, V. D., Buono, M. D., & Lambiase, S. (2024) · International Conference on Intelligent Metaverse Technologies & Applications (iMETA) · doi:10.1109/imeta62882.2024.10808008 · Cited by 7
“Many of these approaches involve using AI to identify and position pre-existing models in a scene based on a textual description [16], [21].”
Scene Graph Masked Variational Autoencoders for 3D Scene Generation
Xu, R., Hui, L., Han, Y., Qian, J., & Xie, J. (2023) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3581783.3612262 · Cited by 6
“Furthermore, Ma et al. [21] designed a novel framework for generating and editing 3D indoor scenes us-ing natural language.”
Background3D-GENERALIST: Vision-Language-Action Models for Crafting 3D Worlds
Sun, F.-Y., Wu, S., Jacobsen, C., Yim, T., Zou, H., Zook, A., Li, S., Chou, Y., Can, E., Wu, X., et al. (2026) · International Conference on 3D Vision (3DV) · doi:10.1109/3dv69130.2026.00177 · Cited by 8
“Methods such as Wordseye [10] and its follow-ups are symbolic and rule-based, requiring substantial manual effort to generalize to new domains and constraints [5, 6, 27, 30, 38].”
DecorIn: An Automatic Method for Plane-Based Decorating
Liang, Y., Fan, L., Ren, P., Xie, X., & Hua, X. (2020) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2020.2972897 · Cited by 4
CLIPXPlore: Coupled CLIP and Shape Spaces for 3D Shape Exploration
Hu, J., Hui, K.-H., Liu, Z., Zhang, H., & Fu, C. (2023) · SIGGRAPH Asia Conference Papers · doi:10.1145/3610548.3618144 · Cited by 12
“…Chaudhuri and Koltun 2010; Ovsjanikov et al. 2011; Umetani et al. 2012; Xu et al. 2012; Yang et al. 2011; Zheng et al. 2016] and the same can be said about language-driven [Coyne and Sproat 2001; Ma et al. 2018] and sketch-based [Ding and Liu 2016; Igarashi et al. 1999; Olsen et al. 2009] modeling.”
BackgroundSelf-citationMAAN: Memory-Augmented Auto-Regressive Network for Text-Driven 3D Indoor Scene Generation
Ye, Z., Liu, Y., & Peng, Y. (2024) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2024.3443657 · Cited by 8
“For example, Ma et al., [30] introduce a natural language framework to edit 3D indoor scenes with an annotated large 3D scene database.”
In Situ 3D Scene Synthesis for Ubiquitous Embodied Interfaces
Jiang, H., Song, L., Weng, D., Sun, Z., Li, H., Dongye, X., & Zhang, Z. (2024) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3664647.3681616 · Cited by 4
“Chang et al. [4–7] and Ma et al. [36] parsed the input text into a knowledge tree or graph for synthesis, where the initial scene could be changed by language.”
Background“Computer, Generate!” – Investigating User-Controlled Generation of Immersive Virtual Environments
Liebers, C., Pfützenreuter, N., Auda, J., Gruenefeld, U., & Schneegaß, S. (2024) · Frontiers in artificial intelligence and applications · doi:10.3233/faia240196 · Cited by 4
CFC: Simulating Character-Fluid Coupling using a Two-Level World Model
Dou, Z., Chen, P., Lu, X., Ye, X., Fang, L., Liu, Y., Wang, W., Gan, C., Liu, L., & Komura, T. (2025) · ACM Transactions on Graphics · doi:10.1145/3763318 · Cited by 6
“Efforts have been made to use multi-task reinforcement learning to control objects influenced by fluid flow [Ma et al. 2018; Ren et al. 2022].”
ScanEdit: Hierarchically-Guided Functional 3D Scan Editing
Boudjoghra, M. E. A., Laptev, I., & Dai, A. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.02516 · Cited by 10
A survey of recent 3D scene analysis and processing methods
Yuan, J., Abdul-Rashid, H., & Li, B. (2021) · Multimedia Tools and Applications · doi:10.1007/s11042-021-10615-7 · Cited by 6
“Ma et al. [37] proposed a sub-scene level framework that can generate 3D indoor scenes by using natural language commands.”
BackgroundText to Scene: A System of Configurable 3D Indoor Scene Synthesis
Yang, X., Hu, F., & Ye, L. (2021) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3474085.3478570 · Cited by 6
“Meanwhile, most of existing semantic-driven indoor scene synthesis tasks concentrate on strictly semantic matching [6][4][9] or linguistic understanding [5][1][7].”
BackgroundFuzzy-based indoor scene modeling with differentiated examples
Fu, Q., He, S., Fu, H., Li, X., & Deng, Z. (2023) · Computational Visual Media · doi:10.1007/s41095-022-0299-z · Cited by 3
Text‐Guided Interactive Scene Synthesis with Scene Prior Guidance
Fang, S., Yang, H., Mooney, R. J., & Huang, Q. (2025) · Computer Graphics Forum · doi:10.1111/cgf.70039 · Cited by 2
“Traditional approaches [CSM14,CMS ∗ 15,CESM17,MPF ∗ 18] extract information about object categories and spatial relationships from language descriptions through sentence parsing and then apply optimization techniques to arrange objects within a scene.”
ChatDC: Geometric-aware Data Center Digital Twin Generation via Large Language Models
Li, M., Wang, R., Zhou, X., Zhu, Z., Wen, Y., Tan, R., Zheng, H., & Kennedy, S. (2026) · ACM Transactions on Internet of Things · doi:10.1145/3772078 · Cited by 4
“Ma et al. [28] utilize NL for generative 3D scene creation, transforming user commands into semantic scene graphs, and synthesizing new 3D scenes from large databases.”
Self-citationFlexPlan: High-flexibility interactive floorplan design based on ArchiGraph
Wang, Z., Wang, J., Li, Z., Su, H., Lv, P., & Xu, M. (2025) · Graphical Models · doi:10.1016/j.gmod.2025.101303 · Cited by 3
InteriorAgent: LLM Agent for Interior Design-Aware 3D Layout Generation
Gupta, K., Mehta, I., Wang, K., Chua, N., Krishna, A., Deng, Y., Ramamoorthi, R., & Chandraker, M. K. (2026) · International Conference on 3D Vision (3DV) · doi:10.1109/3dv69130.2026.00138 · Cited by 3
“Learning-Based Methods Recent works have used Gaussian mixtures [7, 19], autoregressive transformers [13, 17, 22], recursive autoencoder [14], and graph convolution networks [36, 43] to learn indoor scene layouts, as well as diffusion models [34, 38, 40] for indoor scene synthesis.”
A Dataset and Benchmark for 3D Scene Plausibility Assessment
Hu, F., Ma, Y., Zhong, W., Ye, L., Yang, X., Fang, L., & Zhang, Q. (2024) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2024.3353456 · Cited by 3
“Ma et al. [6] propose that the spatial distribution of object pairs in a scene follows a Gaussian mixture distribution.”
BackgroundInfluential3D Software Synthesis Driven by Constraint-Expressive Intermediate Representation
Li, S., Lam, A. Y., Peng, Y., Wang, W., & Lyu, M. R.-T. (2026) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3744916.3787812 · Cited by 3
“Early probabilistic meth-ods [8, 9, 22, 36, 48, 61, 90] suffered from limited diversity, hindering the discovery of edge cases, a primary goal of thorough software quality assurance.”
Li, Y., Wang, X., Wu, Z., Liu, S., & Zhou, M. (2019) · International Conference on Cyberworlds (CW) · doi:10.1109/cw.2019.00014 · Cited by 2
“For the first question, traditional methods of inserting a furniture into an indoor scene rely on various kinds of retrieval methods, including text descriptions[7], an image[8], a sketch[9] and natural language[10].”
BackgroundMethodGraph Neural Network for Generative Furniture Arrangement
Yang, M., Fan, Q., Wang, Y., Duan, Y., & Chen, B. (2021) · Journal of Computer-Aided Design & Computer Graphics · doi:10.3724/sp.j.1089.2021.18457 · Cited by 2
Automated modeling to high level-of-detail composite object using spatial BIM objects and properties
Lee, J.-K., Cho, D., Kim, Y., & Mo, Y. (2025) · Scientific Reports · doi:10.1038/s41598-025-32433-9 · Cited by 1
“By analyzing data and user inputs related to the target space, the necessary interior design objects are identified and integrated, resulting in a filled space recognized as a BIM composite object 55 .”
GenAssist: Interactive Prompt-Driven XR Program Generation
Srinidhi, S., Singh, A., Lu, E., & Rowe, A. (2026) · IEEE Conference on Virtual Reality and 3D User Interfaces (VR) · doi:10.1109/vr67842.2026.00081 · Cited by 2
“These systems typically modified existing scenes based on spatial or contextual cues rather than generating content from scratch [25, 24, 46].”
Transfer of ISOSpace into a 3D Environment for Annotations and Applications
Henlein, A., Abrami, G., Kett, A., & Mehler, A. (2020) · Cited by 3
Fuzzy Speech Driven Indoor Modeling System
Kang, J., Zhu, Y., Lv, C., Xu, S., Xue, Y., Wang, X., & Wu, Z. (2019) · International Conference on Virtual Reality and Visualization · doi:10.1109/icvrv47840.2019.00016 · Cited by 1
Semantic Scene Builder: Towards a Context Sensitive Text-to-3D Scene Framework
Henlein, A., Kett, A., Baumartz, D., Abrami, G., Mehler, A., Bastian, J., Blecher, Y., Budgenhagen, D., Christof, R., Ewald, T.-O., et al. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-35748-0_32 · Cited by 1
Build-It-Here: Utilizing 2D Inpainting Models for On-Site 3D Object Generation
Wu, Z., & Sato, H. (2024) · International Conference on Computational Intelligence and Applications (ICCIA) · doi:10.1109/iccia62557.2024.10719286 · Cited by 1
“Ma et al. [21],Sceneseer [22],Diffuscene [23], parse text command into a semantic scene graph that is used to retrieve corresponding sub-scenes.”
MethodToward context-based text-to-3D scene generation
Henlein, A. (2022) · Publication Server of Goethe University Frankfurt am Main (Goethe University Frankfurt) · doi:10.21248/gups.73448 · Cited by 1
3DSRASG: 3D Scene Retrieval and Augmentation Using Semantic Graphs
Srinivasan, S., Kumar, S., Chockalingam, V., & Chitrakala, S. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-86230-5_25
Ige, I. A., & Oladejo, B. F. (2023) · International Journal of Advanced Computer Science and Applications · doi:10.14569/ijacsa.2023.0140203
“The object‘s color and other characteristics are not emphasized as much work has been done on this area in literature [24], [28].”
BackgroundAsad, R., & Savva, M. (2023) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw59228.2023.00271
“[15] propose a 3D scene synthesis method that takes text as input and generates 3D scenes by retrieving objects from a scene database.”
MethodExperience Graph using Spatio-Temporal Scene Data for Replaying Mixed Reality Interaction
Kim, S. (2024) · IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW) · doi:10.1109/vrw62533.2024.00350
“Research has been conducted to create new scenes or spaces that can perform the same or similar interactions as the given interactions [4].”
BackgroundZeng, C. Y., Yang, H., Wu, Y., Zhang, P. P., Yao, Y., Al-Akaidi, M., & Zhou, Z. (2024) · IEEE International Conference on Digital Health (ICDH) · doi:10.1109/icdh62654.2024.00044
“The initial positioning within our 3D model utilises a spatial relationship determination algorithm, explored by Chang et al. And further developed by Ma et al., extracting spatial dependencies from text descriptions using a node-graph methodology and the Stanford Core NLP package [15][16].”
Method3D indoor scene assessment via layout plausibility
Yang, X., Hu, F., Liu, S., Ye, L., Wang, Y., Zhu, G., & Li, J. (2025) · Displays · doi:10.1016/j.displa.2025.102964
Socket Layout Planning Based on Furniture Layout Examples
Tam, H. I. I., Zhang, S.-K., Xu, M., & Zhang, S. (2023) · Journal of Computer-Aided Design & Computer Graphics · doi:10.3724/sp.j.1089.2023.20039
“如把文字输入转化为场 景图 [21] , 复用已有的场景例子来生成出更多的场 景 [22-23] , 以现实世界中真实存在的街区作为约束 生成场景 [24] .”
DGDiff: Immersive 3D Indoor Scene Synthesis via Dialog-Graph Conditioned Diffusion
Liu, Y., Zhang, X., Geng, Q., & Zhou, Z. (2025) · Proceedings - International Symposium on Mixed and Augmented Reality, ISMAR · doi:10.1109/ismar67309.2025.00055 · Cited by 1
“Early text-based approaches [7, 6, 31, 37] used procedural modeling with handcrafted rules, while transformer-based approaches [45, 34, 41] aligned text and layouts via cross-attention.”
Hong, J.-S., Park, H., Ko, H., & Woo, W. (2025) · IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct) · doi:10.1109/ismar-adjunct68609.2025.00027
“Extensive research has leveraged scene graphs for diverse applications, including interior layout planning [17,23], mutual space reasoning [11], object placement optimization [14,27], and 3D scene synthesis [34,35].”
Mutual Space Representation Standardization for Mixed and Augmented Reality Remote Collaboration
Kim, S., & Woo, W. (2025) · IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct) · doi:10.1109/ismar-adjunct68609.2025.00101
“In particular, studies have been conducted to create and synthesize new virtual spaces by representing scenes as graph structures [15, 16, 17, 20, 12].”
RL-ACD: Reinforcement Learning-based Approximate Convex Decomposition
Luo, Y., Pan, Z., Wu, K., Du, X., Zeng, Y., Tang, X., Wu, Y., Jin, X., & Gao, X. (2025) · ACM Transactions on Graphics · doi:10.1145/3763270 · Cited by 5
Curompt: A Spatially Situated Interface for Generative AI in 3D Design Software
Foo, M., Wannamaker, K., Vermeulen, J., Fitzmaurice, G. W., & Matejka, J. (2025) · IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vl-hcc65237.2025.00065
“Now, with Large Language Models (LLMs) capable of translating ambiguous commands into actions, they can make spatially informed responses when provided structured scene descriptions [4].”
Quan, Y., Wang, H., Lian, W., Wang, Z., & Yang, X. (2026) · ACM Transactions on Graphics · doi:10.1145/3811337
Metadata
- Authors
- Ma, R., Patil, A. G., Fisher, M., Li, M., Pirk, S., Hua, B.-S., Yeung, S.-K., Tong, X., Guibas, L., & Zhang, H.
- Affiliations
- Simon Fraser University (CA)
- Adobe Systems (United States) (US)
- Shandong University (CN)
- Stanford University (US)
- The University of Tokyo (JP)
- Hong Kong University of Science and Technology (HK)
- Microsoft Research Asia (China) (CN)
- ORCID
- Venue
- ACM Transactions on Graphics
- Published
- 2018-12-04
- Volume
- 37
- Issue
- 6
- Pages
- 1-16
- ISSN
- 0730-0301 (ISSN-L)
- 1557-7368
- OpenAlex
- W2902539442
- Cited by
- 100
- References
- 42
- Citations by year
- 2026: 7 · 2025: 15 · 2024: 19 · 2023: 17 · 2022: 6 · 2021: 16 · 2020: 12 · 2019: 7 · 2018: 1
- FWCI
- 6.63
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- ACM Transactions on Graphics (publishedVersion)
- Rare & Special e-Zone (The Hong Kong University of Science and Technology) (submittedVersion)
- Funders
- Natural Sciences and Engineering Research Council of Canada (611370)
- National Science Foundation (IIS-1528025)
- Ministry of Education - Singapore (MOE2016-T2-2-154)
- Amazon Web Services
- Adobe Systems
- Hong Kong University of Science and Technology (R9429)
- Stanford AI Lab-Toyota Center for Artificial Intelligence Research
- Topics
- Physical Sciences › Computer Science › Computer Graphics and Computer-Aided Design › 3D Shape Modeling and Analysis
- Physical Sciences › Computer Science › Computer Graphics and Computer-Aided Design › Computer Graphics and Visualization Techniques
- Physical Sciences › Computer Science › Human-Computer Interaction › Interactive and Immersive Displays
- Keywords
- 3D scene synthesis, indoor scenes, scene semantics, scene graph, natural language ambiguity
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.