An Interactive Scene Generation Using Natural Language
The DOI matches its Crossref record; 26 of the 28 references it cites were found in a registry or an index.
Abstract
Scene generation is an important step of robotic drawing. Recent works have shown success in scene generation conditioned on text using a variety of approaches, with which the generated scenes cannot be revised after its generation. To allow modification on generated scenes, we model the scene generation process as a discrete event system. Instead of training text-to-pixel mappings using large datasets, the proposed approach uses object instances retrieved from the Internet to synthesize scenes. Evaluated on 128 experiments using MSCOCO evaluation dataset, the result shows the scene generation performance has been increased by 197%, 22.3%, and 55.7% compared with the state of the art approach on three standard metrics (CIDEr, ROUGH-L, METEOR), respectively. Human evaluation conducted on Amazon Mechanical Turk shows over 80% of generated scenes are considered to have higher recognizability and better alignment with natural language descriptions than baseline works.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Many works on scene design either learn spatial knowledge priors from existing 3D scene databases [4, 31, 49-51, 54, 59] or leverage user input and refine the 3D scene iteratively [3, 5].”
From Holodeck: Language Guided Generation of 3D Embodied AI Environments (2024) · cites this paper as [4]
Citations
Cites
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460
(2004) · Cited by 9,718
Mayer, R. (2020) · doi:10.1017/9781316941355 · Cited by 7,777
Show and tell: A neural image caption generator
Vinyals, O., Toshev, A., Bengio, S., & Erhan, D. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298935 · Cited by 6,318
CIDEr: Consensus-based image description evaluation
Vedantam, R., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7299087 · Cited by 4,845
(2005) · Cited by 4,152
Caltech-UCSD Birds-200-2011 Dataset
Wah, C. (2024) · CaltechAUTHORS (California Institute of Technology) · doi:10.57702/i9oeewf6 · Cited by 3,525
Automated Flower Classification over a Large Number of Classes
Nilsback, M.-E., & Zisserman, A. (2008) · 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing · doi:10.1109/icvgip.2008.47 · Cited by 3,442 · cited 1×
“These approaches are limited to single objects, such as birds [7] and flowers [8].”
BackgroundAccurate unlexicalized parsing
Klein, D., & Manning, C. D. (2003) · Proceedings of the 41st Annual Meeting on Association for Computational Linguistics - ACL '03 · doi:10.3115/1075096.1075150 · Cited by 3,008
The control of discrete event systems
Ramadge, P. J. G., & Wonham, W. M. (1989) · Proceedings of the IEEE · doi:10.1109/5.21072 · Cited by 2,937
Generative Adversarial Text to Image Synthesis
Reed, S., Akata, Z., Yan, X., Logeswaran, L., Schiele, B., & Lee, H. (2016) · arXiv · doi:10.48550/arxiv.1605.05396 · Cited by 2,494
Conditional Random Fields as Recurrent Neural Networks
Zheng, S., Jayasumana, S., Romera-Paredes, B., Vineet, V., Su, Z., Du, D., Huang, C., & Torr, P. H. S. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.179 · Cited by 2,386 · cited 1×
“The object detector developed in [20] is used in this work, which is able to detect 20 categories of objects.”
MethodAttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks
Xu, T., Zhang, P., Huang, Q., Zhang, H., Gan, Z., Huang, X., & He, X. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00143 · Cited by 1,923
Gomez M, P. J., & Jacome S, J. E. (2018) · DSPACE System - Metalibrary (University of the Coast) · Cited by 979
A Survey of Research on Cloud Robotics and Automation
Kehoe, B., Patil, S., Abbeel, P., & Goldberg, K. (2015) · IEEE Transactions on Automation Science and Engineering · doi:10.1109/tase.2014.2376492 · Cited by 833
Attribute2Image: Conditional Image Generation from Visual Attributes
Yan, X., Yang, J., Sohn, K., & Lee, H. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46493-0_47 · Cited by 688
Generating Images from Captions with AttentionPreprint
Mansimov, E., Parisotto, E., Ba, J. L., & Salakhutdinov, R. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.02793 · Cited by 488
Deep Semantic Role Labeling: What Works and What’s Next
He, L., Lee, K., Lewis, M., & Zettlemoyer, L. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1044 · Cited by 420
Learning the Visual Interpretation of Sentences
Zitnick, C. L., Parikh, D., & Vanderwende, L. (2013) · 2013 IEEE International Conference on Computer Vision · doi:10.1109/iccv.2013.211 · Cited by 153
Learning Spatial Knowledge for Text to 3D Scene Generation
Chang, A., Savva, M., & Manning, C. D. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1217 · Cited by 153
A text-to-picture synthesis system for augmenting communication
Zhu, X., Goldberg, A. B., Eldawy, M., Dyer, C. R., & Strock, B. (2007) · Cited by 96
PropBank Annotation Guidelines
Bonial, C., Babko-Malaya, O., Choi, J. D., & Hwang, J. D. (2010) · Cited by 76
I2T2I: Learning text to image synthesis with textual data augmentation
Dong, H., Zhang, J., McIlwraith, D. G., & Guo, Y. (2017) · Proceedings - International Conference on Image Processing · doi:10.1109/icip.2017.8296635 · Cited by 76
DESUMA: A Tool Integrating GIDDES and UMDES
Ricker, L., Lafortune, S., & Genç, Ş. (2006) · International Workshop on Discrete Event Systems · doi:10.1109/wodes.2006.382402 · Cited by 65
Automatic feature extraction and optimal path planning for robotic drawing
Xinlong, H., Bi, S., Dong, M., Chen, H., Fang, S., & Xi, N. (2016) · IEEE International Conference on Cyber Technology in Automation, Control, and Intelligent Systems · doi:10.1109/cyber.2016.7574788 · Cited by 8
Modeling robotic operations controlled by natural language
Cheng, Y., Bao, J., Jia, Y., Deng, Z., Sun, Z., Bi, S., Li, C., & Xi, N. (2017) · Control Theory and Technology · doi:10.1007/s11768-017-7099-5 · Cited by 7
Self-citation
Research works printed but not linked: 1
- 10.1016/S0079-7421(02)80005-6 · doi:10.1016/s0079-7421(02)80005-6 · Paper page
Cited by
Holodeck: Language Guided Generation of 3D Embodied AI Environments
Yang, Y., Sun, F.-Y., Weihs, L., Vanderbilt, E., Herrasti, A., Han, W., Wu, J., Haber, N., Krishna, R., Liu, L., et al. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.01536 · Cited by 101
“Many works on scene design either learn spatial knowledge priors from existing 3D scene databases [4, 31, 49–51, 54, 59] or leverage user input and refine the 3D scene iteratively [3, 5].”
BackgroundGómez, L. V., & Miura, J. (2021) · Robotics and Autonomous Systems · doi:10.1016/j.robot.2021.103763 · Cited by 35
Fime, A. A., Mahmud, S., Das, A., Islam, M. K., & Kim, J.-H. (2025) · IEEE Access · doi:10.1109/access.2025.3574298 · Cited by 16
“Reference [201] used CIDEr, ROUGH-L, and METEOR as the evaluation metric of their proposed system.”
A Study of NLP and AI-Driven Online Text-to-Video Generation Platforms in Social Media
Kaur, G., & Pratibha, N. A. (2024) · International Conference on Advances in Computation, Communication and Information Technology (ICAICCIT) · doi:10.1109/icaiccit64383.2024.10912131 · Cited by 2
“Platforms like Pictory and Synthesia use NLP to extract important information from text [8].”
Measuring Perceptual and Linguistic Complexity in Multilingual Grounded Language Data
Pillai, N., Matuszek, C., & Ferraro, F. (2021) · Proceedings of the ... International Florida Artificial Intelligence Research Society Conference · doi:10.32473/flairs.v34i1.128450
“…applications to areas including visual navigation (Nguyen et al. 2019; Jain et al. 2019; Hu et al. 2019; Nguyen and Daum´e 2019); scene generation (Cheng et al. 2019); understanding videos (He et al. 2019; Antol et al. 2015); image captioning (Song et al. 2019; Jiang et al. 2019); visual story…”
Background
Metadata
- Authors
- Cheng, Y., Shi, Y., Sun, Z., Feng, D., & Dong, L.
- Affiliations
- Michigan State University (US)
- Venue
- 2019 International Conference on Robotics and Automation (ICRA)
- Publisher
- IEEE
- Published
- 2019-05
- Pages
- 6957-6963
- ISSN
- 1050-4729 (ISSN-L)
- OpenAlex
- W2968389886
- Cited by
- 5
- References
- 28
- Citations by year
- 2025: 1 · 2024: 2 · 2021: 2
- FWCI
- 0.16
- Type
- conference paper
- Language
- en
- Open access
- closed
- Licence
- Locations
- Proceedings - IEEE International Conference on Robotics and Automation/Proceedings (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image Retrieval and Classification Techniques
- Keywords
- scene generation, text-to-image generation, image generation, natural language descriptions, human evaluation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.