A Layer-Based Sequential Framework for Scene Generation with GANs
Open accessAbstract
The visual world we sense, interpret and interact everyday is a complex composition of interleaved physical entities. Therefore, it is a very challenging task to generate vivid scenes of similar complexity using computers. In this work, we present a scene generation framework based on Generative Adversarial Networks (GANs) to sequentially compose a scene, breaking down the underlying problem into smaller ones. Different than the existing approaches, our framework offers an explicit control over the elements of a scene through separate background and foreground generators. Starting with an initially generated background, foreground objects then populate the scene one-by-one in a sequential manner. Via quantitative and qualitative experiments on a subset of the MS-COCO dataset, we show that our proposed framework produces not only more diverse images but also copes better with affine transformations and occlusion artifacts of foreground objects than its counterparts.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Since 2014, generative adversarial training algorithms have been widely used in unimodal applications such as scene generation, image-to-image translation, and image super-resolution.
“Since their development in 2014, generative adversarial training algorithms have been widely used in various unimodal applications such as scene generation [17], image-to-image translation [18], and image super-resolution [224, 225].”
From A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets (2022) · cites this paper as [17]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
U-Net: Convolutional Networks for Biomedical Image Segmentation
Ronneberger, O., Fischer, P., & Brox, T. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24574-4_28 · Cited by 99,404 · see citing works
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · see citing works
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460 · see citing works
Rethinking the Inception Architecture for Computer Vision
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.308 · Cited by 32,138 · see citing works
Unpaired image-to-image translation using cycle-consistent adversarial networks
Zhu, J.-Y., Park, T., Isola, P., & Efros, A. A. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.244 · Cited by 23,602 · see citing works
Chen, L.-C., Papandreou, G., Kokkinos, I., Murphy, K., & Yuille, A. L. (2018) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2017.2699184 · Cited by 22,392 · see citing works
Labaca-Castro, R. (2023) · Springer eBooks · doi:10.1007/978-3-658-40442-0_9 · Cited by 21,100 · see citing works
GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., & Hochreiter, S. (2017) · arXiv · doi:10.48550/arxiv.1706.08500 · Cited by 8,365 · see citing works
Improved Techniques for Training GANs
Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., & Chen, X. (2016) · arXiv · doi:10.48550/arxiv.1606.03498 · Cited by 6,965 · see citing works
Context Encoders: Feature Learning by Inpainting
Pathak, D., Krahenbuhl, P., Donahue, J., Darrell, T., & Efros, A. A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.278 · Cited by 5,691 · see citing works
High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs
Wang, T.-C., Liu, M.-Y., Zhu, J.-Y., Tao, A., Kautz, J., & Catanzaro, B. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00917 · Cited by 4,514 · see citing works
Spectral Normalization for Generative Adversarial Networks
Miyato, T., Kataoka, T., Koyama, M., & Yoshida, Y. (2018) · arXiv · doi:10.48550/arxiv.1802.05957 · Cited by 3,438 · see citing works
StackGAN: Text to Photo-Realistic Image Synthesis with Stacked Generative Adversarial Networks
Zhang, H., Xu, T., Li, H., Zhang, S., Wang, X., Huang, X., & Metaxas, D. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.629 · Cited by 3,021 · see citing works
Conditional Image Synthesis With Auxiliary Classifier GANsPreprint
Odena, A., Olah, C., & Shlens, J. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1610.09585 · Cited by 2,968 · see citing works
Generative Adversarial Text to Image Synthesis
Reed, S., Akata, Z., Yan, X., Logeswaran, L., Schiele, B., & Lee, H. (2016) · arXiv · doi:10.48550/arxiv.1605.05396 · Cited by 2,494 · see citing works
Deep Generative Image Models using a Laplacian Pyramid of Adversarial NetworksPreprint
Denton, E., Chintala, S., Szlam, A., & Fergus, R. (2015) · Neural Information Processing Systems · doi:10.48550/arxiv.1506.05751 · Cited by 2,360 · see citing works
Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial NetworksPreprint
Zhu, J.-Y., Park, T., Isola, P., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.10593 · Cited by 2,278 · see citing works
Globally and locally consistent image completion
Iizuka, S., Simo-Serra, E., & Ishikawa, H. (2017) · ACM Transactions on Graphics · doi:10.1145/3072959.3073659 · Cited by 2,203 · see citing works
Hertzmann, A., Jacobs, C. E., Oliver, N., Curless, B., & Salesin, D. (2001) · ACM SIGGRAPH Conference (SIGGRAPH) · doi:10.1145/383259.383295 · Cited by 1,579 · see citing works
Toward Multimodal Image-to-Image TranslationPreprint
Zhu, J.-Y., Zhang, R., Pathak, D., Darrell, T., Efros, A. A., Wang, O., & Shechtman, E. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.11586 · Cited by 1,159 · see citing works
Image Generation from Scene Graphs
Johnson, J., Gupta, A., & Fei-Fei, L. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00133 · Cited by 836 · see citing works
Learning What and Where to DrawPreprint
Reed, S., Akata, Z., Mohan, S., Tenka, S., Schiele, B., & Lee, H. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1610.02454 · Cited by 411 · see citing works
Inferring Semantic Layout for Hierarchical Text-to-Image Synthesis
Hong, S., Yang, D., Choi, J., & Lee, H. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00833 · Cited by 348 · see citing works
High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANsPreprint
Wang, T.-C., Liu, M.-Y., Zhu, J.-Y., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.11585 · Cited by 301 · see citing works
LR-GAN: Layered Recursive Generative Adversarial Networks for Image GenerationPreprint
Yang, J., Kannan, A., Batra, D., & Parikh, D. (2016) · International Conference on Learning Representations · doi:10.48550/arxiv.1703.01560 · Cited by 249 · see citing works
ST-GAN: Spatial Transformer Generative Adversarial Networks for Image Compositing
Lin, C.-H., Yumer, E., Wang, O., Shechtman, E., & Lucey, S. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00985 · Cited by 223 · see citing works
ExprGAN: Facial Expression Editing With Controllable Expression Intensity
Ding, H., Sricharan, K., & Chellappa, R. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.12277 · Cited by 207 · see citing works
Generative Attribute Controller with Conditional Filtered Generative Adversarial Networks
Kaneko, T., Hiramatsu, K., & Kashino, K. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.741 · Cited by 97 · see citing works
Scene Collaging: Analysis and Synthesis of Natural Images with Semantic Layers
Isola, P., & Liu, C. (2013) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2013.457 · Cited by 57 · see citing works
MC-GAN: Multi-conditional Generative Adversarial Network for Image SynthesisPreprint
Park, H., Yoo, Y., & Kwak, N. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.01123 · Cited by 51 · see citing works
Inferring Semantic Layout for Hierarchical Text-to-Image SynthesisPreprint
Hong, S., Yang, D., Choi, J., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1801.05091 · Cited by 42 · see citing works
ST-GAN: Spatial Transformer Generative Adversarial Networks for Image CompositingPreprint
Lin, C.-H., Yumer, E., Wang, O., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1803.01837 · Cited by 39 · see citing works
Knowledge-based spatial reasoning for scene generation from text descriptions
Tappan, D. (2008) · National Conference on Artificial Intelligence · Cited by 6 · see citing works
r-BTN: Cross-Domain Face Composite and Synthesis From Limited Facial Patches
Song, Y., Zhang, Z., & Qi, H. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11854 · Cited by 6 · see citing works
Flagged first, then by citations
Research works printed but not linked: 3
- Chen, T.; Cheng, M.-M.; Tan, P.; Shamir, A.; and Hu, S.-M. 2009. Sketch2photo: Internet image montage. In ACM TOG.
- Isola, P.; Zhu, J.-Y.; Zhou, T.; and Efros, A. A. 2017. Image- to-image translation with conditional adversarial networks. In CVPR.
- Zhu, J.-Y.; Park, T.; Isola, P.; and Efros, A. A. 2017a. Un- paired image-to-image translation using cycle-consistent ad- versarial networks. In ICCV.
Cited by
Sorted by citations
A Review on Generative Adversarial Networks: Algorithms, Theory, and Applications
Gui, J., Sun, Z., Wen, Y., Tao, D., & Ye, J. (2023) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2021.3130191 · Cited by 1,231 · see citing works
Bayoudh, K., Knani, R., Hamdaoui, F., & Mtibaa, A. (2022) · The Visual Computer · doi:10.1007/s00371-021-02166-7 · Cited by 456 · see citing works
Generative Adversarial Networks in Computer Vision: A Survey and Taxonomy
Wang, Z., She, Q., & Ward, T. E. (2022) · ACM Computing Surveys · doi:10.1145/3439723 · Cited by 267 · see citing works
Generative Adversarial Networks in Computer Vision: A Survey and TaxonomyPreprint
Wang, Z., She, Q., & Ward, T. E. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.01529 · Cited by 254 · see citing works
Adversarial text-to-image synthesis: A review
Frolov, S., Hinz, T., Raue, F., Hees, J., & Dengel, A. R. (2021) · Neural Networks · doi:10.1016/j.neunet.2021.07.019 · Cited by 214 · see citing works
Tang, H., Xu, D., Yan, Y., Torr, P. H. S., & Sebe, N. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00789 · Cited by 166 · see citing works
SketchyCOCO: Image Generation From Freehand Scene Sketches
Gao, C., Liu, Q., Xu, Q., Wang, L., Liu, J., & Zou, C. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00522 · Cited by 147 · see citing works
Towards Unsupervised Learning of Generative Models for 3D Controllable Image Synthesis
Liao, Y., Schwarz, K., Mescheder, L., & Geiger, A. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00591 · Cited by 111 · see citing works
BlockGAN: Learning 3D Object-aware Scene Representations from Unlabelled ImagesPreprint
Nguyen-Phuoc, T., Richardt, C., Mai, L., Yang, Y., & Mitra, N. J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.08988 · Cited by 105 · see citing works
Generative Adversarial Networks: A Survey and Taxonomy.Preprint
Wang, Z., She, Q., & Ward, T. E. (2019) · arXiv (Cornell University) · Cited by 65 · see citing works
Deep learning for complex displacement field measurement
Lan, S., Su, Y. J., Gao, Z., Chen, Y., Tu, H., & Zhang, Q. (2022) · Science China Technological Sciences · doi:10.1007/s11431-022-2122-y · Cited by 34 · see citing works
Neutralizing the impact of heat haze on digital image correlation measurements via deep learning
Liu, Y., Yu, L., Wang, Z., & Pan, B. (2023) · Optics and Lasers in Engineering · doi:10.1016/j.optlaseng.2023.107522 · Cited by 27 · see citing works
Shete, S., Srinivasan, S., & Gonsalves, T. A. (2020) · Plant Phenomics · doi:10.34133/2020/8309605 · Cited by 26 · see citing works
Towards causal generative scene models via competition of expertsPreprint
von Kügelgen, J., Ustyuzhaninov, I., Gehler, P., Bethge, M., & Schölkopf, B. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.12906 · Cited by 20 · see citing works
A Survey on Multimodal Deep Learning for Image Synthesis
Luo, S. (2021) · International Conference on Innovation in Artificial Intelligence (ICIAI) · doi:10.1145/3461353.3461388 · Cited by 13 · see citing works
MotionVideoGAN: A Novel Video Generator Based on the Motion Space Learned From Image Pairs
Zhu, J., Ma, H., Chen, J., & Yuan, J. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3251095 · Cited by 11 · see citing works
Meng, R., Cui, Q., Zhou, Z., Li, Z., Wu, Q. M. J., & Sun, X. (2021) · IEEE Transactions on Network Science and Engineering · doi:10.1109/tnse.2021.3137829 · Cited by 10 · see citing works
Virtual staining for pixel-wise and quantitative analysis of single cell images
Yılmaz, A., Aydin, T., & Varol, R. (2023) · Scientific Reports · doi:10.1038/s41598-023-45150-y · Cited by 9 · see citing works
Research on visual‐tactile cross‐modality based on generative adversarial network
Li, Y., Zhao, H., Liu, H., Lu, S., & Hou, Y. (2021) · Cognitive Computation and Systems · doi:10.1049/ccs2.12008 · Cited by 8 · see citing works
Penava, P., & Buettner, R. (2023) · IEEE Access · doi:10.1109/access.2023.3326720 · Cited by 7 · see citing works
Fime, A. A., Mahmud, S., Das, A., Islam, M. K., & Kim, J.-H. (2025) · IEEE Access · doi:10.1109/access.2025.3574298 · Cited by 11 · see citing works
Li, Y., Chen, H., Dong, S., Zhuang, Y., & Li, L. (2023) · Remote Sensing · doi:10.3390/rs15092470 · Cited by 7 · see citing works
Scene Graph to Image Synthesis via Knowledge Consensus
Wu, Y., Wei, P., & Lin, L. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i3.25387 · Cited by 6 · see citing works
Unsupervised Object Learning via Common FatePreprint
Tangemann, M., Schneider, S., von Kügelgen, J., Locatello, F., Gehler, P., Brox, T., Kümmerer, M., Bethge, M., & Schölkopf, B. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.06562 · Cited by 6 · see citing works
Deep-Sea Biological Image Augmentation: A Generative Adversarial Networks-Based Application
Liu, Y., & Liu, L. (2020) · Global Oceans 2020: Singapore – U.S. Gulf Coast · doi:10.1109/ieeeconf38699.2020.9389026 · Cited by 4 · see citing works
Synthetic Video Generation for Robust Hand Gesture Recognition in Augmented Reality ApplicationsPreprint
Jain, V., Aggarwal, S., Mehta, S., & Hebbalaguppe, R. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.01320 · Cited by 5 · see citing works
PI-Trans: Parallel-Convmlp and Implicit-Transformation Based Gan for Cross-View Image Translation
Ren, B., Tang, H., Wang, Y., Li, X., Wang, W., & Sebe, N. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10095449 · Cited by 4 · see citing works
An Approach Towards Generalization of Automotive Camera Sensor Data using Deep Learning
Ramesh, G. B., Chamas, M., Wang, A., Raghuraman, S., & Sax, E. (2024) · International Congress on Human-Computer Interaction, Optimization and Robotic Applications (ICHORA) · doi:10.1109/hora61326.2024.10550698 · Cited by 3 · see citing works
aRTIC GAN: A Recursive Text-Image-Conditioned GAN
Alati, E., Caracciolo, C. A., Costa, M., Sanzari, M., Russo, P., & Amerini, I. (2022) · Electronics · doi:10.3390/electronics11111737 · Cited by 3 · see citing works
Subset-based Smart-DIC vs. deep learning-based DIC: Which is better?
Zhao, J., Liu, Y., Kemao, Q., & Pan, B. (2026) · Optics and Lasers in Engineering · doi:10.1016/j.optlaseng.2026.109896 · Cited by 2 · see citing works
Generating Abstract Art from Hand-Drawn Sketches Using GAN Models
Chakrabarty, S., Johnson, R. F., Rashmi, M., & Raha, R. (2023) · Algorithms for intelligent systems · doi:10.1007/978-981-99-1435-7_45 · Cited by 2 · see citing works
Bouqata, B., Aswani, K., & Bailey, D. (2021) · International Conference on Arab Women in Computing (ArabWIC) · doi:10.1145/3485557.3485584 · Cited by 1 · see citing works
Sun, M., Singh, G., & Chiang, P. Y. (2020) · IEEE International Symposium on Signal Processing and Information Technology (ISSPIT) · doi:10.1109/isspit51521.2020.9408901 · Cited by 1 · see citing works
Neural Scene Decoration from a Single Photograph
Pang, H.-W., Chen, Y., Le, P.-H., Hua, B., Nguyen, D. T., & Yeung, S.-K. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-20050-2_9 · Cited by 1 · see citing works
International Conference on Recent and Future Trends in Smart Electronics System and Manufacturing
Kotecha, K. V., Shah, P., Kshirsagar, U., Nandan, D., & Kantipudi, M. V. V. P. (2023) · doi:10.13052/rp-9788770229852
Generative Adversarial Networks Based Scene Generation on Indian Driving Dataset
Shastry, K. A., Manjunatha, B. A., Kumar, T., & Karthik, D. U. (2023) · Journal of ICT Research and Applications · doi:10.5614/itbj.ict.res.appl.2023.17.2.4
- Authors
- Turkoglu, M. O., Thong, W., Spreeuwers, L., & Kicanaoglu, B.
- Affiliations
- University of Amsterdam (NL)
- University of Twente (NL)
- Venue
- Proceedings of the AAAI Conference on Artificial Intelligence
- Published
- 2019-07-17
- Volume
- 33
- Issue
- 01
- Pages
- 8901-8908
- ISSN
- 2159-5399 (ISSN-L)
- 2374-3468
- OpenAlex
- W2914914077
- Cited by
- 41
- References
- 30
- Citations by year
- 2026: 1 · 2025: 1 · 2024: 1 · 2023: 10 · 2022: 4 · 2021: 10 · 2020: 11 · 2019: 3
- FWCI
- 2.26
- Type
- conference paper
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- www.aaai.org/ (unspecified)
- Locations
- Proceedings of the AAAI Conference on Artificial Intelligence (publishedVersion)
- University of Twente Research Information (publishedVersion, other-oa)
- UvA-DARE (University of Amsterdam) (submittedVersion)
- AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) (submittedVersion)
- Data Archiving and Networked Services (DANS) (submittedVersion)
- Funders
- Natural Sciences and Engineering Research Council of Canada
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Digital Media Forensic Detection
- Keywords
- scene generation, generative adversarial networks, foreground-background separation, affine transformation, image diversity, MS COCO
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
