Next-generation deep learning based on simulators and synthetic data
The DOI matches its Crossref record; 43 of the 43 references it cites were found in a registry or an index.
Abstract
Deep learning (DL) is being successfully applied across multiple domains, yet these models learn in a most artificial way: they require large quantities of labeled data to grasp even simple concepts. Thus, the main bottleneck is often access to supervised data. Here, we highlight a trend in a potential solution to this challenge: synthetic data. Synthetic data are becoming accessible due to progress in rendering pipelines, generative adversarial models, and fusion models. Moreover, advancements in domain adaptation techniques help close the statistical gap between synthetic and real data. Paradoxically, this artificial solution is also likely to enable more natural learning, as seen in biological systems, including continual, multimodal, and embodied learning. Complementary to this, simulators and deep neural networks (DNNs) will also have a critical role in providing insight into the cognitive and neural functioning of biological systems. We also review the strengths of, and opportunities and novel challenges associated with, synthetic data.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
Unpaired image-to-image translation using cycle-consistent adversarial networks
Zhu, J.-Y., Park, T., Isola, P., & Efros, A. A. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.244 · Cited by 23,602
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
Howard, A. G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., & Adam, H. (2017) · arXiv · doi:10.48550/arxiv.1704.04861 · Cited by 19,015
Learning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293 · cited 1×
“…trend has been to train big generative models (e.g., with billions of parameters and terabytes of data) [53], including language [54] and multimodal models [55], that can subsequently be reused to synthe-size novel content and be integrated with other pipelines to solve domain-speci fi c tasks.”
A Style-Based Generator Architecture for Generative Adversarial Networks
Karras, T., Laine, S., & Aila, T. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00453 · Cited by 11,714 · cited 3×
“One challenge with using GANs is that the synthesized imagery is not produced with the associated ground truth data, as is done for graphics pipelines.”
Playing Atari with Deep Reinforcement Learning
Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., & Riedmiller, M. (2013) · arXiv · doi:10.48550/arxiv.1312.5602 · Cited by 9,471
Conditional Generative Adversarial Nets
Mirza, M., & Osindero, S. (2014) · arXiv · doi:10.48550/arxiv.1411.1784 · Cited by 8,863
Overcoming catastrophic forgetting in neural networks
Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A. A., Milan, K., Quan, J., Ramalho, T., Grabska-Barwinska, A., et al. (2017) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1611835114 · Cited by 7,899
Adversarial Discriminative Domain Adaptation
Tzeng, E., Hoffman, J., Saenko, K., & Darrell, T. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.316 · Cited by 5,154 · cited 1×
“Thus, the integration of DL and differentiable rendering pipelines has the potential to support the generation of customized curricula for more sample-ef fi cient learning.”
Multimodal Machine Learning: A Survey and Taxonomy
Baltrusaitis, T., Ahuja, C., & Morency, L.-P. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2798607 · Cited by 5,018
Learning Transferable Features with Deep Adaptation Networks
Long, M., Cao, Y., Wang, J., & Jordan, M. I. (2015) · arXiv · doi:10.48550/arxiv.1502.02791 · Cited by 4,505 · cited 1×
“Thus, the integration of DL and differentiable rendering pipelines has the potential to support the generation of customized curricula for more sample-ef fi cient learning.”
Flownet: Learning optical flow with convolutional networks
Dosovitskiy, A., Fischer, P., Ilg, E., Hausser, P., Hazirbas, C., Golkov, V., Smagt, P. V. D., Cremers, D., & Brox, T. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.316 · Cited by 4,313 · cited 1×
“…[15 – 17], object tracking [18,19], viewpoint estimation [20], semantic segmentation [21 – 23], robot manipulation and control [24 – 28], pose estimation [29 – 31], gaze estimation [32], and activity recognition [33,34] (for a detailed review of simulators and synthetic data sets, see [35]).”
Deep CORAL: Correlation Alignment for Deep Domain Adaptation
Sun, B., & Saenko, K. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-49409-8_35 · Cited by 3,757 · cited 1×
“Several feature alignment approaches have been explored, including through minimization of some distance between source and target distributions [76,77], weight sharing, and discriminators to encourage the network to learn domain invariant representations [78,79], projecting the distance…”
Large Scale GAN Training for High Fidelity Natural Image Synthesis
Brock, A., Donahue, J., & Simonyan, K. (2018) · arXiv · doi:10.48550/arxiv.1809.11096 · Cited by 3,717
Human3.6M: Large Scale Datasets and Predictive Methods for 3D Human Sensing in Natural Environments
Ionescu, C., Papava, D., Olaru, V., & Sminchisescu, C. (2014) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2013.248 · Cited by 3,653
CyCADA: Cycle-Consistent Adversarial Domain Adaptation
Hoffman, J., Tzeng, E., Park, T., Zhu, J.-Y., Isola, P., Saenko, K., Efros, A. A., & Darrell, T. (2017) · arXiv · doi:10.48550/arxiv.1711.03213 · Cited by 2,656 · cited 1×
“Thus, the integration of DL and differentiable rendering pipelines has the potential to support the generation of customized curricula for more sample-ef fi cient learning.”
Building machines that learn and think like people
Lake, B. M., Ullman, T. D., Tenenbaum, J. B., & Gershman, S. J. (2017) · Behavioral and Brain Sciences · doi:10.1017/s0140525x16001837 · Cited by 2,550
Building Machines That Learn and Think Like People
Lake, B. M., Ullman, T. D., Tenenbaum, J. B., & Gershman, S. J. (2016) · arXiv · doi:10.48550/arxiv.1604.00289 · Cited by 2,550
Ros, G., Sellart, L., Materzynska, J., Vazquez, D., & Lopez, A. M. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.352 · Cited by 2,338
Performance-optimized hierarchical models predict neural responses in higher visual cortex
Yamins, D. L. K., Hong, H., Cadieu, C. F., Solomon, E. A., Seibert, D., & DiCarlo, J. J. (2014) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1403112111 · Cited by 1,924
Self-citationPlaying for Data: Ground Truth from Computer Games
Richter, S. R., Vineet, V., Roth, S., & Koltun, V. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46475-6_7 · Cited by 1,815
Unsupervised Pixel-Level Domain Adaptation with Generative Adversarial Networks
Bousmalis, K., Silberman, N., Dohan, D., Erhan, D., & Krishnan, D. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.18 · Cited by 1,603 · cited 1×
“In contrast to traditional rendering pipelines, neural rendering pipelines are differentiable and can acquire 3D and physicsknowledgefrom2Dtrainingdata.”
Neuroscience-Inspired Artificial Intelligence
Hassabis, D., Kumaran, D., Summerfield, C., & Botvinick, M. (2017) · Neuron · doi:10.1016/j.neuron.2017.06.011 · Cited by 1,557 · cited 4×
“Domain randomization: techniques to create a data set that is diverse and broadly representative of a target domain, for the purpose of increasing the robustness and generalizability of a DL model.”
InfluentialBlascovich, J., Loomis, J., Beall, A. C., Swinth, K. R., Hoyt, C. L., & Bailenson, J. N. (2002) · Psychological Inquiry · doi:10.1207/s15327965pli1302_01 · Cited by 1,280 · cited 2×
“Here, we emphasize the central role of synthetic data and simulators in enabling this next generation of DL and, complemen-tarily, in providing insight into biological systems (Figure 3).”
Habitat: A Platform for Embodied AI Research
Savva, M., Kadian, A., Maksymets, O., Zhao, Y., Wijmans, E., Jain, B., Straub, J., Liu, J., Koltun, V., Malik, J., et al. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00943 · Cited by 1,237 · cited 3×
“A natural extension is the development of simulated open-ended environments [85,111] that would not only enable lifelong, but also structured [117], continual learning.”
Deep supervised, but not unsupervised, models may explain IT cortical representation
Khaligh-Razavi, S.-M., & Kriegeskorte, N. (2014) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1003915 · Cited by 1,204
Learning the signatures of the human grasp using a scalable tactile glove
Sundaram, S., Kellnhofer, P., Li, Y., Zhu, J.-Y., Torralba, A., & Matusik, W. (2019) · Nature · doi:10.1038/s41586-019-1234-z · Cited by 1,162 · cited 1×
“In this paradigm, a physics simulator can be explicitly used to make relevant predictions and the challenge is then transformed into perceiving the environment (e.g., constructing a scene graph representation through inverse rendering [93]) and feeding that information to the simulator (Figure 3A,…”
Self-citationSpelke, E. S., & Kinzler, K. D. (2007) · Developmental Science · doi:10.1111/j.1467-7687.2007.00569.x · Cited by 1,089
VirtualWorlds as Proxy for Multi-object Tracking Analysis
Gaidon, A., Wang, Q., Cabon, Y., & Vig, E. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.470 · Cited by 1,079
Simulation as an engine of physical scene understanding
Battaglia, P. W., Hamrick, J. B., & Tenenbaum, J. B. (2013) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1306572110 · Cited by 681
Reinforcement learning, fast and slow
Botvinick, M., Ritter, S., Wang, J. X., Kurth-Nelson, Z., Blundell, C., & Hassabis, D. (2019) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2019.02.006 · Cited by 667
Ego4D: Around the World in 3,000 Hours of Egocentric Video
Grauman, K., Westbury, A., Byrne, E., Chavis, Z., Furnari, A., Girdhar, R., Hamburger, J., Jiang, H., Liu, M., Liu, X., et al. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01842 · Cited by 630
Self-citationThe Development of Embodied Cognition: Six Lessons from Babies
Smith, L., & Gasser, M. (2005) · Artificial Life · doi:10.1162/1064546053278973 · Cited by 597
If deep learning is the answer, what is the question?
Saxe, A., Nelli, S., & Summerfield, C. (2021) · Nature Reviews Neuroscience · doi:10.1038/s41583-020-00395-8 · Cited by 397 · cited 2×
“We then articulate how synthetic data and simulators can enable DL solutions that can learn richer representations of the world and in more sophisticated ways, while simultaneously providing insight into the biological systems they draw inspiration from.”
Scaling Egocentric Vision: The EPIC-KITCHENS Dataset
Damen, D., Doughty, H., Farinella, G. M., Fidler, S., Furnari, A., Kazakos, E., Moltisanti, D., Munro, J., Perrett, T., Price, W., et al. (2018) · arXiv · doi:10.48550/arxiv.1804.02748 · Cited by 274 · cited 1×
“Whereas data sets have started emerging to support embodied learning [122,123], collecting this type of data is particularly labor intensive [124].”
DatasetGAN: Efficient Labeled Data Factory with Minimal Human Effort
Zhang, Y., Ling, H., Gao, J., Yin, K., Lafleche, J.-F., Barriuso, A., Torralba, A., & Fidler, S. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01001 · Cited by 259 · cited 1×
“However, good progress is being made extending GANs to produce imagery that already comes with detailed annotation, such as images of scenes with automatically generated scene segmentation ground truths [52].”
Self-citationGenerative Adversarial Networks for Image and Video Synthesis: Algorithms and Applications
Liu, M.-Y., Huang, X., Yu, J., Wang, T.-C., & Mallya, A. (2021) · Proceedings of the IEEE · doi:10.1109/jproc.2021.3049196 · Cited by 236 · cited 2×
“…networks (GANs) are a promising technique to synthesize novel images that match the statistical properties of the training data (Figure 1B) [44] (for a recent survey see [45]); for instance, GANs can generate faces of people that do not exist from a training set of existing human faces [46].”
Integrative benchmarking to advance neurally mechanistic models of human intelligence
Schrimpf, M., Kubilius, J., Lee, M. J., Ratan Murty, N. A., Ajemian, R., & DiCarlo, J. J. (2020) · Neuron · doi:10.1016/j.neuron.2020.07.040 · Cited by 233 · cited 2×
“The benchmarking of different DNNs with respect to how well they predict brain activity allows scientists to test current theory and formulate novel hypotheses about cognitive and neural functioning in biological systems [12].”
Self-citationWhat Makes Good Synthetic Training Data for Learning Disparity and Optical Flow Estimation?
Mayer, N., Ilg, E., Fischer, P., Hazirbas, C., Cremers, D., Dosovitskiy, A., & Brox, T. (2018) · International Journal of Computer Vision · doi:10.1007/s11263-018-1082-6 · Cited by 191 · cited 1×
“In general, increasing the realism of the synthesized output tends to improve DL performance [20,36 – 39], although, in some cases, it is less important [17,40 – 43].”
Online Continual Learning with Maximally Interfered Retrieval
Aljundi, R., Caccia, L., Belilovsky, E., Caccia, M., Lin, M., Charlin, L., & Tuytelaars, T. (2019) · arXiv · doi:10.48550/arxiv.1908.04742 · Cited by 140
How Useful Is Photo-Realistic Rendering for Visual Learning?
Movshovitz-Attias, Y., Kanade, T., & Sheikh, Y. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-49409-8_18 · Cited by 118 · cited 2×
“…been used to synthesize training data for a variety of tasks, including object detection [15 – 17], object tracking [18,19], viewpoint estimation [20], semantic segmentation [21 – 23], robot manipulation and control [24 – 28], pose estimation [29 – 31], gaze estimation [32], and activity…”
Meta-Sim2: Unsupervised Learning of Scene Structure for Synthetic Data Generation
Devaranjan, J., Kar, A., & Fidler, S. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58520-4_42 · Cited by 73
Cited by
Wu, Y., Sicard, B., & Gadsden, S. A. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.124678 · Cited by 357
Human-centric artificial intelligence architecture for industry 5.0 applications
Rožanec, J. M., Novalija, I., Zajec, P., Kenda, K., Tavakoli Ghinani, H., Suh, S., Bian, S., Veliou, E., Papamartzivanos, D., Giannetsos, T., et al. (2023) · International Journal of Production Research · doi:10.1080/00207543.2022.2138611 · Cited by 314
DeepFake detection in the AIGC era: A survey, benchmarks, and future perspectives
Xie, S., Qiao, T., Li, S., Zhang, X., Zhou, J., & Feng, G. (2026) · Information Fusion · doi:10.1016/j.inffus.2025.103740 · Cited by 14
Metadata
- Authors
- de Melo, C. M., Torralba, A., Guibas, L., DiCarlo, J., Chellappa, R., & Hodgins, J.
- Venue
- Trends in Cognitive Sciences
- Publisher
- Elsevier BV
- Published
- 2022-02
- Volume
- 26
- Issue
- 2
- Pages
- 174-187
- ISSN
- 1364-6613 (ISSN-L)
- OpenAlex
- W4200579548
- Cited by
- 164
- References
- 43
- FWCI
- 2.35
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.