The DOI matches its Crossref record; 23 of the 23 references it cites were found in a registry or an index.
Driving in the Matrix: Can virtual worlds replace human-generated annotations for real world tasks?
Open accessAbstract
Deep learning has rapidly transformed the state of the art algorithms used to address a variety of problems in computer vision and robotics. These breakthroughs have relied upon massive amounts of human annotated training data. This time consuming process has begun impeding the progress of these deep learning efforts. This paper describes a method to incorporate photo-realistic computer images from a simulation engine to rapidly generate annotated data that can be used for the training of machine learning algorithms. We demonstrate that a state of the art architecture, which is trained only using these synthetic annotations, performs better than the identical architecture trained on human annotated real-world data, when tested on the KITTI data set for vehicle detection. By training machine learning algorithms on a rich virtual world, real objects in real scenes can be learned and classified using synthetic data. This approach offers the possibility of accelerating deep learning's application to sensor-based classification problems like those that appear in self-driving cars. The source code and data to train and validate the networks described in this paper are made available for researchers.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2015) · arXiv · doi:10.48550/arxiv.1506.01497 · Cited by 70,812
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380
Are we ready for autonomous driving? The KITTI vision benchmark suite
Geiger, A., Lenz, P., & Urtasun, R. (2012) · 2012 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2012.6248074 · Cited by 14,897
The Cityscapes Dataset for Semantic Urban Scene Understanding
Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., & Schiele, B. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.350 · Cited by 12,433 · cited 1×
“Cityscapes [20] 2,975 KITTI [6] 7,481 Sim 10k 10,000 Sim 50k 50,000 Sim 200k 200,000”
LabelMe: A Database and Web-Based Tool for Image Annotation
Russell, B. C., Torralba, A., Murphy, K. P., & Freeman, W. T. (2008) · International Journal of Computer Vision · doi:10.1007/s11263-007-0090-8 · Cited by 4,210
Torralba, A., & Efros, A. A. (2011) · CVPR 2011 · doi:10.1109/cvpr.2011.5995347 · Cited by 2,440 · cited 3×
“While data sets have spurred impressive progress in the research community, the lack of generalizability remains an impediment to wide adoption and deployment on fielded robotic systems [9].”
BackgroundRos, G., Sellart, L., Materzynska, J., Vazquez, D., & Lopez, A. M. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.352 · Cited by 2,338
Labeling images with a computer game
von Ahn, L., & Dabbish, L. (2004) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/985692.985733 · Cited by 2,204
MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems
Chen, T., Li, M., Li, Y., Lin, M., Wang, N., Wang, M., Xiao, T., Xu, B., Zhang, C., & Zhang, Z. (2015) · arXiv · doi:10.48550/arxiv.1512.01274 · Cited by 1,819
Cited by
Domain Adaptive Faster R-CNN for Object Detection in the Wild
Chen, Y., Li, W., Sakaridis, C., Dai, D., & Van Gool, L. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00352 · Cited by 1,672
Unsupervised Pixel-Level Domain Adaptation with Generative Adversarial Networks
Bousmalis, K., Silberman, N., Dohan, D., Erhan, D., & Krishnan, D. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.18 · Cited by 1,603
“Unfortunately, models naively trained on synthetic data do not typically generalize to real images.”
BackgroundSemantic Foggy Scene Understanding with Synthetic Data
Sakaridis, C., Dai, D., & Van Gool, L. (2018) · International Journal of Computer Vision · doi:10.1007/s11263-018-1072-8 · Cited by 1,341
Planning and Decision-Making for Autonomous Vehicles
Schwarting, W., Alonso-Mora, J., & Rus, D. (2018) · Annual Review of Control, Robotics, and Autonomous Systems · doi:10.1146/annurev-control-060117-105157 · Cited by 972
Wu, B., Wan, A., Yue, X., & Keutzer, K. (2018) · 2018 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2018.8462926 · Cited by 967
Training Deep Networks with Synthetic Data: Bridging the Reality Gap by Domain Randomization
Tremblay, J., Prakash, A., Acuna, D., Brophy, M., Jampani, V., Anil, C., To, T., Cameracci, E., Boochoon, S., & Birchfield, S. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2018.00143 · Cited by 884
Strong-Weak Distribution Alignment for Adaptive Object Detection
Saito, K., Ushiku, Y., Harada, T., & Saenko, K. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00712 · Cited by 819
“We evaluate our approach on four domain shifts– PASCAL [8] to Clipart [15], PASCAL to Watercolor [15], Cityscapes [6] to FoggyCityscapes [34], and GTA [16] to Cityscapes–to demonstrate that it is effective for adaptation between both dissimilar and similar domains.”
MethodSynthetic data in machine learning for medicine and healthcare
Chen, R. J., Lu, M. Y., Chen, T. Y., Williamson, D. F. K., & Mahmood, F. (2021) · Nature Biomedical Engineering · doi:10.1038/s41551-021-00751-8 · Cited by 728
Wu, B., Zhou, X., Zhao, S., Yue, X., & Keutzer, K. (2019) · 2019 International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2019.8793495 · Cited by 726
Deep Learning for Safe Autonomous Driving: Current Challenges and Future Directions
Muhammad, K., Ullah, A., Lloret, J., Ser, J. D., & de Albuquerque, V. H. C. (2021) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2020.3032227 · Cited by 659
Learning From Synthetic Data for Crowd Counting in the Wild
Wang, Q., Gao, J., Lin, W., & Yuan, Y. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00839 · Cited by 654
Artificial intelligence applications in the development of autonomous vehicles: a survey
Ma, Y., Wang, Z., Yang, H., & Yang, L. (2020) · IEEE/CAA Journal of Automatica Sinica · doi:10.1109/jas.2020.1003021 · Cited by 639
ACDC: The Adverse Conditions Dataset with Correspondences for Semantic Driving Scene Understanding
Sakaridis, C., Dai, D., & Van Gool, L. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01059 · Cited by 572
LGSVL Simulator: A High Fidelity Simulator for Autonomous Driving
Rong, G., Shin, B. H., Tabatabaee, H., Lu, Q., Lemke, S., Mozeiko, M., Boise, E., Uhm, G., Gerow, M., Mehta, S., et al. (2020) · 2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC) · doi:10.1109/itsc45102.2020.9294422 · Cited by 498
Unbiased Mean Teacher for Cross-domain Object Detection
Deng, J., Li, W., Chen, Y., & Duan, L. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00408 · Cited by 401
Multi-Adversarial Faster-RCNN for Unrestricted Object Detection
He, Z., & Zhang, L. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00677 · Cited by 368
A Review of Single-Source Deep Unsupervised Visual Domain Adaptation
Zhao, S., Yue, X., Zhang, S., Li, B., Zhao, H., Wu, B., Krishna, R., Gonzalez, J. E., Sangiovanni-Vincentelli, A. L., Seshia, S. A., et al. (2022) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2020.3028503 · Cited by 352
Harmonizing Transferability and Discriminability for Adapting Object Detectors
Chen, C., Zheng, Z., Ding, X., Huang, Y., & Dou, Q. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00889 · Cited by 350
Exploring Object Relation in Mean Teacher for Cross-Domain Detection
Cai, Q., Pan, Y., Ngo, C.-W., Tian, X., Duan, L., & Yao, T. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.01172 · Cited by 347
Benchmarking Robustness in Object Detection: Autonomous Driving when Winter is Coming
Michaelis, C., Mitzkus, B., Geirhos, R., Rusak, E., Bringmann, O., Ecker, A. S., Bethge, M., & Brendel, W. (2019) · arXiv · doi:10.48550/arxiv.1907.07484 · Cited by 324
Unsupervised Domain Adaptation of Object Detectors: A Survey
Oza, P., Sindagi, V. A., VS, V., & Patel, V. M. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3217046 · Cited by 313
Conditional Generative Adversarial Network for Structured Domain Adaptation
Hong, W., Wang, Z., Yang, M., & Yuan, J. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00145 · Cited by 309
A Robust Learning Approach to Domain Adaptive Object Detection
Khodabandeh, M., Vahdat, A., Ranjbar, M., & Macready, W. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00057 · Cited by 274
Deep Drone Racing: From Simulation to Reality With Domain Randomization
Loquercio, A., Kaufmann, E., Ranftl, R., Dosovitskiy, A., Koltun, V., & Scaramuzza, D. (2020) · IEEE Transactions on Robotics · doi:10.1109/tro.2019.2942989 · Cited by 272
Structured Domain Randomization: Bridging the Reality Gap by Context-Aware Synthetic Data
Prakash, A., Boochoon, S., Brophy, M., Acuna, D., Cameracci, E., State, G., Shapira, O., & Birchfield, S. (2019) · 2019 International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2019.8794443 · Cited by 266
Cross-Domain Detection via Graph-Induced Prototype Alignment
Xu, M., Wang, H., Ni, B., Tian, Q., & Zhang, W. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01237 · Cited by 265
Physics-Based Rendering for Improving Robustness to Rain
Halder, S., Lalonde, J.-F., & Charette, R. D. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.01030 · Cited by 130
Precise Synthetic Image and LiDAR (PreSIL) Dataset for Autonomous Vehicle Perception
Hurl, B., Czarnecki, K., & Waslander, S. (2019) · 2019 IEEE Intelligent Vehicles Symposium (IV) · doi:10.1109/ivs.2019.8813809 · Cited by 122
Scenic: a language for scenario specification and data generation
Fremont, D. J., Kim, E., Dreossi, T., Ghosh, S., Yue, X., Sangiovanni-Vincentelli, A. L., & Seshia, S. A. (2023) · Machine Learning · doi:10.1007/s10994-021-06120-5 · Cited by 112
AlQuraishi, M., & Sorger, P. K. (2021) · Nature Methods · doi:10.1038/s41592-021-01283-4 · Cited by 96
A Photorealistic Terrain Simulation Pipeline for Unstructured Outdoor Environments
Muller, M. G., Durner, M., Gawel, A., Sturzl, W., Triebel, R., & Siegwart, R. (2021) · 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros51168.2021.9636644 · Cited by 37
“We note that the performance is generally high, which confirms previous findings of the suitability of training semantic segmentation tasks on synthetic data [23].”
Background
Metadata
- Authors
- Johnson-Roberson, M., Barto, C., Mehta, R., Sridhar, S. N., Rosaen, K., & Vasudevan, R.
- Venue
- 2017 IEEE International Conference on Robotics and Automation (ICRA)
- Publisher
- IEEE
- Published
- 2017-05
- Pages
- 746-753
- ISSN
- 1050-4729 (ISSN-L)
- OpenAlex
- W2528963632
- Cited by
- 622
- References
- 23
- FWCI
- 15.86
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.