The DOI matches its Crossref record; 55 of the 55 references it cites were found in a registry or an index.
Playing for Data: Ground Truth from Computer Games
Abstract
Recent progress in computer vision has been driven by high-capacity models trained on large datasets. Unfortunately, creating large datasets with pixel-level labels has been extremely costly due to the amount of human effort required. In this paper, we present an approach to rapidly creating pixel-accurate semantic label maps for images extracted from modern computer games. Although the source code and the internal operation of commercial games are inaccessible, we show that associations between image patches can be reconstructed from the communication between the game and the graphics hardware. This enables rapid propagation of semantic labels within and across images synthesized by the game, with no access to the source code or the content. We validate the presented approach by producing dense pixel-level semantic annotations for 25 thousand images synthesized by a photorealistic open-world computer game. Experiments on semantic segmentation datasets show that using the acquired data to supplement real-world images significantly increases accuracy and that the acquired data enables reducing the amount of hand-labeled real-world data: models trained with game data and just \(\tfrac{1}{3}\) of the CamVid training set outperform models trained on the complete CamVid training set.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Fully convolutional networks for semantic segmentation
Long, J., Shelhamer, E., & Darrell, T. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298965 · Cited by 38,834
The Cityscapes Dataset for Semantic Urban Scene Understanding
Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., & Schiele, B. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.350 · Cited by 12,433 · cited 8×
“In a conventional annotation process used for datasets such as CamVid and Cityscapes, the pre-annotated area is a constant 0% for all images.”
BackgroundMethodInfluentialSelf-citationHorn, B. K. P., & Schunck, B. G. (1981) · Artificial Intelligence · doi:10.1016/0004-3702(81)90024-2 · Cited by 10,312
Vision meets robotics: The KITTI dataset
Geiger, A., Lenz, P., Stiller, C., & Urtasun, R. (2013) · The International Journal of Robotics Research · doi:10.1177/0278364913491297 · Cited by 9,913
Fast Algorithms for Mining Association Rules in Large Databases
Agrawal, R., & Srikant, R. (1994) · Very Large Data Bases · Cited by 9,419
Multi-scale context aggregation by dilated convolutions
Yu, F., & Koltun, V. (2015) · arXiv · doi:10.48550/arxiv.1511.07122 · Cited by 7,738 · cited 2×
“As our semantic segmentation model, we use the front-end prediction module of Yu and Koltun [54].”
MethodSelf-citationFlownet: Learning optical flow with convolutional networks
Dosovitskiy, A., Fischer, P., Ilg, E., Hausser, P., Hazirbas, C., Golkov, V., Smagt, P. V. D., Cremers, D., & Brox, T. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.316 · Cited by 4,313
Performance of optical flow techniques
Barron, J. L., Fleet, D. J., & Beauchemin, S. S. (1994) · International Journal of Computer Vision · doi:10.1007/bf01420984 · Cited by 4,305
Efficient Inference in Fully Connected CRFs with Gaussian Edge Potentials
Krähenbühl, P., & Koltun, V. (2012) · arXiv · doi:10.48550/arxiv.1210.5644 · Cited by 2,908
Self-citationMayer, N., Ilg, E., Hausser, P., Fischer, P., Cremers, D., Dosovitskiy, A., & Brox, T. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.438 · Cited by 2,488
A Database and Evaluation Methodology for Optical Flow
Baker, S., Scharstein, D., Lewis, J. P., Roth, S., Black, M. J., & Szeliski, R. (2011) · International Journal of Computer Vision · doi:10.1007/s11263-010-0390-2 · Cited by 2,205 · cited 1×
“The use of synthetic data has been particularly significant in evaluating optical flow estimation due to the difficulty of obtaining accurate ground-truth flow measurements for real-world scenes [6, 7, 18, 32].”
BackgroundSelf-citationDeepDriving: Learning Affordance for Direct Perception in Autonomous Driving
Chen, C., Seff, A., Kornhauser, A., & Xiao, J. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.312 · Cited by 1,876
A Naturalistic Open Source Movie for Optical Flow Evaluation
Butler, D. J., Wulff, J., Stanley, G. B., & Black, M. J. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-33783-3_44 · Cited by 1,842
Learning a classification model for segmentation
Ren, & Malik (2003) · Proceedings Ninth IEEE International Conference on Computer Vision · doi:10.1109/iccv.2003.1238308 · Cited by 1,765
Semantic object classes in video: A high-definition ground truth database
Brostow, G. J., Fauqueur, J., & Cipolla, R. (2009) · Pattern Recognition Letters · doi:10.1016/j.patrec.2008.04.005 · Cited by 1,397
Mayer, N., Ilg, E., Cremers, D., Dosovitskiy, A., & Brox, T. (2016) · Cited by 1,214
Shotton, J., Winn, J., Rother, C., & Criminisi, A. (2009) · International Journal of Computer Vision · doi:10.1007/s11263-007-0109-1 · Cited by 1,026
A benchmark for RGB-D visual odometry, 3D reconstruction and SLAM
Handa, A., Whelan, T., McDonald, J., & Davison, A. J. (2014) · 2014 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2014.6907054 · Cited by 906
Scene Text Recognition using Higher Order Language Priors
Mishra, A., Alahari, K., & Jawahar, C. (2012) · Procedings of the British Machine Vision Conference 2012 · doi:10.5244/c.26.127 · Cited by 831
Multiscale conditional random fields for image labeling
He, X., Zemel, R. S., & Carreira-Perpinan, M. A. (2004) · Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004. · doi:10.1109/cvpr.2004.1315232 · Cited by 828
Comprehensible rendering of 3-D shapes
Saito, T., & Takahashi, T. (1990) · ACM SIGGRAPH Conference (SIGGRAPH) · doi:10.1145/97879.97901 · Cited by 579
Efficient Human Pose Estimation from Single Depth Images
Shotton, J., Girshick, R., Fitzgibbon, A., Sharp, T., Cook, M., Finocchio, M., Moore, R., Kohli, P., Criminisi, A., Kipman, A., et al. (2013) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2012.241 · Cited by 576
FlowNet: Learning Optical Flow with Convolutional Networks
Fischer, P., Dosovitskiy, A., Ilg, E., Haeusser, P., Hazırbaş, C., Golkov, V., van der Smagt, P., Cremers, D., & Brox, T. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1504.06852 · Cited by 562
Seeing 3D Chairs: Exemplar Part-Based 2D-3D Alignment Using a Large Dataset of CAD Models
Aubry, M., Maturana, D., Efros, A. A., Russell, B. C., & Sivic, J. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.487 · Cited by 473
Accurate, Robust, and Flexible Real-time Hand Tracking
Sharp, T., Keskin, C., Robertson, D., Taylor, J., Shotton, J., Kim, D., Rhemann, C., Leichter, I., Vinnikov, A., Wei, Y., et al. (2015) · Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems · doi:10.1145/2702123.2702179 · Cited by 431 · cited 1×
“Rendered depth maps of parametric models have been used prominently in training leading systems for human pose estimation and hand tracking [42, 43].”
MethodDetours: binary interception of Win32 functions
Hunt, G. C., & Brubacher, D. (1999) · Conference on USENIX Windows NT Symposium · Cited by 424
Learning Deep Object Detectors from 3D Models
Peng, X., Sun, B., Ali, K., & Saenko, K. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.151 · Cited by 338
Comprehensible rendering of 3-D shapes
Saito, T., & Takahashi, T. (1990) · ACM SIGGRAPH Computer Graphics · doi:10.1145/97880.97901 · Cited by 320
Real-Time Rendering, Third Edition
Akenine-Möller, T., Haines, E., & Hoffman, N. (2008) · A K Peters/CRC Press eBooks · doi:10.1201/b10644 · Cited by 264
McCane, B., Novins, K., Crannitch, D., & Galvin, B. (2001) · Computer Vision and Image Understanding · doi:10.1006/cviu.2001.0930 · Cited by 250
Understanding RealWorld Indoor Scenes with Synthetic Data
Handa, A., Patraucean, V., Badrinarayanan, V., Stent, S., & Cipolla, R. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.442 · Cited by 230
Viewpoint-independent object class detection using 3D Feature Maps
Liebelt, J., Schmid, C., & Schertler, K. (2008) · 2008 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2008.4587614 · Cited by 200
Superparsing: Scalable Nonparametric Image Parsing with Superpixels
Tighe, J., & Lazebnik, S. (2013) · International Journal of Computer Vision · doi:10.1007/s11263-012-0574-z · Cited by 193
Combining Appearance and Structure from Motion Features for Road Scene Understanding
Sturgess, P., Alahari, K., Ladicky, L., & Torr, P. H. S. (2009) · Procedings of the British Machine Vision Conference 2009 · doi:10.5244/c.23.62 · Cited by 189
Virtual and Real World Adaptation for Pedestrian Detection
Vazquez, D., Lopez, A. M., Marin, J., Ponsa, D., & Geronimo, D. (2014) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2013.163 · Cited by 178
Feature Space Optimization for Semantic Video Segmentation
Kundu, A., Vineet, V., & Koltun, V. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.345 · Cited by 175
Self-citationLearning appearance in virtual scenarios for pedestrian detection
Marin, J., Vazquez, D., Geronimo, D., & Lopez, A. M. (2010) · 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2010.5540218 · Cited by 168
Semantic Instance Annotation of Street Scenes by 3D to 2D Label Transfer
Xie, J., Kiefel, M., Sun, M.-T., & Geiger, A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.401 · Cited by 167
Back to the Future: Learning Shape Models from 3D CAD Data
Stark, M. M., Goesele, M., & Schiele, B. (2010) · British Machine Vision Conference (BMVC) · doi:10.5244/c.24.106 · Cited by 151
Vision-Based Offline-Online Perception Paradigm for Autonomous Driving
Ros, G., Ramos, S., Granados, M., Bakhtiary, A., Vázquez, D., & López, A. M. (2015) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv.2015.38 · Cited by 138
Understanding Deep Features with Computer-Generated Imagery
Aubry, M., & Russell, B. C. (2015) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.329 · Cited by 137
OVVV: Using Virtual Worlds to Design and Evaluate Surveillance Systems
Taylor, G. R., Chosak, A. J., & Brewer, P. C. (2007) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2007.383518 · Cited by 133
Play and Learn: Using Video Games to Train Computer Vision Models
Shafaei, A., Little, J. J., & Schmidt, M. (2016) · British Machine Vision Conference (BMVC) · doi:10.5244/c.30.26 · Cited by 129
Deep Exemplar 2D-3D Detection by Adapting from Real to Rendered Views
Massa, F., Russell, B. C., & Aubry, M. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.648 · Cited by 106
Multi-View and 3D Deformable Part Models
Pepik, B., Stark, M. M., Gehler, P., & Schiele, B. (2015) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2015.2408347 · Cited by 91
Semantic Pose Using Deep Networks Trained on Synthetic RGB-D
Papon, J. A., & Schoeler, M. (2015) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.95 · Cited by 75
Evaluation of image features using a photorealistic virtual world
Kaneva, B., Torralba, A. B., & Freeman, W. T. (2011) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2011.6126508 · Cited by 73
Multiclass semantic video segmentation with object-level active inference
Liu, B., & He, X. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7299057 · Cited by 72
Discriminative shape from shading in uncalibrated illumination
Richter, S. R., & Roth, S. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7298716 · Cited by 65
Self-citationLearning a Part-Based Pedestrian Detector in a Virtual World
Xu, J., Vázquez, D., López, A. M., Marín, J., & Ponsa, D. (2014) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2014.2310138 · Cited by 58
Framework for Generation of Synthetic Ground Truth Data for Driver Assistance Applications
Haltakov, V., Unger, C., & Ilić, S. (2013) · Lecture notes in computer science · doi:10.1007/978-3-642-40602-7_35 · Cited by 57
Semantic video segmentation: Exploring inference efficiency
Tripathi, S., Belongie, S., Hwang, Y., & Nguyen, T. Q. (2015) · International SoC Design Conference · doi:10.1109/isocc.2015.7401766 · Cited by 24
Model-driven Simulations for Deep Convolutional Neural NetworksPreprint
Veeravasarapu, V. S. R., Rothkopf, C. A., & Ramesh, V. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.09582 · Cited by 14
A Discriminative Approach to Perspective Shape from Shading in Uncalibrated Illumination
Richter, S. R., & Roth, S. (2015) · Computers & Graphics · doi:10.1016/j.cag.2015.09.001 · Cited by 8
Self-citation
Cited by
A survey on Image Data Augmentation for Deep Learning
Shorten, C., & Khoshgoftaar, T. M. (2019) · Journal of Big Data · doi:10.1186/s40537-019-0197-0 · Cited by 13,203
Domain randomization for transferring deep neural networks from simulation to the real world
Tobin, J., Fong, R., Ray, A., Schneider, J., Zaremba, W., & Abbeel, P. (2017) · 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2017.8202133 · Cited by 3,124
“Combining data from high-quality simulators with other approaches like fine-tuning can also reduce the number of labeled samples required in the real world [34].”
BackgroundMachine Learning for Fluid Mechanics
Brunton, S. L., Noack, B. R., & Koumoutsakos, P. (2020) · Annual Review of Fluid Mechanics · doi:10.1146/annurev-fluid-010719-060214 · Cited by 3,063
Maximum Classifier Discrepancy for Unsupervised Domain Adaptation
Saito, K., Watanabe, K., Ushiku, Y., & Harada, T. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00392 · Cited by 2,161
The One Hundred Layers Tiramisu: Fully Convolutional DenseNets for Semantic Segmentation
Jegou, S., Drozdzal, M., Vazquez, D., Romero, A., & Bengio, Y. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2017.156 · Cited by 1,736
Metadata
- Authors
- Richter, S. R., Vineet, V., Roth, S., & Koltun, V.
- Venue
- Lecture Notes in Computer Science
- Publisher
- Springer International Publishing
- Published
- 2016
- Pages
- 102-118
- ISSN
- 0302-9743 (ISSN-L)
- 1611-3349
- OpenAlex
- W2487365028
- Cited by
- 1,815
- References
- 55
- FWCI
- 38.65
- Type
- conference paper
- Language
- en
- Open access
- closed
- Licence
- www.springer.com/tdm (tdm)
- www.springer.com/tdm (vor)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.