The DOI matches its Crossref record; 8 of the 8 references it cites were found in a registry or an index.
Learning from Simulated and Unsupervised Images through Adversarial Training
Open accessAbstract
With recent progress in graphics, it has become more tractable to train models on synthetic images, potentially avoiding the need for expensive annotations. However, learning from synthetic images may not achieve the desired performance due to a gap between synthetic and real image distributions. To reduce this gap, we propose Simulated+Unsupervised (S+U) learning, where the task is to learn a model to improve the realism of a simulators output using unlabeled real data, while preserving the annotation information from the simulator. We develop a method for S+U learning that uses an adversarial network similar to Generative Adversarial Networks (GANs), but with synthetic images as inputs instead of random vectors. We make several key modifications to the standard GAN algorithm to preserve annotations, avoid artifacts, and stabilize training: (i) a self-regularization term, (ii) a local adversarial loss, and (iii) updating the discriminator using a history of refined images. We show that this enables generation of highly realistic images, which we demonstrate both qualitatively and with a user study. We quantitatively evaluate the generated images by training models for gaze estimation and hand pose estimation. We show a significant improvement over using synthetic images, and achieve state-of-the-art results on the MPIIGaze dataset without any labeled real data.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
Improved Techniques for Training GANs
Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., & Chen, X. (2016) · arXiv · doi:10.48550/arxiv.1606.03498 · Cited by 6,965
Stacked Hourglass Networks for Human Pose Estimation
Newell, A., Yang, K., & Deng, J. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46484-8_29 · Cited by 5,634
Human3.6M: Large Scale Datasets and Predictive Methods for 3D Human Sensing in Natural Environments
Ionescu, C., Papava, D., Olaru, V., & Sminchisescu, C. (2014) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2013.248 · Cited by 3,653
Ros, G., Sellart, L., Materzynska, J., Vazquez, D., & Lopez, A. M. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.352 · Cited by 2,338
Learning methods for generic object recognition with invariance to pose and lighting
LeCun, Y., Huang, F. J., & Bottou, L. (2004) · Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004. · doi:10.1109/cvpr.2004.1315150 · Cited by 1,324
VirtualWorlds as Proxy for Multi-object Tracking Analysis
Gaidon, A., Wang, Q., Cabon, Y., & Vig, E. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.470 · Cited by 1,079
Cited by
Unpaired image-to-image translation using cycle-consistent adversarial networks
Zhu, J.-Y., Park, T., Isola, P., & Efros, A. A. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.244 · Cited by 23,602
Generative adversarial networks
Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., & Bengio, Y. (2020) · Communications of the ACM · doi:10.1145/3422622 · Cited by 18,384
Generative Adversarial Networks: An Overview
Creswell, A., White, T., Dumoulin, V., Arulkumaran, K., Sengupta, B., & Bharath, A. A. (2018) · IEEE Signal Processing Magazine · doi:10.1109/msp.2017.2765202 · Cited by 4,540
CyCADA: Cycle-Consistent Adversarial Domain Adaptation
Hoffman, J., Tzeng, E., Park, T., Zhu, J.-Y., Isola, P., Saenko, K., Efros, A. A., & Darrell, T. (2017) · arXiv · doi:10.48550/arxiv.1711.03213 · Cited by 2,656
Deep visual domain adaptation: A survey
Wang, M., & Deng, W. (2018) · Neurocomputing · doi:10.1016/j.neucom.2018.05.083 · Cited by 2,209
Instructpix2pix: Learning to follow image editing instructions
Brooks, T., Holynski, A., & Efros, A. A. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01764 · Cited by 1,466
In Ictu Oculi: Exposing AI Created Fake Videos by Detecting Eye Blinking
Li, Y., Chang, M.-C., & Lyu, S. (2018) · 2018 IEEE International Workshop on Information Forensics and Security (WIFS) · doi:10.1109/wifs.2018.8630787 · Cited by 1,089
A survey of unsupervised deep domain adaptation
Wilson, G., & Cook, D. J. (2020) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3400066 · Cited by 987
Exposing DeepFake Videos By Detecting Face Warping Artifacts
Li, Y., & Lyu, S. (2018) · arXiv · doi:10.48550/arxiv.1811.00656 · Cited by 785
Deep Reinforcement Learning: An Overview
Li, Y. (2017) · arXiv · doi:10.48550/arxiv.1701.07274 · Cited by 762
Sandfort, V., Yan, K., Pickhardt, P. J., & Summers, R. M. (2019) · Scientific Reports · doi:10.1038/s41598-019-52737-x · Cited by 684
Using Simulation and Domain Adaptation to Improve Efficiency of Deep Robotic Grasping
Bousmalis, K., Irpan, A., Wohlhart, P., Bai, Y., Kelcey, M., Kalakrishnan, M., Downs, L., Ibarz, J., Pastor, P., Konolige, K., et al. (2018) · 2018 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2018.8460875 · Cited by 636
Cross-Domain Weakly-Supervised Object Detection Through Progressive Domain Adaptation
Inoue, N., Furuta, R., Yamasaki, T., & Aizawa, K. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00525 · Cited by 616
GANimation: Anatomically-Aware Facial Animation from a Single Image
Pumarola, A., Agudo, A., Martinez, A. M., Sanfeliu, A., & Moreno-Noguer, F. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01249-6_50 · Cited by 621
Wang, T.-C., Liu, M.-Y., Zhu, J.-Y., Liu, G., Tao, A., Kautz, J., & Catanzaro, B. (2018) · arXiv · doi:10.48550/arxiv.1808.06601 · Cited by 594
How to train your robot with deep reinforcement learning: Lessons we have learned
Ibarz, J., Tan, J., Finn, C., Kalakrishnan, M., Pastor, P., & Levine, S. (2021) · The International Journal of Robotics Research · doi:10.1177/0278364920987859 · Cited by 570
Implicit 3D Orientation Learning for 6D Object Detection from RGB Images
Sundermeyer, M., Marton, Z.-C., Durner, M., Brucker, M., & Triebel, R. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01231-1_43 · Cited by 564
James, S., Wohlhart, P., Kalakrishnan, M., Kalashnikov, D., Irpan, A., Ibarz, J., Levine, S., Hadsell, R., & Bousmalis, K. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.01291 · Cited by 420
Detection of GAN-Generated Fake Images over Social Networks
Marra, F., Gragnaniello, D., Cozzolino, D., & Verdoliva, L. (2018) · 2018 IEEE Conference on Multimedia Information Processing and Retrieval (MIPR) · doi:10.1109/mipr.2018.00084 · Cited by 399
Dimiduk, D. M., Holm, E. A., & Niezgoda, S. R. (2018) · Integrating Materials and Manufacturing Innovation · doi:10.1007/s40192-018-0117-8 · Cited by 347
Shmelkov, K., Schmid, C., & Alahari, K. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01216-8_14 · Cited by 341
Kellenberger, B., Marcos, D., & Tuia, D. (2018) · Remote Sensing of Environment · doi:10.1016/j.rse.2018.06.028 · Cited by 335
Self-Supervised GANs via Auxiliary Rotation Loss
Chen, T., Zhai, X., Ritter, M., Lucic, M., & Houlsby, N. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.01243 · Cited by 328
Synthetic data augmentation for surface defect detection and classification using deep learning
Jain, S., Seth, G., Paruthi, A., Soni, U., & Kumar, G. (2022) · Journal of Intelligent Manufacturing · doi:10.1007/s10845-020-01710-x · Cited by 302
On Aliased Resizing and Surprising Subtleties in GAN Evaluation
Parmar, G., Zhang, R., & Zhu, J.-Y. (2021) · arXiv · doi:10.48550/arxiv.2104.11222 · Cited by 293
Weakly Supervised Object Localization and Detection: A Survey
Zhang, D., Han, J., Cheng, G., & Yang, M.-H. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3074313 · Cited by 277
Unsupervised Reverse Domain Adaptation for Synthetic Medical Images via Adversarial Training
Mahmood, F., Chen, R., & Durr, N. J. (2018) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2018.2842767 · Cited by 257
“This complication makes it difficult for methods like SimGAN [15] to work both accurately and generally because the span of realistic images produced will be similar to the real images used for training.”
BackgroundMethodInfluentialCombining Noise-to-Image and Image-to-Image GANs: Brain MR Image Augmentation for Tumor Detection
Han, C., Rundo, L., Araki, R., Nagano, Y., Furukawa, Y., Mauri, G., Nakayama, H., & Hayashi, H. (2019) · IEEE Access · doi:10.1109/access.2019.2947606 · Cited by 253
Deep metric learning for few-shot image classification: A Review of recent developments
Li, X., Yang, X., Ma, Z., & Xue, J.-H. (2023) · Pattern Recognition · doi:10.1016/j.patcog.2023.109381 · Cited by 243
RoboTHOR: An Open Simulation-to-Real Embodied AI Platform
Deitke, M., Han, W., Herrasti, A., Kembhavi, A., Kolve, E., Mottaghi, R., Salvador, J., Schwenk, D., VanderBilt, E., Wallingford, M., et al. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00323 · Cited by 220
GAN Compression: Efficient Architectures for Interactive Conditional GANs
Li, M., Lin, J., Ding, Y., Liu, Z., Zhu, J.-Y., & Han, S. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00533 · Cited by 216
Artificial Intelligence and Biosensors in Healthcare and Its Clinical Relevance: A Review
(2023) · doi:10.1109/access.2023.3285596 · Cited by 205
Neural Rendering and Reenactment of Human Actor Videos
Liu, L., Xu, W., Zollhöfer, M., Kim, H., Bernard, F., Habermann, M., Wang, W., & Theobalt, C. (2019) · ACM Transactions on Graphics · doi:10.1145/3333002 · Cited by 163
Self-citationA survey on face data augmentation for the training of deep neural networks
Wang, X., Wang, K., & Lian, S. (2020) · Neural Computing and Applications · doi:10.1007/s00521-020-04748-3 · Cited by 141
On Learning Invariant Representation for Domain Adaptation
Zhao, H., Combes, R. T. D., Zhang, K., & Gordon, G. J. (2019) · arXiv · doi:10.48550/arxiv.1901.09453 · Cited by 134
CAM-Convs: Camera-Aware Multi-Scale Convolutions for Single-View Depth
Facil, J. M., Ummenhofer, B., Zhou, H., Montesano, L., Brox, T., & Civera, J. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.01210 · Cited by 130
“Among others, there has been recent relevant results on problems such as 6D object pose detection [17, 30, 32], 3D model reconstruction [9, 34], depth estimation from single [21, 10, 22] and multiple views [38, 16], 6D camera pose recovery [19, 18] or camera tracking and mapping [46, 2, 33, 35].”
BackgroundSelf-Supervised Feature Learning by Learning to Spot Artifacts
Jenni, S., & Favaro, P. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00289 · Cited by 115
DreamSim: Learning New Dimensions of Human Visual Similarity using Synthetic Data
Fu, S., Tamir, N., Sundaram, S., Chai, L., Zhang, R., Dekel, T., & Isola, P. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-2208 · Cited by 108
Synthetic Examples Improve Generalization for Rare Classes
Beery, S., Liu, Y., Morris, D., Piavis, J., Kapoor, A., Meister, M., Joshi, N., & Perona, P. (2020) · 2020 IEEE Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv45572.2020.9093570 · Cited by 103
Triguero, I., Molina, D., Poyatos, J., Del Ser, J., & Herrera, F. (2024) · Information Fusion · doi:10.1016/j.inffus.2023.102135 · Cited by 90
Enhancing scientific discoveries in molecular biology with deep generative models
Lopez, R., Gayoso, A., & Yosef, N. (2020) · Molecular Systems Biology · doi:10.15252/msb.20199198 · Cited by 82
Sim-to-Real Transfer with Neural-Augmented Robot Simulation
(2018) · HAL · Cited by 63
Yuan, W., Hang, K., Kragic, D., Wang, M. Y., & Stork, J. A. (2019) · Robotics and Autonomous Systems · doi:10.1016/j.robot.2019.06.007 · Cited by 54
Human Gaze Assisted Artificial Intelligence: A Review
Zhang, R., Saran, A., Liu, B., Zhu, Y., Guo, S., Niekum, S., Ballard, D., & Hayhoe, M. (2020) · Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2020/689 · Cited by 48
Rizvi, S. K. J., Azad, M. A., & Fraz, M. M. (2021) · Archives of Computational Methods in Engineering · doi:10.1007/s11831-021-09543-4 · Cited by 37
Pérez-Villar, J. I. B., García-Martín, Á., Bescós, J., & Escudero-Viñolo, M. (2024) · IEEE Transactions on Aerospace and Electronic Systems · doi:10.1109/taes.2023.3306731 · Cited by 22
Ma, T., & Tian, W. (2021) · The Visual Computer · doi:10.1007/s00371-020-01843-3 · Cited by 16
Market Manipulation: An Adversarial Learning Framework for Detection and Evasion
Wang, X., & Wellman, M. P. (2020) · Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2020/638 · Cited by 13
Sim-to-Real Transfer for Object Detection in Aerial Inspections of Transmission Towers
Peterlevitz, A. J., Chinelatto, M. A., Menezes, A. G., Motta, C. A. M., Pereira, G. A. B., Lopes, G. L., Souza, G. D. M., Rodrigues, J., Godoy, L. C., Koller, M. A. F. F., et al. (2023) · IEEE Access · doi:10.1109/access.2023.3322374 · Cited by 7
“…explored the application of style transfer and domain adaption techniques (techniques that aim to improve the performance of a model trained on data that belongs to a different domain) to compensate for these differences and improve results for applications in the real-world [8], [9], [10], [11].”
Background
Metadata
- Authors
- Shrivastava, A., Pfister, T., Tuzel, O., Susskind, J., Wang, W., & Webb, R.
- Venue
- 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2017-07
- Pages
- 2242-2251
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W2567101557
- Cited by
- 1,814
- References
- 8
- FWCI
- 83.65
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
