Generalization in Reinforcement Learning by Soft Data Augmentation
Open accessAbstract
Extensive efforts have been made to improve the generalization ability of Reinforcement Learning (RL) methods via domain randomization and data augmentation. However, as more factors of variation are introduced during training, optimization becomes increasingly challenging, and empirically may result in lower sample efficiency and unstable training. Instead of learning policies directly from augmented data, we propose SOft Data Augmentation (SODA), a method that decouples augmentation from policy learning. Specifically, SODA imposes a soft constraint on the encoder that aims to maximize the mutual information between latent representations of augmented and non-augmented data, while the RL optimization process uses strictly non-augmented data. Empirical evaluations are performed on diverse tasks from DeepMind Control suite as well as a robotic manipulation task, and we find SODA to significantly advance sample efficiency, generalization, and stability in training over state-of-the-art vision-based RL methods.1
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Soft Data Augmentation improves sample efficiency and stabilises reinforcement learning optimisation by decoupling data augmentation from policy learning.
“Soft Data Augmentation (SODA) [142] further improves sample efficiency and stabilises the RL optimisation by decoupling data augmentation from policy learning.”
From Data-Driven Robotic Manipulation of Cloth-like Deformable Objects: The Present, Challenges and Future Prospects (2023) · cites this paper as [142]
Deep reinforcement learning encoders suffer from observation overfitting, so data augmentation can improve data efficiency and generalisation ability of a DRL agent.
“In general, DRL encoders suffer from observation overfitting [160,161], so DA can also be used to improve data efficiency and generalisation ability of a DRL agent [141,142].”
From Data-Driven Robotic Manipulation of Cloth-like Deformable Objects: The Present, Challenges and Future Prospects (2023) · cites this paper as [142]
Many benchmarks and datasets with perturbations have been specifically defined or curated as part of various BC or reinforcement learning works, but they do not provide a unified framework to evaluate all potential perturbations for generalization.
“Additionally, many of the benchmarks and datasets with perturbations have been specifically defined or curated as part of various BC or reinforcement learning works [76, 24, 71]. However, they do not provide an unified framework to evaluate all potential perturbations for generalization, primarily because they are not the main focus of these works.”
From THE COLOSSEUM: A Benchmark for Evaluating Generalization for Robotic Manipulation (2024) · cites this paper as [24]
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · see citing works
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S., & Szegedy, C. (2015) · arXiv · doi:10.48550/arxiv.1502.03167 · Cited by 27,818 · see citing works
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate ShiftPreprint
Ioffe, S. (2024) · arXiv (Cornell University) · doi:10.57702/o9raffed · Cited by 15,273 · see citing works
Momentum Contrast for Unsupervised Visual Representation Learning
He, K., Fan, H., Wu, Y., Xie, S., & Girshick, R. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00975 · Cited by 13,347 · see citing works
A simple framework for contrastive learning of visual representations
Chen, T., Kornblith, S., Norouzi, M., & Hinton, G. (2020) · arXiv · doi:10.48550/arxiv.2002.05709 · Cited by 10,857 · see citing works
Continuous control with deep reinforcement learning
Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., & Wierstra, D. (2015) · arXiv · doi:10.48550/arxiv.1509.02971 · Cited by 10,782 · see citing works
Playing Atari with Deep Reinforcement Learning
Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., & Riedmiller, M. (2013) · arXiv · doi:10.48550/arxiv.1312.5602 · Cited by 9,471 · see citing works
Representation Learning with Contrastive Predictive Coding
Oord, A. V. D., Li, Y., & Vinyals, O. (2018) · arXiv · doi:10.48550/arxiv.1807.03748 · Cited by 6,242 · see citing works
Context Encoders: Feature Learning by Inpainting
Pathak, D., Krahenbuhl, P., Donahue, J., Darrell, T., & Efros, A. A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.278 · Cited by 5,691 · see citing works
Places: A 10 Million Image Database for Scene Recognition
Zhou, B., Lapedriza, A., Khosla, A., Oliva, A., & Torralba, A. (2018) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2017.2723009 · Cited by 4,042 · see citing works
End-to-End Training of Deep Visuomotor PoliciesPreprint
Levine, S., Finn, C., Darrell, T., & Abbeel, P. (2015) · Journal of machine learning research · doi:10.48550/arxiv.1504.00702 · Cited by 4,011 · see citing works
Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning
Grill, J.-B., Strub, F., Altché, F., Tallec, C., Richemond, P. H., Buchatskaya, E., Doersch, C., Pires, B. A., Guo, Z. D., Azar, M. G., et al. (2020) · arXiv · doi:10.48550/arxiv.2006.07733 · Cited by 3,846 · see citing works
Unsupervised Feature Learning via Non-parametric Instance Discrimination
Wu, Z., Xiong, Y., Yu, S. X., & Lin, D. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00393 · Cited by 3,664 · see citing works
Zhang, R., Isola, P., & Efros, A. A. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46487-9_40 · Cited by 3,361 · see citing works
Domain randomization for transferring deep neural networks from simulation to the real world
Tobin, J., Fong, R., Ray, A., Schneider, J., Zaremba, W., & Abbeel, P. (2017) · 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2017.8202133 · Cited by 3,124 · cited 2× · see citing works
“An example of the former is random cropping [9], [39], whereas examples of the latter are domain randomization [12], [13] and random convolution [9], [16].”
BackgroundUnsupervised Learning of Visual Representations by Solving Jigsaw Puzzles
Noroozi, M., & Favaro, P. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46466-4_5 · Cited by 2,775 · see citing works
Unsupervised Visual Representation Learning by Context Prediction
Doersch, C., Gupta, A., & Efros, A. A. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.167 · Cited by 2,629 · see citing works
End-to-end training of deep visuomotor policies
Levine, S., Finn, C., Darrell, T., & Abbeel, P. (2016) · Journal of Machine Learning Research · Cited by 2,192 · see citing works
Tian, Y., Krishnan, D., & Isola, P. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58621-8_45 · Cited by 1,997 · see citing works
Soft Actor-Critic Algorithms and Applications
Haarnoja, T., Zhou, A., Hartikainen, K., Tucker, G., Ha, S., Tan, J., Kumar, V., Zhu, H., Gupta, A., Abbeel, P., et al. (2018) · arXiv · doi:10.48550/arxiv.1812.05905 · Cited by 1,994 · see citing works
Target-driven visual navigation in indoor scenes using deep reinforcement learning
Zhu, Y., Mottaghi, R., Kolve, E., Lim, J. J., Gupta, A., Fei-Fei, L., & Farhadi, A. (2017) · 2017 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2017.7989381 · Cited by 1,533 · see citing works
Self-Supervised Learning of Pretext-Invariant Representations
Misra, I., & van der Maaten, L. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00674 · Cited by 1,435 · see citing works
Sim-to-Real Transfer of Robotic Control with Dynamics Randomization
Peng, X. B., Andrychowicz, M., Zaremba, W., & Abbeel, P. (2018) · 2018 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2018.8460528 · Cited by 1,309 · see citing works
Unsupervised Learning of Visual Representations Using Videos
Wang, X., & Gupta, A. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.320 · Cited by 1,005 · see citing works
Self-citationWhat makes for good views for contrastive learning
Tian, Y., Sun, C., Poole, B., Krishnan, D., Schmid, C., & Isola, P. (2020) · arXiv · doi:10.48550/arxiv.2005.10243 · Cited by 773 · see citing works
QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation
Kalashnikov, D., Irpan, A., Pastor, P., Ibarz, J., Herzog, A., Jang, E., Quillen, D., Holly, E., Kalakrishnan, M., Vanhoucke, V., et al. (2018) · arXiv · doi:10.48550/arxiv.1806.10293 · Cited by 697 · see citing works
Visual Reinforcement Learning with Imagined GoalsPreprint
Nair, A., Pong, V. H., Dalal, M., Bahl, S., Lin, S., & Levine, S. (2018) · Neural Information Processing Systems · doi:10.48550/arxiv.1807.04742 · Cited by 623 · see citing works
Tassa, Y., Doron, Y., Muldal, A., Erez, T., Li, Y., Casas, D. D. L., Budden, D., Abdolmaleki, A., Merel, J., Lefrancq, A., et al. (2018) · arXiv · doi:10.48550/arxiv.1801.00690 · Cited by 497 · see citing works
Tian, Y., Krishnan, D., & Isola, P. (2019) · arXiv · doi:10.48550/arxiv.1906.05849 · Cited by 465 · see citing works
Asymmetric Actor Critic for Image-Based Robot Learning
Pinto, L., Andrychowicz, M., Welinder, P., Zaremba, W., & Abbeel, P. (2018) · Robotics: Science and Systems · doi:10.15607/rss.2018.xiv.008 · Cited by 369 · see citing works
Quantifying Generalization in Reinforcement LearningPreprint
Cobbe, K., Klimov, O., Hesse, C., Kim, T., & Schulman, J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1812.02341 · Cited by 348 · see citing works
Reinforcement Learning with Augmented Data
Laskin, M., Lee, K., Stooke, A., Pinto, L., Abbeel, P., & Srinivas, A. (2020) · arXiv · doi:10.48550/arxiv.2004.14990 · Cited by 299 · see citing works
Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels
Kostrikov, I., Yarats, D., & Fergus, R. (2020) · arXiv · doi:10.48550/arxiv.2004.13649 · Cited by 267 · see citing works
Self-Supervised Policy Adaptation during Deployment
Hansen, N., Jangir, R., Sun, Y., Alenyà, G., Abbeel, P., Efros, A. A., Pinto, L., & Wang, X.-L. (2020) · International Conference on Learning Representations · doi:10.48550/arxiv.2007.04309 · Cited by 212 · see citing works
Self-citationLeveraging Procedural Generation to Benchmark Reinforcement Learning
Cobbe, K., Hesse, C., Hilton, J., & Schulman, J. (2019) · arXiv · doi:10.48550/arxiv.1912.01588 · Cited by 196 · see citing works
Decoupling Representation Learning from Reinforcement Learning
Stooke, A., Lee, K., Abbeel, P., & Laskin, M. (2020) · arXiv · doi:10.48550/arxiv.2009.08319 · Cited by 123 · see citing works
BayesSim: Adaptive Domain Randomization Via Probabilistic Inference for Robotics Simulators
Ramos, F. T., Possas, R., & Fox, D. (2019) · Robotics: Science and Systems · doi:10.15607/rss.2019.xv.029 · Cited by 121 · see citing works
Assessing Generalization in Deep Reinforcement Learning
Packer, C., Gao, K., Kos, J., Krähenbühl, P., Koltun, V., & Song, D. (2018) · arXiv · doi:10.48550/arxiv.1810.12282 · Cited by 114 · see citing works
Improving Sample Efficiency in Model-Free Reinforcement Learning from Images
Yarats, D., Zhang, A., Kostrikov, I., Amos, B., Pineau, J., & Fergus, R. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i12.17276 · Cited by 108 · see citing works
Data-Efficient Reinforcement Learning with Self-Predictive RepresentationsPreprint
Schwarzer, M., Anand, A., Goel, R., Hjelm, R. D., Courville, A. C., & Bachman, P. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.05929 · Cited by 103 · see citing works
Transfer Learning for Related Reinforcement Learning Tasks via Image-to-Image Translation
Gamrian, S., & Goldberg, Y. (2018) · arXiv · doi:10.48550/arxiv.1806.07377 · Cited by 70 · see citing works
Automatic Data Augmentation for Generalization in Deep Reinforcement LearningPreprint
Răileanu, R., Goldstein, M., Yarats, D., Kostrikov, I., & Fergus, R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.12862 · Cited by 65 · see citing works
Never Stop Learning: The Effectiveness of Fine-Tuning in Robotic Reinforcement LearningPreprint
Julian, R., Swanson, B., Sukhatme, G. S., Levine, S., Finn, C., & Hausman, K. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.10190 · Cited by 50 · see citing works
Observational Overfitting in Reinforcement Learning
Song, X., Jiang, Y., Tu, S., Du, Y., & Neyshabur, B. (2019) · arXiv · doi:10.48550/arxiv.1912.02975 · Cited by 48 · see citing works
Single Episode Policy Transfer in Reinforcement LearningPreprint
Yang, J., Petersen, B. K., Hongyuan Zha, B., & Faissol, D. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.07719 · Cited by 25 · see citing works
Adversarial Feature Training for Generalizable Robotic Visuomotor Control
Chen, X., Ghadirzadeh, A., Björkman, M., & Jensfelt, P. (2020) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra40945.2020.9197505 · Cited by 14 · see citing works
Domain Adaptation for Reinforcement Learning on the Atari
Carr, T. A., Chli, M., & Vogiatzis, G. (2019) · Adaptive Agents and Multi-Agents Systems · doi:10.65109/ukyv4725 · Cited by 10 · see citing works
A Simple Randomization Technique for Generalization in Deep Reinforcement LearningPreprint
Lee, K., Lee, K., Shin, J., & Lee, H. (2019) · arXiv (Cornell University) · Cited by 3 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
A Survey of Zero-shot Generalisation in Deep Reinforcement Learning
Kirk, R., Zhang, A., Grefenstette, E., & Rocktäschel, T. (2023) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.14174 · Cited by 193 · see citing works
“…continuous control environments (RoboSuite, DMC-Remastered, DMC-GB, DCS, KitchenShift, NaturalEnvs MuJoCo; Fan et al., 2021; Grigsby & Qi, 2020; Hansen & Wang, 2021; Stone et al., 2021; Xing et al., 2021a; Zhang et al., 2018a), and multi-task continuous control benchmarks which could be…”
BackgroundMethodInfluentialMastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learning
Yarats, D., Fergus, R., Lazaric, A., & Pinto, L. (2021) · arXiv · doi:10.48550/arxiv.2107.09645 · Cited by 89 · see citing works
Towards Trustworthy Machine Learning in Production: An Overview of the Robustness in MLOps Approach
Bayram, F., & Ahmed, B. S. (2025) · ACM Computing Surveys · doi:10.1145/3708497 · Cited by 48 · see citing works
THE COLOSSEUM: A Benchmark for Evaluating Generalization for Robotic Manipulation
Pumacay, W., Singh, I., Duan, J., Krishna, R., Thomason, J., & Fox, D. (2024) · Robotics: Science and Systems XX · doi:10.15607/rss.2024.xx.133 · Cited by 29 · see citing works
Kadi, H. A., & Terzić, K. (2023) · Sensors · doi:10.3390/s23052389 · Cited by 19 · see citing works
“However, optimisation of RL becomes increasingly challenging with the increasing variation in the observation due to the limited expressiveness of NNs [142].”
BackgroundMethodInfluential
- Authors
- Hansen, N., & Wang, X.
- Affiliations
- University of California San Diego (US)
- Venue
- 2021 IEEE International Conference on Robotics and Automation (ICRA)
- Publisher
- IEEE
- Published
- 2021-05-30
- Pages
- 13611-13617
- ISSN
- 1050-4729 (ISSN-L)
- OpenAlex
- W3205321526
- Cited by
- 111
- References
- 48
- Citations by year
- 2026: 6 · 2025: 26 · 2024: 29 · 2023: 28 · 2022: 15 · 2021: 7
- FWCI
- 9.12
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- Proceedings - IEEE International Conference on Robotics and Automation/Proceedings (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- eScholarship (California Digital Library) (submittedVersion, cc-by)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1730158, 1541349)
- Defense Advanced Research Projects Agency
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
- Keywords
- generalization, reinforcement learning, domain randomization, mutual information maximization, vision-based reinforcement learning, sample efficiency, robotic manipulation, DeepMind Control Suite
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
