Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learning
Open accessAbstract
We present DrQ-v2, a model-free reinforcement learning (RL) algorithm for visual continuous control. DrQ-v2 builds on DrQ, an off-policy actor-critic approach that uses data augmentation to learn directly from pixels. We introduce several improvements that yield state-of-the-art results on the DeepMind Control Suite. Notably, DrQ-v2 is able to solve complex humanoid locomotion tasks directly from pixel observations, previously unattained by model-free RL. DrQ-v2 is conceptually simple, easy to implement, and provides significantly better computational footprint compared to prior work, with the majority of tasks taking just 8 hours to train on a single GPU. Finally, we publicly release DrQ-v2's implementation to provide RL practitioners with a strong and computationally efficient baseline.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Dreamer establishes a new state-of-the-art on Visual Control, outperforming DrQ-v2 and CURL.
“Dreamer establishes a new state-of-the-art on this benchmark, outperforming DrQ-v2 and CURL 47 , which are specialized to visual environments and leverage data augmentation.”
From Mastering Diverse Domains through World Models (2023) · cites this paper as [47]
The Introduction (p. 2) says DrQ-v2 matches Dreamer-v2 in sample efficiency and performance, while training four times faster—not that Dreamer generally outperforms it. Section 4.3 (p. 9) reports that Dreamer-v2 outperforms DrQ-v2 on several individual tasks, but does not support the claimed overall ranking against DrQ-v2 and CURL. (LLM, conf=0.87)
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe magnitude of the value estimate and entropy can vary greatly between datasets and different stages of training, making it necessary to balance the two losses to prevent premature entropy collapse.
“As magnitude of the value estimate Q ( z t , p ( z t )) and entropy H can vary greatly between datasets and different stages of training, it is necessary to balance the two losses to prevent premature entropy collapse (Yarats et al., 2021).”
From TD-MPC2: Scalable, Robust World Models for Continuous Control (2023) · cites this paper as Yarats et al., 2021
The Ablation Study (p. 9) reports that automatic entropy adjustment can lead to premature entropy collapse. It does not say that the magnitudes of Q estimates and entropy vary across datasets or training stages, or that balancing their losses prevents collapse; it instead examines switching from SAC to DDPG. (LLM, conf=0.86)
Experimental — accuracy not yet measuredThe state-of-the-art baseline for the Sphero navigation task is DrQv2.
“For the Sphero navigation task, inputs are images and actions are continuous. The state-of-the-art baseline in this category is DrQv2 (Yarats et al., 2021), which uses image augmentation to increase sample-efficiency.”
From DayDreamer: World Models for Physical Robot Learning (2022) · cites this paper as Yarats et al., 2021
The Abstract (p. 1) presents DrQ-v2 as a strong baseline for visual continuous control, and Section 3.1 (p. 3) describes its image augmentation. But the experiments evaluate DeepMind Control Suite tasks (Section 4, p. 5), not Sphero navigation, so the source does not establish that DrQ-v2 is state of the art for that specific task. (LLM, conf=0.86)
Experimental — accuracy not yet measuredThe similar performance of Dreamer and DrQv2 on the Sphero navigation task matches the simulated experiments of Yarats et al. (2021), which showed the two algorithms to perform similarly for continuous control tasks from images.
“This result matches the simulated experiments of Yarats et al. (2021) that showed the two algorithms to perform similarly for continuous control tasks from images.”
From DayDreamer: World Models for Physical Robot Learning (2022) · cites this paper as Yarats et al. (2021)
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Continuous control with deep reinforcement learning
Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., & Wierstra, D. (2015) · arXiv · doi:10.48550/arxiv.1509.02971 · Cited by 10,782 · see citing works
Playing Atari with Deep Reinforcement Learning
Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., & Riedmiller, M. (2013) · arXiv · doi:10.48550/arxiv.1312.5602 · Cited by 9,471 · see citing works
Extracting and composing robust features with denoising autoencoders
Vincent, P., Larochelle, H., Bengio, Y., & Manzagol, P.-A. (2008) · Proceedings of the 25th international conference on Machine learning - ICML '08 · doi:10.1145/1390156.1390294 · Cited by 7,436 · see citing works
Asynchronous Methods for Deep Reinforcement Learning
Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lillicrap, T. P., Harley, T., Silver, D., & Kavukcuoglu, K. (2016) · arXiv · doi:10.48550/arxiv.1602.01783 · Cited by 7,132 · see citing works
Watkins, C. (1989) · OpenGrey (Institut de l'Information Scientifique et Technique) · doi:10.17863/cam.132164 · Cited by 5,406 · see citing works
MuJoCo: A physics engine for model-based control
Todorov, E., Erez, T., & Tassa, Y. (2012) · 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems · doi:10.1109/iros.2012.6386109 · Cited by 4,767 · see citing works
Deterministic Policy Gradient Algorithms
Silver, D., Lever, G., Heess, N., Degris, T., Wierstra, D., & Riedmiller, M. (2014) · HAL · Cited by 3,275 · see citing works
Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles
Noroozi, M., & Favaro, P. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46466-4_5 · Cited by 2,775 · see citing works
Unsupervised Visual Representation Learning by Context Prediction
Doersch, C., Gupta, A., & Efros, A. A. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.167 · Cited by 2,629 · see citing works
Unsupervised Representation Learning by Predicting Image Rotations
Gidaris, S., Singh, P., & Komodakis, N. (2018) · arXiv · doi:10.48550/arxiv.1803.07728 · Cited by 2,477 · see citing works
Bellman, R. (1957) · Indiana University Mathematics Journal · doi:10.1512/iumj.1957.6.56038 · Cited by 2,408 · see citing works
Soft Actor-Critic Algorithms and Applications
Haarnoja, T., Zhou, A., Hartikainen, K., Tucker, G., Ha, S., Tan, J., Kumar, V., Zhu, H., Gupta, A., Abbeel, P., et al. (2018) · arXiv · doi:10.48550/arxiv.1812.05905 · Cited by 1,994 · see citing works
Rainbow: Combining Improvements in Deep Reinforcement Learning
Hessel, M., Modayil, J., Van Hasselt, H., Schaul, T., Ostrovski, G., Dabney, W., Horgan, D., Piot, B., Azar, M., & Silver, D. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11796 · Cited by 1,846 · see citing works
Unsupervised Learning of Visual Representations Using Videos
Wang, X., & Gupta, A. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.320 · Cited by 1,005 · see citing works
Reinforcement Learning with Unsupervised Auxiliary TasksPreprint
Jaderberg, M., Mnih, V., Czarnecki, W. M., Schaul, T., Leibo, J. Z., Silver, D., & Kavukcuoglu, K. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.05397 · Cited by 826 · see citing works
Synthesis and stabilization of complex behaviors through online trajectory optimization
Tassa, Y., Erez, T., & Todorov, E. (2012) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2012.6386025 · Cited by 774 · see citing works
Split-Brain Autoencoders: Unsupervised Learning by Cross-Channel Prediction
Zhang, R., Isola, P., & Efros, A. A. (2016) · Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2017.76 · Cited by 724 · see citing works
Dream to Control: Learning Behaviors by Latent Imagination
Hafner, D., Lillicrap, T., Ba, J., & Norouzi, M. (2019) · arXiv · doi:10.48550/arxiv.1912.01603 · Cited by 558 · see citing works
CURL: Contrastive Unsupervised Representations for Reinforcement LearningPreprint
Srinivas, A., Laskin, M., & Abbeel, P. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.04136 · Cited by 521 · see citing works
Tassa, Y., Doron, Y., Muldal, A., Erez, T., Li, Y., Casas, D. D. L., Budden, D., Abdolmaleki, A., Merel, J., Lefrancq, A., et al. (2018) · arXiv · doi:10.48550/arxiv.1801.00690 · Cited by 497 · see citing works
Addressing Function Approximation Error in Actor-Critic Methods
Fujimoto, S., van Hoof, H., & Meger, D. (2018) · UvA-DARE (University of Amsterdam) · Cited by 385 · see citing works
Learning to Poke by Poking: Experiential Learning of Intuitive PhysicsPreprint
Agrawal, P., Nair, A., Abbeel, P., Malik, J., & Levine, S. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.07419 · Cited by 383 · see citing works
Safe and Efficient Off-Policy Reinforcement LearningPreprint
Munos, R., Stepleton, T., Harutyunyan, A., & Bellemare, M. G. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.02647 · Cited by 320 · see citing works
Reinforcement Learning with Augmented Data
Laskin, M., Lee, K., Stooke, A., Pinto, L., Abbeel, P., & Srinivas, A. (2020) · arXiv · doi:10.48550/arxiv.2004.14990 · Cited by 299 · see citing works
Self-citationIncremental multi-step Q-learning
Peng, J., & Williams, R. J. (1996) · Machine Learning · doi:10.1007/bf00114731 · Cited by 289 · see citing works
Mastering Atari with Discrete World Models
Hafner, D., Lillicrap, T., Norouzi, M., & Ba, J. (2020) · arXiv · doi:10.48550/arxiv.2010.02193 · Cited by 247 · see citing works
Revisiting Fundamentals of Experience ReplayPreprint
Fedus, W., Ramachandran, P., Agarwal, R., Bengio, Y., Larochelle, H., Rowland, M., & Dabney, W. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.06700 · Cited by 166 · see citing works
The Curious Robot: Learning Visual Representations via Physical Interactions
Pinto, L., Gandhi, D., Han, Y., Park, Y., & Gupta, A. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46475-6_1 · Cited by 160 · see citing works
Self-citationImage Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels
Yarats, D., Kostrikov, I., & Fergus, R. (2021) · International Conference on Learning Representations · Cited by 156 · see citing works
Self-citationDecoupling Representation Learning from Reinforcement Learning
Stooke, A., Lee, K., Abbeel, P., & Laskin, M. (2020) · arXiv · doi:10.48550/arxiv.2009.08319 · Cited by 123 · see citing works
Learning Predictive Representations for Deformable Objects Using Contrastive Estimation
Yan, W., Vangipuram, A., Abbeel, P., & Pinto, L. (2020) · arXiv · doi:10.48550/arxiv.2003.05436 · Cited by 122 · see citing works
Self-citationGeneralization in Reinforcement Learning by Soft Data Augmentation
Hansen, N., & Wang, X. (2021) · 2021 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra48506.2021.9561103 · Cited by 111 · see citing works
Improving Sample Efficiency in Model-Free Reinforcement Learning from Images
Yarats, D., Zhang, A., Kostrikov, I., Amos, B., Pineau, J., & Fergus, R. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i12.17276 · Cited by 108 · see citing works
Self-citationData-Efficient Reinforcement Learning with Self-Predictive RepresentationsPreprint
Schwarzer, M., Anand, A., Goel, R., Hjelm, R. D., Courville, A. C., & Bachman, P. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.05929 · Cited by 103 · see citing works
Automatic Data Augmentation for Generalization in Deep Reinforcement LearningPreprint
Răileanu, R., Goldstein, M., Yarats, D., Kostrikov, I., & Fergus, R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.12862 · Cited by 65 · see citing works
Self-citationReinforcement Learning with Prototypical RepresentationsPreprint
Yarats, D., Fergus, R., Lazaric, A., & Pinto, L. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.11271 · Cited by 64 · see citing works
Self-citationYoung, S., Gandhi, D., Tulsiani, S., Gupta, A., Abbeel, P., & Pinto, L. (2020) · arXiv · doi:10.48550/arxiv.2008.04899 · Cited by 42 · see citing works
Self-citationLearning Visual Feature Spaces for Robotic Manipulation with Deep Spatial Autoencoders.Preprint
Finn, C., Tan, X., Duan, Y., Darrell, T., Levine, S., & Abbeel, P. (2015) · arXiv (Cornell University) · Cited by 39 · see citing works
Learning Visual Robotic Control Efficiently with Contrastive Pre-training and Data Augmentation
Zhan, A., Zhao, R., Pinto, L., Abbeel, P., & Laskin, M. (2022) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros47612.2022.9981055 · Cited by 34 · see citing works
Self-citationOn the model-based stochastic value gradient for continuous reinforcement learningPreprint
Amos, B., Stanton, S., Yarats, D., & Wilson, A. G. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.12775 · Cited by 25 · see citing works
Self-citationChen, B., (Sasha), A. S., Lewis, G., Armeni, I., Savarese, S., Zamir, A., Malik, J., & Pinto, L. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.06698 · Cited by 23 · see citing works
Self-citation
Flagged first, then by citations
2 of the 43 references couldn’t be matched to a known paper.
Research works printed but not linked: 8
- Gabriel Barth-Maron, Matthew W. Hoffman, David Budden, Will Dabney, Dan Horgan, Dhruva TB, Alistair Muldal, Nicolas Heess, and Timothy Lillicrap. Distributional policy gradients. In International Conference on Learning Representations , 2018.
- Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta a nd Pieter Abbeel, and Sergey Levine. Soft actor-critic algorithms and applications. CoRR , 2018a.
- Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. arXiv preprint arXiv:1811.04551 , 2018. · arXiv
- A. X. Lee, A. Nagabandi, P. Abbeel, and S. Levine. Stochastic latent actor-critic: Deep reinforcement learning with a latent variable model. arXiv e-prints , 2019.
- Alex X Lee, Anusha Nagabandi, Pieter Abbeel, and Sergey Levine. Stochastic latent actor-critic: Deep reinforcement learning with a latent variable model. arXiv preprint arXiv:1907.00953 , 2019. · arXiv
- Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. CoRR , 2015a.
- Max Schwarzer, Ankesh Anand, Rishab Goel, R. Devon Hjelm, Aaron C. Courville, and Philip Bachman. Data-efficient reinforcement learning with momentum predictive representations. CoRR , 2020b.
- Albert Zhan, Philip Zhao, Lerrel Pinto, Pieter Abbeel, and Michael Laskin. A framework for efficient robotic manipulation. CoRR , 2020.
Cited by
Sorted by citations
A comprehensive survey on pretrained foundation models: a history from BERT to ChatGPT
Zhou, C., Li, Q., Li, C., Yu, J., Liu, Y., Wang, G., Zhang, K., Ji, C., Yan, Q., He, L., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.09419 · Cited by 574 · see citing works
Mastering Diverse Domains through World Models
Hafner, D., Pasukonis, J., Ba, J., & Lillicrap, T. (2023) · arXiv · doi:10.48550/arxiv.2301.04104 · Cited by 124 · see citing works
“Dreamer establishes a new state-of-the-art on this benchmark, outperforming DrQ-v2 and CURL 47 , which are specialized to visual environments and leverage data augmentation.”
BackgroundManipLLM: Embodied Multimodal Large Language Model for Object-Centric Robotic Manipulation
Li, X., Zhang, M., Geng, Y., Geng, H., Long, Y., Shen, Y. F., Zhang, R., Liu, J., & Dong, H. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01710 · Cited by 97 · see citing works
The Surprising Effectiveness of Representation Learning for Visual Imitation
Pari, J., Shafiullah, N. M. (., Arunachalam, S. P., & Pinto, L. (2022) · Robotics: Science and Systems XVIII · doi:10.15607/rss.2022.xviii.010 · Cited by 95 · see citing works
Self-citationWhat matters in language conditioned robotic imitation learning over unstructured data
Mees, O., Hermann, L., & Burgard, W. (2022) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2022.3196123 · Cited by 77 · see citing works
PIRLNav: Pretraining with Imitation and RL Finetuning for OBJECTNAV
Ramrakhya, R., Batra, D., Wijmans, E., & Das, A. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01716 · Cited by 71 · see citing works
Holo-Dex: Teaching Dexterity with Immersive Mixed Reality
Arunachalam, S. P., Guzey, I., Chintala, S., & Pinto, L. (2023) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra48891.2023.10160547 · Cited by 49 · see citing works
Self-citationDayDreamer: World Models for Physical Robot Learning
Wu, P., Escontrela, A., Hafner, D., Goldberg, K., & Abbeel, P. (2022) · arXiv · doi:10.48550/arxiv.2206.14176 · Cited by 76 · see citing works
“This result matches the simulated experiments of Yarats et al. (2021) that showed the two algorithms to perform similarly for continuous control tasks from images.”
BackgroundMethodResultInfluentialVisuotactile-RL: Learning Multimodal Manipulation Policies with Deep Reinforcement Learning
Hansen, J., Hogan, F. R., Rivkin, D., Meger, D., Jenkin, M. R. M., & Dudek, G. (2022) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra46639.2022.9812019 · Cited by 37 · see citing works
THE COLOSSEUM: A Benchmark for Evaluating Generalization for Robotic Manipulation
Pumacay, W., Singh, I., Duan, J., Krishna, R., Thomason, J., & Fox, D. (2024) · Robotics: Science and Systems XX · doi:10.15607/rss.2024.xx.133 · Cited by 29 · see citing works
Decomposing the Generalization Gap in Imitation Learning for Visual Robotic Manipulation
Xie, A., Lee, L., Xiao, T., & Finn, C. (2024) · 2024 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra57147.2024.10611331 · Cited by 28 · see citing works
DreamingV2: Reinforcement Learning with Discrete World Models without Reconstruction
Okada, M., & Taniguchi, T. (2022) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros47612.2022.9981405 · Cited by 26 · see citing works
Liu, S., Chen, Z., Liu, Y., Wang, Y., Yang, D., Zhao, Z., Zhou, Z., Yi, X., Li, W., Zhang, W., et al. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.02142 · Cited by 23 · see citing works
Learning Visual Quadrupedal Loco-Manipulation from Demonstrations
He, Z., Lei, K., Ze, Y., Sreenath, K., Li, Z., & Xu, H. (2024) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros58592.2024.10802742 · Cited by 19 · see citing works
Autonomous Landing on a Moving Platform Using Vision-Based Deep Reinforcement Learning
Ładosz, P., Mammadov, M., Shin, H., Shin, W., & Oh, H. (2024) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2024.3379837 · Cited by 19 · see citing works
RGBManip: Monocular Image-based Robotic Manipulation through Active Object Pose Estimation
An, B., Geng, Y., Chen, K., Li, X., Dou, Q., & Dong, H. (2024) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra57147.2024.10610690 · Cited by 17 · see citing works
CCLF: A Contrastive-Curiosity-Driven Learning Framework for Sample-Efficient Reinforcement Learning
Sun, C., Qian, H., & Miao, C. (2022) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2022/478 · Cited by 16 · see citing works
RLAD: Reinforcement Learning From Pixels for Autonomous Driving in Urban Environments
Coelho, D., de Oliveira, M. A. R., & Santos, V. M. F. (2023) · IEEE Transactions on Automation Science and Engineering · doi:10.1109/tase.2023.3342419 · Cited by 15 · see citing works
Off-policy RL algorithms can be sample-efficient for continuous control via sample multiple reuse
Lyu, J., Wan, L., Li, X., & Lu, Z. (2024) · Information Sciences · doi:10.1016/j.ins.2024.120371 · Cited by 15 · see citing works
Efficient Multitask Reinforcement Learning Without Performance Loss
Baek, J., Baek, S., & Han, S. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3281473 · Cited by 14 · see citing works
RLfOLD: Reinforcement Learning from Online Demonstrations in Urban Autonomous Driving
Coelho, D., Oliveira, M., & Santos, V. M. F. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i10.29049 · Cited by 13 · see citing works
He, Z., Chen, P., Shi, H., Li, J., & Hwang, K. (2024) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2024.3468453 · Cited by 13 · see citing works
Noh, S., Lee, W., & Myung, H. (2025) · Neural Networks · doi:10.1016/j.neunet.2025.107202 · Cited by 9 · see citing works
Learning Pessimism for Reinforcement Learning
Cetin, E., & Çeliktutan, O. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i6.25852 · Cited by 9 · see citing works
Albilani, M., & Bouzeghoub, A. (2022) · IEEE Intelligent Vehicles Symposium · doi:10.1109/iv51971.2022.9827194 · Cited by 9 · see citing works
TD-MPC2: Scalable, Robust World Models for Continuous Control
Hansen, N., Su, H., & Wang, X. (2023) · arXiv · doi:10.48550/arxiv.2310.16828 · Cited by 8 · see citing works
“…of rewards by performing discrete regression in a transformed space, and (3) model-free algorithms DrQ (Kostrikov et al., 2020) and DrQ-v2 (Yarats et al., 2021) improve training stability and exploration, respectively, through use of data augmentation and several other minor but important…”
BackgroundMethodInfluentialGAIT: Generating Aesthetic Indoor Tours with Deep Reinforcement Learning
Xie, D., Hu, P. F., Sun, X., Pirk, S., Zhang, J., Měch, R., & Kaufman, A. E. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00681 · Cited by 8 · see citing works
Ma, Z., Liu, X., & Huang, Y. (2024) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2024.3400224 · Cited by 8 · see citing works
Sim-to-real via latent prediction: Transferring visual non-prehensile manipulation policies
Rizzardo, C., Chen, F., & Caldwell, D. G. (2023) · Frontiers in Robotics and AI · doi:10.3389/frobt.2022.1067502 · Cited by 10 · see citing works
Towards Deeper Deep Reinforcement Learning with Spectral NormalizationPreprint
Bjorck, N., Gomes, C. P., & Weinberger, K. Q. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.01151 · Cited by 12 · see citing works
Point Cloud Models Improve Visual Robustness in Robotic Learners
Peri, S. V., Lee, I., Kim, C., Li, F., Hermans, T., & Lee, S. (2024) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra57147.2024.10610710 · Cited by 7 · see citing works
Zero-shot sim-to-real transfer using Siamese-Q-Based reinforcement learning
Zhang, Z., Xie, S., Zhang, H., Luo, X., & Yu, H. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102664 · Cited by 7 · see citing works
Is High Variance Unavoidable in RL? A Case Study in Continuous ControlPreprint
Bjorck, J., Gomes, C. P., & Weinberger, K. Q. (2021) · International Conference on Learning Representations · doi:10.48550/arxiv.2110.11222 · Cited by 46 · see citing works
RMBench: Benchmarking Deep Reinforcement Learning for Robotic Manipulator Control
Xiang, Y., Wang, X., Hu, S., Zhu, B., Huang, X., Wu, X., & Lyu, S. (2023) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros55552.2023.10342479 · Cited by 6 · see citing works
Masked and Inverse Dynamics Modeling for Data-Efficient Reinforcement Learning
Lee, Y. J., Kim, J., Park, Y. J., Kwak, M. G., & Kim, S. B. (2024) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2024.3439261 · Cited by 6 · see citing works
Counterfactual experience augmented off-policy reinforcement learning
Lee, S., Gong, Y., & Deng, C. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.130017 · Cited by 6 · see citing works
Reinforcement Learning and Multi-Access Edge Computing for 6G-Based Underwater Wireless Networks
Cepeda-Pacheco, J. C., & Domingo, M. C. (2025) · IEEE Access · doi:10.1109/access.2025.3557158 · Cited by 6 · see citing works
Altmann, P. J., Ritz, F., Feuchtinger, L., Nüßlein, J., Linnhoff–Popien, C., & Phan, T. (2023) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2023/380 · Cited by 6 · see citing works
Learning State Representations via Retracing in Reinforcement LearningPreprint
Yu, C., Li, D., Hao, J., Wang, Y.-X., & Burgess, N. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.12600 · Cited by 6 · see citing works
Rank2Reward: Learning Shaped Reward Functions from Passive Video
Yang, D. Z., Tjia, D., Berg, J., Damen, D., Agrawal, P., & Gupta, A. (2024) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra57147.2024.10610873 · Cited by 5 · see citing works
Liu, Q., Zhou, Q., Yang, R., & Wang, J. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i7.26063 · Cited by 5 · see citing works
Metrics for Assessing Generalization of Deep Reinforcement Learning in Parameterized Environments
Aleksandrowicz, M., & Jaworek-Korjakowska, J. (2023) · Journal of Artificial Intelligence and Soft Computing Research · doi:10.2478/jaiscr-2024-0003 · Cited by 5 · see citing works
Learning Pessimism for Robust and Efficient Off-Policy Reinforcement LearningPreprint
Cetin, E., & Çeliktutan, O. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.03375 · Cited by 5 · see citing works
Curiosity-driven learning in artificial intelligence and its applications
Sun, C. (2023) · doi:10.32657/10356/172831 · Cited by 5 · see citing works
Improving Sample Efficiency Through Stability Enhancement in Deep-Reinforcement Learning
Wang, Z., Jiang, W., Peng, R., Kou, Q., Wan, L., & Lan, X. (2025) · IEEE Transactions on Systems Man and Cybernetics Systems · doi:10.1109/tsmc.2025.3578050 · Cited by 4 · see citing works
Improving Generalization in Game Agents with Data Augmentation in Imitation Learning
Yadgaroff, D., Sestini, A., Tollmar, K., Özçelikkale, A., & Gisslén, L. (2024) · IEEE Congress on Evolutionary Computation (CEC) · doi:10.1109/cec60901.2024.10612178 · Cited by 4 · see citing works
Clustering-Driven State Embedding for Reinforcement Learning Under Visual Distractions
Wang, R., Cheng, Y., & Wang, X. (2024) · IEEE Transactions on Systems Man and Cybernetics Systems · doi:10.1109/tsmc.2024.3449294 · Cited by 4 · see citing works
AREPO: Uncertainty-Aware Robot Ensemble Learning Under Extreme Partial Observability
Du, Y., Hanut, L., Bruyninckx, H., & Detry, R. J. (2025) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2025.3554451 · Cited by 3 · see citing works
CTPR: Contrastive transition predictive representation for reinforcement learning
Sun, H., & Wang, C. (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2025.111124 · Cited by 4 · see citing works
Hippocampus supports multi-task reinforcement learning under partial observability
Pedamonti, D., Mohinta, S., Dimitrov, M. V., Malagon‐Vina, H., Ciocchi, S., & Costa, R. P. (2025) · Nature Communications · doi:10.1038/s41467-025-64591-9 · Cited by 3 · see citing works
Pore, A., Muradore, R., & Dall’Alba, D. (2024) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros58592.2024.10801730 · Cited by 3 · see citing works
Wang, C., Wang, L., Lu, Z., Wu, C., Shou, G., & Wen, X. (2025) · IEEE Internet of Things Journal · doi:10.1109/jiot.2025.3559407 · Cited by 3 · see citing works
Robust Scuba Diver Tracking and Recovery in Open Water Using YOLOv7, SORT, and Spiral Search
Lotfi, F., Virji, K., & Dudek, G. (2023) · Canadian Conference on Computer and Robot Vision · doi:10.1109/crv60082.2023.00037 · Cited by 3 · see citing works
Off-policy Imitation Learning from Visual Inputs
Cheng, Z., Shen, L., & Tao, D. (2023) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra48891.2023.10161566 · Cited by 3 · see citing works
Havens, A., Kevian, D., Seiler, P. J., Dullerud, G. E., & Hu, B. (2022) · IEEE Control Systems Letters · doi:10.1109/lcsys.2022.3188016 · Cited by 3 · see citing works
Dual Policy-Based TD-Learning for Model Predictive Control
Ji, C.-H., Choi, H.-B., Heo, J.-S., Kim, J.-B., Lim, H.-K., & Han, Y. (2023) · International Conference on Artificial Intelligence in Information and Communication (ICAIIC) · doi:10.1109/icaiic57133.2023.10067037 · Cited by 2 · see citing works
CURLing the Dream: Contrastive Representations for World Modeling in Reinforcement Learning
Kich, V. A., Bottega, J. A., Steinmetz, R., Grando, R. B., Yorozu, A., & Ohya, A. (2024) · International Conference on Control, Automation and Systems (ICCAS) · doi:10.23919/iccas63016.2024.10773008 · Cited by 2 · see citing works
State-novelty guided action persistence in deep reinforcement learning
Hu, J., Weng, P., & Ban, Y. (2025) · Machine Learning · doi:10.1007/s10994-024-06675-z · Cited by 2 · see citing works
Ryoo, S., & Han, S. (2023) · International Conference on Control, Automation and Systems (ICCAS) · doi:10.23919/iccas59377.2023.10317047 · Cited by 2 · see citing works
Effective Representation Learning is More Effective in Reinforcement Learning than You Think
Zheng, J., & Song, Y. (2024) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra57147.2024.10611330 · Cited by 2 · see citing works
Image Quality Assessment in Visual Reinforcement Learning for Fast-moving Targets
Ryoo, S., Jeong, J., & Han, S. (2024) · International Journal of Control Automation and Systems · doi:10.1007/s12555-024-0045-7 · Cited by 2 · see citing works
Dynamic bottleneck with a predictable prior for image-based deep reinforcement learning
You, B., Chen, B., Yao, L., Chen, Y., & Xie, J. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.126094 · Cited by 2 · see citing works
Predicting Long-Term Human Behaviors in Discrete Representations via Physics-Guided Diffusion
Zhang, Z., Li, A., Lim, A., & Chen, M. (2024) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros58592.2024.10802068 · Cited by 2 · see citing works
Balancing State Exploration and Skill Diversity in Unsupervised Skill Discovery
Liu, X., Chen, Y., Chen, G., Li, H., & Zhao, D. (2025) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2025.3548821 · Cited by 2 · see citing works
Learning generalizable agents via self-supervised exploration
Liang, B., Xu, L., & Deng, Z. (2025) · Neural Networks · doi:10.1016/j.neunet.2025.107787 · Cited by 2 · see citing works
Invariant Adversarial Imitation Learning From Visual Inputs
Zhang, H., Tian, Y., Yuan, L., & Lu, Y. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10096070 · Cited by 2 · see citing works
Maximum Entropy Model-based Reinforcement LearningPreprint
Svidchenko, O., & Shpilman, A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2112.01195 · Cited by 3 · see citing works
Tracking Control using standalone Reinforcement Learning for a Robot Manipulator
Siddique, T., Choutri, K., Fareh, R., Dylov, D. V., & Bettayeb, M. (2024) · Advances in Science and Engineering Technology International Conferences (ASET) · doi:10.1109/aset60340.2024.10708704 · Cited by 1 · see citing works
Sheng, L., Chen, H., & Chen, X. (2024) · Algorithms · doi:10.3390/a17120579 · Cited by 1 · see citing works
A Champion-Level Vision-Based Reinforcement Learning Agent for Competitive Racing in Gran Turismo 7
Lee, H., Seno, T., Tai, J. J., Subramanian, K., Kawamoto, K., Stone, P., & Wurman, P. R. (2025) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2025.3560873 · Cited by 1 · see citing works
Actively learning costly reward functions for reinforcement learning
Eberhard, A., Metni, H., Fahland, G., Stroh, A., & Friederich, P. (2024) · Machine Learning Science and Technology · doi:10.1088/2632-2153/ad33e0 · Cited by 1 · see citing works
Focus-Then-Decide: Segmentation-Assisted Reinforcement Learning
Chen, C., Xu, J., Liao, W., Ding, H., Zhang, Z., Yu, Y., & Zhao, R. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i10.29002 · Cited by 1 · see citing works
Differentially Encoded Observation Spaces for Perceptive Reinforcement Learning
Grossman, L., & Plancher, B. (2024) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra57147.2024.10611215 · Cited by 1 · see citing works
Wang, R., Cheng, Y., & Wang, X. (2025) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2025.3556838 · Cited by 1 · see citing works
Liu, X., Chen, Y., Li, H., & Zhao, D. (2025) · Science China Information Sciences · doi:10.1007/s11432-024-4380-4 · Cited by 1 · see citing works
Invariance Through Latent Alignment
Yoneda, T., Yang, G., Walter, M. R., & Stadie, B. C. (2022) · Robotics: Science and Systems · doi:10.15607/rss.2022.xviii.064 · Cited by 1 · see citing works
Beyond Simple Argumentation: Image Learnable Transformation For Efficient Reinforcement Learning
Dai, F., & Zhang, C. (2022) · International Conference on Signal Processing · doi:10.1109/icsp56322.2022.9965348
VMBRL3: A Simple Visual Model-Based Reinforcement Learning Framework for Continuous Control
Wang, J., Wang, H., & Wu, H. (2023) · Frontiers in artificial intelligence and applications · doi:10.3233/faia230550
Improving Visual Reinforcement Learning with Discrete Information Bottleneck Approach
Wang, H., & Wu, H. (2023) · Frontiers in artificial intelligence and applications · doi:10.3233/faia230548
Chebykin, A., Dushatskiy, A., Alderliesten, T., & Bosman, P. A. N. (2023) · Frontiers in artificial intelligence and applications · doi:10.3233/faia230294
An Empirical Investigation of Visual Reinforcement Learning for 3D Continuous Control
Noh, S., Kim, S., Jang, I., & Lee, D. (2023) · International Conference on Information and Communication Technology Convergence (ICTC) · doi:10.1109/ictc58733.2023.10393822
NISHINA, K., & Fujita, S. (2024) · Journal of Japan Society for Fuzzy Theory and Intelligent Informatics · doi:10.3156/jsoft.36.1_571
Learning to Grasp through World Models with Backward Prediction
Wang, S., Yuan, Y., Mei, Y., Sun, J. Z., & Wang, G. (2024) · Chinese Control Conference (CCC) · doi:10.23919/ccc63176.2024.10661609
Visual Reinforcement Learning Using Dynamic State Representation for Continuous Motion Control
Tang, Z.-Y., Liu, Y., & Qin, P. (2024) · Chinese Control Conference (CCC) · doi:10.23919/ccc63176.2024.10662669
Improving Self-Supervised Vision Transformers for Visual Control
Song, W., Sohn, K., & Min, D. (2024) · Proceedings - International Conference on Image Processing · doi:10.1109/icip51287.2024.10647522
Atanov, A., Fu, J., Singh, R., Yu, I., Spielberg, A., & Zamir, A. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72904-1_27 · Cited by 1 · see citing works
Zheng, J., Song, Y., Lin, G., Duan, J., Lin, H., & Li, S. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.129139
Exploring Visual Reinforcement Learning for Sample-Efficient Robotic Manipulation
Noh, S., Kim, S., & Jang, I. (2024) · International Conference on Information and Communication Technology Convergence (ICTC) · doi:10.1109/ictc62082.2024.10827272
A Feedback-based Decision-Making Mechanism for Actor-Critic Deep Reinforcement Learning
Yang, G., Geng, Z., Li, J., Zhao, Y., Abdelwahed, S., & Luo, C. (2024) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata62323.2024.10826018
Introducing an Adapter into the Pre-trained Encoder for Visual Reinforcement Learning
Yang, S., Wang, S., Wang, J., Han, S., Zou, J., Lin, Y., & Lv, K. (2024) · IEEE International Conference on Unmanned Systems (ICUS) · doi:10.1109/icus61736.2024.10839600
Improving Visual Reinforcement Learning via Spatial-Temporal Data Augmentation
Li, J., Hou, Q., Zhang, J., & Chen, X. (2025) · International Conference on Consumer Electronics and Computer Engineering (ICCECE) · doi:10.1109/iccece65250.2025.10984654
Regularizing Model Predictive Control for pixel-based long-horizon tasks
Li, Y.-H., Zhang, F., Hua, Q., & Dong, C.-R. (2025) · Applied Soft Computing · doi:10.1016/j.asoc.2025.113377
Analyzing Latent Entropy in Deep Q-Learning
Kooi, J. E., Hoogendoorn, M., & François-Lavet, V. (2025) · doi:10.1109/cism64958.2025.11060867
- Authors
- Yarats, D., Fergus, R., Lazaric, A., & Pinto, L.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2021
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2107.09645
- OpenAlex
- W3186672455
- Cited by
- 89
- References
- 43
- Citations by year
- 2025: 17 · 2024: 35 · 2023: 22 · 2022: 10 · 2021: 5
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Computational Theory and Mathematics › Adaptive Dynamic Programming Control
- Keywords
- data augmentation, model-free reinforcement learning, off-policy actor-critic, humanoid locomotion, DeepMind Control Suite, computational efficiency
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
