Robust Domain Randomised Reinforcement Learning through Peer-to-Peer Distillation
Open accessAbstract
In reinforcement learning, domain randomisation is an increasingly popular technique for learning more general policies that are robust to domain-shifts at deployment. However, naively aggregating information from randomised domains may lead to high variance in gradient estimation and unstable learning process. To address this issue, we present a peer-to-peer online distillation strategy for RL termed P2PDRL, where multiple workers are each assigned to a different environment, and exchange knowledge through mutual regularisation based on Kullback-Leibler divergence. Our experiments on continuous control tasks show that P2PDRL enables robust learning across a wider randomisation distribution than baselines, and more robust generalisation to new environments at testing.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Some methods for sim-to-real transfer rely on data from the testing environment (reality).
“some methods for sim-to-real transfer rely on data from the testing environment (reality), which we do not assume here.”
From A Survey of Zero-shot Generalisation in Deep Reinforcement Learning (2023) · cites this paper as Zhao et al. (2020)
In sim-to-real transfer, there is always dynamics variation between the simulator and reality.
“In sim-to-real transfer (Zhao et al., 2020), there is always dynamics variation between the simulator and reality”
From A Survey of Zero-shot Generalisation in Deep Reinforcement Learning (2023) · cites this paper as Zhao et al., 2020
Cites
Human-level control through deep reinforcement learning
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015) · Nature · doi:10.1038/nature14236 · Cited by 30,357 · see citing works
Mastering the game of Go with deep neural networks and tree search
Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., van den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al. (2016) · Nature · doi:10.1038/nature16961 · Cited by 15,612 · see citing works
Distilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204 · see citing works
Proximal Policy Optimization Algorithms
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017) · arXiv · doi:10.48550/arxiv.1707.06347 · Cited by 10,625 · see citing works
Domain randomization for transferring deep neural networks from simulation to the real world
Tobin, J., Fong, R., Ray, A., Schneider, J., Zaremba, W., & Abbeel, P. (2017) · 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2017.8202133 · Cited by 3,124 · cited 4× · see citing works
“With domain randomisation, we first generate a random set of, or distribution over, domains with different properties such as observation function [25], dynamics [16], or both [1].”
InfluentialZhang, Y., Xiang, T., Hospedales, T. M., & Lu, H. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00454 · Cited by 1,933 · see citing works
Self-citationLearning dexterous in-hand manipulation
Andrychowicz, O. M., Baker, B., Chociej, M., Józefowicz, R., McGrew, B., Pachocki, J., Petron, A., Plappert, M., Powell, G., Ray, A., et al. (2020) · The International Journal of Robotics Research · doi:10.1177/0278364919887447 · Cited by 1,643 · see citing works
Sim-to-Real Transfer of Robotic Control with Dynamics Randomization
Peng, X. B., Andrychowicz, M., Zaremba, W., & Abbeel, P. (2018) · 2018 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2018.8460528 · Cited by 1,309 · see citing works
Sim-to-Real: Learning Agile Locomotion For Quadruped Robots
Tan, J., Zhang, T., Coumans, E., Iscen, A., Bai, Y., Hafner, D., Bohez, S., & Vanhoucke, V. (2018) · Robotics: Science and Systems XIV · doi:10.15607/rss.2018.xiv.010 · Cited by 735 · see citing works
Emergence of Locomotion Behaviours in Rich Environments
Heess, N., TB, D., Sriram, S., Lemmon, J., Merel, J., Wayne, G., Tassa, Y., Erez, T., Wang, Z., Eslami, S. M. A., et al. (2017) · arXiv · doi:10.48550/arxiv.1707.02286 · Cited by 657 · see citing works
CAD2RL: Real Single-Image Flight Without a Single Real Image
Sadeghi, F., & Levine, S. (2017) · Robotics: Science and Systems XIII · doi:10.15607/rss.2017.xiii.034 · Cited by 618 · see citing works
Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning
Yu, T., Quillen, D., He, Z., Julian, R., Narayan, A., Shively, H., Bellathur, A., Hausman, K., Finn, C., & Levine, S. (2019) · arXiv · doi:10.48550/arxiv.1910.10897 · Cited by 450 · see citing works
Sim-to-Real Robot Learning from Pixels with Progressive Nets
Rusu, A. A., Vecerik, M., Rothörl, T., Heess, N., Pascanu, R., & Hadsell, R. (2016) · arXiv · doi:10.48550/arxiv.1610.04286 · Cited by 385 · see citing works
Learning to Adapt in Dynamic, Real-World Environments through Meta-Reinforcement Learning
Nagabandi, A., Clavera, I., Liu, S., Fearing, R. S., Abbeel, P., Levine, S., & Finn, C. (2018) · arXiv · doi:10.48550/arxiv.1803.11347 · Cited by 370 · see citing works
Large scale distributed neural network training through online distillationPreprint
Anil, R., Pereyra, G., Passos, A., Ormándi, R., Dahl, G. E., & Hinton, G. E. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1804.03235 · Cited by 315 · see citing works
The Transferability Approach: Crossing the Reality Gap in Evolutionary Robotics
Koos, S., Mouret, J.-B., & Doncieux, S. (2012) · IEEE Transactions on Evolutionary Computation · doi:10.1109/tevc.2012.2185849 · Cited by 270 · see citing works
Rusu, A. A., Colmenarejo, S. G., Gulcehre, C., Desjardins, G., Kirkpatrick, J., Pascanu, R., Mnih, V., Kavukcuoglu, K., & Hadsell, R. (2015) · arXiv · doi:10.48550/arxiv.1511.06295 · Cited by 270 · see citing works
Learning Invariant Feature Spaces to Transfer Skills with Reinforcement LearningPreprint
Gupta, A., Devin, C., Liu, Y., Abbeel, P., & Levine, S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.02949 · Cited by 209 · see citing works
Distral: Robust Multitask Reinforcement Learning
Teh, Y. W. (2025) · TIB Data Manager · doi:10.57702/nufubi9l · Cited by 126 · see citing works
Assessing Generalization in Deep Reinforcement Learning
Packer, C., Gao, K., Kos, J., Krähenbühl, P., Koltun, V., & Song, D. (2018) · arXiv · doi:10.48550/arxiv.1810.12282 · Cited by 114 · see citing works
A Dissection of Overfitting and Generalization in Continuous Reinforcement Learning
Zhang, A., Ballas, N., & Pineau, J. (2018) · arXiv · doi:10.48550/arxiv.1806.07937 · Cited by 88 · see citing works
Distilling Policy DistillationPreprint
Czarnecki, W. M., Pascanu, R., Osindero, S., Jayakumar, S. M., Świrszcz, G., & Jaderberg, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.02186 · Cited by 72 · see citing works
Been There, Done That: Meta-Learning with Episodic RecallPreprint
Ritter, S., Wang, J. X., Kurth-Nelson, Z. L., Jayakumar, S. M., Blundell, C., Pascanu, R., & Botvinick, M. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.09692 · Cited by 60 · see citing works
Investigating Generalisation in Continuous Deep Reinforcement LearningPreprint
Zhao, C., Sigaud, O., Stulp, F., & Hospedales, T. M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.07015 · Cited by 32 · see citing works
Self-citation
Flagged first, then by citations
3 of the 27 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Knowledge Distillation: A Survey
Gou, J., Yu, B., Maybank, S. J., & Tao, D. (2021) · International Journal of Computer Vision · doi:10.1007/s11263-021-01453-z · Cited by 3,931 · see citing works
A Survey of Zero-shot Generalisation in Deep Reinforcement Learning
Kirk, R., Zhang, A., Grefenstette, E., & Rocktäschel, T. (2023) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.14174 · Cited by 193 · see citing works
A Survey of Generalisation in Deep Reinforcement LearningPreprint
Kirk, R., Zhang, A., Grefenstette, E., & Rocktäschel, T. (2021) · arXiv (Cornell University) · Cited by 63 · see citing works
Semantic-Aware Collaborative Deep Reinforcement Learning Over Wireless Cellular Networks
Lotfi, F., Semiari, O., & Saad, W. (2022) · International Conference on Communications · doi:10.1109/icc45855.2022.9839122 · Cited by 36 · see citing works
Electric Load Forecasting for Individual Households via Spatial-Temporal Knowledge Distillation
Lin, W., Wu, D., & Jenkin, M. R. M. (2024) · IEEE Transactions on Power Systems · doi:10.1109/tpwrs.2024.3393926 · Cited by 26 · see citing works
Yang, J., & Zhang, J. (2024) · IEEE Transactions on Vehicular Technology · doi:10.1109/tvt.2024.3379972 · Cited by 6 · see citing works
LHADRO: A Robust Control Framework for Autonomous Vehicles Under Cyber-Physical Attacks
Yang, J., & Zhang, J. (2024) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2024.3497808 · Cited by 4 · see citing works
RAP Vol: Robust Adversary Populations With Volume Diversity Measure
Yang, J., & Zhang, J. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3317145 · Cited by 3 · see citing works
DD-NeuS: Dense depth self-supervised neural surface reconstruction from few images
Qiu, Y., Wu, G., Wei, Z., Li, X., Li, Y., & Sun, Y. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.114022 · Cited by 2 · see citing works
Domain Randomization with Adaptive Weight Distillation
Wang, S., Wu, F., Lin, Y., & Zhao, J. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-23609-9_13 · Cited by 1 · see citing works
Kerbel, L., Ayalew, B., & Ivančo, A. (2025) · Applied Energy · doi:10.1016/j.apenergy.2025.126495
- Authors
- Zhao, C., & Hospedales, T.
- Affiliations
- University of Edinburgh (GB)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2020
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2012.04839
- OpenAlex
- W3111065475
- Cited by
- 11
- References
- 27
- Citations by year
- 2025: 2 · 2024: 3 · 2023: 2 · 2022: 2 · 2021: 2
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- Edinburgh Research Explorer (University of Edinburgh) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
- Keywords
- domain randomization, knowledge distillation, domain shift, Kullback-Leibler divergence, continuous control, online distillation, gradient variance
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
