CoChat
Sign Up
PreprintCited by 2Reinforcement Learning in RoboticsDomain Adaptation and Few-Shot LearningMachine Learning and Algorithms

The DOI matches its DataCite record; the references it cites are not checked yet.

PLASTIC: Improving Input and Label Plasticity for Sample Efficient Reinforcement Learning

Open access
Lee, Hojoon, Cho, Hanseul, Kim, Hyunseung, Gwak, Daehoon, Kim, Joonkee, Choo, Jaegul, Yun, Se-Young, Yun, Chulhee·2023·arXiv
DOI 10.48550/arxiv.2306.10711arXiv 2306.10711
Open this paper in CoChat — free accountOpen PDF

Abstract

In Reinforcement Learning (RL), enhancing sample efficiency is crucial, particularly in scenarios when data acquisition is costly and risky. In principle, off-policy RL algorithms can improve sample efficiency by allowing multiple updates per environment interaction. However, these multiple updates often lead the model to overfit to earlier interactions, which is referred to as the loss of plasticity. Our study investigates the underlying causes of this phenomenon by dividing plasticity into two aspects. Input plasticity, which denotes the model's adaptability to changing input data, and label plasticity, which denotes the model's adaptability to evolving input-output relationships. Synthetic experiments on the CIFAR-10 dataset reveal that finding smoother minima of loss landscape enhances input plasticity, whereas refined gradient propagation improves label plasticity. Leveraging these findings, we introduce the PLASTIC algorithm, which harmoniously combines techniques to address both concerns. With minimal architectural modifications, PLASTIC achieves competitive performance on benchmarks including Atari-100k and Deepmind Control Suite. This result emphasizes the importance of preserving the model's plasticity to elevate the sample efficiency in RL. The code is available at https://github.com/dojeon-ai/plastic.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2306.10711

Cites

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

The citation lists haven’t been fetched yet

Cited by 2

Research works printed but not linked: 77

  1. Momin Abbas, Quan Xiao, Lisha Chen, Pin-Yu Chen, and Tianyi Chen. Sharp-maml: Sharpness- aware model-agnostic meta learning. In Proc. the International Conference on Machine Learning (ICML), 2022. 19
  2. Zaheer Abbas, Rosie Zhao, Joseph Modayil, Adam White, and Marlos C Machado. Loss of plasticity in continual deep reinforcement learning. arXiv preprint arXiv:2303.07507, 2023. 2, 3, 6 · arXiv
  3. Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Marc Belle- mare. Deep reinforcement learning at the edge of the statistical precipice. Proc. the Advances in Neural Information Processing Systems (NeurIPS), 34:29304-29320, 2021. 6
  4. Jordan Ash and Ryan P Adams. On warm-starting neural network training. Proc. the Advances in Neural Information Processing Systems (NeurIPS), 33:3884-3894, 2020. 7
  5. Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016. 2 · arXiv
  6. Dara Bahri, Hossein Mobahi, and Yi Tay. Sharpness-aware minimization improves language model generalization. In Proc. the Annual Meeting of the Association for Computational Linguistics (ACL), pages 7360-7371, 2022. 19
  7. Bowen Baker, Ilge Akkaya, Peter Zhokhov, Joost Huizinga, Jie Tang, Adrien Ecoffet, Brandon Houghton, Raul Sampedro, and Jeff Clune. Video pretraining (vpt): Learning to act by watching unlabeled online videos. Proc. the Advances in Neural Information Processing Systems (NeurIPS), 2022. 8
  8. Marc G Bellemare, Will Dabney, and Rémi Munos. A distributional perspective on reinforce- ment learning. In Proc. the International Conference on Machine Learning (ICML), pages 449-458. PMLR, 2017. 3
  9. Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-279, 2013. 5
  10. David Bertoin and Emmanuel Rachelson. Local feature swapping for generalization in rein- forcement learning. Proc. the International Conference on Learning Representations (ICLR), 2022. 3
  11. Nils Bjorck, Carla P Gomes, and Kilian Q Weinberger. Towards deeper deep reinforcement learning with spectral normalization. Proc. the Advances in Neural Information Processing Systems (NeurIPS), 34:8242-8255, 2022. 3
  12. Edoardo Cetin, Philip J Ball, Steve Roberts, and Oya Celiktutan. Stabilizing off-policy deep reinforcement learning from pixels. arXiv preprint arXiv:2207.00986, 2022. 3 · arXiv
  13. Xiangning Chen, Cho-Jui Hsieh, and Boqing Gong. When vision transformers outperform resnets without pre-training or strong data augmentations. Proc. the International Conference on Learning Representations (ICLR), 2022. 3
  14. Xiangning Chen, Cho-Jui Hsieh, and Boqing Gong. When vision transformers outperform resnets without pre-training or strong data augmentations. In International Conference on Learning Representations, 2022. 19
  15. Karl Cobbe, Chris Hesse, Jacob Hilton, and John Schulman. Leveraging procedural generation to benchmark reinforcement learning. In Proc. the International Conference on Machine Learning (ICML), pages 2048-2056. PMLR, 2020. 9
  16. Terrance DeVries and Graham W Taylor. Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv:1708.04552, 2017. 3 · arXiv
  17. Shibhansh Dohare, Richard S Sutton, and A Rupam Mahmood. Continual backprop: Stochastic gradient descent with persistent randomness. arXiv preprint arXiv:2108.06325, 2021. 2, 3, 6 · arXiv
  18. Pierluca D'Oro, Max Schwarzer, Evgenii Nikishin, Pierre-Luc Bacon, Marc G Bellemare, and Aaron Courville. Sample-efficient reinforcement learning by breaking the replay ratio barrier. In Proc. the International Conference on Learning Representations (ICLR), 2023. 1, 3, 7, 8, 16
  19. Jiameng Fan and Wenchao Li. Dribo: Robust deep reinforcement learning via multi-view information bottleneck. In Proc. the International Conference on Machine Learning (ICML), pages 6074-6102. PMLR, 2022. 3
  20. Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware min- imization for efficiently improving generalization. In Proc. the International Conference on Learning Representations (ICLR), 2021. 2, 3, 6, 19
  21. Meire Fortunato, Mohammad Gheshlaghi Azar, Bilal Piot, Jacob Menick, Ian Osband, Alex Graves, Vlad Mnih, Remi Munos, Demis Hassabis, Olivier Pietquin, et al. Noisy networks for exploration. arXiv preprint arXiv:1706.10295, 2017. 3, 4 · arXiv
  22. Florin Gogianu, Tudor Berariu, Mihaela C Rosca, Claudia Clopath, Lucian Busoniu, and Razvan Pascanu. Spectral normalisation for deep reinforcement learning: an optimisation perspective. In International Conference on Machine Learning, pages 3734-3744. PMLR, 2021. 3
  23. Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Proc. the International Conference on Machine Learning (ICML), pages 1861-1870. PMLR, 2018. 1, 3, 4, 6
  24. Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models. arXiv preprint arXiv:2301.04104, 2023. 8 · arXiv
  25. Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improve- ments in deep reinforcement learning. In Proc. the AAAI Conference on Artificial Intelligence (AAAI), 2018. 1, 3, 4, 6, 16
  26. Takuya Hiraoka, Takahisa Imagawa, Taisei Hashimoto, Takashi Onishi, and Yoshimasa Tsu- ruoka. Dropout q-functions for doubly efficient reinforcement learning. Proc. the International Conference on Learning Representations (ICLR), 2022. 3
  27. Minyoung Huh, Hossein Mobahi, Richard Zhang, Brian Cheung, Pulkit Agrawal, and Phillip Isola. The low-rank simplicity bias in deep networks. Transactions on Machine Learning Research (TMLR), 2023. 3
  28. Maximilian Igl, Gregory Farquhar, Jelena Luketina, Wendelin Boehmer, and Shimon White- son. Transient non-stationarity and generalisation in deep reinforcement learning. Proc. the International Conference on Learning Representations (ICLR), 2021. 3
  29. Pavel Izmailov, Dmitrii Podoprikhin, T. Garipov, Dmitry P. Vetrov, and Andrew Gordon Wilson. Averaging weights leads to wider optima and better generalization. In Proc. the Conference on Uncertainty in Artificial Intelligence (UAI), 2018. 3, 19
  30. Jean Kaddour, Linqing Liu, Ricardo Silva, and Matt J Kusner. When do flat minima optimizers work? Proc. the Advances in Neural Information Processing Systems (NeurIPS), 35:16577- 16595, 2022. 19
  31. Łukasz Kaiser, Mohammad Babaeizadeh, Piotr Miłos, Błażej Osiński, Roy H Campbell, Konrad Czechowski, Dumitru Erhan, Chelsea Finn, Piotr Kozakowski, Sergey Levine, et al. Model based reinforcement learning for atari. In Proc. the International Conference on Learning Representations (ICLR), 2019. 5
  32. Simran Kaur, Jeremy Cohen, and Zachary Chase Lipton. On the maximum hessian eigenvalue and generalization. In I Can't Believe It's Not Better Workshop: Understanding Deep Learning Through Empirical Falsification, 2023. 6
  33. Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. In Proc. the International Conference on Learning Representations (ICLR), 2017. 3, 19
  34. Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. Proc. the International Conference on Learning Representations (ICLR), 2015. 19
  35. Jens Kober, J Andrew Bagnell, and Jan Peters. Reinforcement learning in robotics: A survey. The International Journal of Robotics Research, 2013. 1, 3
  36. Michael Laskin, Aravind Srinivas, and Pieter Abbeel. Curl: Contrastive unsupervised represen- tations for reinforcement learning. In Proc. the International Conference on Machine Learning (ICML), 2020. 3
  37. Misha Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, and Aravind Srinivas. Reinforcement learning with augmented data. Proc. the Advances in Neural Information Processing Systems (NeurIPS), 2020. 3
  38. Hojoon Lee, Koanho Lee, Dongyoon Hwang, Hyunho Lee, Byungkun Lee, and Jaegul Choo. On the importance of feature decorrelation for unsupervised representation learning in reinforcement learning. arXiv preprint arXiv:2306.05637, 2023. 3, 8 · arXiv
  39. Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643, 2020. 1, 3 · arXiv
  40. Qiyang Li, Aviral Kumar, Ilya Kostrikov, and Sergey Levine. Efficient deep reinforcement learning requires regulating overfitting. arXiv preprint arXiv:2304.10466, 2023. 1, 3 · arXiv
  41. Yong Liu, Siqi Mai, Xiangning Chen, Cho-Jui Hsieh, and Yang You. Towards efficient and scalable sharpness-aware minimization. In Proc. of the IEEE conference on computer vision and pattern recognition (CVPR), pages 12360-12370, 2022. 3
  42. Zhuang Liu, Xuanlin Li, Bingyi Kang, and Trevor Darrell. Regularization matters in policy optimization-an empirical study on continuous control. Proc. the International Conference on Learning Representations (ICLR), 2021. 3
  43. Clare Lyle, Mark Rowland, and Will Dabney. Understanding and preventing capacity loss in reinforcement learning. Proc. the International Conference on Learning Representations (ICLR), 2022. 1
  44. Clare Lyle, Zeyu Zheng, Evgenii Nikishin, Bernardo Avila Pires, Razvan Pascanu, and Will Dabney. Understanding plasticity in neural networks. Proc. the International Conference on Machine Learning (ICML), 2023. 1, 3, 6
  45. Bogdan Mazoure, Remi Tachet des Combes, Thang Long Doan, Philip Bachman, and R Devon Hjelm. Deep reinforcement and infomax learning. Advances in Neural Information Processing Systems, 33:3686-3698, 2020. 3
  46. Vincent Micheli, Eloi Alonso, and François Fleuret. Transformers are sample efficient world models. arXiv preprint arXiv:2209.00588, 2022. 8 · arXiv
  47. Clara Na, Sanket Vaibhav Mehta, and Emma Strubell. Train flat, then compress: Sharpness- aware minimization learns more compressible models. In Proc. of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4909-4936, 2022. 19
  48. Evgenii Nikishin, Max Schwarzer, Pierluca D'Oro, Pierre-Luc Bacon, and Aaron Courville. The primacy bias in deep reinforcement learning. In Proc. the International Conference on Machine Learning (ICML), pages 16828-16847. PMLR, 2022. 1, 2, 3, 6, 16, 17, 18
  49. Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748, 2018. 3 · arXiv
  50. Boris T Polyak. Some methods of speeding up the convergence of iteration methods. Ussr computational mathematics and mathematical physics, 4(5):1-17, 1964. 4, 19
  51. Roberta Raileanu, Max Goldstein, Denis Yarats, Ilya Kostrikov, and Rob Fergus. Auto- matic data augmentation for generalization in deep reinforcement learning. arXiv preprint arXiv:2006.12862, 2020. 3 · arXiv
  52. Herbert Robbins and Sutton Monro. A Stochastic Approximation Method. The Annals of Mathematical Statistics, 22(3):400 -407, 1951. 4, 19
  53. Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay. Proc. the International Conference on Learning Representations (ICLR), 2016. 3
  54. Max Schwarzer, Ankesh Anand, Rishab Goel, R Devon Hjelm, Aaron Courville, and Philip Bachman. Data-efficient reinforcement learning with self-predictive representations. In Proc. the International Conference on Learning Representations (ICLR), 2020. 3, 5
  55. Max Schwarzer, Nitarshan Rajkumar, Michael Noukhovitch, Ankesh Anand, Laurent Charlin, R Devon Hjelm, Philip Bachman, and Aaron C Courville. Pretraining representations for data-efficient reinforcement learning. Proc. the Advances in Neural Information Processing Systems (NeurIPS), 2021. 8
  56. Younggyo Seo, Kimin Lee, Stephen L James, and Pieter Abbeel. Reinforcement learning with action-free pre-training from videos. In Proc. the International Conference on Machine Learning (ICML), pages 19561-19579. PMLR, 2022. 8
  57. Wenling Shang, Kihyuk Sohn, Diogo Almeida, and Honglak Lee. Understanding and improving convolutional neural networks via concatenated rectified linear units. In Proc. the International Conference on Machine Learning (ICML), pages 2217-2225. PMLR, 2016. 2, 3, 6
  58. Dongkuk Si and Chulhee Yun. Practical sharpness-aware minimization cannot converge all the way to optima. Proc. the Advances in Neural Information Processing Systems (NeurIPS), 2023. 19
  59. Ghada Sokar, Rishabh Agarwal, Pablo Samuel Castro, and Utku Evci. The dormant neuron phenomenon in deep reinforcement learning. Proc. the International Conference on Machine Learning (ICML), 2023. 3
  60. Adam Stooke, Kimin Lee, Pieter Abbeel, and Michael Laskin. Decoupling representation learning from reinforcement learning. In Proc. the International Conference on Machine Learning (ICML), 2021. 3
  61. Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv:1801.00690, 2018. 5 · arXiv
  62. Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q-learning. In Proc. the AAAI Conference on Artificial Intelligence (AAAI), volume 30, 2016. 3
  63. Hado P Van Hasselt, Matteo Hessel, and John Aslanides. When to use parametric models in reinforcement learning? Proc. the Advances in Neural Information Processing Systems (NeurIPS), 32, 2019. 16
  64. Kaixin Wang, Bingyi Kang, Jie Shao, and Jiashi Feng. Improving generalization in reinforce- ment learning with mixture regularization. Proc. the Advances in Neural Information Processing Systems (NeurIPS), 33:7968-7978, 2020. 3
  65. Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Hasselt, Marc Lanctot, and Nando Freitas. Dueling network architectures for deep reinforcement learning. In Proc. the International Conference on Machine Learning (ICML), pages 1995-2003. PMLR, 2016. 3
  66. Kaiyue Wen, Tengyu Ma, and Zhiyuan Li. How sharpness-aware minimization minimizes sharpness? In Proc. the International Conference on Learning Representations (ICLR), 2023. 19
  67. Dongxian Wu, Shu-Tao Xia, and Yisen Wang. Adversarial weight perturbation helps robust generalization. Advances in Neural Information Processing Systems, 33:2958-2969, 2020. 3, 19
  68. Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Mastering visual continuous control: Improved data-augmented reinforcement learning. arXiv preprint arXiv:2107.09645, 2021. 3, 22 · arXiv
  69. Denis Yarats, Ilya Kostrikov, and Rob Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. In Proc. the International Conference on Machine Learning (ICML), 2020. 3, 4, 5, 6, 16, 22
  70. Denis Yarats, Amy Zhang, Ilya Kostrikov, Brandon Amos, Joelle Pineau, and Rob Fergus. Improving sample efficiency in model-free reinforcement learning from images. In Proc. the AAAI Conference on Artificial Intelligence (AAAI), volume 35, pages 10674-10681, 2021. 3
  71. Weirui Ye, Shaohuai Liu, Thanard Kurutach, Pieter Abbeel, and Yang Gao. Mastering atari games with limited data. Proc. the Advances in Neural Information Processing Systems (NeurIPS), 34:25476-25488, 2021. 8
  72. Tao Yu, Cuiling Lan, Wenjun Zeng, Mingxiao Feng, Zhizheng Zhang, and Zhibo Chen. Playvir- tual: Augmenting cycle-consistent virtual trajectories for reinforcement learning. Proc. the Advances in Neural Information Processing Systems (NeurIPS), 34:5276-5289, 2021. 7, 8
  73. Tao Yu, Zhizheng Zhang, Cuiling Lan, Yan Lu, and Zhibo Chen. Mask-based latent reconstruc- tion for reinforcement learning. Proc. the Advances in Neural Information Processing Systems (NeurIPS), 35:25117-25131, 2022. 3
  74. Tao Yu, Zhizheng Zhang, Cuiling Lan, Yan Lu, and Zhibo Chen. Mask-based latent reconstruc- tion for reinforcement learning. Proc. the Advances in Neural Information Processing Systems (NeurIPS), 35:25117-25131, 2022. 7, 8
  75. Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on robot learning, pages 1094-1100. PMLR, 2020. 9
  76. Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.09412, 2017. 3 · arXiv
  77. Hattie Zhou, Ankit Vani, Hugo Larochelle, and Aaron Courville. Fortuitous forgetting in connectionist networks. Proc. the International Conference on Learning Representations (ICLR), 2022. 2

Metadata

Authors
Lee, H., Cho, H., Kim, H., Gwak, D., Kim, J., Choo, J., Yun, S.-Y., & Yun, C.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2023
Pages
62270-62295
ISSN
1049-5258 (ISSN-L)
DOI
10.48550/arxiv.2306.10711
arXiv
2306.10711
OpenAlex
W7133193816
Cited by
2
References
0
Citations by year
2025: 1 · 2024: 1
FWCI
0.36
Type
conference paper
Language
en
Open access
Licence
Locations
  • neural information processing systems (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Funders
  • National Research Foundation
  • National Research Foundation of Korea (2019-0-00075)
  • Korea Advanced Institute of Science and Technology (2019-0-00075)
  • Ministry of Science and ICT, South Korea (2019-0-00075, No.2019-0-00075, 2022-0-00184)
  • Institute for Information and Communications Technology Promotion (2022-0-00184, 2022-, 2019-0-00075)
  • National Supercomputing Center, Korea Institute of Science and Technology Information
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
Keywords
sample efficiency, reinforcement learning, neuroplasticity

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: PLASTIC: Improving Input and Label Plasticity for Sample Efficient Reinforcement Learning. https://app.cochat.ai/papers/10.48550/arxiv.2306.10711
The paper
Lee, H., Cho, H., Kim, H., Gwak, D., Kim, J., Choo, J., Yun, S.-Y., & Yun, C. (2023). PLASTIC: Improving Input and Label Plasticity for Sample Efficient Reinforcement Learning. arXiv. https://doi.org/10.48550/arxiv.2306.10711