CoChat
Sign Up
PreprintCited by 9Reinforcement Learning in RoboticsArtificial Intelligence in GamesDomain Adaptation and Few-Shot Learning

The DOI matches its DataCite record; the references it cites are not checked yet.

Bigger, Better, Faster: Human-level Atari with human-level efficiency

Open access
Schwarzer, Max, Obando-Ceron, Johan, Courville, Aaron, Bellemare, Marc, Agarwal, Rishabh, Castro, Pablo Samuel·2023·arXiv
DOI 10.48550/arxiv.2305.19452arXiv 2305.19452
Open this paper in CoChat — free accountOpen PDF

Abstract

We introduce a value-based RL agent, which we call BBF, that achieves super-human performance in the Atari 100K benchmark. BBF relies on scaling the neural networks used for value estimation, as well as a number of other design choices that enable this scaling in a sample-efficient manner. We conduct extensive analyses of these design choices and provide insights for future work. We end with a discussion about updating the goalposts for sample-efficient RL research on the ALE. We make our code and data publicly available at https://github.com/google-research/google-research/tree/master/bigger_better_faster.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2305.19452

Cites

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

The citation lists haven’t been fetched yet

Cited by 9

Research works printed but not linked: 72

  1. Agarwal, R., Schuurmans, D., and Norouzi, M. An opti- mistic perspective on offline reinforcement learning. In International Conference on Machine Learning, pp. 104- 114. PMLR, 2020.
  2. Agarwal, R., Machado, M. C., Castro, P. S., and Bellemare, M. G. Contrastive behavioral similarity embeddings for generalization in reinforcement learning. arXiv preprint arXiv:2101.05265, 2021a. · arXiv
  3. Agarwal, R., Schwarzer, M., Castro, P. S., Courville, A., and Bellemare, M. G. Deep reinforcement learning at the edge of the statistical precipice. Advances in Neural Information Processing Systems, 2021b.
  4. Agarwal, R., Schwarzer, M., Castro, P. S., Courville, A., and Bellemare, M. G. Reincarnating reinforcement learn- ing: Reusing prior computation to accelerate progress. In Advances in Neural Information Processing Systems, 2022.
  5. Bellemare, M. G., Naddaf, Y., Veness, J., and Bowling, M. The arcade learning environment: An evaluation plat- form for general agents. Journal of Artificial Intelligence Research, 47:253-279, 2013.
  6. Bellemare, M. G., Dabney, W., and Munos, R. A distribu- tional perspective on reinforcement learning. In ICML, 2017.
  7. Berner, C., Brockman, G., Chan, B., Cheung, V., Debiak, P., Dennison, C., Farhi, D., Fischer, Q., Hashme, S., Hesse, C., et al. Dota 2 with large scale deep reinforcement learning. arXiv preprint arXiv:1912.06680, 2019. · arXiv
  8. Castro, P. S., Moitra, S., Gelada, C., Kumar, S., and Belle- mare, M. G. Dopamine: A Research Framework for Deep Reinforcement Learning. 2018. URL http: //arxiv.org/abs/1812.06110.
  9. Castro, P. S., Kastner, T., Panangaden, P., and Rowland, M. MICo: Improved representations via sampling-based state similarity for markov decision processes. In Beygelz- imer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems, 2021. URL https://openreview.net/forum? id=wFp6kmQELgu.
  10. Ceron, J. S. O. and Castro, P. S. Revisiting rainbow: Pro- moting more insightful and inclusive deep reinforcement learning research. In International Conference on Ma- chine Learning, pp. 1373-1383. PMLR, 2021.
  11. Chen, X. and He, K. Exploring simple siamese represen- tation learning. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition, pp. 15750-15758, 2021.
  12. Cobbe, K., Hesse, C., Hilton, J., and Schulman, J. Lever- aging procedural generation to benchmark reinforcement learning. In International conference on machine learn- ing, pp. 2048-2056. PMLR, 2020.
  13. Dabney, W., Ostrovski, G., Silver, D., and Munos, R. Im- plicit quantile networks for distributional reinforcement learning. In International conference on machine learn- ing, pp. 1096-1105. PMLR, 2018.
  14. Degrave, J., Felici, F., Buchli, J., Neunert, M., Tracey, B. D., Carpanese, F., Ewalds, T., Hafner, R., Abdolmaleki, A., de Las Casas, D., Donner, C., Fritz, L., Galperti, C., Hu- ber, A., Keeling, J., Tsimpoukelli, M., Kay, J., Merle, A., Moret, J., Noury, S., Pesamosca, F., Pfau, D., Sauter, O., Sommariva, C., Coda, S., Duval, B., Fasoli, A., Kohli, P., Kavukcuoglu, K., Hassabis, D., and Riedmiller, M. A. Magnetic control of tokamak plasmas through deep reinforcement learning. Nature, 602(7897):414-419, 2022. doi: 10.1038/s41586-021-04301-9. URL https: //doi.org/10.1038/s41586-021-04301-9. · DOI · Paper page
  15. D'Oro, P., Schwarzer, M., Nikishin, E., Bacon, P.-L., Belle- mare, M. G., and Courville, A. Sample-efficient rein- forcement learning by breaking the replay ratio barrier. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview. net/forum?id=OpC-9aBBVJe.
  16. Espeholt, L., Soyer, H., Munos, R., Simonyan, K., Mnih, V., Ward, T., Doron, Y., Firoiu, V., Harley, T., Dunning, I., et al. Impala: Scalable distributed deep-rl with im- portance weighted actor-learner architectures. In Interna- tional conference on machine learning, pp. 1407-1416.
  17. PMLR, 2018.
  18. Fan, L., Wang, G., Jiang, Y., Mandlekar, A., Yang, Y., Zhu, H., Tang, A., Huang, D.-A., Zhu, Y., and Anandkumar, A. Minedojo: Building open-ended embodied agents with internet-scale knowledge. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
  19. Farebrother, J., Greaves, J., Agarwal, R., Lan, C. L., Goroshin, R., Castro, P. S., and Bellemare, M. G. Proto-value networks: Scaling representation learning with auxiliary tasks. In Submitted to The Eleventh In- ternational Conference on Learning Representations, 2023. URL https://openreview.net/forum? id=oGDKSt9JrZi. under review.
  20. Fedus, W., Ramachandran, P., Agarwal, R., Bengio, Y., Larochelle, H., Rowland, M., and Dabney, W. Revisiting fundamentals of experience replay. In International Con- ference on Machine Learning, pp. 3061-3071.
  21. PMLR, 2020a.
  22. Fedus, W., Ramachandran, P., Agarwal, R., Bengio, Y., Larochelle, H., Rowland, M., and Dabney, W. Revisit- ing fundamentals of experience replay. In III, H. D. and Singh, A. (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Pro- ceedings of Machine Learning Research, pp. 3061-3071. PMLR, 13-18 Jul 2020b.
  23. Fortunato, M., Azar, M. G., Piot, B., Menick, J., Hes- sel, M., Osband, I., Graves, A., Mnih, V., Munos, R., Hassabis, D., Pietquin, O., Blundell, C., and Legg, S. Noisy networks for exploration. In International Confer- ence on Learning Representations, 2018a. URL https: //openreview.net/forum?id=rywHCPkAW.
  24. Fortunato, M., Azar, M. G., Piot, B., Menick, J., Osband, I., Graves, A., Mnih, V., Munos, R., Hassabis, D., Pietquin, O., Blundell, C., and Legg, S. Noisy networks for explo- ration. 2018b. Franc ¸ois-Lavet, V., Fonteneau, R., and Ernst, D. How to discount deep reinforcement learning: Towards new dy- namic strategies. arXiv preprint arXiv:1512.02011, 2015. · arXiv
  25. Grill, J.-B., Strub, F., Altché, F., Tallec, C., Richemond, P., Buchatskaya, E., Doersch, C., Avila Pires, B., Guo, Z., Gheshlaghi Azar, M., et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems, 33:21271-21284, 2020.
  26. Hafner, D., Lillicrap, T., Norouzi, M., and Ba, J. Mas- tering atari with discrete world models. arXiv preprint arXiv:2010.02193, 2020. · arXiv
  27. Hafner, D., Pasukonis, J., Ba, J., and Lillicrap, T. Master- ing diverse domains through world models, 2023. URL https://arxiv.org/abs/2301.04104. · arXiv
  28. Harris, C. R., Millman, K. J., Van Der Walt, S. J., Gommers, R., Virtanen, P., Cournapeau, D., Wieser, E., Taylor, J., Berg, S., Smith, N. J., et al. Array programming with numpy. Nature, 585(7825):357-362, 2020.
  29. Hessel, M., Modayil, J., van Hasselt, H., Schaul, T., Ostro- vski, G., Dabney, W., Horgan, D., Piot, B., Azar, M., and Silver, D. Rainbow: Combining improvements in deep reinforcement learning. arXiv preprint arXiv:1710.02298, 2017. · arXiv
  30. Hessel, M., Modayil, J., Hasselt, H. V., Schaul, T., Ostrovski, G., Dabney, W., Horgan, D., Piot, B., Azar, M. G., and Silver, D. Rainbow: Combining improvements in deep reinforcement learning. In AAAI, 2018.
  31. Hunter, J. D. Matplotlib: A 2d graphics environment. Com- puting in science & engineering, 9(03):90-95, 2007.
  32. Kaiser, L., Babaeizadeh, M., Milos, P., Osinski, B., Camp- bell, R. H., Czechowski, K., Erhan, D., Finn, C., Koza- kowski, P., Levine, S., et al. Model-based reinforcement learning for atari. International Conference on Learning Representations, 2020.
  33. Kapturowski, S., Campos, V., Jiang, R., Rakićević, N., van Hasselt, H., Blundell, C., and Badia, A. P. Human-level atari 200x faster. arXiv preprint arXiv:2209.07550, 2022. · arXiv
  34. Kearns, M. J. and Singh, S. Bias-variance error bounds for temporal difference updates. In COLT, pp. 142-147, 2000.
  35. Kostrikov, I., Yarats, D., and Fergus, R. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. arXiv preprint arXiv:2004.13649, 2020. · arXiv
  36. Kumar, A., Agarwal, R., Ghosh, D., and Levine, S. Implicit under-parameterization inhibits data-efficient deep rein- forcement learning. arXiv preprint arXiv:2010.14498, 2020. · arXiv
  37. Kumar, A., Agarwal, R., Geng, X., Tucker, G., and Levine, S. Offline q-learning on diverse multi-task data both scales and generalizes, 2022. URL https://arxiv. org/abs/2211.15144.
  38. Levine, S., Kumar, A., Tucker, G., and Fu, J. Offline re- inforcement learning: Tutorial, review, and perspectives on open problems. CoRR, abs/2005.01643, 2020. URL https://arxiv.org/abs/2005.01643. · arXiv
  39. Li, X., Shang, J., Das, S., and Ryoo, M. S. Does self- supervised learning really improve reinforcement learn- ing from pixels? In Oh, A. H., Agarwal, A., Bel- grave, D., and Cho, K. (eds.), Advances in Neural In- formation Processing Systems, 2022. URL https: //openreview.net/forum?id=fVslVNBfjd8.
  40. Loshchilov, I. and Hutter, F. Decoupled weight decay reg- ularization. In International Conference on Learning Representations, 2019. URL https://openreview. net/forum?id=Bkg6RiCqY7.
  41. Machado, M. C., Bellemare, M. G., Talvitie, E., Veness, J., Hausknecht, M., and Bowling, M. Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents. J. Artif. Int. Res., 61(1): 523-562, jan 2018. ISSN 1076-9757.
  42. Mazoure, B., Tachet des Combes, R., Doan, T. L., Bachman, P., and Hjelm, R. D. Deep reinforcement and infomax learning. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 3686-3698.
  43. Curran Associates, Inc., 2020. URL https://proceedings. neurips.cc/paper/2020/file/ 26588e932c7ccfa1df309280702fe1b5-Paper. pdf.
  44. Micheli, V., Alonso, E., and Fleuret, F. Transformers are sample-efficient world models. In To appear in The Eleventh International Conference on Learning Represen- tations, 2023. URL https://openreview.net/ forum?id=vhFu1Acb0xb.
  45. Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., and Riedmiller, M. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, 2013. · arXiv
  46. Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Ve- ness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., Petersen, S., Beattie, C., Sadik, A., Antonoglou, I., King, H., Kumaran, D., Wier- stra, D., Legg, S., and Hassabis, D. Human-level control through deep reinforcement learning. Nature, 518(7540): 529-533, February 2015a.
  47. Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidje- land, A. K., Ostrovski, G., et al. Human-level control through deep reinforcement learning. nature, 518(7540): 529-533, 2015b.
  48. Nikishin, E., Schwarzer, M., D'Oro, P., Bacon, P.-L., and Courville, A. The primacy bias in deep reinforcement learning. In Chaudhuri, K., Jegelka, S., Song, L., Szepes- vari, C., Niu, G., and Sabato, S. (eds.), Proceedings of the 39th International Conference on Machine Learning, vol- ume 162 of Proceedings of Machine Learning Research, pp. 16828-16847. PMLR, 17-23 Jul 2022.
  49. Oliphant, T. E. Python for scientific computing. Computing in Science & Engineering, 9(3):10-20, 2007. doi: 10. 1109/MCSE.2007. · DOI
  50. Ostrovski, G., Castro, P. S., and Dabney, W. The difficulty of passive learning in deep reinforcement learning. In Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems, 2021. URL https://openreview.net/ forum?id=nPHA8fGicZk.
  51. Ota, K., Jha, D. K., and Kanezaki, A. Training larger net- works for deep reinforcement learning. arXiv preprint arXiv:2102.07920, 2021. · arXiv
  52. Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Gray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P., Leike, J., and Lowe, R. Training language models to follow instruc- tions with human feedback. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), Advances in Neural Information Processing Systems, 2022. URL https: //openreview.net/forum?id=TG8KACxEON.
  53. Puterman, M. L. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014.
  54. Robine, J., Höftmann, M., Uelwer, T., and Harmeling, S. Transformer-based world models are happy with 100k interactions. In The Eleventh International Conference on Learning Representations, 2023. URL https:// openreview.net/forum?id=TdBaDGCpjly.
  55. Schaul, T., Quan, J., Antonoglou, I., and Silver, D. Priori- tized experience replay. CoRR, abs/1511.05952, 2016. · arXiv
  56. Schaul, T., Barreto, A., Quan, J., and Ostrovski, G. The phenomenon of policy churn. Advances in Neural Infor- mation Processing Systems, 2022.
  57. Schmidt, D. and Schmied, T. Fast and data-efficient training of rainbow: an experimental study on atari. arXiv preprint arXiv:2111.10247, 2021. · arXiv
  58. Schrittwieser, J., Antonoglou, I., Hubert, T., Simonyan, K., Sifre, L., Schmitt, S., Guez, A., Lockhart, E., Hassabis, D., Graepel, T., et al. Mastering atari, go, chess and shogi by planning with a learned model. Nature, 588(7839): 604-609, 2020.
  59. Schrittwieser, J., Hubert, T., Mandhane, A., Barekatain, M., Antonoglou, I., and Silver, D. Online and offline reinforcement learning by planning with a learned model. Advances in Neural Information Processing Systems, 34: 27580-27591, 2021.
  60. Schwarzer, M., Anand, A., Goel, R., Hjelm, R. D., Courville, A., and Bachman, P. Data-efficient reinforce- ment learning with self-predictive representations. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum? id=uCQfPZwRaUu.
  61. Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., van den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., Dieleman, S., Grewe, D., Nham, J., Kalchbrenner, N., Sutskever, I., Lillicrap, T., Leach, M., Kavukcuoglu, K., Graepel, T., and Hassabis, D. Mastering the game of go with deep neural networks and tree search. Nature, 529:484-503, 2016. URL http://www.nature.com/nature/journal/ v529/n7587/full/nature16961.html.
  62. Sinha, S., Bharadhwaj, H., Srinivas, A., and Garg, A. D2rl: Deep dense architectures in reinforcement learning. arXiv preprint arXiv:2010.09163, 2020. · arXiv
  63. Sokar, G., Agarwal, R., Castro, P. S., and Evci, U. The dor- mant neuron phenomenon in deep reinforcement learning. In ICML, 2023.
  64. Sutton, R. S. Learning to predict by the methods of temporal differences. Machine Learning, 3(1):9-44, August 1988.
  65. Sutton, R. S. and Barto, A. G. Introduction to Reinforcement Learning. MIT Press, Cambridge, MA, USA, 1st edition, 1998. ISBN 0262193981.
  66. Taiga, A. A., Agarwal, R., Farebrother, J., Courville, A., and Bellemare, M. G. Investigating multi-task pretraining and generalization in reinforcement learning. In Submitted to The Eleventh International Conference on Learning Representations, 2023. URL https://openreview. net/forum?id=sSt9fROSZRO. under review.
  67. Team, A. A., Bauer, J., Baumli, K., Baveja, S., Behba- hani, F., Bhoopchand, A., Bradley-Schmieg, N., Chang, M., Clay, N., Collister, A., et al. Human-timescale adaptation in an open-ended task space. arXiv preprint arXiv:2301.07608, 2023. · arXiv
  68. van Hasselt, H., Guez, A., and Silver, D. Deep reinforce- ment learning with double q-learning. In Proceedings of the Thirthieth AAAI Conference On Artificial Intelligence (AAAI), 2016, 2016. cite arxiv:1509.06461Comment: AAAI 2016.
  69. Van Hasselt, H. P., Hessel, M., and Aslanides, J. When to use parametric models in reinforcement learning? Ad- vances in Neural Information Processing Systems, 32, 2019.
  70. Vinyals, O., Babuschkin, I., Czarnecki, W. M., Mathieu, M., Dudzik, A., Chung, J., Choi, D. H., Powell, R., Ewalds, T., Georgiev, P., et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575 (7782):350-354, 2019.
  71. Wang, Z., Schaul, T., Hessel, M., Hasselt, H., Lanctot, M., and Freitas, N. Dueling network architectures for deep reinforcement learning. In Proceedings of the 33rd Inter- national Conference on Machine Learning, volume 48, pp. 1995-2003, 2016.
  72. Ye, W., Liu, S., Kurutach, T., Abbeel, P., and Gao, Y. Mas- tering atari games with limited data. Advances in Neural Information Processing Systems, 34:25476-25488, 2021.

Metadata

Authors
Schwarzer, M., Obando-Ceron, J., Courville, A., Bellemare, M., Agarwal, R., & Castro, P. S.
Affiliations
Université de Montréal (CA)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2023
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2305.19452
arXiv
2305.19452
OpenAlex
W4379087150
Cited by
9
References
0
Citations by year
2025: 3 · 2024: 6
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence in Games
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords
sample efficiency, value-based reinforcement learning, human-level performance, value estimation, game-playing agents

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review