The DOI matches its DataCite record; 9 of the 9 references it cites were found in a registry or an index.
Distributed Prioritized Experience Replay
Abstract
We propose a distributed architecture for deep reinforcement learning at scale, that enables agents to learn effectively from orders of magnitude more data than previously possible. The algorithm decouples acting from learning: the actors interact with their own instances of the environment by selecting actions according to a shared neural network, and accumulate the resulting experience in a shared experience replay memory; the learner replays samples of experience and updates the neural network. The architecture relies on prioritized experience replay to focus only on the most significant data generated by the actors. Our architecture substantially improves the state of the art on the Arcade Learning Environment, achieving better final performance in a fraction of the wall-clock training time.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Decoupling the agent that gathers experience from the one that fits the model is a long-standing design in reinforcement learning [Sutton, 1991, Konda and Tsitsiklis, 1999, Horgan et al., 2018], and cooperating language-model agents with specialized roles have outperformed a single agent on multi-step tasks [Hong et al., 2024, Wu et al., 2023, Qian et al., 2024].”
From OPINE-World: Programmatic World Modeling with Ontology-error-Prioritized Interactive Exploration for ARC-AGI-3 (2026) · cites this paper as Horgan et al., 2018
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
Mastering the game of Go with deep neural networks and tree search
Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., van den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al. (2016) · Nature · doi:10.1038/nature16961 · Cited by 15,612
Self-citationTensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G. S., Davis, A., Dean, J., Devin, M., et al. (2016) · arXiv · doi:10.48550/arxiv.1603.04467 · Cited by 9,408
Asynchronous Methods for Deep Reinforcement Learning
Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lillicrap, T. P., Harley, T., Silver, D., & Kavukcuoglu, K. (2016) · arXiv · doi:10.48550/arxiv.1602.01783 · Cited by 7,132
Self-citationLearning to predict by the methods of temporal differences
Sutton, R. S. (1988) · Machine Learning · doi:10.1007/bf00115009 · Cited by 5,148
The Arcade Learning Environment: An Evaluation Platform for General Agents
Bellemare, M. G., Naddaf, Y., Veness, J., & Bowling, M. (2013) · Journal of Artificial Intelligence Research · doi:10.1613/jair.3912 · Cited by 2,161
Rainbow: Combining Improvements in Deep Reinforcement Learning
Hessel, M., Modayil, J., Van Hasselt, H., Schaul, T., Ostrovski, G., Dabney, W., Horgan, D., Piot, B., Azar, M., & Silver, D. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11796 · Cited by 1,846
Self-citationEmergence of Locomotion Behaviours in Rich Environments
Heess, N., TB, D., Sriram, S., Lemmon, J., Merel, J., Wayne, G., Tassa, Y., Erez, T., Wang, Z., Eslami, S. M. A., et al. (2017) · arXiv · doi:10.48550/arxiv.1707.02286 · Cited by 657
Self-citation
Cited by
Mastering Atari, Go, chess and shogi by planning with a learned model
Schrittwieser, J., Antonoglou, I., Hubert, T., Simonyan, K., Sifre, L., Schmitt, S., Guez, A., Lockhart, E., Hassabis, D., Graepel, T., et al. (2020) · Nature · doi:10.1038/s41586-020-03051-4 · Cited by 1,214
Self-citationDota 2 with Large Scale Deep Reinforcement Learning
OpenAI, :, Berner, C., Brockman, G., Chan, B., Cheung, V., Dębiak, P., Dennison, C., Farhi, D., Fischer, Q., et al. (2019) · arXiv · doi:10.48550/arxiv.1912.06680 · Cited by 1,065
An Intriguing Failing of Convolutional Neural Networks and the CoordConv Solution
Liu, R., Lehman, J., Molino, P., Such, F. P., Frank, E., Sergeev, A., & Yosinski, J. (2018) · arXiv · doi:10.48550/arxiv.1807.03247 · Cited by 784
QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation
Kalashnikov, D., Irpan, A., Pastor, P., Ibarz, J., Herzog, A., Jang, E., Quillen, D., Holly, E., Kalakrishnan, M., Vanhoucke, V., et al. (2018) · arXiv · doi:10.48550/arxiv.1806.10293 · Cited by 697
Exploration by Random Network Distillation
Burda, Y., Edwards, H., Storkey, A., & Klimov, O. (2018) · arXiv · doi:10.48550/arxiv.1810.12894 · Cited by 527
Ecoffet, A., Huizinga, J., Lehman, J., Stanley, K. O., & Clune, J. (2021) · Nature · doi:10.1038/s41586-020-03157-9 · Cited by 330
Mastering Atari with Discrete World Models
Hafner, D., Lillicrap, T., Norouzi, M., & Ba, J. (2020) · arXiv · doi:10.48550/arxiv.2010.02193 · Cited by 247
Go-Explore: a New Approach for Hard-Exploration Problems
Ecoffet, A., Huizinga, J., Lehman, J., Stanley, K. O., & Clune, J. (2019) · arXiv · doi:10.48550/arxiv.1901.10995 · Cited by 205
Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domain
Hao, J., Yang, T., Tang, H., Bai, C., Liu, J., Meng, Z., Liu, P., & Wang, Z. (2021) · arXiv · doi:10.48550/arxiv.2109.06668 · Cited by 192
Learning structures: Predictive representations, replay, and generalization
Momennejad, I. (2020) · Current Opinion in Behavioral Sciences · doi:10.1016/j.cobeha.2020.02.017 · Cited by 159
Never Give Up: Learning Directed Exploration Strategies
Badia, A. P., Sprechmann, P., Vitvitskyi, A., Guo, D., Piot, B., Kapturowski, S., Tieleman, O., Arjovsky, M., Pritzel, A., Bolt, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2002.06038 · Cited by 170
Scalable agent alignment via reward modeling: a research direction
Leike, J., Krueger, D., Everitt, T., Martic, M., Maini, V., & Legg, S. (2018) · arXiv · doi:10.48550/arxiv.1811.07871 · Cited by 121
CleanRL: High-quality Single-file Implementations of Deep Reinforcement Learning Algorithms
Huang, S., Dossa, R. F. J., Ye, C., & Braga, J. (2021) · arXiv · doi:10.48550/arxiv.2111.08819 · Cited by 116
Morales, E. F., Murrieta-Cid, R., Becerra, I., & Esquivel-Basaldua, M. A. (2021) · Intelligent Service Robotics · doi:10.1007/s11370-021-00398-z · Cited by 101
An empirical investigation of the challenges of real-world reinforcement learning
Dulac-Arnold, G., Levine, N., Mankowitz, D. J., Li, J., Paduraru, C., Gowal, S., & Hester, T. (2020) · arXiv · doi:10.48550/arxiv.2003.11881 · Cited by 78
Reinforcement Learning with an Ensemble of Binary Action Deep Q-Networks
M. Hafiz, A., Hassaballah, M., Alqahtani, A., Alsubai, S., & Abdel Hameed, M. (2023) · Computer Systems Science and Engineering · doi:10.32604/csse.2023.031720 · Cited by 33
Synthetic Returns for Long-Term Credit Assignment
Raposo, D., Ritter, S., Santoro, A., Wayne, G., Weber, T., Botvinick, M., van Hasselt, H., & Song, F. (2021) · arXiv · doi:10.48550/arxiv.2102.12425 · Cited by 12
Podracer architectures for scalable Reinforcement Learning
Hessel, M., Kroiss, M., Clark, A., Kemaev, I., Quan, J., Keck, T., Viola, F., & van Hasselt, H. (2021) · arXiv · doi:10.48550/arxiv.2104.06272 · Cited by 8
Self-citationMeasuring Progress in Deep Reinforcement Learning Sample Efficiency
Dorner, F. E. (2021) · arXiv · doi:10.48550/arxiv.2102.04881 · Cited by 6
XLand-MiniGrid: Scalable Meta-Reinforcement Learning Environments in JAX
Nikulin, A., Kurenkov, V., Zisman, I., Agarkov, A., Sinii, V., & Kolesnikov, S. (2024) · Advances in Neural Information Processing Systems 37 · doi:10.52202/079017-1390 · Cited by 3
“For example, many of the early breakthroughs on the Atari benchmark were driven by more efficient distributed experience collection (Horgan et al., 2018; Espeholt et al., 2018; Kapturowski et al., 2018), eventually reducing training time to just a few minutes (Stooke & Abbeel, 2018; Adamski et al.,…”
BackgroundExploration-Driven Generative Interactive Environments
Savov, N., Kazemi, N., Mahdi, M., Paudel, D. P., Wang, X., & Van Gool, L. (2025) · 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52734.2025.02570 · Cited by 2
Courtis, D., Li, W., & Sanner, S. (2026) · arXiv · doi:10.48550/arxiv.2607.01531
Wang, R., Lehman, J., Clune, J., & Stanley, K. O. (2019) · arXiv
Accelerating Reinforcement Learning through GPU Atari Emulation
Dalton, S., Frosio, I., & Garland, M. (2019) · arXiv · doi:10.48550/arxiv.1907.08467
Metadata
- Authors
- Horgan, D., Quan, J., Budden, D., Barth-Maron, G., Hessel, M., van Hasselt, H., & Silver, D.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2018
- arXiv
- 1803.00933
- References
- 9
- Type
- Preprint
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
