The DOI matches its Crossref record; 78 of the 93 references it cites were found in a registry or an index.
Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agents
Open accessAbstract
The Arcade Learning Environment (ALE) is an evaluation platform that poses the challenge of building AI agents with general competency across dozens of Atari 2600 games. It supports a variety of different problem settings and it has been receiving increasing attention from the scientific community, leading to some high-profile success stories such as the much publicized Deep Q-Networks (DQN). In this article we take a big picture look at how the ALE is being used by the research community. We show how diverse the evaluation methodologies in the ALE have become with time, and highlight some key concerns when evaluating agents in the ALE. We use this discussion to present some methodological best practices and provide new benchmark results using these best practices. To further the progress in the field, we introduce a new version of the ALE that supports multiple game modes and provides a form of stochasticity we call sticky actions. We conclude this big picture look by revisiting challenges posed when the ALE was introduced, summarizing the state-of-the-art in various problems and highlighting problems that remain open.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“For instance, the standard protocol for the Atari benchmark (a set of environments that require limited generalisation (Cobbe et al., 2020)) is to use 200M environment interactions per game (Machado et al., 2018)”
From Craftax: A Lightning-Fast Benchmark for Open-Ended Reinforcement Learning (2024) · cites this paper as Machado et al., 2018
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“We abide by Machado et al. (2018)'s recommendations of evaluation protocol, including the use of stochasticity, ignoring the lives signal, and reporting the average performance during training.”
From Loss of Plasticity in Continual Deep Reinforcement Learning (2023) · cites this paper as Machado et al. (2018)
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Human-level control through deep reinforcement learning
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015) · Nature · doi:10.1038/nature14236 · Cited by 30,357
Self-citationReinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. (2005) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2004.842673 · Cited by 25,268
Mastering the game of Go with deep neural networks and tree search
Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., van den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al. (2016) · Nature · doi:10.1038/nature16961 · Cited by 15,612
Bellman, R. E. (2010) · doi:10.1515/9781400835386 · Cited by 14,588
Asynchronous Methods for Deep Reinforcement Learning
Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lillicrap, T. P., Harley, T., Silver, D., & Kavukcuoglu, K. (2016) · arXiv · doi:10.48550/arxiv.1602.01783 · Cited by 7,132
Deep Reinforcement Learning with Double Q-Learning
Van Hasselt, H., Guez, A., & Silver, D. (2016) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v30i1.10295 · Cited by 3,318
Dueling Network Architectures for Deep Reinforcement Learning
Wang, Z., Schaul, T., Hessel, M., van Hasselt, H., Lanctot, M., & de Freitas, N. (2015) · arXiv · doi:10.48550/arxiv.1511.06581 · Cited by 3,300
Schaul, T., Quan, J., Antonoglou, I., & Silver, D. (2015) · arXiv · doi:10.48550/arxiv.1511.05952 · Cited by 3,120
The Arcade Learning Environment: An Evaluation Platform for General Agents
Bellemare, M. G., Naddaf, Y., Veness, J., & Bowling, M. (2013) · Journal of Artificial Intelligence Research · doi:10.1613/jair.3912 · Cited by 2,161
Self-citation
Flagged first, then by citations
15 of the 93 references couldn’t be matched to a known paper.
Research works printed but not linked: 8
- Hausknecht, M.; Lehman, J.; Miikkulainen, R.; and Stone, P. 2013. A neuroevolution approach to general atari game playing. In IEEE Transactions on Computational Intelli- gence and AI in Games.
- Mnih, V.; Kavukcuoglu, K.; Silver, D.; Graves, A.; Antonoglou, I.; Wierstra, D.; and Riedmiller, M. 2013. Playing atari with deep reinforcement learning. CoRR abs/1312.5602. · arXiv
- Albrecht, S. V., L., J. C., Buckeridge, D. L., Botea, A., Caragea, C., Chi, C., Damoulas, T., Dilkina, B. N., Eaton, E., Fazli, P., Ganzfried, S., Lindauer, M. T., Machado, M. C., Malitsky, Y., Marcus, G., Meijer, S., Rossi, F., Shaban-Nejad, A., Thiebaux, S., Veloso, M. M., Walsh, T., Wang, C., Zhang, J., & Zheng, Y. (2015). Reports on the 2015 AAAI Workshop Program. AI Magazine, 36 (2), 90-101.
- Campbell, M., Hoane, Jr., A. J., & Hsu, F.-h. (2002). Deep Blue. Artificial Intelligence, 134 (1-2), 57-83.
- Harutyunyan, A., Bellemare, M. G., Stepleton, T., & Munos, R. (2016). Q(λ) with Off- Policy Corrections. In Proceedings of the International Conference on Algorithmic Learning Theory (ALT), pp. 305-320.
- Hutter, M. (2005). Universal Artificial Intelligence: Sequential Decisions based on Algorith- mic Probability. Springer.
- Kearns, M. J., & Singh, S. P. (2002). Near-Optimal Reinforcement Learning in Polynomial Time. Machine Learning, 49 (2-3), 209-232.
- Nair, A., Srinivasan, P., Blackwell, S., Alcicek, C., Fearon, R., Maria, A. D., Panneershelvam, V., Suleyman, M., Beattie, C., Petersen, S., Legg, S., Mnih, V., Kavukcuoglu, K., & Silver, D. (2015). Massively Parallel Methods for Deep Reinforcement Learning. In ICML Deep Learning Workshop.
Cited by
Sorted by citations
Deep Reinforcement Learning That Matters
Henderson, P., Islam, R., Bachman, P., Pineau, J., Precup, D., & Meger, D. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11694 · Cited by 1,685
Mastering Atari, Go, chess and shogi by planning with a learned model
Schrittwieser, J., Antonoglou, I., Hubert, T., Simonyan, K., Sifre, L., Schmitt, S., Guez, A., Lockhart, E., Hassabis, D., Graepel, T., et al. (2020) · RePEc: Research Papers in Economics · Cited by 1,310
Mastering Atari, Go, chess and shogi by planning with a learned model
Schrittwieser, J., Antonoglou, I., Hubert, T., Simonyan, K., Sifre, L., Schmitt, S., Guez, A., Lockhart, E., Hassabis, D., Graepel, T., et al. (2020) · Nature · doi:10.1038/s41586-020-03051-4 · Cited by 1,214
Meta-Learning in Neural Networks: A Survey
Hospedales, T. M., Antoniou, A., Micaelli, P., & Storkey, A. J. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3079209 · Cited by 886
QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation
Kalashnikov, D., Irpan, A., Pastor, P., Ibarz, J., Herzog, A., Jang, E., Quillen, D., Holly, E., Kalakrishnan, M., Vanhoucke, V., et al. (2018) · arXiv · doi:10.48550/arxiv.1806.10293 · Cited by 697
Metadata
- Authors
- Machado, M. C., Bellemare, M. G., Talvitie, E., Veness, J., Hausknecht, M., & Bowling, M.
- Affiliations
- University of Alberta (CA)
- Franklin & Marshall College (US)
- Google DeepMind (United Kingdom) (GB)
- Microsoft (United States) (US)
- Venue
- Journal of Artificial Intelligence Research
- Publisher
- AI Access Foundation
- Published
- 2018-03-19
- Volume
- 61
- Pages
- 523-562
- ISSN
- 1076-9757 (ISSN-L)
- OpenAlex
- W2754879180
- Cited by
- 359
- References
- 93
- Citations by year
- 2026: 9 · 2025: 16 · 2024: 21 · 2023: 27 · 2022: 30 · 2021: 71 · 2020: 71 · 2019: 73 · 2018: 39
- FWCI
- 35.74
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- cc-by
- Locations
- Journal of Artificial Intelligence Research (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1552533)
- Compute Canada
- Alberta Machine Intelligence Institute
- DeepMind
- Alberta Innovates - Technology Futures
- Alberta Innovates
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence in Games
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
- Keywords
- Arcade Learning Environment, evaluation methodology, benchmark results, deep Q-network, game modes, evaluation best practices
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
