The DOI matches its Crossref record; 78 of the 93 references it cites were found in a registry or an index.
Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agents
Open accessAbstract
The Arcade Learning Environment (ALE) is an evaluation platform that poses the challenge of building AI agents with general competency across dozens of Atari 2600 games. It supports a variety of different problem settings and it has been receiving increasing attention from the scientific community, leading to some high-profile success stories such as the much publicized Deep Q-Networks (DQN). In this article we take a big picture look at how the ALE is being used by the research community. We show how diverse the evaluation methodologies in the ALE have become with time, and highlight some key concerns when evaluating agents in the ALE. We use this discussion to present some methodological best practices and provide new benchmark results using these best practices. To further the progress in the field, we introduce a new version of the ALE that supports multiple game modes and provides a form of stochasticity we call sticky actions. We conclude this big picture look by revisiting challenges posed when the ALE was introduced, summarizing the state-of-the-art in various problems and highlighting problems that remain open.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“For instance, the standard protocol for the Atari benchmark (a set of environments that require limited generalisation (Cobbe et al., 2020)) is to use 200M environment interactions per game (Machado et al., 2018)”
From Craftax: A Lightning-Fast Benchmark for Open-Ended Reinforcement Learning (2024) · cites this paper as Machado et al., 2018
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“We abide by Machado et al. (2018)'s recommendations of evaluation protocol, including the use of stochasticity, ignoring the lives signal, and reporting the average performance during training.”
From Loss of Plasticity in Continual Deep Reinforcement Learning (2023) · cites this paper as Machado et al. (2018)
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Human-level control through deep reinforcement learning
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015) · Nature · doi:10.1038/nature14236 · Cited by 30,357
Self-citationReinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. (2005) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2004.842673 · Cited by 25,268
Mastering the game of Go with deep neural networks and tree search
Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., van den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al. (2016) · Nature · doi:10.1038/nature16961 · Cited by 15,612
Bellman, R. E. (2010) · doi:10.1515/9781400835386 · Cited by 14,588
Asynchronous Methods for Deep Reinforcement Learning
Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lillicrap, T. P., Harley, T., Silver, D., & Kavukcuoglu, K. (2016) · arXiv · doi:10.48550/arxiv.1602.01783 · Cited by 7,132
Deep Reinforcement Learning with Double Q-Learning
Van Hasselt, H., Guez, A., & Silver, D. (2016) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v30i1.10295 · Cited by 3,318
Dueling Network Architectures for Deep Reinforcement Learning
Wang, Z., Schaul, T., Hessel, M., van Hasselt, H., Lanctot, M., & de Freitas, N. (2015) · arXiv · doi:10.48550/arxiv.1511.06581 · Cited by 3,300
Schaul, T., Quan, J., Antonoglou, I., & Silver, D. (2015) · arXiv · doi:10.48550/arxiv.1511.05952 · Cited by 3,120
The Arcade Learning Environment: An Evaluation Platform for General Agents
Bellemare, M. G., Naddaf, Y., Veness, J., & Bowling, M. (2013) · Journal of Artificial Intelligence Research · doi:10.1613/jair.3912 · Cited by 2,161
Self-citation
Flagged first, then by citations
15 of the 93 references couldn’t be matched to a known paper.
Research works printed but not linked: 8
- Hausknecht, M.; Lehman, J.; Miikkulainen, R.; and Stone, P. 2013. A neuroevolution approach to general atari game playing. In IEEE Transactions on Computational Intelli- gence and AI in Games.
- Mnih, V.; Kavukcuoglu, K.; Silver, D.; Graves, A.; Antonoglou, I.; Wierstra, D.; and Riedmiller, M. 2013. Playing atari with deep reinforcement learning. CoRR abs/1312.5602. · arXiv
- Albrecht, S. V., L., J. C., Buckeridge, D. L., Botea, A., Caragea, C., Chi, C., Damoulas, T., Dilkina, B. N., Eaton, E., Fazli, P., Ganzfried, S., Lindauer, M. T., Machado, M. C., Malitsky, Y., Marcus, G., Meijer, S., Rossi, F., Shaban-Nejad, A., Thiebaux, S., Veloso, M. M., Walsh, T., Wang, C., Zhang, J., & Zheng, Y. (2015). Reports on the 2015 AAAI Workshop Program. AI Magazine, 36 (2), 90-101.
- Campbell, M., Hoane, Jr., A. J., & Hsu, F.-h. (2002). Deep Blue. Artificial Intelligence, 134 (1-2), 57-83.
- Harutyunyan, A., Bellemare, M. G., Stepleton, T., & Munos, R. (2016). Q(λ) with Off- Policy Corrections. In Proceedings of the International Conference on Algorithmic Learning Theory (ALT), pp. 305-320.
- Hutter, M. (2005). Universal Artificial Intelligence: Sequential Decisions based on Algorith- mic Probability. Springer.
- Kearns, M. J., & Singh, S. P. (2002). Near-Optimal Reinforcement Learning in Polynomial Time. Machine Learning, 49 (2-3), 209-232.
- Nair, A., Srinivasan, P., Blackwell, S., Alcicek, C., Fearon, R., Maria, A. D., Panneershelvam, V., Suleyman, M., Beattie, C., Petersen, S., Legg, S., Mnih, V., Kavukcuoglu, K., & Silver, D. (2015). Massively Parallel Methods for Deep Reinforcement Learning. In ICML Deep Learning Workshop.
Cited by
Sorted by citations
Items 201–300 of 358
Measuring Progress in Deep Reinforcement Learning Sample Efficiency
Dorner, F. E. (2021) · arXiv · doi:10.48550/arxiv.2102.04881 · Cited by 6
The Difficulty of Passive Learning in Deep Reinforcement Learning
Ostrovski, G., Castro, P. S., & Dabney, W. (2021) · arXiv · doi:10.48550/arxiv.2110.14020 · Cited by 15
Policy Search through Genetic Programming and LLM-assisted Curriculum Learning
Jorgensen, S., Nadizar, G., Pietropolli, G., Manzoni, L., Medvet, E., O’Reilly, U.-M., & Hemberg, E. (2025) · ACM Transactions on Evolutionary Learning and Optimization · doi:10.1145/3772718 · Cited by 6
Approximate Model-Based Shielding for Safe Reinforcement Learning
Goodall, A. W., & Belardinelli, F. (2023) · Frontiers in artificial intelligence and applications · doi:10.3233/faia230357 · Cited by 5
On the Effectiveness of Regularization Methods for Soft Actor-Critic in Discrete-Action Domains
Le, B. G., & Ta, V. C. (2024) · IEEE Transactions on Systems Man and Cybernetics Systems · doi:10.1109/tsmc.2024.3505613 · Cited by 5
Application of DQN-IRL Framework in Doudizhu’s Sparse Reward
Kong, Y., Shi, H., Wu, X., & Rui, Y. (2023) · Neural Processing Letters · doi:10.1007/s11063-023-11209-0 · Cited by 6
Exploration Policies for On-the-Fly Controller Synthesis: A Reinforcement Learning Approach
Delgado, T., Sorondo, M. S., Braberman, V. A., & Uchitel, S. (2023) · Proceedings of the International Conference on Automated Planning and Scheduling · doi:10.1609/icaps.v33i1.27238 · Cited by 5
Addressing maximization bias in reinforcement learning with two-sample testing
Waltz, M., & Okhrin, O. (2024) · Artificial Intelligence · doi:10.1016/j.artint.2024.104204 · Cited by 5
Learning Robust Helpful Behaviors in Two-Player Cooperative Atari Environments
Tylkin, P., Radanović, G., & Parkes, D. C. (2021) · Adaptive Agents and Multi-Agents Systems · doi:10.65109/mjqh9483 · Cited by 5
Behavioral Cloning in Atari Games Using a Combined Variational Autoencoder and Predictor Model
Chen, B., Tandon, S., Gorsich, D., Gorodetsky, A. A., & Veerapaneni, S. (2021) · IEEE Congress on Evolutionary Computation (CEC) · doi:10.1109/cec45853.2021.9505001 · Cited by 5
Project Thyia: A Forever Gameplayer
Gaina, R. D., Lucas, S. M., & Pérez-Liébana, D. (2019) · IEEE Conference on Games (CoG) · doi:10.1109/cig.2019.8848047 · Cited by 7
Evolving a Dota 2 Hero Bot with a Probabilistic Shared Memory Model
Smith, R. J., & Heywood, M. I. (2020) · Genetic and evolutionary computation · doi:10.1007/978-3-030-39958-0_17 · Cited by 5
Metatrace: Online Step-size Tuning by Meta-gradient Descent for Reinforcement Learning Control.Preprint
Young, K., Wang, B., & Taylor, M. E. (2018) · arXiv (Cornell University) · Cited by 5
Variational Inference for Data-Efficient Model Learning in POMDPsPreprint
Tschiatschek, S., Arulkumaran, K., Stühmer, J., & Hofmann, K. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.09281 · Cited by 12
Catalyst.RL: A Distributed Framework for Reproducible RL ResearchPreprint
Kolesnikov, S., & Hrinchuk, O. (2019) · arXiv.org · doi:10.48550/arxiv.1903.00027 · Cited by 8
A Visual Communication Map for Multi-Agent Deep Reinforcement LearningPreprint
Nguyen, N. D., Nguyen, T. T., Creighton, D., & Nahavandi, S. (2020) · arXiv (Cornell University) · doi:10.13140/rg.2.2.13433.62563 · Cited by 7
Deep Reinforcement Learning with Weighted Q-LearningPreprint
Cini, A., D'Eramo, C., Peters, J., & Alippi, C. (2020) · arXiv.org · doi:10.48550/arxiv.2003.09280 · Cited by 12
Learning to Represent Action Values as a Hypergraph on the Action VerticesPreprint
Tavakoli, A., Fatemi, M., & Kormushev, P. S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.14680 · Cited by 5
Convergent and Efficient Deep Q Network AlgorithmPreprint
Wang, Z. T., & Ueda, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.15419 · Cited by 5
Craftax: A Lightning-Fast Benchmark for Open-Ended Reinforcement Learning
Matthews, M., Beukman, M., Ellis, B., Samvelyan, M., Jackson, M., Coward, S., & Foerster, J. (2024) · arXiv · doi:10.48550/arxiv.2402.16801 · Cited by 4
“…instance, the standard protocol for the Atari benchmark (a set of environments that require limited generalisation (Cobbe et al., 2020)) is to use 200M environment interactions per game (Machado et al., 2018), while The NetHack Learning Environment suggests using 1 billion (Küttler et al., 2020).”
MethodFrAG: A Framework for the Analysis of Games
Ganesh, S., Aycock, J., & Biittner, K. M. (2023) · IEEE Conference on Games (CoG) · doi:10.1109/cog57401.2023.10333209 · Cited by 4
Learning Switched Koopman Models for Control of Entity-Based Systems
Blischke, M., & Hespanha, J. P. (2023) · IEEE Conference on Decision and Control (CDC) · doi:10.1109/cdc49753.2023.10384106 · Cited by 4
Improving the Diversity of Bootstrapped DQN by Replacing Priors With Noise
Meng, L., Goodwin, M., Yazidi, A., & Engelstad, P. E. (2022) · IEEE Transactions on Games · doi:10.1109/tg.2022.3185330 · Cited by 4
Draw on advantages and avoid disadvantages by making a multi-step prediction
Zhu, G., & Zhu, F. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.121345 · Cited by 4
Unsupervised Task Clustering for Multi-task Reinforcement Learning
Ackermann, J., Richter, O., & Wattenhofer, R. P. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-86486-6_14 · Cited by 4
Improving the Generalization of Visual Navigation Policies using Invariance Regularization
Aractingi, M., Dance, C. R., Pérez, J., & Silander, T. (2019) · Cited by 4
Estimates for the Branching Factors of Atari Games
Nelson, M. J. (2021) · IEEE Conference on Games (CoG) · doi:10.1109/cog52621.2021.9619137 · Cited by 4
Temporally-Extended ε-Greedy ExplorationPreprint
Dabney, W., Ostrovski, G., & Barreto, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.01782 · Cited by 41
Selective Dyna-style Planning Under Limited Model CapacityPreprint
Abbas, Z., Sokota, S., Talvitie, E. J., & White, M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.02418 · Cited by 7
Self-citationMultiplayer Support for the Arcade Learning EnvironmentPreprint
Terry, J. K., Black, B., & Santos, L. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.09341 · Cited by 4
The act of remembering: a study in partially observable reinforcement learningPreprint
Icarte, R. T., Valenzano, R., Klassen, T. Q., Christoffersen, P. J. K., Farahmand, A., & McIlraith, S. A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.01753 · Cited by 5
DeepAveragers: Offline Reinforcement Learning by Solving Derived Non-Parametric MDPsPreprint
Shrestha, A., Lee, S., Tadepalli, P. V., & Fern, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.08891 · Cited by 4
Large Batch Experience ReplayPreprint
Lahire, T., Geist, M., & Rachelson, E. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.01528 · Cited by 4
Fast and Data-Efficient Training of Rainbow: an Experimental Study on AtariPreprint
Schmidt, D. V., & Schmied, T. (2021) · arXiv (Cornell University) · doi:10.34726/3909 · Cited by 13
A survey of benchmarks for reinforcement learning algorithms
Stapelberg, B., & Malan, K. M. (2020) · South African Computer Journal · doi:10.18489/sacj.v32i2.746 · Cited by 9
Bairaktaris, J. A., & Johannssen, A. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.127251 · Cited by 3
AI Generality and Spearman’s Law of Diminishing Returns
Hernández‐Orallo, J. (2019) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.11388 · Cited by 3
From eye-blinks to state construction: Diagnostic benchmarks for online representation learning
Rafiee, B., Abbas, Z., Ghiassian, S., Kumaraswamy, R., Sutton, R. S., Ludvig, E. A., & White, A. (2022) · Adaptive Behavior · doi:10.1177/10597123221085039 · Cited by 8
Deep Value-Based Reinforcement Learning
Plaat, A. (2022) · Springer eBooks · doi:10.1007/978-981-19-0638-1_3 · Cited by 4
Evolutionary Ensemble Learning
Heywood, M. I. (2023) · Genetic and evolutionary computation · doi:10.1007/978-981-99-3814-8_8 · Cited by 3
Calibration of Continual Learning Models
Li, L., Piccoli, E., Cossu, A., Bacciu, D., & Lomonaco, V. (2024) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw63382.2024.00419 · Cited by 3
NeuroPong: the event-based camera driven embedded neuromorphic system
Rizzo, C. P., Gullett, B., Crumley, A., Marcum, M., Hyman, M., Earheart-Brown, C., Steed, J., Standaert, F., Schuman, C. D., & Plank, J. S. (2025) · Neuromorphic Computing and Engineering · doi:10.1088/2634-4386/add0db · Cited by 4
RL Perceptron: Generalization Dynamics of Policy Learning in High Dimensions
Patel, N., Lee, S., Mannelli, S. S., Goldt, S., & Saxe, A. (2025) · Physical Review X · doi:10.1103/physrevx.15.021051 · Cited by 7
Emergent Tangled Program Graphs in Multi-Task Learning
Kelly, S., & Heywood, M. I. (2018) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2018/740 · Cited by 3
Parallel Random Embedding with Negatively Correlated Search
Yang, Q., Yang, P., & Tang, K. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-78811-7_33 · Cited by 3
Machado, M. C., Bellemare, M. G., Talvitie, E., Veness, J., Hausknecht, M., & Bowling, M. (2018) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2018/787 · Cited by 3
Self-citationEnhanced Pub/Sub Communications for Massive IoT Traffic with SARSA Reinforcement Learning
Arruda, C. E., Moraes, P. F., Agoulmine, N., & Martins, J. S. B. (2020) · HAL · doi:10.5281/zenodo.4413404 · Cited by 3
Fast and Data Efficient Reinforcement Learning from Pixels via Non-parametric Value Approximation
Long, A., Blair, A., & van Hoof, H. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i7.20728 · Cited by 3
Data augmented Approach to Optimizing Asynchronous Actor-Critic Methods
Sandeep Varma, N., Pradyumna Rahul, K., & Sinha, V. (2022) · International Conference on Distributed Computing and Electrical Circuits and Electronics (ICDCECE) · doi:10.1109/icdcece53908.2022.9792764 · Cited by 3
Azad, A. S., Kim, E., Wu, Q., Lee, K., Stoica, I. G., Abbeel, P., Sangiovanni-Vincentelli, A. L., & Seshia, S. A. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i6.20549 · Cited by 3
ToyBox: Better Atari Environments for Testing Reinforcement Learning AgentsPreprint
Foley, J., Tosch, E., Clary, K., & Jensen, D. D. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1812.02850 · Cited by 5
Beyond Exponentially Discounted Sum: Automatic Learning of Return FunctionPreprint
Wang, Y., Ye, Q., & Liu, T. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.11591 · Cited by 5
Game AI Research with Fast Planet Wars VariantsPreprint
Lucas, S. M. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.08544 · Cited by 3
Optimistic Proximal Policy OptimizationPreprint
Imagawa, T., Hiraoka, T., & Tsuruoka, Y. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.11075 · Cited by 6
GPU-Accelerated Atari Emulation for Reinforcement LearningPreprint
Dalton, S., Frosio, I., & Garland, M. (2019) · arXiv (Cornell University) · Cited by 3
Navigating the Landscape of Games.Preprint
Omidshafiei, S., Tuyls, K., Czarnecki, W. M., Santos, F. C., Rowland, M., Connor, J. T., Hennes, D., Müller, P. F., Pérolat, J., Vylder, B. D., et al. (2020) · arXiv (Cornell University) · Cited by 3
Learning Abstract Models for Strategic Exploration and Fast Reward TransferPreprint
Liu, E. Z., Keramati, R., Seshadri, S., Guu, K., Pasupat, P., Brunskill, E., & Liang, P. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.05896 · Cited by 7
Scenic4RL: Programmatic Modeling and Generation of Reinforcement Learning EnvironmentsPreprint
Azad, A. S., Kim, E., Wu, Q., Lee, K., Stoica, I. G., Abbeel, P., & Seshia, S. A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.10365 · Cited by 3
An evaluation of unity ML-Agents toolkit for learning boss strategies
Metz, L. A. P. (2020) · Skemman · Cited by 3
When Less May Be More: Exploring Similarity to Improve Experience Replay
Neves, D. E., Ishitani, L., & Patrocínio, Z. K. G. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-21689-3_8 · Cited by 2
Learning to Generalize With Object-Centric Agents in the Open World Survival Game Crafter
Stanić, A., Tang, Y., Ha, D., & Schmidhuber, J. (2023) · IEEE Transactions on Games · doi:10.1109/tg.2023.3276849 · Cited by 9
KUROSU, H., Kurodo, Y., & Manabe, Y. (2023) · Journal of Japan Society for Fuzzy Theory and Intelligent Informatics · doi:10.3156/jsoft.35.4_759 · Cited by 2
Zhang, Y., Shi, D., Wu, Y., Zhang, Y., Wang, L., & She, F. (2021) · Lecture notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering · doi:10.1007/978-3-030-67537-0_36 · Cited by 2
Do Deep Reinforcement Learning Agents Model Intentions?
Matiisen, T., Labash, A., Majoral, D., Aru, J., & Vicente, R. (2022) · Stats · doi:10.3390/stats6010004 · Cited by 7
Embodied Cognition and Multi-Agent Behavioral Emergence
Silvey, P. E., & Norman, M. D. (2018) · Springer proceedings in complexity · doi:10.1007/978-3-319-96661-8_20 · Cited by 2
Stochastic Activation Actor Critic Methods
Shang, W., van der Wal, D., van Hoof, H., & Welling, M. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-46133-1_7 · Cited by 2
LevDoom: A Benchmark for Generalization on Level Difficulty in Reinforcement Learning
Tomilin, T., Dai, T., Fang, M., & Pechenizkiy, M. (2022) · IEEE Conference on Games (CoG) · doi:10.1109/cog51982.2022.9893707 · Cited by 2
Gathering Physical Particles with a Global Magnetic Field Using Reinforcement Learning
Konitzny, M., Lu, Y., Leclerc, J., Fekete, S. P., & Becker, A. T. (2022) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros47612.2022.9982256 · Cited by 2
A Data-Efficient Training Method for Deep Reinforcement Learning
Feng, W., Han, C., Lian, F., & Liu, X. (2022) · Electronics · doi:10.3390/electronics11244205 · Cited by 5
Learning Uncertainty-Aware Temporally-Extended Actions
Lee, J., Park, S. J., Tang, Y., & Oh, M.-H. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i12.29241 · Cited by 2
Exploration by Maximizing Rényi Entropy for Reward-Free RL FrameworkPreprint
Zhang, C., Cai, Y., Huang, L., & Li, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.06193 · Cited by 2
Softmax with Regularization: Better Value Estimation in Multi-Agent Reinforcement Learning.Preprint
Pan, L., Rashid, T., Peng, B., Huang, L., & Whiteson, S. (2021) · arXiv (Cornell University) · Cited by 3
Self-Consistent Models and ValuesPreprint
Farquhar, G., Baumli, K., Marinho, Z., Filos, A., Hessel, M., van Hasselt, H. P., & Silver, D. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.12840 · Cited by 3
Learning state-action features for general game playing
Soemers, D. J. N. J. (2023) · Research Publications (Maastricht University) · doi:10.26481/dis.20230425ds · Cited by 1
Iterative improvements from feedback for language modelsPreprint
Li, Y. (2023) · doi:10.14293/pr2199.000220.v1 · Cited by 3
Dizon-Paradis, O., Wormald, S., Capecci, D., Bhandarkar, A., & Woodard, D. (2023) · doi:10.36227/techrxiv.23739099 · Cited by 1
A multi-step on-policy deep reinforcement learning method assisted by off-policy policy evaluation
Zhang, H., Ma, H., Mersha, B. W., & Jin, Y. (2024) · Applied Intelligence · doi:10.1007/s10489-024-05508-9 · Cited by 1
Yang, T., Hou, J., Zhang, P., Hu, J., & Zhang, Y. (2026) · Neural Networks · doi:10.1016/j.neunet.2026.108735 · Cited by 1
Analyzing Interpretable Visual Control Policy Search and Synthesis
Berghegger, C., Torre, C. D. L., Nadizar, G., Lavinas, Y. C., Simoncini, D., Luga, H., Wilson, D. G., & Cussat‐Blanc, S. (2026) · ACM Transactions on Evolutionary Learning and Optimization · doi:10.1145/3799432 · Cited by 1
CR-Eyes: A Computational Rational Model of Visual Sampling Behavior in Atari Games
Lorenz, M., Konzack, N., Lingler, A., Wintersberger, P., & Ebel, P. (2026) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3772363.3798498 · Cited by 1
Maybe a few considerations in Reinforcement Learning Research
Azizzadenesheli, K. (2019) · Cited by 1
Source tasks selection for transfer deep reinforcement learning: a case of study on Atari games
García-Ramírez, J., Morales, E. F., & Escalante, H. J. (2021) · Neural Computing and Applications · doi:10.1007/s00521-021-06419-3 · Cited by 1
NAEM: Noisy Attention Exploration Module for Deep Reinforcement Learning
Cai, Z., Lee, F., Hu, C., Kotani, K., & Chen, Q. (2021) · IEEE Access · doi:10.1109/access.2021.3128558 · Cited by 1
Deterministic policy optimization with clipped value expansion and long-horizon planning
Gao, S., Shi, H., Wang, F., Wang, Z., Zhang, S., Li, Y., & Sun, Y. (2022) · Neurocomputing · doi:10.1016/j.neucom.2022.02.022 · Cited by 1
Utilizing Hidden Observations to Enhance the Performance of the Trained Agent
Jang, S., & Lee, J. (2022) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2022.3186508 · Cited by 1
Revisiting Deep Attention Recurrent Networks
Duarte, F. F., Lau, N., Pereira, A., & Reis, L. P. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-49008-8_10 · Cited by 1
Li, J., & Allan, V. H. (2024) · IEEE Annual Computing and Communication Workshop and Conference (CCWC) · doi:10.1109/ccwc60891.2024.10427616 · Cited by 1
Distributional Reinforcement Learning with Sample-set Bellman Update
Zhang, W., Wang, J., & Yu, Y. (2024) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra57147.2024.10610740 · Cited by 1
Guo, J., Han, C., Ma, Y., & Zhang, C. (2024) · IISE Transactions · doi:10.1080/24725854.2024.2425292 · Cited by 1
Roy, S., Saha, S., & Das, S. (2025) · IEEE Transactions on Emerging Topics in Computational Intelligence · doi:10.1109/tetci.2025.3593841 · Cited by 1
Agent-based Learning for Auto-Navigation within the Virtual City
Dawn, S., Saraogi, U., & Thakur, U. S. (2020) · International Conference on Computational Performance Evaluation (ComPE) · doi:10.1109/compe49325.2020.9199993 · Cited by 1
Sensitivity and robustness of end-to-end data-driven approach for building performance optimization
Naug, A., Quiñones-Grueiro, M., & Biswas, G. (2021) · ACM International Conference on Systems for Energy-Efficient Buildings, Cities, and Transportation (BuildSys) · doi:10.1145/3486611.3488728 · Cited by 1
Show me the Way: Intrinsic Motivation from DemonstrationsPreprint
Hussenot, L., Dadashi, R., Geist, M., & Pietquin, O. (2020) · arXiv (Cornell University) · Cited by 1
The Intersection of Planning and Learning
Moerland, T. M. (2021) · Research Repository (Delft University of Technology) · doi:10.4233/uuid:5437884e-0078-4b36-b2c7-c6edfea3b418 · Cited by 1
Correcting Momentum in Temporal Difference LearningPreprint
Bengio, E., Pineau, J., & Precup, D. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.03955 · Cited by 1
Utilizing Skipped Frames in Action Repeats via Pseudo-ActionsPreprint
Hashimoto, T., & Tsuruoka, Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.03041 · Cited by 1
Castro, P. S., Kastner, T., Panangaden, P., & Rowland, M. (2021) · arXiv (Cornell University) · Cited by 1
Towards robust and domain agnostic reinforcement learning competitionsPreprint
Guss, W. H., Milani, S., Topin, N., Houghton, B., Mohanty, S. P., Melnik, A., Harter, A., Buschmaas, B., Jaster, B., Berganski, C., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.03748 · Cited by 1
Improve Agents without Retraining: Parallel Tree Search with Off-Policy CorrectionPreprint
Hallak, A., Dalal, G., Dalton, S., Frosio, I., Mannor, S., & Chechik, G. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.01715 · Cited by 2
Improving Experience Replay through Modeling of Similar Transitions' SetsPreprint
Neves, D. E., Batisteli, J. P. O., Lopes, E. F., Ishitani, L., & Patrocínio, Z. K. G. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.06907 · Cited by 1
Metadata
- Authors
- Machado, M. C., Bellemare, M. G., Talvitie, E., Veness, J., Hausknecht, M., & Bowling, M.
- Affiliations
- University of Alberta (CA)
- Franklin & Marshall College (US)
- Google DeepMind (United Kingdom) (GB)
- Microsoft (United States) (US)
- Venue
- Journal of Artificial Intelligence Research
- Publisher
- AI Access Foundation
- Published
- 2018-03-19
- Volume
- 61
- Pages
- 523-562
- ISSN
- 1076-9757 (ISSN-L)
- OpenAlex
- W2754879180
- Cited by
- 359
- References
- 93
- Citations by year
- 2026: 9 · 2025: 16 · 2024: 21 · 2023: 27 · 2022: 30 · 2021: 71 · 2020: 71 · 2019: 73 · 2018: 39
- FWCI
- 35.74
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- cc-by
- Locations
- Journal of Artificial Intelligence Research (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1552533)
- Compute Canada
- Alberta Machine Intelligence Institute
- DeepMind
- Alberta Innovates - Technology Futures
- Alberta Innovates
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence in Games
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
- Keywords
- Arcade Learning Environment, evaluation methodology, benchmark results, deep Q-network, game modes, evaluation best practices
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
