RUDDER: Return Decomposition for Delayed Rewards
Open accessThe DOI matches its OpenAlex record; 29 of the 29 references it cites were found in a registry or an index.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Self-citationDeep learning in neural networks: an overview
Schmidhuber, J. (2015) · Neural Networks · doi:10.1016/j.neunet.2014.09.003 · Cited by 18,570
Mastering the game of Go with deep neural networks and tree search
Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., van den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al. (2016) · Nature · doi:10.1038/nature16961 · Cited by 15,612
Proximal Policy Optimization Algorithms
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017) · arXiv · doi:10.48550/arxiv.1707.06347 · Cited by 10,625
Playing Atari with Deep Reinforcement Learning
Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., & Riedmiller, M. (2013) · arXiv · doi:10.48550/arxiv.1312.5602 · Cited by 9,471 · cited 1×
“to receive the delayed reward to secure it for the future. Recent successful reinforcement learning methods provide solutions to one or more of these phases. Most prominent are Deep Q-Networks (DQNs) [81, 82], which combine Q-learning with convolutional neural networks for visual reinforcement learning [69]. The succe…”
BackgroundSpeech recognition with deep recurrent neural networks
Graves, A., Mohamed, A.-R., & Hinton, G. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6638947 · Cited by 8,893 · cited 1×
“hnique in speech and language processing. LSTM networks have been overwhelming successful in different speech and language applications, including handwriting recognition [37], generation of writings [38], language modeling and identification [35, 146], automatic language translation [126], speech recognition [107, 29]…”
MethodAsynchronous Methods for Deep Reinforcement Learning
Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lillicrap, T. P., Harley, T., Silver, D., & Kavukcuoglu, K. (2016) · arXiv · doi:10.48550/arxiv.1602.01783 · Cited by 7,132 · cited 5×
“A3C with asynchronous gradient descent [80] and Ape-X DPG [58] also rely on parallel policies.”
BackgroundMethodInfluentialGreff, K., Srivastava, R. K., Koutník, J., Steunebrink, B. R., & Schmidhuber, J. (2015) · arXiv · doi:10.48550/arxiv.1503.04069 · Cited by 7,073
Greff, K., Srivastava, R. K., Koutnik, J., Steunebrink, B. R., & Schmidhuber, J. (2017) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2016.2582924 · Cited by 6,955 · cited 1×
“rning, or biological sequences. That peephole connections can be removed without performance decrease, was recently confirmed in a large assessment, where different LSTM architectures have been tested [40]. While LSTM networks are highly successful in various applications, the central memory cell architecture was not m…”
BackgroundLearning to predict by the methods of temporal differences
Sutton, R. S. (1988) · Machine Learning · doi:10.1007/bf00115009 · Cited by 5,148 · cited 1×
“ature. The function can generalize this correlation to the beginning of an episode and predict already high reward for states possessing the same feature. Multi-step temporal difference (TD) learning [127, 128] improved both DQNs and policy gradients [47, 80]. AlphaGo and AlphaZero learned to play Go and Chess better…”
BackgroundTrust region policy optimization
Schulman, J., Levine, S., Moritz, P., Jordan, M. I., & Abbeel, P. (2015) · arXiv · doi:10.48550/arxiv.1502.05477 · Cited by 4,578
On Pixel-Wise Explanations for Non-Linear Classifier Decisions by Layer-Wise Relevance Propagation
Bach, S., Binder, A., Montavon, G., Klauschen, F., Müller, K.-R., & Samek, W. (2015) · PLOS ONE · doi:10.1371/journal.pone.0130140 · Cited by 4,524 · cited 3×
“We can use any contribution analysis method, but we specifically consider three methods: (A) differences of return predictions, (B) integrated gradients (IG) [40], and (C) layer-wise relevance propagation (LRP) [3].”
MethodAxiomatic attribution for deep networks
Sundararajan, M., Taly, A., & Yan, Q. (2017) · arXiv · doi:10.48550/arxiv.1703.01365 · Cited by 4,087
Deep Reinforcement Learning with Double Q-Learning
Van Hasselt, H., Guez, A., & Silver, D. (2016) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v30i1.10295 · Cited by 3,318
Dueling Network Architectures for Deep Reinforcement Learning
Wang, Z., Schaul, T., Hessel, M., van Hasselt, H., Lanctot, M., & de Freitas, N. (2015) · arXiv · doi:10.48550/arxiv.1511.06581 · Cited by 3,300
Long short-term memory recurrent neural network architectures for large scale acoustic modeling
Sak, H., Senior, A., & Beaufays, F. (2014) · Interspeech 2014 · doi:10.21437/interspeech.2014-80 · Cited by 2,973
The Vanishing Gradient Problem During Learning Recurrent Neural Nets and Problem Solutions
Hochreiter, S. (1998) · International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems · doi:10.1142/s0218488598000094 · Cited by 2,939
Self-citationMethods for interpreting and understanding deep neural networks
Montavon, G., Samek, W., & Müller, K.-R. (2018) · Digital Signal Processing · doi:10.1016/j.dsp.2017.10.011 · Cited by 2,665 · cited 4×
“sis. For decomposing a linear gone can use the Taylor decomposition (a linear approximation) of gwith respect to the h[3, 75]. A non-linear gcan be decomposed by layerwise relevance propagation (LRP) [3,76] or integrated gradients (IG) [112]. Reward Redistribution based on Return Decomposition. The optimal policies do…”
BackgroundMethodInfluentialThe Arcade Learning Environment: An Evaluation Platform for General Agents
Bellemare, M. G., Naddaf, Y., Veness, J., & Bowling, M. (2013) · Journal of Artificial Intelligence Research · doi:10.1613/jair.3912 · Cited by 2,161 · cited 2×
“Atari Games Bowling and Venture: We investigated the Atari games supported by the Arcade 245 Learning Environment [7] and OpenAI Gym [8] for games with delayed reward.”
MethodRainbow: Combining Improvements in Deep Reinforcement Learning
Hessel, M., Modayil, J., Van Hasselt, H., Schaul, T., Ostrovski, G., Dabney, W., Horgan, D., Piot, B., Azar, M., & Silver, D. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11796 · Cited by 1,846
IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
Espeholt, L., Soyer, H., Munos, R., Simonyan, K., Mnih, V., Ward, T., Doron, Y., Firoiu, V., Harley, T., Dunning, I., et al. (2018) · arXiv · doi:10.48550/arxiv.1802.01561 · Cited by 1,708
Explaining nonlinear classification decisions with deep taylor decomposition
Montavon, G., Lapuschkin, S., Binder, A., Samek, W., & Müller, K.-R. (2017) · Pattern Recognition · doi:10.1016/j.patcog.2016.11.008 · Cited by 1,357
Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm
Silver, D., Hubert, T., Schrittwieser, J., Antonoglou, I., Lai, M., Guez, A., Lanctot, M., Sifre, L., Kumaran, D., Graepel, T., et al. (2017) · arXiv · doi:10.48550/arxiv.1712.01815 · Cited by 1,102 · cited 2×
“methods like temporal difference (TD), Monte Carlo (MC), and Monte Carlo Tree Search (MCTS) take a forward view approach2 by estimating the expected future return [115]. MCTS is the core of AlphaZero [105, 106], which learned to play Chess and Go better than human professionals. The future return is estimated by rolli…”
MethodDistributed Prioritized Experience Replay
Horgan, D., Quan, J., Budden, D., Barth-Maron, G., Hessel, M., van Hasselt, H., & Silver, D. (2018) · arXiv · doi:10.48550/arxiv.1803.00933 · Cited by 963
Simple statistical gradient-following algorithms for connectionist reinforcement learning
Williams, R. J. (1992) · Machine Learning · doi:10.1023/a:1022672621406 · Cited by 817
Explaining Recurrent Neural Network Predictions in Sentiment Analysis
Arras, L., Montavon, G., Müller, K.-R., & Samek, W. (2017) · Proceedings of the 8th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis · doi:10.18653/v1/w17-5221 · Cited by 361 · cited 5×
“uality = 1 jg bg Tj 1 1 ; (A267) with as quality threshold of = 80% and the maximum possible error as = 10 due to the reward scaling applied. quality is furthermore clipped to be within range [0;1]. PPO model. The ppo2 implementation [21] samples from the environment using multiple agents in parallel. These agents pla…”
BackgroundMethodInfluentialHa, D., & Schmidhuber, J. (2018) · Zenodo · doi:10.5281/zenodo.1207631 · Cited by 187 · cited 2×
“ls. The future return is estimated by rolling out games until their end and, therefore, capturing delayed rewards. Recently, also forward world models using an evolution strategy have been successful [35]. However, forward view approaches fail at MDPs with delayed rewards if rewards, state transitions, or policies are…”
MethodEvolving large-scale neural networks for vision-based reinforcement learning
Koutník, J., Cuccu, G., Schmidhuber, J., & Gomez, F. (2013) · Proceedings of the 15th annual conference on Genetic and evolutionary computation · doi:10.1145/2463372.2463509 · Cited by 130
Optimizing Agent Behavior over Long Time Scales by Transporting Value
Hung, C.-C., Lillicrap, T., Abramson, J., Wu, Y., Mirza, M., Carnevale, F., Ahuja, A., & Wayne, G. (2018) · arXiv · doi:10.48550/arxiv.1810.06721 · Cited by 75
Cited by
Explaining Deep Neural Networks and Beyond: A Review of Methods and Applications
Samek, W., Montavon, G., Lapuschkin, S., Anders, C. J., & Muller, K.-R. (2021) · Proceedings of the IEEE · doi:10.1109/jproc.2021.3060483 · Cited by 1,477
“Furthermore, initial steps have been taken to use XAI beyond validation to arrive at better and more predictive models e.g. [152], [6], [8], [7].”
MethodSNAS: Stochastic Neural Architecture Search
Xie, S., Zheng, H., Liu, C., & Lin, L. (2018) · arXiv · doi:10.48550/arxiv.1812.09926 · Cited by 731
“Details are given in Appendix E. Arjona-Medina et al. (2018) prove that decomposing delayed reward to each step with gradients could resolve the delayed bias since no delayed reward exists.”
BackgroundInfluentialA survey and critique of multiagent deep reinforcement learning
Hernandez-Leal, P., Kartal, B., & Taylor, M. E. (2019) · Autonomous Agents and Multi-Agent Systems · doi:10.1007/s10458-019-09421-1 · Cited by 671
Challenges of real-world reinforcement learning: definitions, benchmarks and analysis
Dulac-Arnold, G., Levine, N., Mankowitz, D. J., Li, J., Paduraru, C., Gowal, S., & Hester, T. (2021) · Machine Learning · doi:10.1007/s10994-021-05961-4 · Cited by 645
Towards Explainable Artificial Intelligence
Samek, W., & Müller, K.-R. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-28954-6_1 · Cited by 580
Metadata
- Authors
- Arjona-Medina, J. A., Gillhofer, M., Widrich, M., Unterthiner, T., Brandstetter, J., & Hochreiter, S.
- Venue
- arXiv (Cornell University)
- Publisher
- Cornell University
- Published
- 2018-06-20
- Volume
- 32
- Pages
- 13544-13555
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1806.07857
- OpenAlex
- W2809013025
- Cited by
- 104
- References
- 29
- Type
- posted content
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.