Addressing Function Approximation Error in Actor-Critic Methods
Open accessAbstract
In value-based reinforcement learning methods such as deep Q-learning, function approximation errors are known to lead to overestimated value estimates and suboptimal policies. We show that this problem persists in an actor-critic setting and propose novel mechanisms to minimize its effects on both the actor and the critic. Our algorithm builds on Double Q-learning, by taking the minimum value between a pair of critics to limit overestimation. We draw the connection between target networks and overestimation bias, and suggest delaying policy updates to reduce per-update error and further improve performance. We evaluate our method on the suite of OpenAI gym tasks, outperforming the state of the art in every environment tested.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Twin Delayed DDPG mitigates DDPG overestimation by introducing clipped double Q-learning, delayed policy updates and target policy smoothing.
“Twin Delayed DDPG (TD3) [128] mitigates this by introducing clipped double Q-learning, delayed policy updates and target policy smoothing.”
From Data-Driven Robotic Manipulation of Cloth-like Deformable Objects: The Present, Challenges and Future Prospects (2023) · cites this paper as [128]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Bellman, R. E. (2010) · doi:10.1515/9781400835386 · Cited by 14,588 · see citing works
Proximal Policy Optimization Algorithms
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017) · arXiv · doi:10.48550/arxiv.1707.06347 · Cited by 10,625 · see citing works
Deep Reinforcement Learning with Double Q-Learning
Van Hasselt, H., Guez, A., & Silver, D. (2016) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v30i1.10295 · Cited by 3,318 · see citing works
Deep Reinforcement Learning That Matters
Henderson, P., Islam, R., Bachman, P., Pineau, J., Precup, D., & Meger, D. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11694 · Cited by 1,685 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 36
- Anschel, O., Baram, N., and Shimkin, N. Averaged-dqn: Variance reduction and stabilization for deep reinforcement learning. In International Conference on Machine Learning , pp. 176-185, 2017.
- Barth-Maron, G., Hoffman, M. W., Budden, D., Dabney, W., Horgan, D., TB, D., Muldal, A., Heess, N., and Lillicrap, T. Distributional policy gradients. International Conference on Learning Representations , 2018.
- Bellemare, M. G., Dabney, W., and Munos, R. A distributional perspective on reinforcement learning. In International Conference on Machine Learning , pp. 449-458, 2017.
- Bellman, R. Dynamic Programming . Princeton University Press, 1957.
- Espeholt, L., Soyer, H., Munos, R., Simonyan, K., Mnih, V., Ward, T., Doron, Y., Firoiu, V., Harley, T., Dunning, I., et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. arXiv preprint arXiv:1802.01561 , 2018. · arXiv
- Fox, R., Pakman, A., and Tishby, N. Taming the noise in reinforcement learning via soft updates. In Proceedings of the Thirty-Second Conference on Uncertainty in Artificial Intelligence , pp. 202-211. AUAI Press, 2016.
- Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. arXiv preprint arXiv:1801.01290 , 2018. · arXiv
- He, F. S., Liu, Y., Schwing, A. G., and Peng, J. Learning to play in a day: Faster deep reinforcement learning by optimality tightening. arXiv preprint arXiv:1611.01606 , 2016. · arXiv
- Horgan, D., Quan, J., Budden, D., Barth-Maron, G., Hessel, M., van Hasselt, H., and Silver, D. Distributed prioritized experience replay. International Conference on Learning Representations , 2018.
- Kingma, D. and Ba, J. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014. · arXiv
- Konda, V. R. and Tsitsiklis, J. N. On actor-critic algorithms. SIAM journal on Control and Optimization , 42(4):11431166, 2003.
- Lee, D., Defourny, B., and Powell, W. B. Bias-corrected q-learning to control max-operator bias in q-learning. In Adaptive Dynamic Programming And Reinforcement Learning (ADPRL), 2013 IEEE Symposium on , pp. 93-99. IEEE, 2013.
- Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., and Wierstra, D. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971 , 2015. · arXiv
- Lin, L.-J. Self-improving reactive agents based on reinforcement learning, planning and teaching. Machine learning , 8(3-4):293-321, 1992.
- Mannor, S. and Tsitsiklis, J. N. Mean-variance optimization in markov decision processes. In International Conference on Machine Learning , pp. 177-184, 2011.
- Mannor, S., Simester, D., Sun, P., and Tsitsiklis, J. N. Bias and variance approximation in value function estimates. Management Science , 53(2):308-322, 2007.
- Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. Human-level control through deep reinforcement learning. Nature , 518(7540): 529-533, 2015.
- Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lillicrap, T., Harley, T., Silver, D., and Kavukcuoglu, K. Asynchronous methods for deep reinforcement learning. In International Conference on Machine Learning , pp. 19281937, 2016.
- Munos, R., Stepleton, T., Harutyunyan, A., and Bellemare, M. Safe and efficient off-policy reinforcement learning. In Advances in Neural Information Processing Systems , pp. 1054-1062, 2016.
- Nachum, O., Norouzi, M., Tucker, G., and Schuurmans, D. Smoothed action value functions for learning gaussian policies. arXiv preprint arXiv:1803.02348 , 2018. · arXiv
- O'Donoghue, B., Osband, I., Munos, R., and Mnih, V. The uncertainty bellman equation and exploration. arXiv preprint arXiv:1709.05380 , 2017. · arXiv
- Popov, I., Heess, N., Lillicrap, T., Hafner, R., BarthMaron, G., Vecerik, M., Lampe, T., Tassa, Y., Erez, T., and Riedmiller, M. Data-efficient deep reinforcement learning for dexterous manipulation. arXiv preprint arXiv:1704.03073 , 2017. · arXiv
- Precup, D., Sutton, R. S., and Dasgupta, S. Off-policy temporal-difference learning with function approximation. In International Conference on Machine Learning , pp. 417-424, 2001.
- Schaul, T., Quan, J., Antonoglou, I., and Silver, D. Prioritized experience replay. In International Conference on Learning Representations , Puerto Rico, 2016.
- Schulman, J., Levine, S., Abbeel, P., Jordan, M., and Moritz, P. Trust region policy optimization. In International Conference on Machine Learning , pp. 1889-1897, 2015.
- Silver, D., Lever, G., Heess, N., Degris, T., Wierstra, D., and Riedmiller, M. Deterministic policy gradient algorithms. In International Conference on Machine Learning , pp. 387-395, 2014.
- Singh, S., Jaakkola, T., Littman, M. L., and Szepesv´ ari, C. Convergence results for single-step on-policy reinforcement-learning algorithms. Machine learning , 38(3):287-308, 2000.
- Sutton, R. S. Learning to predict by the methods of temporal differences. Machine learning , 3(1):9-44, 1988.
- Sutton, R. S. and Barto, A. G. Reinforcement learning: An introduction , volume 1. MIT press Cambridge, 1998.
- Thrun, S. and Schwartz, A. Issues in using function approximation for reinforcement learning. In Proceedings of the 1993 Connectionist Models Summer School Hillsdale, NJ. Lawrence Erlbaum , 1993.
- Todorov, E., Erez, T., and Tassa, Y. Mujoco: A physics engine for model-based control. In Intelligent Robots and Systems (IROS), 2012 IEEE/RSJ International Conference on , pp. 5026-5033. IEEE, 2012.
- Uhlenbeck, G. E. and Ornstein, L. S. On the theory of the brownian motion. Physical review , 36(5):823, 1930.
- Van Hasselt, H. Double q-learning. In Advances in Neural Information Processing Systems , pp. 2613-2621, 2010.
- Van Seijen, H., Van Hasselt, H., Whiteson, S., and Wiering, M. A theoretical and empirical analysis of expected sarsa. In Adaptive Dynamic Programming and Reinforcement Learning, 2009. ADPRL'09. IEEE Symposium on , pp. 177-184. IEEE, 2009.
- Watkins, C. J. C. H. Learning from delayed rewards . PhD thesis, King's College, Cambridge, 1989.
- Wu, Y., Mansimov, E., Grosse, R. B., Liao, S., and Ba, J. Scalable trust-region method for deep reinforcement learning using kronecker-factored approximation. In Advances in Neural Information Processing Systems , pp. 5285-5294, 2017.
Cited by
Sorted by citations
Soft Actor-Critic Algorithms and Applications
Haarnoja, T., Zhou, A., Hartikainen, K., Tucker, G., Ha, S., Tan, J., Kumar, V., Zhu, H., Gupta, A., Abbeel, P., et al. (2018) · arXiv · doi:10.48550/arxiv.1812.05905 · Cited by 1,994 · see citing works
Transfer Learning in Deep Reinforcement Learning: A Survey
Zhu, Z., Lin, K., Jain, A. K., & Zhou, J. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3292075 · Cited by 861 · see citing works
How to train your robot with deep reinforcement learning: Lessons we have learned
Ibarz, J., Tan, J., Finn, C., Kalakrishnan, M., Pastor, P., & Levine, S. (2021) · The International Journal of Robotics Research · doi:10.1177/0278364920987859 · Cited by 570 · see citing works
Learning to Walk Via Deep Reinforcement Learning
Haarnoja, T., Ha, S., Zhou, A., Tan, J., Tucker, G., & Levine, S. (2019) · Robotics: Science and Systems XV · doi:10.15607/rss.2019.xv.011 · Cited by 430 · see citing works
Single and multi-agent deep reinforcement learning for AI-enabled wireless networks: A tutorial
Feriani, A., & Hossain, E. (2021) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2021.3063822 · Cited by 410 · see citing works
RLBench: The Robot Learning Benchmark & Learning Environment
James, S., Ma, Z., Arrojo, D. R., & Davison, A. J. (2020) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2020.2974707 · Cited by 405 · see citing works
Wu, J., Huang, Z., Hu, Z., & Lv, C. (2023) · Engineering · doi:10.1016/j.eng.2022.05.017 · Cited by 205 · see citing works
Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domain
Hao, J., Yang, T., Tang, H., Bai, C., Liu, J., Meng, Z., Liu, P., & Wang, Z. (2021) · arXiv · doi:10.48550/arxiv.2109.06668 · Cited by 192
Reinforcement learning for HVAC control in intelligent buildings: A technical and conceptual review
Al Sayed, K., Boodi, A., Sadeghian Broujeny, R., & Beddiar, K. (2024) · Journal of Building Engineering · doi:10.1016/j.jobe.2024.110085 · Cited by 140 · see citing works
Deep active inference as variational policy gradients
Millidge, B. (2020) · Journal of Mathematical Psychology · doi:10.1016/j.jmp.2020.102348 · Cited by 115 · see citing works
Xie, J., Ajagekar, A., & You, F. (2023) · Applied Energy · doi:10.1016/j.apenergy.2023.121162 · Cited by 109 · see citing works
Optimization for Reinforcement Learning: From a single agent to cooperative agents
Lee, D., He, N., Kamalaruban, P., & Cevher, V. (2020) · IEEE Signal Processing Magazine · doi:10.1109/msp.2020.2976000 · Cited by 113 · see citing works
Liu, X.-Y., Yang, H., Gao, J., & Wang, C. D. (2021) · Proceedings of the Second ACM International Conference on AI in Finance · doi:10.1145/3490354.3494366 · Cited by 87 · see citing works
FinRL: Deep Reinforcement Learning Framework to Automate Trading in Quantitative Finance
Liu, X.-Y., Yang, H., Gao, J., & Wang, C. (2021) · doi:10.2139/ssrn.3955949 · Cited by 46 · see citing works
Human-in-the-loop reinforcement learning in continuous-action space
Luo, B., Wu, Z., Zhou, F., & Wang, B.-C. (2024) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3289315 · Cited by 41 · see citing works
Deep reinforcement learning and its applications in medical imaging and radiation therapy: a survey
Xu, L., Zhu, S., & Wen, N. (2022) · Physics in Medicine & Biology · doi:10.1088/1361-6560/ac9cb3 · Cited by 35 · see citing works
Liu, J., Ma, Y., Chen, Y., Zhao, C., Meng, X., & Wu, J. (2025) · Energy · doi:10.1016/j.energy.2025.135056 · Cited by 35 · see citing works
Optimal control of gene regulatory networks for morphogen-driven tissue patterning
Pezzotta, A., & Briscoe, J. (2023) · Cell Systems · doi:10.1016/j.cels.2023.10.004 · Cited by 34 · see citing works
LazyDAgger: Reducing Context Switching in Interactive Imitation Learning
Hoque, R., Balakrishna, A., Putterman, C., Luo, M., Brown, D. S., Seita, D., Thananjeyan, B., Novoseller, E., & Goldberg, K. (2021) · 2021 IEEE 17th International Conference on Automation Science and Engineering (CASE) · doi:10.1109/case49439.2021.9551469 · Cited by 31 · see citing works
“Experimental Setup: For all MuJoCo environments, we use a reinforcement learning agent trained with TD3 [16] as an algorithmic supervisor.”
MethodDeep Reinforcement Active Learning for Medical Image Classification
Wang, J., Yan, Y., Zhang, Y., Cao, G., Yang, M., & Ng, M. K. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-59710-8_4 · Cited by 31 · see citing works
Abbas, A. N., Amazu, C. W., Mietkiewicz, J., Briwa, H., Perez, A. A., Baldissone, G., Demichela, M., Chasparis, G. C., Kelleher, J. D., & Leva, M. C. (2025) · International Journal of Human–Computer Interaction · doi:10.1080/10447318.2024.2391605 · Cited by 34 · see citing works
SAGCI-System: Towards Sample-Efficient, Generalizable, Compositional, and Incremental Robot Learning
Lv, J., Yu, Q., Shao, L., Liu, W., Xu, W., & Lu, C. (2022) · 2022 International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra46639.2022.9811859 · Cited by 22 · see citing works
Kadi, H. A., & Terzić, K. (2023) · Sensors · doi:10.3390/s23052389 · Cited by 19 · see citing works
“Twin Delayed DDPG (TD3) [128] mitigates this by introducing clipped double Q-learning , delayed policy updates and target policy smoothing.”
BackgroundMethodC-Learning: Learning to Achieve Goals via Recursive Classification
Eysenbach, B., Salakhutdinov, R., & Levine, S. (2020) · arXiv · doi:10.48550/arxiv.2011.08909 · Cited by 18 · see citing works
Nguyen, C. H., Shihua, J. M., & Liem, R. P. (2025) · Communications in Transportation Research · doi:10.1016/j.commtr.2025.100165 · Cited by 12 · see citing works
Measuring Visual Generalization in Continuous Control from Pixels
Grigsby, J., & Qi, Y. (2020) · arXiv · doi:10.48550/arxiv.2010.06740 · Cited by 10 · see citing works
MoDem-V2: Visuo-Motor World Models for Real-World Robot Manipulation
Lancaster, P., Hansen, N., Rajeswaran, A., & Kumar, V. (2024) · 2024 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra57147.2024.10611121 · Cited by 8 · see citing works
FOCUS: object-centric world models for robotic manipulation
Ferraro, S., Mazzaglia, P., Verbelen, T., & Dhoedt, B. (2025) · Frontiers in Neurorobotics · doi:10.3389/fnbot.2025.1585386 · Cited by 9 · see citing works
“Among RL algorithms, model-based approaches aspire to provide greater data efficiency, compared to the model-free counterparts (Fujimoto et al., 2018; Haarnoja et al., 2018).”
When Do Transformers Shine in RL? Decoupling Memory from Credit Assignment
Ni, T., Ma, M., Eysenbach, B., & Bacon, P.-L. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-2194 · Cited by 7 · see citing works
Chen, P., Liu, Q., Li, Y., & Ma, S. (2024) · 2024 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra57147.2024.10610970 · Cited by 6 · see citing works
Li, G., Zhang, G., & Qin, C. (2021) · Sensors · doi:10.3390/s21186100 · Cited by 2 · see citing works
“TD3 [26] developed a novel variant of Double Q-learning algorithm to solve the overestimation problem in actor-critic methods.”
MethodA Hybrid Pure Pursuit–PID and TD3 Control Framework for Differential-Drive Mobile Robots
Demircioglu, U., Bakir, H., Almarri, B., & Abdul Hafez, A. H. (2026) · IEEE Access · doi:10.1109/access.2026.3691502
Space-sampled Value Decay: Forgetting Mechanisms for Non-stationary Reinforcement Learning
Störck, F., Hartmann, P., Hinder, F., Neumann, K., & Hammer, B. (2026) · arXiv · doi:10.48550/arxiv.2606.11797
- Authors
- Fujimoto, S., van Hoof, H., & Meger, D.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2018
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1802.09477
- OpenAlex
- W2787938642
- Cited by
- 3,650
- References
- 4
- Citations by year
- 2026: 16 · 2025: 448 · 2024: 949 · 2023: 826 · 2022: 565 · 2021: 489 · 2020: 267 · 2019: 84 · 2018: 6
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Computational Theory and Mathematics › Adaptive Dynamic Programming Control
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence in Games
- Keywords
- actor-critic, overestimation bias, double Q-learning, target network, value-based reinforcement learning, deep Q-learning, value function estimation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
