Interactive Learning from Policy-Dependent Human Feedback
Open accessAbstract
This paper investigates the problem of interactively learning behaviors communicated by a human teacher using positive and negative feedback. Much previous work on this problem has made the assumption that people provide feedback for decisions that is dependent on the behavior they are teaching and is independent from the learner's current policy. We present empirical results that show this assumption to be false -- whether human trainers give a positive or negative feedback for a decision is influenced by the learner's current policy. Based on this insight, we introduce {\em Convergent Actor-Critic by Humans} (COACH), an algorithm for learning from policy-dependent feedback that converges to a local optimum. Finally, we demonstrate that COACH can successfully learn multiple behaviors on a physical robot.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Markov decision processes: Discrete stochastic dynamic programming
Hazeghi, K., & Puterman, M. L. (1995) · Journal of the American Statistical Association · doi:10.2307/2291177 · Cited by 8,037 · see citing works
A survey of robot learning from demonstration
Argall, B. D., Chernova, S., Veloso, M., & Browning, B. (2009) · Robotics and Autonomous Systems · doi:10.1016/j.robot.2008.10.024 · Cited by 3,227 · see citing works
Flagged first, then by citations
Research works printed but not linked: 24
- Baird, Leemon. Residual algorithms: Reinforcement learning with function approximation. In Proceedings of the twelfth international conference on machine learning , pp. 30-37, 1995.
- Barto, A.G., Sutton, R.S., and Anderson, C.W. Neuronlike adaptive elements that can solve difficult learning control problems. Systems, Man and Cybernetics, IEEE Transactions on , SMC-13(5):834 -846, sept.-oct. 1983.
- Bhatnagar, Shalabh, Sutton, Richard S, Ghavamzadeh, Mohammad, and Lee, Mark. Natural actor-critic algorithms. Automatica , 45(11):2471-2482, 2009.
- Clouse, Jeffery A and Utgoff, Paul E. A teaching method for reinforcement learning. In Proceedings of the Ninth International Conference on Machine Learning (ICML'92) , pp. 92-101, 1992.
- Griffith, Shane, Subramanian, Kaushik, Scholz, Jonathan, Isbell, Charles, and Thomaz, Andrea L. Policy shaping: Integrating human feedback with reinforcement learning. In Advances in Neural Information Processing Systems , pp. 2625-2633, 2013.
- Ho, Mark K, Littman, Michael L., Cushman, Fiery, and Austerweil, Joseph L. Teaching with rewards and punishments: Reinforcement or communication? In Proceedings of the 37th Annual Meeting of the Cognitive Science Society , 2015.
- Isbell, Charles, Shelton, Christian R, Kearns, Michael, Singh, Satinder, and Stone, Peter. A social reinforcement learning agent. In Proceedings of the fifth international conference on Autonomous agents , pp. 377-384. ACM, 2001.
- Knox, W Bradley and Stone, Peter. Interactively shaping agents via human reinforcement: The TAMER framework. In Proceedings of the Fifth International Conference on Knowledge Capture , pp. 9-16, 2009a.
- Knox, W Bradley and Stone, Peter. Interactively shaping agents via human reinforcement: The tamer framework. In Proceedings of the fifth international conference on Knowledge capture , pp. 9-16. ACM, 2009b.
- Knox, W Bradley and Stone, Peter. Learning nonmyopically from human-generated reward. In Proceedings of the 2013 international conference on Intelligent user interfaces , pp. 191-202. ACM, 2013.
- Knox, W Bradley, Glass, Brian D, Love, Bradley C, Maddox, W Todd, and Stone, Peter. How humans teach agents. International Journal of Social Robotics , 4(4): 409-421, 2012.
- Knox, W Bradley, Stone, Peter, and Breazeal, Cynthia. Training a robot via human feedback: A case study. In Social Robotics , pp. 460-470. Springer, 2013.
- Knox, W. Bradley Knox and Stone, Peter. Combining manual feedback with subsequent MDP reward signals for reinforcement learning. In Proc. of 9th Int. Conf. on Autonomous Agents and Multiagent Systems (AAMAS 2010) , May 2010.
- Knox, William Bradley. Learning from human-generated reward . PhD thesis, University of Texas at Austin, 2012.
- Loftin, Robert, Peng, Bei, MacGlashan, James, Littman, Michael L., Taylor, Matthew E., Huang, Jeff, and Roberts, David L. Learning behaviors via humandelivered discrete feedback: modeling implicit feedback strategies to speed up learning. Autonomous Agents and Multi-Agent Systems , 30(1):30-59, 2015.
- Maclin, Richard, Shavlik, Jude, Torrey, Lisa, Walker, Trevor, and Wild, Edward. Giving advice about preferred actions to reinforcement learners via knowledgebased kernel regression. In Proceedings of the National Conference on Artificial intelligence , volume 20, pp. 819, 2005.
- Pilarski, Patrick M, Dawson, Michael R, Degris, Thomas, Fahimi, Farbod, Carey, Jason P, and Sutton, Richard S. Online human training of a myoelectric prosthesis controller via actor-critic reinforcement learning. In 2011 IEEE International Conference on Rehabilitation Robotics , pp. 1-7. IEEE, 2011.
- Puterman, Martin L. Markov Decision ProcessesDiscrete Stochastic Dynamic Programming . John Wiley &Sons, Inc., New York, NY, 1994.
- Sutton, Richard S and Barto, Andrew G. Reinforcement learning: An introduction , volume 1. MIT press Cambridge, 1998.
- Sutton, Richard S, McAllester, David A, Singh, Satinder P, Mansour, Yishay, et al. Policy gradient methods for reinforcement learning with function approximation. In NIPS , volume 99, pp. 1057-1063, 1999.
- Tenorio-Gonzalez, Ana C, Morales, Eduardo F, and Villase˜ nor-Pineda, Luis. Dynamic reward shaping: training a robot by voice. In Advances in Artificial IntelligenceIBERAMIA 2010 , pp. 483-492. Springer, 2010.
- Thomaz, Andrea L and Breazeal, Cynthia. Robot learning via socially guided exploration. In Development and Learning, 2007. ICDL 2007. IEEE 6th International Conference on , pp. 82-87. IEEE, 2007.
- Thomaz, Andrea Lockerd and Breazeal, Cynthia. Reinforcement learning with human teachers: Evidence of feedback and guidance with implications for learning performance. In AAAI , volume 6, pp. 1000-1005, 2006.
- Watkins, Christopher J. C. H. and Dayan, Peter. Q-learning. Machine Learning , 8(3):279-292, 1992.
Cited by
Sorted by citations
Deep reinforcement learning from human preferences
Christiano, P., Leike, J., Brown, T. B., Martic, M., Legg, S., & Amodei, D. (2017) · arXiv · doi:10.48550/arxiv.1706.03741 · Cited by 1,058 · see citing works
A Survey on Transfer Learning for Multiagent Reinforcement Learning Systems
Silva, F. L. D., & Costa, A. H. R. (2019) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.11396 · Cited by 273 · see citing works
Integrating Machine Learning with Human Knowledge
Deng, C., Ji, X., Rainey, C., Zhang, J., & Lu, W. (2020) · iScience · doi:10.1016/j.isci.2020.101656 · Cited by 243 · see citing works
Wu, J., Huang, Z., Hu, Z., & Lv, C. (2023) · Engineering · doi:10.1016/j.eng.2022.05.017 · Cited by 205 · see citing works
Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback
Casper, S., Davies, X., Shi, C., Gilbert, T. K., Scheurer, J., Rando, J., Freedman, R., Korbak, T., Lindner, D., Freire, P., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.15217 · Cited by 101 · see citing works
“Although learning a reward model is efficient, it might be necessary to directly provide rewards [MacGlashan et al., 2017] for RL training in certain safety-critical situations.”
BackgroundMethodPreference Ranking Optimization for Human Alignment
Song, F., Yu, B., Li, M., Yu, H., Huang, F., Li, Y., & Wang, H. (2023) · arXiv · doi:10.48550/arxiv.2306.17492 · Cited by 77 · see citing works
Preference Ranking Optimization for Human Alignment
Song, F., Yu, B., Li, M., Yu, H., Huang, F., Li, Y., & Wang, H. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i17.29865 · Cited by 77 · see citing works
Kreutzer, J., Uyheng, J., & Riezler, S. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1165 · Cited by 38 · see citing works
Human-in-the-Loop Imitation Learning using Remote Teleoperation
Mandlekar, A., Xu, D., Martín-Martín, R., Zhu, Y., Fei-Fei, L., & Savarese, S. (2020) · arXiv · doi:10.48550/arxiv.2012.06733 · Cited by 35 · see citing works
When to Show a Suggestion? Integrating Human Feedback in AI-Assisted Programming
Mozannar, H., Bansal, G., Fourney, A., & Horvitz, E. (2023) · arXiv · doi:10.48550/arxiv.2306.04930 · Cited by 22 · see citing works
“Integrating human preferences when training machine learning models has long been studied in the literature (Knox and Stone 2008; MacGlashan et al. 2017).”
BackgroundWhen to Show a Suggestion? Integrating Human Feedback in AI-Assisted Programming
Mozannar, H., Bansal, G., Fourney, A., & Horvitz, E. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i9.28878 · Cited by 23 · see citing works
Habitat 3.0: A Co-Habitat for Humans, Avatars, and Robots
Puig, X., Undersander, E., Szot, A., Cote, M. D., Yang, T.-Y., Partsey, R., Desai, R., Clegg, A. W., Hlavac, M., Min, S. Y., et al. (2023) · arXiv · doi:10.48550/arxiv.2310.13724 · Cited by 15 · see citing works
- Authors
- MacGlashan, J., Ho, M. K., Loftin, R., Peng, B., Wang, G., Roberts, D., Taylor, M. E., & Littman, M. L.
- Affiliations
- Brown University (US)
- North Carolina State University (US)
- Washington State University (US)
- ORCID
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2017
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1701.06049
- OpenAlex
- W2580300496
- Cited by
- 163
- References
- 2
- Citations by year
- 2025: 5 · 2024: 26 · 2023: 21 · 2022: 19 · 2021: 31 · 2020: 33 · 2019: 22 · 2018: 5 · 2017: 1
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1149917, 1643614)
- National Aeronautics and Space Administration (NNX16CD07C)
- U.S. Department of Agriculture (2014-67021, 2014-67021-22174)
- Washington State University
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
- Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
- Keywords
- interactive learning, human feedback, positive and negative feedback, actor-critic, human-robot interaction, robot learning from demonstration, physical robots
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
