Shared Autonomy via Deep Reinforcement Learning
Open accessAbstract
In shared autonomy, user input is combined with semi-autonomous control to achieve a common goal.The goal is often unknown ex-ante, so prior work enables agents to infer the goal from user input and assist with the task.Such methods tend to assume some combination of knowledge of the dynamics of the environment, the user's policy given their goal, and the set of possible goals the user might target, which limits their application to real-world scenarios.We propose a deep reinforcement learning framework for model-free shared autonomy that lifts these assumptions.We use human-in-the-loop reinforcement learning with neural network function approximation to learn an end-to-end mapping from environmental observation and user input to agent action values, with task reward as the only form of supervision.This approach poses the challenge of following user commands closely enough to provide the user with real-time action feedback and thereby ensure high-quality user input, but also deviating from the user's actions when they are suboptimal.We balance these two needs by discarding actions whose values fall below some threshold, then selecting the remaining action closest to the user's input.Controlled studies with users (n = 12) and synthetic pilots playing a video game, and a pilot study with users (n = 4) flying a real quadrotor, demonstrate the ability of our algorithm to assist users with real-time control tasks in which the agent cannot directly access the user's private information through observations, but receives a reward signal and user input that both depend on the user's intent.The agent learns to assist the user without access to this private information, implicitly inferring it from the user's input.This enables the assisted user to complete the task more effectively than the user or an autonomous agent could on their own.This paper is a proof of concept that illustrates the potential for deep reinforcement learning to enable flexible and practical assistive systems.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Deep reinforcement learning has been widely applied to the problem of instructional sequencing.
“As knowledge delivery platforms are increasingly becoming virtual and thereby generating more data, deep reinforcement learning has been widely applied to the problem of instructional sequencing (Reddy et al., 2017; Upadhyay et al., 2018; Pu et al., 2020; Islam et al., 2021).”
From Proactive and reactive engagement of artificial intelligence methods for education: a review (2023) · cites this paper as Reddy et al., 2017
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Human-level control through deep reinforcement learning
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015) · Nature · doi:10.1038/nature14236 · Cited by 30,357 · see citing works
Deep Reinforcement Learning with Double Q-Learning
Van Hasselt, H., Guez, A., & Silver, D. (2016) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v30i1.10295 · Cited by 3,318 · see citing works
A survey of robot learning from demonstration
Argall, B. D., Chernova, S., Veloso, M., & Browning, B. (2009) · Robotics and Autonomous Systems · doi:10.1016/j.robot.2008.10.024 · Cited by 3,227 · see citing works
Maximum Entropy Inverse Reinforcement Learning
Ziebart, B. D., Maas, A. L., Bagnell, J. A., & Dey, A. K. (2018) · doi:10.1184/r1/6555512 · Cited by 1,868 · see citing works
Rainbow: Combining Improvements in Deep Reinforcement Learning
Hessel, M., Modayil, J., Van Hasselt, H., Schaul, T., Ostrovski, G., Dabney, W., Horgan, D., Piot, B., Azar, M., & Silver, D. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11796 · Cited by 1,846 · see citing works
(2000) · doi:10.2460/ajvr.67.2.323 · Cited by 1,621 · see citing works
Deep Recurrent Q-Learning for Partially Observable MDPsPreprint
Hausknecht, M., & Stone, P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1507.06527 · Cited by 1,224 · see citing works
Reinforcement learning for robots using neural networks
Lin, L.-J. (1992) · Defense Technical Information Center (DTIC) · Cited by 836 · see citing works
Interactively shaping agents via human reinforcement: the TAMER framework
Knox, W. B., & Stone, P. (2009) · International Conference on Knowledge Capture · doi:10.1145/1597735.1597738 · Cited by 601 · see citing works
Anticipating Human Activities Using Object Affordances for Reactive Robotic Response
Koppula, H. S., & Saxena, A. (2016) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2015.2430335 · Cited by 596 · see citing works
A policy-blending formalism for shared control
Dragan, A. D., & Srinivasa, S. S. (2013) · The International Journal of Robotics Research · doi:10.1177/0278364913490324 · Cited by 387 · see citing works
Self-citationRainbow: Combining Improvements in Deep Reinforcement LearningPreprint
Hessel, M., Modayil, J., van Hasselt, H. P., Schaul, T., Ostrovski, G., Dabney, W., Horgan, D., Piot, B., Azar, M. G., & Silver, D. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1710.02298 · Cited by 377 · see citing works
Human-Robot Collaboration: A Literature Review and Augmented Reality Approach in Design
Green, S. A., Billinghurst, M., Chen, X., & Chase, J. G. (2008) · International Journal of Advanced Robotic Systems · doi:10.5772/5664 · Cited by 306 · see citing works
Reinforcement learning for mapping instructions to actions
Branavan, S. R. K., Chen, H., Zettlemoyer, L., & Barzilay, R. (2009) · doi:10.3115/1687878.1687892 · Cited by 245 · see citing works
Shared Autonomy via Hindsight Optimization
Javdani, S., Srinivasa, S. S., & Bagnell, A. (2015) · Robotics: Science and Systems · doi:10.15607/rss.2015.xi.032 · Cited by 169 · see citing works
Online human training of a myoelectric prosthesis controller via actor-critic reinforcement learning
Pilarski, P. M., Dawson, M. R. W., Degris, T., Fahimi, F., Carey, J. P., & Sutton, R. S. (2011) · International Conference on Rehabilitation Robotics · doi:10.1109/icorr.2011.5975338 · Cited by 159 · see citing works
Reinforcement learning from simultaneous human and MDP reward
Knox, W. B., & Stone, P. (2012) · doi:10.5555/2343576.2343644 · Cited by 157 · see citing works
Pérez-D’Arpino, C., & Shah, J. (2015) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2015.7140066 · Cited by 149 · see citing works
Layered acting for character animation
Dontcheva, M., Yngve, G., & Popović, Z. (2003) · ACM SIGGRAPH Conference Papers (SIGGRAPH) · doi:10.1145/1201775.882285 · Cited by 139 · see citing works
Learning Language Games through Interaction
Wang, S. I., Liang, P., & Manning, C. D. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1224 · Cited by 132 · see citing works
Autonomy infused teleoperation with application to brain computer interface controlled manipulation
Muelling, K., Venkatraman, A., Valois, J., Downey, J. E., Weiss, J. M., Javdani, S., Hebert, M., Schwartz, A. B., Collinger, J. L., & Bagnell, J. A. (2017) · Autonomous Robots · doi:10.1007/s10514-017-9622-4 · Cited by 122 · see citing works
Shanechi, M. M., Orsborn, A. L., & Carmena, J. M. (2016) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1004730 · Cited by 114 · see citing works
Recognition, prediction, and planning for assisted teleoperation of freeform tasks
Hauser, K. K. (2013) · Autonomous Robots · doi:10.1007/s10514-013-9350-3 · Cited by 102 · see citing works
Parallel autonomy in automated vehicles: Safe motion generation with minimal intervention
Schwarting, W., Alonso–Mora, J., Pauli, L., Karaman, S., & Rus, D. L. (2017) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2017.7989224 · Cited by 99 · see citing works
Human integration into robot control utilising potential fields
Aigner, P., & McCarragher, B. J. (2002) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/robot.1997.620053 · Cited by 63 · see citing works
Learning Models for Shared Control of Human-Machine Systems with Unknown Dynamics
Broad, A., Murphey, T. D., & Argall, B. (2017) · Robotics: Science and Systems · doi:10.15607/rss.2017.xiii.037 · Cited by 58 · see citing works
Comparing human-centric and robot-centric sampling for robot deep learning from demonstrations
Laskey, M., Chuck, C., Lee, J., Mahler, J., Krishnan, S., Jamieson, K., Dragan, A. D., & Goldberg, K. Y. (2017) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2017.7989046 · Cited by 57 · see citing works
Self-citationDeep TAMER: Interactive Agent Shaping in High-Dimensional State Spaces
Warnell, G., Waytowich, N. R., Lawhern, V. J., & Stone, P. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11485 · Cited by 37 · see citing works
Formal methods for semi-autonomous driving
Seshia, S. A., Sadigh, D., & Sastry, S. S. (2015) · Proceedings - ACM IEEE Design Automation Conference · doi:10.1145/2744769.2747927 · Cited by 36 · see citing works
Lin, Z., Harrison, B. E., Keech, A., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1709.03969 · Cited by 27 · see citing works
Flagged first, then by citations
Research works printed but not linked: 4
- Ray C Goertz. Manipulators used for handling radioac- tive materials. Human factors in technology, pages 425- 443, 1963.
- Stefanos Nikolaidis, Yu Xiang Zhu, David Hsu, and Siddhartha Srinivasa. Human-robot mutual adaptation in shared autonomy. arXiv preprint arXiv:1701.07851, 2017. · arXiv
- Martin Riedmiller. Neural fitted q iteration-first experi- ences with a data efficient neural reinforcement learn- ing method. In ECML, volume 3720, pages 317-328. Springer, 2005.
- Christopher JCH Watkins and Peter Dayan. Q-learning. Machine learning, 8(3-4):279-292, 1992.
Cited by
Sorted by citations
Proactive and reactive engagement of artificial intelligence methods for education: a review
Mallik, S., & Gangopadhyay, A. (2023) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2023.1151391 · Cited by 139 · see citing works
LazyDAgger: Reducing Context Switching in Interactive Imitation Learning
Hoque, R., Balakrishna, A., Putterman, C., Luo, M., Brown, D. S., Seita, D., Thananjeyan, B., Novoseller, E., & Goldberg, K. (2021) · 2021 IEEE 17th International Conference on Automation Science and Engineering (CASE) · doi:10.1109/case49439.2021.9551469 · Cited by 31 · see citing works
“of human advice can be unintuitive, since the learner may visit states that are significantly far from those the human supervisor would visit, making it difficult for humans to judge what correct behavior looks like without interacting with the environment themselves [38, 45].”
Background
- Authors
- Reddy, S., Dragan, A., & Levine, S.
- Affiliations
- University of California, Berkeley (US)
- Venue
- Robotics: Science and Systems XIV
- Published
- 2018-06-26
- OpenAlex
- W2786110872
- Cited by
- 175
- References
- 30
- Citations by year
- 2026: 8 · 2025: 19 · 2024: 24 · 2023: 22 · 2022: 20 · 2021: 29 · 2020: 29 · 2019: 19 · 2018: 5
- FWCI
- 8.37
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Robotics: Science and Systems (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation
- Air Force Office of Scientific Research
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
- Social Sciences › Psychology › Social Psychology › Human-Automation Interaction and Safety
- Keywords
- shared autonomy, human-in-the-loop reinforcement learning, model-free reinforcement learning, assistive systems, user intention recognition, human-robot interaction, video game control
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
