The DOI matches its Crossref record; 18 of the 18 references it cites were found in a registry or an index.
Active Preference-Based Learning of Reward Functions
Open accessAbstract
Our goal is to efficiently learn reward functions encoding a human's preferences for how a dynamical system should act. There are two challenges with this. First, in many problems it is difficult for people to provide demonstrations of the desired system trajectory (like a high-DOF robot arm motion or an aggressive driving maneuver), or to even assign how much numerical reward an action or trajectory should get. We build on work in label ranking and propose to learn from preferences (or comparisons) instead: the person provides the system a relative preference between two trajectories. Second, the learned reward function strongly depends on what environments and trajectories were experienced during the training phase. We thus take an active learning approach, in which the system decides on what preference queries to make. A novel aspect of our work is the complexity and continuous nature of the queries: continuous trajectories of a dynamical system in environments with other moving agents (humans or robots). We contribute a method for actively synthesizing queries that satisfy the dynamics of the system. Further, we learn the reward function from a continuous hypothesis space by maximizing the volume removed from the hypothesis space by each query. We assign weights to the hypothesis space in the form of a log-concave distribution and provide a bound on the number of iterations required to converge. We show that our algorithm converges faster to the desired reward compared to approaches that are not active or that do not synthesize queries in an autonomous driving domain. We then run a user study to put our method to the test with real people.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
An Adaptive Metropolis Algorithm
Haario, H., Saksman, E., & Tamminen, J. (2001) · Bernoulli · doi:10.2307/3318737 · Cited by 2,631
Maximum Entropy Inverse Reinforcement Learning
Ziebart, B. D., Maas, A. L., Bagnell, J. A., & Dey, A. K. (2018) · doi:10.1184/r1/6555512 · Cited by 1,869
Label ranking by learning pairwise preferences
Hüllermeier, E., Fürnkranz, J., Cheng, W., & Brinker, K. (2008) · Artificial Intelligence · doi:10.1016/j.artint.2008.08.002 · Cited by 536
Scalable training of L1-regularized log-linear models
Andrew, G., & Gao, J. (2007) · Proceedings of the 24th international conference on Machine learning · doi:10.1145/1273496.1273501 · Cited by 506
Adaptive Submodularity: Theory and Applications in Active Learning and Stochastic Optimization
Golovin, D., & Krause, A. (2010) · arXiv (Cornell University) · doi:10.48550/arxiv.1003.3967 · Cited by 461
Planning for Autonomous Cars that Leverage Effects on Human Actions
Sadigh, D., Sastry, S., A. Seshia, S., & D. Dragan, A. (2016) · Robotics: Science and Systems XII · doi:10.15607/rss.2016.xii.029 · Cited by 404
Self-citationKeyframe-based Learning from Demonstration: Method and Evaluation
Akgun, B., Cakmak, M., Jiang, K., & Thomaz, A. L. (2012) · International Journal of Social Robotics · doi:10.1007/s12369-012-0160-0 · Cited by 197
Active Learning for Reward Estimation in Inverse Reinforcement Learning
Lopes, M. C., Melo, F. S., & Montesano, L. (2009) · Lecture notes in computer science · doi:10.1007/978-3-642-04174-7_3 · Cited by 162
Continuous Inverse Optimal Control with Locally Optimal Examples
Levine, S., & Koltun, V. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1206.4617 · Cited by 158
Preference-based reinforcement learning: a formal framework and a policy iteration algorithm
Fürnkranz, J., Hüllermeier, E., Cheng, W., & Park, S.-H. (2012) · Machine Learning · doi:10.1007/s10994-012-5313-8 · Cited by 130
Cited by
Planning and Decision-Making for Autonomous Vehicles
Schwarting, W., Alonso-Mora, J., & Rus, D. (2018) · Annual Review of Control, Robotics, and Autonomous Systems · doi:10.1146/annurev-control-060117-105157 · Cited by 972
How to train your robot with deep reinforcement learning: Lessons we have learned
Ibarz, J., Tan, J., Finn, C., Kalakrishnan, M., Pastor, P., & Levine, S. (2021) · The International Journal of Robotics Research · doi:10.1177/0278364920987859 · Cited by 570
Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback
Casper, S., Davies, X., Shi, C., Gilbert, T. K., Scheurer, J., Rando, J., Freedman, R., Korbak, T., Lindner, D., Freire, P., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.15217 · Cited by 101
“Two examples are using active reward learning to determine when to collect feedback and which feedback to request first [Sadigh et al., 2017], and leveraging dialogue models to learn desired feedback-seeking patterns [Krasheninnikov et al., 2022].”
BackgroundSelf-citationCognitive Architectures for Language Agents
Sumers, T. R., Yao, S., Narasimhan, K., & Griffiths, T. L. (2023) · arXiv · doi:10.48550/arxiv.2309.02427 · Cited by 798
“…language may ask for help (Ren et al., 2023; Nguyen et al., 2022b; 2019; Nguyen and Daumé III, 2019) or clarification (Biyik and Palan, 2019; Sadigh et al., 2017; Padmakumar et al., 2022; Thomason et al., 2020; Narayan-Chen et al., 2019) – or entertain or emotionally help people (Zhang et…”
BackgroundLazyDAgger: Reducing Context Switching in Interactive Imitation Learning
Hoque, R., Balakrishna, A., Putterman, C., Luo, M., Brown, D. S., Seita, D., Thananjeyan, B., Novoseller, E., & Goldberg, K. (2021) · 2021 IEEE 17th International Conference on Automation Science and Engineering (CASE) · doi:10.1109/case49439.2021.9551469 · Cited by 31
“There has also been significant recent interest in active preference queries for learning reward functions from pairwise preferences over demonstrations [7, 10, 13, 19, 34, 41].”
BackgroundThriftyDAgger: Budget-Aware Novelty and Risk Gating for Interactive Imitation Learning
Hoque, R., Balakrishna, A., Novoseller, E., Wilcox, A., Brown, D. S., & Goldberg, K. (2021) · arXiv · doi:10.48550/arxiv.2109.08273 · Cited by 31
RoboCLIP: One Demonstration is Enough to Learn Robot Policies
Sontakke, S. A., Zhang, J., Arnold, S. M. R., Pertsch, K., Bıyık, E., Sadigh, D., Finn, C., & Itti, L. (2023) · arXiv · doi:10.48550/arxiv.2310.07899 · Cited by 23
Self-citationChoice Set Misspecification in Reward Inference
Freedman, R., Shah, R., & Dragan, A. (2021) · arXiv · doi:10.48550/arxiv.2101.07691 · Cited by 6
Self-citationFeedback Loops With Language Models Drive In-Context Reward Hacking
Pan, A., Jones, E., Jagadeesan, M., & Steinhardt, J. (2024) · arXiv · doi:10.48550/arxiv.2402.06627 · Cited by 3
“Several works (Steinhardt, 2023; Casper et al., 2023; Carroll et al., 2023) discuss one possible failure mode with RLHF, where the model alters the preferences of users to improve its reward.”
BackgroundOn The Fragility of Learned Reward Functions
McKinney, L., Duan, Y., Krueger, D., & Gleave, A. (2023) · arXiv · doi:10.48550/arxiv.2301.03652 · Cited by 1
Metadata
- Authors
- Sadigh, D., Dragan, A., Sastry, S., & Seshia, S.
- Venue
- Robotics: Science and Systems XIII
- Published
- 2017-07-12
- OpenAlex
- W2735318784
- Cited by
- 255
- References
- 18
- FWCI
- 10.04
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.