CoChat
Sign Up
PreprintCited by 31Reinforcement Learning in RoboticsRobot Manipulation and LearningMachine Learning and Algorithms

ThriftyDAgger: Budget-Aware Novelty and Risk Gating for Interactive Imitation Learning

Open access
Hoque, Ryan, Balakrishna, Ashwin, Novoseller, Ellen, Wilcox, Albert, Brown, Daniel S., Goldberg, Ken·2021·arXiv
DOI 10.48550/arxiv.2109.08273arXiv 2109.08273
Open this paper in CoChat — free accountOpen PDF

Abstract

Effective robot learning often requires online human feedback and interventions that can cost significant human time, giving rise to the central challenge in interactive imitation learning: is it possible to control the timing and length of interventions to both facilitate learning and limit burden on the human supervisor? This paper presents ThriftyDAgger, an algorithm for actively querying a human supervisor given a desired budget of human interventions. ThriftyDAgger uses a learned switching policy to solicit interventions only at states that are sufficiently (1) novel, where the robot policy has no reference behavior to imitate, or (2) risky, where the robot has low confidence in task completion. To detect the latter, we introduce a novel metric for estimating risk under the current robot policy. Experiments in simulation and on a physical cable routing experiment suggest that ThriftyDAgger's intervention criteria balances task performance and supervisor burden more effectively than prior algorithms. ThriftyDAgger can also be applied at execution time, where it achieves a 100% success rate on both the simulation and physical tasks. A user study (N=10) in which users control a three-robot fleet while also performing a concentration task suggests that ThriftyDAgger increases human and robot performance by 58% and 80% respectively compared to the next best algorithm while reducing supervisor burden.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2109.08273

Cites

5 of 5 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Prior work has attempted to develop systems that can selectively query humans for demonstrations when they are needed, in order to reduce human operator time and burden.

    “Prior work has also attempted to develop systems that can selectively query humans for demonstrations when they are needed, in order to reduce human operator time and burden [67-70].”

    From MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrations (2023) · cites this paper as [69]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 42

  1. [3][3] T. Osa, J. Pajarinen, G. Neumann, J. A. Bagnell, P. Abbeel, and J. Peters. An algorithmic perspective on imitation learning. arXiv preprint arXiv:1811.06711 , 2018. - · arXiv
  2. [4][4] D. A. Pomerleau. Efficient training of artificial neural networks for autonomous navigation. Neural Computation , 3(1), 1991. -
  3. [5][5] Y. Pan, C.-A. Cheng, K. Saigol, K. Lee, X. Yan, E. Theodorou, and B. Boots. Agile autonomous driving using end-to-end deep imitation learning. In Proc. Robotics: Science and Systems (RSS) , 2018. -
  4. [6][6] F. Codevilla, M. M¨ uller, A. L´ opez, V. Koltun, and A. Dosovitskiy. End-to-end driving via conditional imitation learning. In 2018 IEEE International Conference on Robotics and Automation (ICRA) , pages 4693-4700. IEEE, 2018. -
  5. [7][7] C. Finn, T. Yu, T. Zhang, P. Abbeel, and S. Levine. One-shot visual imitation learning via meta-learning. Conf. on Robot Learning (CoRL) , 2017. -
  6. [8][8] B. Fang, S. Jia, D. Guo, M. Xu, S. Wen, and F. Sun. Survey of imitation learning for robotic manipulation. International Journal of Intelligent Robotics and Applications , 3(4):362-369, 2019. -
  7. [9][9] A. Ganapathi, P. Sundaresan, B. Thananjeyan, A. Balakrishna, D. Seita, J. Grannen, M. Hwang, R. Hoque, J. E. Gonzalez, N. Jamali, K. Yamane, S. Iba, and K. Goldberg. Learning Dense Visual Correspondences in Simulation to Smooth and Fold Real Fabrics. In Proc. IEEE Int. Conf. Robotics and Automation (ICRA) , 2021. -
  8. [10][10] P. Sundaresan, P. Grannen, B. Thananjeyan, A. Balakrishna, M. Laskey, K. Stone, J. E. Gonzalez, and K. Goldberg. Learning interpretable and transferable rope manipulation policies using depth sensing and dense object descriptors. In International Conference on Robotics and Automation (ICRA) . IEEE, 2020. -
  9. [11][11] O. Kroemer, S. Niekum, and G. Konidaris. A review of robot learning for manipulation: Challenges, representations, and algorithms. J. Mach. Learn. Res. , 22:30-1, 2021. -
  10. [12][12] S. Ross, G. J. Gordon, and J. A. Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In International Conference on Artificial Intelligence and Statistics (AISTATS) , 2011. -
  11. [13][13] M. Laskey, J. Lee, R. Fox, A. Dragan, and K. Goldberg. DART: Noise injection for robust imitation learning. In Conf. on Robot Learning (CoRL) , 2017. -
  12. [15][15] M. Kelly, C. Sidrane, K. Driggs-Campbell, and M. J. Kochenderfer. HG-DAgger: Interactive imitation learning with human experts. In Proc. IEEE Int. Conf. Robotics and Automation (ICRA) , 2019. -
  13. [16][16] J. Spencer, S. Choudhury, M. Barnes, M. Schmittle, M. Chiang, P. Ramadge, and S. Srinivasa. Learning from interventions: Human-robot interaction as both explicit and implicit feedback. In Proc. Robotics: Science and Systems (RSS) , 2020. -
  14. [17][17] S. Jauhri, C. Celemin, and J. Kober. Interactive imitation learning in state-space. In Conference on Robot Learning (CoRL) . PMLR, 2020. -
  15. [19][19] J. Zhang and K. Cho. Query-efficient imitation learning for end-to-end autonomous driving. In Association for the Advancement of Artificial Intelligence (AAAI) , 2017. -
  16. [20][20] J. W. Crandall, M. A. Goodrich, D. R. Olsen, and C. W. Nielsen. Validating human-robot interaction schemes in multitasking environments. IEEE Transactions on Systems, Man, and Cybernetics-Part A: Systems and Humans , 35(4):438-449, 2005. -
  17. [21][21] J. Y. Chen, M. J. Barnes, and M. Harper-Sciarini. Supervisory control of multiple robots: Human-performance issues and user-interface design. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews) , 41(4):435-454, 2010. -
  18. [22][22] G. Swamy, S. Reddy, S. Levine, and A. D. Dragan. Scaled autonomy: Enabling human operators to control robot fleets. In 2020 IEEE International Conference on Robotics and Automation (ICRA) , pages 5942-5948. IEEE, 2020. -
  19. [23][23] D. Kalashnikov, A. Irpan, P. Pastor, J. Ibarz, A. Herzog, E. Jang, D. Quillen, E. Holly, M. Kalakrishnan, V. Vanhoucke, and S. Levine. Qt-opt: Scalable deep reinforcement learning for visionbased robotic manipulation. Conference on Robot Learning (CoRL) , 2018. -
  20. [25][25] M. Laskey, S. Staszak, W. Hsieh, J. Mahler, F. Pokorny, A. Dragan, and K. Goldberg. SHIV: Reducing supervisor burden using support vectors for efficient learning from demonstrations in high dimensional state spaces. In Proc. IEEE Int. Conf. Robotics and Automation (ICRA) , 2016. -
  21. [26][26] K. Menda, K. Driggs-Campbell, and M. J. Kochenderfer. EnsembleDAgger: A Bayesian Approach to Safe Imitation Learning. In Proc. IEEE/RSJ Int. Conf. on Intelligent Robots and Systems (IROS) , 2019. -
  22. [28][28] S. Arora and P. Doshi. A survey of inverse reinforcement learning: Challenges, methods and progress. arXiv preprint arXiv:1806.06877 , 2018. - · arXiv
  23. [29][29] A. J. Ijspeert, J. Nakanishi, H. Hoffmann, P. Pastor, and S. Schaal. Dynamical movement primitives: learning attractor models for motor behaviors. Neural computation , 25(2), 2013. -
  24. [30][30] A. Paraschos, C. Daniel, J. R. Peters, and G. Neumann. Probabilistic movement primitives. In Advances in neural information processing systems , pages 2616-2624, 2013. -
  25. [31][31] F. Torabi, G. Warnell, and P. Stone. Behavioral cloning from observation. In Proceedings of the 27th International Joint Conference on Artificial Intelligence (IJCAI) , 7 2018. -
  26. [32][32] F. Codevilla, E. Santana, L. A. M., and A. Gaidon. Exploring the limitations of behavior cloning for autonomous driving. International Conference on Computer Vision , 2019. -
  27. [33][33] P. Abbeel and A. Y. Ng. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the twenty-first international conference on Machine learning , page 1, 2004. -
  28. [34][34] D. S. Brown, W. Goo, and S. Niekum. Better-than-demonstrator imitation learning via automaticaly-ranked demonstrations. In Conference on Robot Learning (CoRL) , 2019. -
  29. [35][35] A. Balakrishna*, B. Thananjeyan*, J. Lee, F. Li, A. Zahed, J. E. Gonzalez, and K. Goldberg. On-policy robot imitation learning from a converging supervisor. In Conference on Robot Learning (CoRL) . PMLR, 2019. -
  30. [36][36] K. Judah, A. Fern, and T. Dietterich. Active imitation learning via state queries. In Proceedings of the ICML workshop on combining learning strategies to reduce label cost . Citeseer, 2011. -
  31. [37][37] D. Sadigh, A. D. Dragan, S. S. Sastry, and S. A. Seshia. Active preference-based learning of reward functions. In Proceedings of Robotics: Science and Systems (RSS) , 2017. -
  32. [38][38] P. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei. Deep reinforcement learning from human preferences. In Proc. Advances in Neural Information Processing Systems (NeurIPS) , 2017. -
  33. [39][39] B. Ibarz, J. Leike, T. Pohlen, G. Irving, S. Legg, and D. Amodei. Reward learning from human preferences and demonstrations in Atari. In Advances in Neural Information Processing Systems , 2018. -
  34. [40][40] M. Palan, N. C. Landolfi, G. Shevchuk, and D. Sadigh. Learning reward functions by integrating human demonstrations and preferences. In Proceedings of Robotics: Science and Systems (RSS) , 2019. -
  35. [41][41] E. Bıyık, M. Palan, N. C. Landolfi, D. P. Losey, and D. Sadigh. Asking easy questions: A user-friendly approach to active reward learning. arXiv preprint arXiv:1910.04365 , 2019. - · arXiv
  36. [42][42] D. Brown, R. Coleman, R. Srinivasan, and S. Niekum. Safe imitation learning via fast Bayesian reward inference from preferences. In International Conference on Machine Learning , pages 1165-1177. PMLR, 2020. -
  37. [43][43] S. Reddy, A. D. Dragan, and S. Levine. Shared autonomy via deep reinforcement learning. Proc. Robotics: Science and Systems (RSS) , 2018. -
  38. [45][45] M. L. Puterman. Markov decision processes: discrete stochastic dynamic programming . John Wiley & Sons, 2014. -
  39. [47][47] B. K. Bose. An adaptive hysteresis-band current control technique of a voltage-fed PWM inverter for machine drive system. IEEE Transactions on Industrial Electronics , 37(5), 1990. doi:10.1109/41.103436. - · DOI
  40. [48][48] Y. Zhu, J. Wong, A. Mandlekar, and R. Mart´ ın-Mart´ ın. Robosuite: A modular simulation framework and benchmark for robot learning. In arXiv preprint arXiv:2009.12293 , 2020. - · arXiv
  41. [50][50] P. Kazanzides, Z. Chen, A. Deguet, G. S. Fischer, R. H. Taylor, and S. P. DiMaio. An opensource research kit for the da Vinci® surgical system. In 2014 IEEE international conference on robotics and automation (ICRA) , pages 6434-6439. IEEE, 2014. -
  42. [51][51] E. Todorov, T. Erez, and Y. Tassa. MuJoCo: A Physics Engine for Model-Based Control. In Proc. IEEE/RSJ Int. Conf. on Intelligent Robots and Systems (IROS) , 2012.

Cited by

Sorted by citations

Authors
Hoque, R., Balakrishna, A., Novoseller, E., Wilcox, A., Brown, D. S., & Goldberg, K.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2021
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2109.08273
arXiv
2109.08273
OpenAlex
W3200105264
Cited by
31
References
5
Citations by year
2025: 3 · 2024: 9 · 2023: 14 · 2022: 5
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
Keywords
human oversight, novelty detection, risk estimation, switching strategy, robot learning from demonstration, human-robot interaction, cable routing

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review