PreprintCited by 39Reinforcement Learning in RoboticsRobot Manipulation and LearningTeleoperation and Haptic Systems

Human-in-the-Loop Imitation Learning using Remote Teleoperation

Open access
Mandlekar, Ajay, Xu, Danfei, Martín-Martín, Roberto, Zhu, Yuke, Fei-Fei, Li, Savarese, Silvio·2020·arXiv

Abstract

Imitation Learning is a promising paradigm for learning complex robot manipulation skills by reproducing behavior from human demonstrations. However, manipulation tasks often contain bottleneck regions that require a sequence of precise actions to make meaningful progress, such as a robot inserting a pod into a coffee machine to make coffee. Trained policies can fail in these regions because small deviations in actions can lead the policy into states not covered by the demonstrations. Intervention-based policy learning is an alternative that can address this issue -- it allows human operators to monitor trained policies and take over control when they encounter failures. In this paper, we build a data collection system tailored to 6-DoF manipulation settings, that enables remote human operators to monitor and intervene on trained policies. We develop a simple and effective algorithm to train the policy iteratively on new data collected by the system that encourages the policy to learn how to traverse bottlenecks through the interventions. We demonstrate that agents trained on data collected by our intervention-based system and algorithm outperform agents trained on an equivalent number of samples collected by non-interventional demonstrators, and further show that our method outperforms multiple state-of-the-art baselines for learning from the human interventions on a challenging robot threading task and a coffee making task. Additional results and videos at https://sites.google.com/stanford.edu/iwr .

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2012.06733

Cites

45 of 45 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • RoboTurk is a system for crowdsourcing task demonstrations from human operators using smartphone-based teleoperation and video streams provided in web browsers.

    “RoboTurk [2,23-26] is a system for crowdsourcing task demonstrations from human operators using smartphone-based teleoperation and video streams provided in web browsers.”

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 3

  1. [2][2] --, 'Inverse reinforcement learning', in Encyclopedia of machine learning , Springer, 2011, pp. 554-558. -
  2. [6][6] A. Bajcsy, D. P. Losey, M. K. O'Malley, and A. D. Dragan, 'Learning robot objectives from physical human interaction', Proceedings of Machine Learning Research , vol. 78, pp. 217-226, 2017. -
  3. [11][11] P. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei, 'Deep reinforcement learning from human preferences', in Advances in Neural Information Processing Systems , 2017, pp. 42994307. -

Cited by

Sorted by citations

Authors
Mandlekar, A., Xu, D., Martín-Martín, R., Zhu, Y., Fei-Fei, L., & Savarese, S.
Affiliations
  • Stanford University (US)
  • The University of Texas at Austin (US)
Venue
arXiv
Publisher
arXiv
Published
2020
ISSN
2331-8422 (ISSN-L)
OpenAlex
W3110682054
Cited by
35
References
42
Citations by year
2026: 1 · 2025: 4 · 2024: 8 · 2023: 8 · 2022: 7 · 2021: 7
Type
posted content
Language
en
Open access
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • U.S. Department of Defense
  • Toyota Research Institute
  • National Defense Science and Engineering Graduate
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
  • Physical Sciences › Engineering › Mechanical Engineering › Teleoperation and Haptic Systems
Keywords
imitation learning, human-in-the-loop learning, robotic manipulation, data collection system

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review