CoChat
Sign Up
Cited by 405Reinforcement Learning in RoboticsRobot Manipulation and LearningMachine Learning and Algorithms

An algorithmic perspective on imitation learning

Open access
Osa, Takayuki, Pajarinen, Joni, Neumann, Gerhard, Bagnell, J. Andrew, Abbeel, Pieter, Peters, Jan·2018·Foundations and Trends in Robotics
DOI 10.1561/2300000053
Open this paper in CoChat — free accountOpen PDF

Abstract

As robots and other intelligent agents move from simple environments and problems to more complex, unstructured settings, manually programming their behavior has become increasingly challenging and expensive. Often, it is easier for a teacher to demonstrate a desired behavior rather than attempt to manually engineer it. This process of learning from demonstrations, and the study of algorithms to do so, is called imitation learning. This work provides an introduction to imitation learning. It covers the underlying assumptions, approaches, and how they relate; the rich set of algorithms developed to tackle the problem; and advice on effective tools and implementation. We intend this paper to serve two audiences. First, we want to familiarize machine learning experts with the challenges of imitation learning, particularly those arising in robotics, and the interesting theoretical and practical distinctions between it and more familiar frameworks like statistical supervised learning theory and reinforcement learning. Second, we want to give roboticists and experts in applied artificial intelligence a broader appreciation for the frameworks and tools available for imitation learning. We organize our work by dividing imitation learning into directly replicating desired behavior (sometimes called behavioral cloning [Bain and Sammut, 1996]) and learning the hidden objectives of the desired behavior from demonstrations (called inverse optimal control [Kalman, 1964] or inverse reinforcement learning [Russell, 1998]). In addition to method analysis, we discuss the design decisions a practitioner must make when selecting an imitation learning approach. Moreover, application examples—such as robots that play table tennis Kober and Peters, 2009 and programs that play the game of Go Silver et al. 2016—illustrate the properties and motivations behind different forms of imitation learning. We conclude by presenting a set of open questions and point towards possible future research directions.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1561/2300000053

Cites

233 of 233 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • IL suffers from several issues: demonstration data can be suboptimal, contain noise, and have a different domain compared to the application domain.

    “The major bottlenecks are demonstration data that can (1) be suboptimal, (2) contain noise and (3) have a different domain (different MDP/POMDP) compared to the application domain [8].”

    From Data-Driven Robotic Manipulation of Cloth-like Deformable Objects: The Present, Challenges and Future Prospects (2023) · cites this paper as [8]

    Section 1.3 describes a mismatch between the distributions of expert demonstrations and learner reproductions, and §1.4 says differences in embodiment and observability can require adaptation. The copy does not establish that this means different MDPs or POMDPs. It gives an example of noisy observations and control inputs, but does not clearly identify noisy or suboptimal demonstration data as major bottlenecks; §1.3 instead says imitation learning often assumes optimal or at least good demonstrations. Page numbers were not available in the indexed passages. (LLM, conf=0.83)

    Experimental — accuracy not yet measured

  • Training an approximated dynamic model using supervised learning with transitions induced by a base policy generally does not work well in practice, especially in high-dimensional settings, mainly due to the distributional shift problem.

    “This approach generally does not work well in practice, especially in high-dimensional settings [37,38], mainly due to the distributional shift problem, i.e., ρπP^(st)≠ρπbase(st)[8]”

    From Data-Driven Robotic Manipulation of Cloth-like Deformable Objects: The Present, Challenges and Future Prospects (2023) · cites this paper as [8]

    Section 1.3 (page not provided) explains that a learner can encounter states absent from expert demonstrations, creating covariate shift between training and deployment distributions. That supports the distribution-shift explanation attributed to this source, but the passage concerns imitation learning, not a supervised dynamics model trained on base-policy transitions; it does not establish that this approach generally fails in high-dimensional settings. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • Data-driven approaches with deep learning, such as deep reinforcement learning and deep imitation learning, have demonstrated advantages in performing task-level and skill-level control problems.

    “In recent years, data-driven approaches with deep learning (DL) [6], such as deep reinforcement learning (DRL) [7] and deep imitation learning (DIL) [8], have demonstrated many advantages in performing task-level and skill-level control problems (Section 3.3).”

    From Data-Driven Robotic Manipulation of Cloth-like Deformable Objects: The Present, Challenges and Future Prospects (2023) · cites this paper as [8]

  • Imitation learning is the first type of data-driven control method that has been used considerably in all four families of the CDO domain.

    “Imitation learning (IL) [8] is the first type of data-driven control method that has been used considerably in all four families of the CDO domain (Section 6).”

    From Data-Driven Robotic Manipulation of Cloth-like Deformable Objects: The Present, Challenges and Future Prospects (2023) · cites this paper as [8]

  • A hidden Markov model cannot produce long smooth action sequences because it is a discrete process.

    “However, an HMM cannot produce long smooth action sequences as it is a discrete process [8].”

    From Data-Driven Robotic Manipulation of Cloth-like Deformable Objects: The Present, Challenges and Future Prospects (2023) · cites this paper as [8]

    The passages in §1.5.2–1.5.3 define the Markov property and finite Markov decision processes, but do not discuss whether HMMs can generate long, smooth action sequences. Searches for “hidden Markov model,” “HMM,” and smooth trajectories found no passage in our copy making the claimed connection. (LLM, conf=0.72)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 22

  1. M. Arjovsky, S. Chintala, and L. Bottou. Wasserstein generative adversar- ial networks. In Proceedings of the International Conference on Machine Learning (ICML), pages 214-223, 2017.
  2. A. Billard, S. Calinon, R. Dillmann, and S. Schaal. Springer handbook of robotics, chapter Robot programming by demonstration, pages 1371-1394. Springer Berlin Heidelberg, 2008.
  3. A. Billard, S. Calinon, and R. Dillmann. Handbook of robotics, chapter Learn- ing from Humans, pages 1995-2014. Springer, 2016.
  4. K. Bogert, J. F. Lin, P. Doshi, and D. Kulic. Expectation-maximization for inverse reinforcement learning with hidden data. In Proceedings of the In- ternational Conference on Autonomous Agents & Multiagent Systems (AA- MAS), pages 1034-1042, 2016.
  5. H. Daumé III and J. Langford. Advances in structured prediction. In Tutorials in the International Conference on Machine Learning (ICML), July 2015. References
  6. A. D. Dragan, K. Muelling, J. Andrew Bagnell, and S. S. Srinivasa. Move- ment primitives via optimization. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), pages 2339-2346, May 2015. .
  7. D. B. Grimes and R. P. N. Rao. Creating brain-like intelligence: From basic principles to complex intelligent systems, chapter Learning Actions through Imitation and Exploration: Towards Humanoid Robots That Learn from Humans, pages 103-138. Springer Berlin Heidelberg, 2009.
  8. D. Hadfield-Menell, A. Dragan, P. Abbeel, and S. Russell. Cooperative in- verse reinforcement learning. In Advances in Neural Information Processing Systems (NIPS), 2016.
  9. J. Ho, J. K. Gupta, and S. Ermon. Model-free imitation learning with policy optimization. In Proceedings of the International Conference on Interna- tional Conference on Machine Learning (ICML), 2016.
  10. S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural compu- tation, 9(8):1735-1780, 1997.
  11. S. Huang, J. Pan, G. Mulcaire, and P. Abbeel. Leveraging appearance priors in non-rigid registration with applications to manipulation of deformable objects. In Proceedings of the IEEE/RSJ International Conference on In- telligent Robots and Systems (IROS), 2015.
  12. H. Khalil. Nonlinear systems. Prentice Hall, Upper Saddle River, NJ, 1996.
  13. K. M. Kitani, B. D. Ziebart, J. A. Bagnell, and M. Hebert. Activity forecast- ing. In European Conference on Computer Vision (ECCV), pages 201-214. Springer, 2012.
  14. P. C. Mahalanobis. On the generalised distance in statistics. In Proceedings of the National Institute of Sciences of India, 1936. References
  15. A. Y. Ng and S. J. Russell. Algorithms for inverse reinforcement learning. In Proceedings of the International Conference on Machine Learning (ICML), pages 663-670, 2000.
  16. J. Rust. Handbook of Econometrics, chapter Structural estimation of Markov decision processes, pages 3082-3139. Elsevier, 1994.
  17. C. Sammut, S. Hurst, D. Kedzier, and D. Michie. Learning to fly. In Proceed- ings of the International Conference on Machine Learning (ICML), pages 385-393, 1992.
  18. S. Schaal, J. Peters, J. Nakanishi, and A. Ijspeert. Learning movement primi- tives. In Proceedings of the International Symposium on Robotics Research (ISRR), 2004.
  19. K. Shiarlis, J. Messias, and S. Whiteson. Inverse reinforcement learning from failure. In Proceedings of the International Conference on Autonomous Agents & Multiagent Systems, pages 1060-1068. International Foundation for Autonomous Agents and Multiagent Systems, 2016.
  20. B. Taskar. Learning structured prediction models: a large margin approach. PhD thesis, Stanford University, 2005.
  21. B. Widrow and F. W. Smith. Pattern recognising control systems. Computer and Information Sciences Clever Hume Press, 1964.
  22. M. Wiering and M. van Otterlo, editors. Reinforcement Learning: State-of- the-Art. Springer, 2012.

Cited by

Sorted by citations

Related

Authors
Osa, T., Pajarinen, J., Neumann, G., Bagnell, J. A., Abbeel, P., & Peters, J.
Affiliations
  • The University of Tokyo (JP)
  • Technische Universität Darmstadt (DE)
  • University of Lincoln (GB)
  • Carnegie Mellon University (US)
  • University of California, Berkeley (US)
ORCID
Venue
Foundations and Trends in Robotics
Publisher
Emerald
Published
2018-03-27
Volume
7
Issue
1-2
Pages
1-179
ISSN
  • 1935-8253 (ISSN-L)
  • 1935-8261
DOI
10.1561/2300000053
OpenAlex
W3101780148
Cited by
405
References
233
Citations by year
2026: 37 · 2025: 45 · 2024: 53 · 2023: 86 · 2022: 61 · 2021: 56 · 2020: 42 · 2019: 23 · 2018: 2
FWCI
17.31
Type
journal article
Language
en
Open access
Locations
  • Foundations and Trends in Robotics (publishedVersion)
  • Lincoln Repository (University of Lincoln) (acceptedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
Keywords
imitation learning, behavior cloning, inverse reinforcement learning, inverse optimal control, robotics, algorithm selection, table tennis robot

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review