CoChat
Sign Up
PreprintCited by 17Reinforcement Learning in RoboticsRobot Manipulation and LearningDomain Adaptation and Few-Shot Learning

The DOI matches its DataCite record; 2 of the 2 references it cites were found in a registry or an index.

Generalization Guarantees for Imitation Learning

Open access
Ren, Allen Z., Veer, Sushant, Majumdar, Anirudha·2020·arXiv
DOI 10.48550/arxiv.2008.01913arXiv 2008.01913
Open this paper in CoChat — free accountOpen PDF

Abstract

Control policies from imitation learning can often fail to generalize to novel environments due to imperfect demonstrations or the inability of imitation learning algorithms to accurately infer the expert's policies. In this paper, we present rigorous generalization guarantees for imitation learning by leveraging the Probably Approximately Correct (PAC)-Bayes framework to provide upper bounds on the expected cost of policies in novel environments. We propose a two-stage training method where a latent policy distribution is first embedded with multi-modal expert behavior using a conditional variational autoencoder, and then "fine-tuned" in new training environments to explicitly optimize the generalization bound. We demonstrate strong generalization bounds and their tightness relative to empirical performance in simulation for (i) grasping diverse mugs, (ii) planar pushing with visual feedback, and (iii) vision-based indoor navigation, as well as through hardware experiments for the two manipulation tasks.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2008.01913

Cites

2 of 2 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 36

  1. [1][1] T. Osa, J. Pajarinen, G. Neumann, J. A. Bagnell, P. Abbeel, and J. Peters. An algorithmic perspective on imitation learning. arXiv preprint arXiv:1811.06711 , 2018. - · arXiv
  2. [2][2] Y. Zhu, Z. Wang, J. Merel, A. Rusu, T. Erez, S. Cabi, S. Tunyasuvunakool, J. Kram´ ar, R. Hadsell, N. de Freitas, et al. Reinforcement and imitation learning for diverse visuomotor skills. arXiv preprint arXiv:1802.09564 , 2018. - · arXiv
  3. [3][3] V. N. Vapnik. An overview of statistical learning theory. IEEE Transactions on Neural Networks , 10(5):988-999, 1999. -
  4. [4][4] D. A. McAllester. Some PAC-Bayesian theorems. Machine Learning , 37(3):355-363, 1999. -
  5. [5][5] M. Seeger. PAC-Bayesian generalisation error bounds for Gaussian process classification. Journal of Machine Learning Research , 3(Oct):233-269, 2002. -
  6. [6][6] J. Langford and J. Shawe-Taylor. PAC-Bayes & margins. In Proceedings of the Advances in Neural Information Processing Systems , pages 439-446, 2003. -
  7. [7][7] G. K. Dziugaite and D. M. Roy. Computing nonvacuous generalization bounds for deep (stochastic) neural networks with many more parameters than training data. arXiv preprint arXiv:1703.11008 , 2017. - · arXiv
  8. [8][8] B. Neyshabur, S. Bhojanapalli, D. McAllester, and N. Srebro. A PAC-Bayesian approach to spectrally-normalized margin bounds for neural networks. arXiv preprint arXiv:1707.09564 , 2017. - · arXiv
  9. [9][9] S. Arora, R. Ge, B. Neyshabur, and Y. Zhang. Stronger generalization bounds for deep nets via a compression approach. arXiv preprint arXiv:1802.05296 , 2018. - · arXiv
  10. [10][10] J. Morton and M. J. Kochenderfer. Simultaneous policy learning and latent state inference for imitating driver behavior. In Proceedings of the IEEE International Conference on Intelligent Transportation Systems (ITSC) , pages 1-6, 2017. -
  11. [11][11] K. Sohn, H. Lee, and X. Yan. Learning structured output representation using deep conditional generative models. In Proceedings of the Advances in Neural Information Processing Systems , pages 3483-3491, 2015. -
  12. [12][12] D. P. Kingma, S. Mohamed, D. J. Rezende, and M. Welling. Semi-supervised learning with deep generative models. In Proceedings of the Advances in Neural Information Processing Systems , pages 3581-3589, 2014. -
  13. [13][13] P. Florence, L. Manuelli, and R. Tedrake. Self-supervised correspondence in visuomotor policy learning. IEEE Robotics and Automation Letters , 5(2):492-499, 2019. -
  14. [14][14] T. Zhang, Z. McCarthy, O. Jow, D. Lee, X. Chen, K. Goldberg, and P. Abbeel. Deep imitation learning for complex manipulation tasks from virtual reality teleoperation. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA) , pages 1-8, 2018. -
  15. [15][15] M. Bansal, A. Krizhevsky, and A. Ogale. Chauffeurnet: Learning to drive by imitating the best and synthesizing the worst. arXiv preprint arXiv:1812.03079 , 2018. - · arXiv
  16. [16][16] F. Codevilla, M. Miiller, A. L´ opez, V. Koltun, and A. Dosovitskiy. End-to-end driving via conditional imitation learning. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA) , pages 1-9, 2018. -
  17. [17][17] K. Hausman, Y. Chebotar, S. Schaal, G. Sukhatme, and J. J. Lim. Multi-modal imitation learning from unstructured demonstrations using generative adversarial nets. In Proceedings of the Advances in Neural Information Processing Systems , pages 1235-1245, 2017. -
  18. [18][18] Z. Wang, J. S. Merel, S. E. Reed, N. de Freitas, G. Wayne, and N. Heess. Robust imitation of diverse behaviors. In Proceedings of the Advances in Neural Information Processing Systems , pages 5320-5329, 2017. -
  19. [19][19] R. Rahmatizadeh, P. Abolghasemi, L. B¨ ol¨ oni, and S. Levine. Vision-based multi-task manipulation for inexpensive robots using end-to-end learning from demonstration. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA) , pages 3758-3765, 2018. -
  20. [20][20] F.-I. Hsiao, J.-H. Kuo, and M. Sun. Learning a multi-modal policy via imitating demonstrations with mixed behaviors. arXiv preprint arXiv:1903.10304 , 2019. - · arXiv
  21. [21][21] Y. Gao, H. Xu, J. Lin, F. Yu, S. Levine, and T. Darrell. Reinforcement learning from imperfect demonstrations. arXiv preprint arXiv:1802.05313 , 2018. - · arXiv
  22. [22][22] S. Ross, G. Gordon, and D. Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the International Conference on Artificial Intelligence and Statistics , pages 627-635, 2011. -
  23. [23][23] A. Gupta, V. Kumar, C. Lynch, S. Levine, and K. Hausman. Relay policy learning: Solving long-horizon tasks via imitation and reinforcement learning. arXiv preprint arXiv:1910.11956 , 2019. - · arXiv
  24. [24][24] M. Laskey, J. Lee, R. Fox, A. Dragan, and K. Goldberg. Dart: Noise injection for robust imitation learning. arXiv preprint arXiv:1703.09327 , 2017. - · arXiv
  25. [25][25] A. Mandlekar, D. Xu, R. Mart´ ın-Mart´ ın, S. Savarese, and L. Fei-Fei. Learning to generalize across long-horizon tasks from human demonstrations. arXiv preprint arXiv:2003.06085 , 2020. - · arXiv
  26. [26][26] A. Majumdar and M. Goldstein. PAC-Bayes Control: synthesizing controllers that provably generalize to novel environments. In Proceedings of the Conference on Robot Learning (CoRL) , 2018. -
  27. [27][27] A. Majumdar, A. Farid, and A. Sonar. PAC-Bayes Control: Learning policies that provably generalize to novel environments. arXiv preprint arXiv:1806.04225 , 2019. - · arXiv
  28. [28][28] S. Veer and A. Majumdar. Probably approximately correct vision-based planning using motion primitives. arXiv preprint arXiv:2002.12852 , 2020. - · arXiv
  29. [29][29] A. Maurer. A note on the PAC-Bayesian theorem. arXiv preprint cs/0411099 , 2004. - · arXiv
  30. [30][30] H.-G. Beyer and H.-P. Schwefel. Evolution strategies-a comprehensive introduction. Natural computing , 1(1):3-52, 2002. -
  31. [31][31] D. Wierstra, T. Schaul, T. Glasmachers, Y. Sun, J. Peters, and J. Schmidhuber. Natural evolution strategies. The Journal of Machine Learning Research , 15(27):949-980, 2014. -
  32. [32][32] S.-I. Amari. Natural gradient works efficiently in learning. Neural computation , 10(2):251276, 1998. -
  33. [34][34] J. Langford and R. Caruana. (not) bounding the true error. In Proceedings of the Advances in Neural Information Processing Systems , pages 809-816, 2002. -
  34. [36][36] A. X. Chang, T. Funkhouser, L. Guibas, P. Hanrahan, Q. Huang, Z. Li, S. Savarese, M. Savva, S. Song, H. Su, et al. Shapenet: An information-rich 3D model repository. arXiv preprint arXiv:1512.03012 , 2015. - · arXiv
  35. [37][37] F. Xia, W. B. Shen, C. Li, P. Kasimbeg, M. E. Tchapmi, A. Toshev, R. Mart´ ın-Mart´ ın, and S. Savarese. Interactive gibson benchmark: A benchmark for interactive navigation in cluttered environments. IEEE Robotics and Automation Letters , 5(2):713-720, 2020. -
  36. [38][38] T. Salimans, J. Ho, X. Chen, S. Sidor, and I. Sutskever. Evolution strategies as a scalable alternative to reinforcement learning. arXiv preprint arXiv:1703.03864 , 2017. - · arXiv

Cited by

Sorted by citations

See all 17

Metadata

Authors
Ren, A. Z., Veer, S., & Majumdar, A.
Affiliations
Princeton University (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2020
Pages
1426-1442
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2008.01913
arXiv
2008.01913
OpenAlex
W3106612012
Cited by
17
References
2
Citations by year
2025: 2 · 2024: 3 · 2023: 1 · 2022: 5 · 2021: 6
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
  • Conference on Robot Learning
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords
imitation learning, generalization guarantees, PAC-Bayesian bounds, conditional variational autoencoder, fine-tuning, robotic manipulation, grasping, planar pushing, visual navigation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review