PreprintCited by 131Reinforcement Learning in RoboticsRobot Manipulation and LearningDomain Adaptation and Few-Shot Learning

Implicit Behavioral Cloning

Open access
Florence, Pete, Lynch, Corey, Zeng, Andy, Ramirez, Oscar, Wahid, Ayzaan, Downs, Laura, Wong, Adrian, Lee, Johnny +2 more·2021·arXiv

Abstract

We find that across a wide range of robot policy learning scenarios, treating supervised policy learning with an implicit model generally performs better, on average, than commonly used explicit models. We present extensive experiments on this finding, and we provide both intuitive insight and theoretical arguments distinguishing the properties of implicit models compared to their explicit counterparts, particularly with respect to approximating complex, potentially discontinuous and multi-valued (set-valued) functions. On robotic policy learning tasks we show that implicit behavioral cloning policies with energy-based models (EBM) often outperform common explicit (Mean Square Error, or Mixture Density) behavioral cloning policies, including on tasks with high-dimensional action spaces and visual image inputs. We find these policies provide competitive results or outperform state-of-the-art offline reinforcement learning methods on the challenging human-expert tasks from the D4RL benchmark suite, despite using no reward information. In the real world, robots with implicit policies can learn complex and remarkably subtle behaviors on contact-rich tasks from human demonstrations, including tasks with high combinatorial complexity and tasks requiring 1mm precision.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2109.00137

Cites

3 of 3 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • IBC is prone to training instability.

    “In contrast, IBC Florence et al. (2021) is prone to training instability.”

    From Diffusion policy: Visuomotor policy learning via action diffusion (2025) · cites this paper as Florence et al. (2021)

    Section B.3.1 (page unavailable) says the authors use spectral normalization and gradient penalties for additional training stability, which supports stability being a concern for their Langevin-based approach. It does not say that IBC generally is prone to training instability; the Limitations section instead names computational cost and implementation complexity. (LLM, conf=0.82)

    Experimental — accuracy not yet measured
  • Existing implicit policies are unstable to train due to the necessity of drawing negative samples when computing the underlying Info-NCE loss.

    “However, existing implicit policies (Florence et al. 2021) are unstable to train due to the necessity of drawing negative samples when computing the underlying Info-NCE loss.”

    From Diffusion policy: Visuomotor policy learning via action diffusion (2025) · cites this paper as Florence et al. 2021

    Section 2 says the implicit models are trained with negative counter-examples and an InfoNCE-style loss. Appendix B.3.1 discusses measures for additional training stability, but neither passage says that drawing negative samples makes these policies unstable to train. The claimed cause of instability is therefore unsupported. (LLM, conf=0.87)

    Experimental — accuracy not yet measured
  • Training energy-based policies often requires negative sampling to estimate an intractable normalization constant, which is known to cause training instability.

    “Training energy-based policies often requires negative sampling to estimate an intractable normalization constant, which is known to cause training instability [11, 12].”

    From Diffusion Policy: Visuomotor Policy Learning via Action Diffusion (2023) · cites this paper as [12]

    Section 2 (page not available) says EBM training generates negative counter-examples to estimate the normalization term Z. Section B.3.1 discusses measures for training stability, but does not say that negative sampling or estimating Z causes instability. That causal part of the claim is unsupported by this copy. (LLM, conf=0.86)

    Experimental — accuracy not yet measured
  • Existing implicit policies are unstable to train due to the necessity of drawing negative samples when computing the underlying Info-NCE loss.

    “However, existing implicit policies [12] are unstable to train due to the necessity of drawing negative samples when computing the underlying Info-NCE loss.”

    From Diffusion Policy: Visuomotor Policy Learning via Action Diffusion (2023) · cites this paper as [12]

    Section 2 says training generates negative counter-examples and uses an InfoNCE-style loss. Appendix B.3.1 discusses measures for additional training stability, but does not say that drawing negative samples makes implicit policies unstable to train. The claimed cause of instability is unsupported. (LLM, conf=0.86)

    Experimental — accuracy not yet measured
  • IBC is prone to training instability.

    “In contrast, IBC [12] is prone to training instability.”

    From Diffusion Policy: Visuomotor Policy Learning via Action Diffusion (2023) · cites this paper as [12]

    Section B.3.1 (page unavailable) says the authors use spectral normalization and gradient penalties for additional training stability in their Langevin-based method. That supports training stability as a concern, but the paper does not say IBC generally is prone to unstable training; its Limitations section (page unavailable) instead identifies computational cost and implementation complexity. (LLM, conf=0.79)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 63

  1. [2][2] S. Ross, G. Gordon, and D. Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics , pages 627-635. JMLR Workshop and Conference Proceedings, 2011. -
  2. [3][3] S. Tu, A. Robey, T. Zhang, and N. Matni. On the sample complexity of stability constrained imitation learning. arXiv preprint arXiv:2102.09161 , 2021. - · arXiv
  3. [4][4] T. Zhang, Z. McCarthy, O. Jow, D. Lee, X. Chen, K. Goldberg, and P. Abbeel. Deep imitation learning for complex manipulation tasks from virtual reality teleoperation. In 2018 IEEE International Conference on Robotics and Automation (ICRA) , pages 5628-5635. IEEE, 2018. -
  4. [5][5] P. Florence, L. Manuelli, and R. Tedrake. Self-supervised correspondence in visuomotor policy learning. IEEE Robotics and Automation Letters , 5(2):492-499, 2019. -
  5. [6][6] A. Zeng, S. Song, J. Lee, A. Rodriguez, and T. Funkhouser. Tossingbot: Learning to throw arbitrary objects with residual physics. IEEE Transactions on Robotics , 2020. -
  6. [8][8] P. Abbeel and A. Y . Ng. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the twenty-first international conference on Machine learning , 2004. -
  7. [9][9] J. Ho, J. Gupta, and S. Ermon. Model-free imitation learning with policy optimization. In International Conference on Machine Learning . PMLR, 2016. -
  8. [10][10] Y. LeCun, S. Chopra, R. Hadsell, M. Ranzato, and F. Huang. A tutorial on energy-based learning. Predicting structured data , 1(0), 2006. -
  9. [11][11] M. Welling and Y . W. Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th international conference on machine learning (ICML-11) , pages 681-688. Citeseer, 2011. -
  10. [12][12] Y. Du and I. Mordatch. Implicit generation and modeling with energy based models. Advances in Neural Information Processing Systems , 32:3608-3618, 2019. -
  11. [13][13] T. Haarnoja, H. Tang, P . Abbeel, and S. Levine. Reinforcement learning with deep energy-based policies. In International Conference on Machine Learning , pages 1352-1361. PMLR, 2017. -
  12. [14][14] Y. Du, T. Lin, and I. Mordatch. Model-based planning with energy-based models. In L. P . Kaelbling, D. Kragic, and K. Sugiura, editors, Proceedings of the Conference on Robot Learning , volume 100 of Proceedings of Machine Learning Research , pages 374-383. PMLR, 30 Oct-01 Nov 2020. -
  13. [15][15] I. Kostrikov, J. Tompson, R. Fergus, and O. Nachum. Offline reinforcement learning with fisher divergence critic regularization. arXiv preprint arXiv:2103.08050 , 2021. - · arXiv
  14. [16][16] O. Nachum and M. Yang. Provable representation learning for imitation with contrastive fourier features. arXiv preprint arXiv:2105.12272 , 2021. - · arXiv
  15. [17][17] J. Fu, A. Kumar, O. Nachum, G. Tucker, and S. Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219 , 2020. - · arXiv
  16. [18][18] A. v. d. Oord, Y . Li, and O. Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748 , 2018. - · arXiv
  17. [19][19] Y. Song and D. P . Kingma. How to train your energy-based models. arXiv preprint arXiv:2101.03288 , 2021. - · arXiv
  18. [20][20] A. Jolicoeur-Martineau and I. Mitliagkas. Gradient penalty from a maximum margin perspective. arXiv preprint arXiv:1910.06922 , 2021. - · arXiv
  19. [22][22] K. Xu, M. Zhang, J. Li, S. S. Du, K.-i. Kawarabayashi, and S. Jegelka. How neural networks extrapolate: From feedforward to graph neural networks. arXiv preprint arXiv:2009.11848 , 2020. - · arXiv
  20. [23][23] R. Liu, J. Lehman, P . Molino, F. Petroski Such, E. Frank, A. Sergeev, and J. Y osinski. An intriguing failing of convolutional neural networks and the coordconv solution. Advances in Neural Information Processing Systems , 31, 2018. -
  21. [24][24] S. Levine, C. Finn, T. Darrell, and P . Abbeel. End-to-end training of deep visuomotor policies. The Journal of Machine Learning Research (JMLR) , 2016. -
  22. [25][25] A. Zeng, P. Florence, J. Tompson, S. Welker, J. Chien, M. Attarian, T. Armstrong, I. Krasin, D. Duong, V. Sindhwani, et al. Transporter networks: Rearranging the visual world for robotic manipulation. In Conference on Robot Learning , 2020. -
  23. [26][26] A. Kumar, A. Zhou, G. Tucker, and S. Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems (NeurIPS) , 2020. -
  24. [27][27] S. Sinha and A. Garg. S4rl: Surprisingly simple self-supervision for offline reinforcement learning. arXiv preprint arXiv:2103.06326 , 2021. - · arXiv
  25. [28][28] J. Peters and S. Schaal. Reinforcement learning by reward-weighted regression for operational space control. In Proceedings of the 24th international conference on Machine learning , pages 745-750, 2007. -
  26. [30][30] R. Rahmatizadeh, P . Abolghasemi, L. Bölöni, and S. Levine. Vision-based multi-task manipulation for inexpensive robots using end-to-end learning from demonstration. In 2018 IEEE international conference on robotics and automation (ICRA) , pages 3758-3765. IEEE, 2018. -
  27. [31][31] D. Ha and J. Schmidhuber. Recurrent world models facilitate policy evolution. In Advances in Neural Information Processing Systems 31 , pages 2451-2463, 2018. -
  28. [32][32] H. Suh and R. Tedrake. The surprising effectiveness of linear models for visual foresight in object pile manipulation. Workshop on Algorithmic Foundations of Robotics (WAFR) , 2020. -
  29. [33][33] S. Marx, E. Pauwels, T. Weisser, D. Henrion, and J. B. Lasserre. Semi-algebraic approximation using christoffeldarboux kernel. Constructive Approximation , pages 1-39, 2021. -
  30. [34][34] I. Mordatch. Concept learning with energy-based models. arXiv preprint arXiv:1811.02486 , 2018. - · arXiv
  31. [35][35] F. K. Gustafsson, M. Danelljan, G. Bhat, and T. B. Schön. Energy-based models for deep probabilistic regression. In European Conference on Computer Vision , pages 325-343. Springer, 2020. -
  32. [36][36] F. K. Gustafsson, M. Danelljan, R. Timofte, and T. B. Schön. How to train your energy-based model for regression. BMVC , 2020. -
  33. [37][37] Y. Du, S. Li, B. J. Tenenbaum, and I. Mordatch. Improved contrastive divergence training of energy based models. In Proceedings of the 38th International Conference on Machine Learning (ICML-21) , 2021. -
  34. [38][38] Y. Deng, A. Bakhtin, M. Ott, A. Szlam, and M. Ranzato. Residual energy-based models for text generation. ICLR , 2020. -
  35. [39][39] B. Amos and J. Z. Kolter. Optnet: Differentiable optimization as a layer in neural networks. In International Conference on Machine Learning , pages 136-145. PMLR, 2017. -
  36. [40][40] V. Niculae, A. Martins, M. Blondel, and C. Cardie. Sparsemap: Differentiable sparse structured inference. In International Conference on Machine Learning , pages 3799-3808. PMLR, 2018. -
  37. [41][41] P.-W. Wang, P. Donti, B. Wilder, and Z. Kolter. Satnet: Bridging deep learning and logical reasoning using a differentiable satisfiability solver. In International Conference on Machine Learning , pages 6545-6554. PMLR, 2019. -
  38. [42][42] S. Bai, J. Z. Kolter, and V . Koltun. Deep equilibrium models. NeurIPS 2019 , 2019. -
  39. [43][43] J. J. Park, P . Florence, J. Straub, R. Newcombe, and S. Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 165-174, 2019. -
  40. [44][44] L. Mescheder, M. Oechsle, M. Niemeyer, S. Nowozin, and A. Geiger. Occupancy networks: Learning 3d reconstruction in function space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4460-4470, 2019. -
  41. [45][45] Z. Chen and H. Zhang. Learning implicit fields for generative shape modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5939-5948, 2019. -
  42. [46][46] S. Saito, Z. Huang, R. Natsume, S. Morishima, A. Kanazawa, and H. Li. Pifu: Pixel-aligned implicit function for high-resolution clothed human digitization. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 2304-2314, 2019. -
  43. [47][47] S. Pfrommer, M. Halm, and M. Posa. Contactnets: Learning discontinuous contact dynamics with smooth, implicit representations. arXiv preprint arXiv:2009.11193 , 2020. - · arXiv
  44. [48][48] M. Liu, T. He, M. Xu, and W. Zhang. Energy-based imitation learning. arXiv preprint arXiv:2004.09395 , 2020. - · arXiv
  45. [49][49] T. Osa, J. Pajarinen, G. Neumann, J. A. Bagnell, P . Abbeel, and J. Peters. An algorithmic perspective on imitation learning. arXiv preprint arXiv:1811.06711 , 2018. - · arXiv
  46. [50][50] X. B. Peng, P. Abbeel, S. Levine, and M. van de Panne. Deepmimic: Example-guided deep reinforcement learning of physics-based character skills. ACMTransactions on Graphics (TOG) , 37(4):1-14, 2018. -
  47. [51][51] X. B. Peng, Z. Ma, P. Abbeel, S. Levine, and A. Kanazawa. Amp: Adversarial motion priors for stylized physics-based character control. arXiv preprint arXiv:2104.02180 , 2021. - · arXiv
  48. [52][52] C. G. Atkeson and S. Schaal. Robot learning from demonstration. In ICML , volume 97, pages 12-20. Citeseer, 1997. -
  49. [54][54] A. Rajeswaran, V . Kumar, A. Gupta, G. V ezzani, J. Schulman, E. Todorov, and S. Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. arXiv preprint arXiv:1709.10087 , 2017. - · arXiv
  50. [55][55] I. Kostrikov, O. Nachum, and J. Tompson. Imitation learning via off-policy distribution matching. In International Conference on Learning Representations , 2020. -
  51. [57][57] G. Cybenko. Approximation by superpositions of a sigmoidal function. Mathematics of control, signals and systems , 2(4):303-314, 1989. -
  52. [58][58] P. Butzer, S. Ries, and R. Stens. Approximation of continuous and discontinuous functions by generalized sampling series. Journal of approximation theory , 50(1):25-39, 1987. -
  53. [59][59] A. L. Tampos, J. E. C. Lope, and J. S. Hesthaven. Accurate reconstruction of discontinuous functions using the singular pade-chebyshev method. IAENG International Journal of Applied Mathematics , 42(ARTICLE): 242-249, 2012. -
  54. [60][60] G. Kvernadze. Approximation of the discontinuities of a function by its classical orthogonal polynomial fourier coefficients. Mathematics of computation , 79(272):2265-2285, 2010. -
  55. [61][61] E. Stella, C. Ladera, and G. Donoso. A very accurate method to approximate discontinuous functions with a finite number of discontinuities. arXiv preprint arXiv:1601.05132 , 2016. - · arXiv
  56. [62][62] R. R. Selmic and F. L. Lewis. Neural-network approximation of piecewise continuous functions: application to friction compensation. IEEE transactions on neural networks , 13(3):745-751, 2002. -
  57. [63][63] P.-T. De Boer, D. P . Kroese, S. Mannor, and R. Y . Rubinstein. A tutorial on the cross-entropy method. Annals of operations research , 134(1):19-67, 2005. -
  58. [64][64] C. Nash and C. Durkan. Autoregressive energy machines. In International Conference on Machine Learning , pages 1735-1744. PMLR, 2019. -
  59. [65][65] W. Grathwohl, K.-C. Wang, J.-H. Jacobsen, D. Duvenaud, M. Norouzi, and K. Swersky. Y our classifier is secretly an energy based model and you should treat it like one. arXiv preprint arXiv:1912.03263 , 2019. - · arXiv
  60. [66][66] T. Miyato, T. Kataoka, M. Koyama, and Y . Y oshida. Spectral normalization for generative adversarial networks. In International Conference on Learning Representations , 2018. -
  61. [67][67] E. Bingham and H. Mannila. Random projection in dimensionality reduction: applications to image and text data. In Proceedings of the seventh ACM SIGKDD international conference on Knowledge discovery and data mining , pages 245-250, 2001. -
  62. [68][68] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research , 15(1):1929-1958, 2014. -
  63. [69][69] K. He, X. Zhang, S. Ren, and J. Sun. Identity mappings in deep residual networks. In European conference on computer vision , pages 630-645. Springer, 2016. -

Cited by

Sorted by citations

Authors
Florence, P., Lynch, C., Zeng, A., Ramirez, O., Wahid, A., Downs, L., Wong, A., Lee, J., Mordatch, I., & Tompson, J.
Venue
arXiv
Publisher
arXiv
Published
2021
ISSN
2331-8422 (ISSN-L)
OpenAlex
W3197820606
Cited by
131
References
3
Citations by year
2025: 31 · 2024: 47 · 2023: 35 · 2022: 15 · 2021: 3
Type
posted content
Language
en
Open access
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords
implicit models, energy-based models, robot policy learning, offline reinforcement learning, human demonstrations, D4RL benchmark, set-valued mappings, high-dimensional action spaces, visual input

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review