Synthesis and stabilization of complex behaviors through online trajectory optimization
The DOI matches its Crossref record; 12 of the 12 references it cites were found in a registry or an index.
Abstract
We present an online trajectory optimization method and software platform applicable to complex humanoid robots performing challenging tasks such as getting up from an arbitrary pose on the ground and recovering from large disturbances using dexterous acrobatic maneuvers. The resulting behaviors, illustrated in the attached video, are computed only 7 × slower than real time, on a standard PC. The video also shows results on the acrobot problem, planar swimming and one-legged hopping. These simpler problems can already be solved in real time, without pre-computing anything.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
MuJoCo: A physics engine for model-based control
Todorov, E., Erez, T., & Tassa, Y. (2012) · 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems · doi:10.1109/iros.2012.6386109 · Cited by 4,767
Self-citationRigid-Body Dynamics with Friction and Impact
Stewart, D. E. (2000) · SIAM Review · doi:10.1137/s0036144599360110 · Cited by 628
Todorov, E., & Li, W. (2005) · Proceedings of the 2005, American Control Conference, 2005. · doi:10.1109/acc.2005.1469949 · Cited by 611
Self-citationEfficient Numerical Methods for Nonlinear MPC and Moving Horizon Estimation
Diehl, M., Ferreau, H. J., & Haverbeke, N. (2009) · Lecture Notes in Control and Information Sciences · doi:10.1007/978-3-642-01094-1_32 · Cited by 539
An Application of Reinforcement Learning to Aerobatic Helicopter Flight
Abbeel, P., Coates, A., Quigley, M., & Ng, A. Y. (2007) · Advances in Neural Information Processing Systems 19 · doi:10.7551/mitpress/7503.003.0006 · Cited by 535
Differential dynamic programming
Denham, W. (1971) · IEEE Transactions on Automatic Control · doi:10.1109/tac.1971.1099760 · Cited by 530
Differential dynamic programming
Jacobson, D. H., & Mayne, D. Q. (1970) · CERN Document Server (European Organization for Nuclear Research) · Cited by 521
A convex, smooth and invertible contact model for trajectory optimization
Todorov, E. V. (2011) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2011.5979814 · Cited by 111
Self-citationInfinite-Horizon Model Predictive Control for Periodic Tasks with Contacts
Erez, T., Tassa, Y., & Todorov, E. (2011) · Robotics: Science and Systems VII · doi:10.15607/rss.2011.vii.010 · Cited by 61
Self-citationStochastic Complementarity for Local Control of Discontinuous Dynamics
Tassa, Y., & Todorov, E. (2010) · Robotics: Science and Systems VI · doi:10.15607/rss.2010.vi.022 · Cited by 49
Self-citation
Cited by
Continuous control with deep reinforcement learning
Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., & Wierstra, D. (2015) · arXiv · doi:10.48550/arxiv.1509.02971 · Cited by 10,782
Self-citationMuJoCo: A physics engine for model-based control
Todorov, E., Erez, T., & Tassa, Y. (2012) · 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems · doi:10.1109/iros.2012.6386109 · Cited by 4,767
Self-citationDeepMimic: example-guided deep reinforcement learning of physics-based character skills
Peng, X. B., Abbeel, P., Levine, S., & van de Panne, M. (2018) · ACM Transactions on Graphics · doi:10.1145/3197517.3201311 · Cited by 1,784
Benchmarking Deep Reinforcement Learning for Continuous Control
Duan, Y., Chen, X., Houthooft, R., Schulman, J., & Abbeel, P. (2016) · arXiv · doi:10.48550/arxiv.1604.06778 · Cited by 1,114
Motion planning with sequential convex optimization and convex collision checking
Schulman, J., Duan, Y., Ho, J., Lee, A., Awwal, I., Bradlow, H., Pan, J., Patil, S., Goldberg, K., & Abbeel, P. (2014) · The International Journal of Robotics Research · doi:10.1177/0278364914528132 · Cited by 866
Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
Levine, S., Kumar, A., Tucker, G., & Fu, J. (2020) · arXiv · doi:10.48550/arxiv.2005.01643 · Cited by 824
“22: for each (si,ai), compute Â(si,ai) = Qφk+1(si,ai)− Ea∼πk,g(a|si)[Qφk+1(si,a)] 23: ∇θk,gJ(πθk,g ) ≈ 1N∇θk,g log πθk,g (si,ai)Â(si,ai) 24: θk,g+1 ← θk,g + απ∇θk,gJ(πθk,g ) 25: end for 26: θk + 1← θk,Gπ . update policy parameters 27: end for utilize it for planning at test time, often by means of model-predictive c…”
MethodInfluentialDynamic Locomotion in the MIT Cheetah 3 Through Convex Model-Predictive Control
Di Carlo, J., Wensing, P. M., Katz, B., Bledt, G., & Kim, S. (2018) · 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2018.8594448 · Cited by 758
Emergence of Locomotion Behaviours in Rich Environments
Heess, N., TB, D., Sriram, S., Lemmon, J., Merel, J., Wayne, G., Tassa, Y., Erez, T., Wang, Z., Eslami, S. M. A., et al. (2017) · arXiv · doi:10.48550/arxiv.1707.02286 · Cited by 657
“They include a variety of obstacle courses for agents with different bodies ( Quadruped , Planar Walker , and Humanoid [5, 6]).”
BackgroundSelf-citationA Tour of Reinforcement Learning: The View from Continuous Control
Recht, B. (2019) · Annual Review of Control, Robotics, and Autonomous Systems · doi:10.1146/annurev-control-053018-023825 · Cited by 609
Deep visual foresight for planning robot motion
Finn, C., & Levine, S. (2017) · 2017 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2017.7989324 · Cited by 599
“The high dimensionality of image observations presents a substantial challenge to model-based approaches, which have been most successful for low-dimensional non-visual tasks [21] such as helicopter control [22], locomotion [23], and robotic cutting [24].”
BackgroundA direct method for trajectory optimization of rigid bodies through contact
Posa, M., Cantu, C., & Tedrake, R. (2014) · The International Journal of Robotics Research · doi:10.1177/0278364913506757 · Cited by 597
“Robust, contact invariant MPC of underactuated systems remains an open problem, although there has been significant recent work in this area, such as in [35].”
Deep learning, reinforcement learning, and world models
Matsuo, Y., LeCun, Y., Sahani, M., Precup, D., Silver, D., Sugiyama, M., Uchibe, E., & Morimoto, J. (2022) · Neural Networks · doi:10.1016/j.neunet.2022.03.037 · Cited by 543
peng (2017) · doi:10.1145/3072959.3073602 · Cited by 502
“The policy (actor) is trained using a positive-temporal di erence update scheme modeled after CACLA [Van Hasselt 2012], and the value function (critic) is trained using Bellman backups.”
MethodTassa, Y., Doron, Y., Muldal, A., Erez, T., Li, Y., Casas, D. D. L., Budden, D., Abdolmaleki, A., Merel, J., Lefrancq, A., et al. (2018) · arXiv · doi:10.48550/arxiv.1801.00690 · Cited by 497
“Humanoid (54, 21, 67): A simplified humanoid with 21 joints, based on the model in (Tassa et al., 2012).”
BackgroundSelf-citationLearning deep control policies for autonomous aerial vehicles with MPC-guided policy search
Zhang, T., Kahn, G., Levine, S., & Abbeel, P. (2016) · 2016 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2016.7487175 · Cited by 452
“While this method is not as general as policy gradient RL algorithms, which can optimize arbitrary parameterized policies with arbitrary observations under unknown dynamics [32], it is fast, simple, and often effective [39].”
MethodControl-limited differential dynamic programming
Tassa, Y., Mansard, N., & Todorov, E. (2014) · 2014 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2014.6907001 · Cited by 440
Self-citationFinding Locally Optimal, Collision-Free Trajectories with Sequential Convex Optimization
Schulman, J., Ho, J., Lee, A., Awwal, I., Bradlow, H., & Abbeel, P. (2013) · Robotics: Science and Systems IX · doi:10.15607/rss.2013.ix.031 · Cited by 430
An algorithmic perspective on imitation learning
Osa, T., Pajarinen, J., Neumann, G., Bagnell, J. A., Abbeel, P., & Peters, J. (2018) · Foundations and Trends in Robotics · doi:10.1561/2300000053 · Cited by 405
“Iterative LQR (iLQR) is often employed to con- 108 Behavioral Cloning trol a system of which the dynamics is not accurately known [Todorov and Li, 2005, Abbeel et al., 2010, Tassa et al., 2012]. iLQR learns a linear feedback controller to follow a trajectory through an iterative learning process.”
MethodAn Algorithmic Perspective on Imitation Learning
Osa, T., Pajarinen, J., Neumann, G., Bagnell, J. A., Abbeel, P., & Peters, J. (2018) · now publishers, Inc. eBooks · doi:10.1561/9781680834116 · Cited by 368
A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems
Figueiredo Prudencio, R., Maximo, M. R. O. A., & Colombini, E. L. (2024) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3250269 · Cited by 361
Continuous Deep Q-Learning with Model-based Acceleration
Gu, S., Lillicrap, T., Sutskever, I., & Levine, S. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1603.00748 · Cited by 304
Hierarchical motor control in mammals and machines
Merel, J., Botvinick, M., & Wayne, G. (2019) · Nature Communications · doi:10.1038/s41467-019-13239-6 · Cited by 274
Perceptive Locomotion Through Nonlinear Model-Predictive Control
Grandia, R., Jenelten, F., Yang, S., Farshidian, F., & Hutter, M. (2023) · IEEE Transactions on Robotics · doi:10.1109/tro.2023.3275384 · Cited by 265
Benchmarking Model-Based Reinforcement Learning
Wang, T., Bao, X., Clavera, I., Hoang, J., Wen, Y., Langlois, E., Zhang, S., Zhang, G., Abbeel, P., & Ba, J. (2019) · arXiv · doi:10.48550/arxiv.1907.02057 · Cited by 226
Ensemble-CIO: Full-body dynamic motion planning that transfers to physical humanoids
Mordatch, I., Lowrey, K., & Todorov, E. (2015) · 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2015.7354126 · Cited by 135
Self-citationTowards Generalization and Simplicity in Continuous Control
Rajeswaran, A., Lowrey, K., Todorov, E., & Kakade, S. (2017) · arXiv · doi:10.48550/arxiv.1703.02660 · Cited by 94
Self-citationMastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learning
Yarats, D., Fergus, R., Lazaric, A., & Pinto, L. (2021) · arXiv · doi:10.48550/arxiv.2107.09645 · Cited by 89
“The humanoid control problem first presented in Tassa et al. [2012], has been studied as one of the hardest control problems due to its large state and action spaces.”
BackgroundMethodFrom motor control to team play in simulated humanoid football
Liu, S., Lever, G., Wang, Z., Merel, J., Eslami, S. M. A., Hennes, D., Czarnecki, W. M., Tassa, Y., Omidshafiei, S., Abdolmaleki, A., et al. (2022) · Science Robotics · doi:10.1126/scirobotics.abo0235 · Cited by 88
Self-citationModel Predictive Path Integral Control using Covariance Variable Importance Sampling
Williams, G., Aldrich, A., & Theodorou, E. (2015) · arXiv · doi:10.48550/arxiv.1509.01149 · Cited by 71
SAGCI-System: Towards Sample-Efficient, Generalizable, Compositional, and Incremental Robot Learning
Lv, J., Yu, Q., Shao, L., Liu, W., Xu, W., & Lu, C. (2022) · 2022 International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra46639.2022.9811859 · Cited by 22
ADAPT: Zero-Shot Adaptive Policy Transfer for Stochastic Dynamical Systems
Harrison, J., Garg, A., Ivanovic, B., Zhu, Y., Savarese, S., Fei-Fei, L., & Pavone, M. (2017) · arXiv · doi:10.48550/arxiv.1707.04674 · Cited by 21
Trajectory-wise Multiple Choice Learning for Dynamics Generalization in Reinforcement Learning
Seo, Y., Lee, K., Clavera, I., Kurutach, T., Shin, J., & Abbeel, P. (2020) · arXiv · doi:10.48550/arxiv.2010.13303 · Cited by 20
“Such a learned dynamics model can be used as a simulator for model-free RL methods [16, 18, 40], providing a prior or additional features to a policy [9, 47], or planning ahead to select actions by predicting the future consequences of actions [1, 22, 42].”
Method
Metadata
- Authors
- Tassa, Y., Erez, T., & Todorov, E.
- Venue
- 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems
- Publisher
- IEEE
- Published
- 2012-10
- Pages
- 4906-4913
- OpenAlex
- W2087617385
- Cited by
- 744
- References
- 12
- FWCI
- 40.63
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.