Hierarchical generative modelling for autonomous robots
Open accessPublished version of a preprint
Abstract
Humans generate intricate whole-body motions by planning, executing and combining individual limb movements. We investigated this fundamental aspect of motor control and approached the problem of autonomous task completion by hierarchical generative modelling with multi-level planning, emulating the deep temporal architecture of human motor control. We explored the temporal depth of nested timescales, where successive levels of a forward or generative model unfold, for example, object delivery requires both global planning and local coordination of limb movements. This separation of temporal scales suggests the advantage of hierarchically organizing the global planning and local control of individual limbs. We validated our proposed formulation extensively through physics simulation. Using a hierarchical generative model, we showcase that an embodied artificial intelligence system, a humanoid robot, can autonomously complete a complex task requiring a holistic use of locomotion, manipulation and grasping: the robot adeptly retrieves and transports a box, opens and walks through a door, kicks a football and exhibits robust performance even in the presence of body damage and ground irregularities. Our findings demonstrated the efficacy and feasibility of human-inspired motor control for an embodied artificial intelligence robot, highlighting the viability of the formulized hierarchical architecture for achieving autonomous completion of challenging goal-directed tasks.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Interactive healthcare simulations can inform future healthcare robots and ease the global healthcare burden.
“Furthermore, these simulations could inform future healthcare robots (Yuan et al. 2023) and ease the global healthcare burden (McIntyre and Chow 2020), (i.e., 40% of the global population (WHO 2018).”
From Human-Clinical AI Agent Collaboration (2025) · cites this paper as Yuan et al. 2023
The Future directions section says the authors plan to transfer their simulation-validated model to physical robots, supporting the idea that simulations could inform future robots. It does not discuss healthcare simulations or healthcare robots. The separate claim about easing the global healthcare burden is attributed to McIntyre and Chow, not Yuan et al. (LLM, conf=0.84)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning
Sutton, R. S., Precup, D., & Singh, S. (1999) · Artificial Intelligence · doi:10.1016/s0004-3702(99)00052-1 · Cited by 3,119 · see citing works
Active inference: a process theory
Friston, K., FitzGerald, T., Rigoli, F., Schwartenbeck, P., & Pezzulo, G. (2017) · Neural Computation · doi:10.1162/neco_a_00912 · Cited by 1,156 · see citing works
Self-citationIs the cerebellum a Smith predictor?
Miall, R. C., Weir, D. J., Wolpert, D. M., & Stein, J. F. (1993) · Journal of Motor Behavior · doi:10.1080/00222895.1993.9942050 · Cited by 1,071 · see citing works
Slow cortical dynamics and the accumulation of information over long timescales
Honey, C. J., Thesen, T., Donner, T. H., Silbert, L. J., Carlson, C. E., Devinsky, O., Doyle, W. K., Rubin, N., Heeger, D. J., & Hasson, U. (2012) · Neuron · doi:10.1016/j.neuron.2012.08.011 · Cited by 532 · see citing works
The graphical brain: Belief propagation and active inference
Friston, K. J., Parr, T., & de Vries, B. (2017) · Network Neuroscience · doi:10.1162/netn_a_00018 · Cited by 415 · see citing works
Self-citationNeural bases of goal-directed locomotion in vertebrates—An overview
Grillner, S., Wallén, P., Saitoh, K., Kozlov, A., & Robertson, B. (2008) · Brain Research Reviews · doi:10.1016/j.brainresrev.2007.06.027 · Cited by 341 · see citing works
Hierarchical active inference: A theory of motivated control
Pezzulo, G., Rigoli, F., & Friston, K. J. (2018) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2018.01.009 · Cited by 334 · see citing works
Self-citationActive inference on discrete state-spaces: A synthesis
Da Costa, L., Parr, T., Sajid, N., Veselic, S., Neacsu, V., & Friston, K. (2020) · Journal of Mathematical Psychology · doi:10.1016/j.jmp.2020.102447 · Cited by 226 · cited 1× · see citing works
“For example, replacing our Q-learning planner with more sophisticated schemes which are designed to handle aleatoric and epistemic uncertainties (that is, expected free energy 50–52 ).”
Self-citationNew insights into action–perception coupling
Feldman, A. G. (2009) · Experimental Brain Research · doi:10.1007/s00221-008-1667-3 · Cited by 162 · see citing works
Hierarchical MOSAIC for movement generation.
Haruno, M., Wolpert, D. M., & Kawato, M. (2003) · International Congress Series · doi:10.1016/s0531-5131(03)00190-0 · Cited by 132 · see citing works
Automatic and controlled processing in the corticocerebellar system
Ramnani, N. (2014) · Progress in Brain Research · doi:10.1016/b978-0-444-63356-9.00010-8 · Cited by 70 · see citing works
Flagged first, then by citations
Research works printed but not linked: 29
- Merel, J. et al. Hierarchical visuomotor control of humanoids. Preprint at https://doi.org/10.48550/arXiv.1811.09656 (2018). · DOI
- Merel, J., Botvinick, M. & Wayne, G. Hierarchical motor control in mammals and machines. Nat. Commun. 10, 5489 (2019).
- Kheddar, A. et al. Humanoid robots in aircraft manufacturing: the airbus use cases. IEEE Robot. Autom. Mag. 26, 30-45 (2019).
- Schmaus, P. et al. IEEE Aerospace Conference (IEEE, 2019).
- Oliver, G., Lanillos, P. & Cheng, G. An empirical study of active inference on a humanoid robot. IEEE Trans. Cogn. Develop. Syst., 14, 462-471 (2021).
- Kumagai, I. et al. Toward industrialization of humanoid robots: autonomous plasterboard installation to improve safety and efficiency. IEEE Robot. Autom. Mag. 26, 20-29 (2019).
- Winkler, A. W., Bellicoso, C. D., Hutter, M. & Buchli, J. Gait and trajectory optimization for legged systems through phase- based end-effector parameterization. IEEE Robot. Autom. Let. 3, 1560-1567 (2018).
- Jain, D., Iscen, A. & Caluwaerts, K. 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (IEEE, 2019).
- Li, C., Xia, F., Martin-Martin, R. & Savarese, S. Hrl4in: hierarchical reinforcement learning for interactive navigation with mobile manipulators. In Proc. Conference on Robot Learning (eds.
- Kaelbling, L. P., Kragic, D. & Sugiura, K.) 603-616 (PMLR, 2020).
- Findeisen, W. et al. Control and Coordination in Hierarchical Systems (Wiley, 1980).
- Wolpert, D. M., Ghahramani, Z. & Jordan, M. Forward dynamic models in human motor control: psychophysical evidence. Adv. Neural Inf. Process. Syst. 7, 43-50 (1994).
- Attias, H. Planning by probabilistic inference. In Proc. Ninth International Workshop on Artificial Intelligence and Statistics (eds Bishop, C. M. and Frey, B. J.) 9-16 (PMLR, 2003).
- Matsumoto, T. & Tani, J. Goal-directed planning for habituated agents by active inference using a variational recurrent neural network. Entropy 22, 564 (2020).
- Baltieri, M. & Buckley, C. L. Generative models as parsimonious descriptions of sensorimotor loops. Behav. Brain Sci. 42, e218 (2019).
- Feldman, A. G. & Levin, M. F. in Progress in Motor Control (ed. Sternad, D.) 699-726 (Springer, 2009).
- Parr, T., Sajid, N. & Friston, K. J. Modules or mean-fields? Entropy 22, 552 (2020).
- Lanillos, P. et al. Active inference in robotics and artificial agents: survey and challenges. Preprint at https://doi.org/10.48550/ arXiv.2112.01871 (2021). · DOI
- Atkeson, C. G. et al. 2015 IEEE-RAS 15th International Conference on Humanoid Robots (Humanoids) (IEEE, 2015).
- Yuan, K. & Li, Z. Multi-expert synthesis for versatile locomotion and manipulation skills. Front. Robot. AI 9, 970890 (2022).
- Sajid, N. et al. Simulating lesion-dependent functional recovery mechanisms. Sci. Rep. 11, 7475 (2021).
- Akiba, T., Sano, S., Yanase, T., Ohta, T. & Koyama, M. Proc. 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (Association for Computing Machinery, 2019).
- Yuan, K., Chatzinikolaidis, I. & Li, Z. Bayesian optimization for whole-body control of high-degree-of-freedom robots through reduction of dimensionality. IEEE Robot. Autom. Lett. 4, 2268-2275 (2019).
- Hasselt, H. Double Q-learning. In Advances in Neural Information Processing Systems (eds Lafferty, J. et al.) 2613-2621 (Curran Associates Inc., 2010).
- Siciliano, B., Khatib, O. & Kröger, T. Springer Handbook of Robotics, Vol. 200 (Springer, 2008).
- Yuan, K., McGreavy, C., Yang, C., Wolfslag, W. & Li, Z. Decoding motor skills of artificial intelligence and human policies: a study on humanoid and human balance control. IEEE Robot. Autom. Mag. 27, 87-101 (2020).
- Haarnoja, T., Zhou, A., Abbeel, P. & Levine, S. Soft actor- critic: off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Proc. 35th International Conference on Machine Learning (eds Dy, J. & Krause, A.) 1861-1870 (PMLR, 2018).
- Yang, C., Yuan, K., Heng, S., Komura, T. & Li, Z. Learning natural locomotion behaviors for humanoid robots using human bias. IEEE Rob. Autom. Lett. 5, 2610-2617 (2020).
- McGreavy, C. et al. 2020 IEEE International Conference on Robotics and Automation (ICRA) (IEEE, 2020).
Cited by
Sorted by citations
Human-Clinical AI Agent Collaboration
Kadem, M., & Al-Khazraji, B. (2025) · Proceedings of the AAAI Symposium Series · doi:10.1609/aaaiss.v6i1.36060
“Furthermore, these simulations could inform future healthcare robots (Yuan et al. 2023) and ease the global healthcare burden (McIntyre and Chow 2020), (i.e., 40% of the global population (WHO 2018).”
- Authors
- Yuan, K., Sajid, N., Friston, K., & Li, Z.
- Affiliations
- Intel (Germany) (DE)
- Wellcome Centre for Human Neuroimaging (GB)
- University College London (GB)
- Corresponding author
- Li, Zhibin
- Venue
- Nature Machine Intelligence
- Published
- 2023-11-02
- Volume
- 5
- Issue
- 12
- Pages
- 1402-1414
- ISSN
- 2522-5839 (ISSN-L)
- OpenAlex
- W4388216531
- Cited by
- 23
- References
- 11
- Citations by year
- 2026: 7 · 2025: 10 · 2024: 6
- FWCI
- 2.17
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- Nature Machine Intelligence (publishedVersion, cc-by)
- Research Square (acceptedVersion, cc-by)
- Funders
- University College London (2021-2022)
- Medical Research Council Canada
- Medical Research Council (MR/S502522/1)
- Engineering and Physical Sciences Research Council (EP/L016834/1)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
- Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
- Keywords
- hierarchical generative models, autonomous robots, humanoid robots, motor control, multi-level planning, time scales, limb coordination, embodied AI, grasping, human-inspired control, physics simulation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
