CoChat
Sign Up
Cited by 134Robot Manipulation and LearningAI-based Problem Solving and PlanningReinforcement Learning in Robotics

The DOI matches its Crossref record; 59 of the 59 references it cites were found in a registry or an index.

Embodied large language models enable robots to complete complex tasks in unpredictable environments

Open access
Mon-Williams, Ruaridh, Li, Gen, Long, Ran, Du, Wenqian, Lucas, Christopher G.·2025·Nature Machine Intelligence
DOI 10.1038/s42256-025-01005-xPMC PMC12088599

Published version of a preprint

Open this paper in CoChat — free account

Abstract

Completing complex tasks in unpredictable settings challenges robotic systems, requiring a step change in machine intelligence. Sensorimotor abilities are considered integral to human intelligence. Thus, biologically inspired machine intelligence might usefully combine artificial intelligence with robotic sensorimotor capabilities. Here we report an embodied large-language-model-enabled robot (ELLMER) framework, utilizing GPT-4 and a retrieval-augmented generation infrastructure, to enable robots to complete long-horizon tasks in unpredictable settings. The method extracts contextually relevant examples from a knowledge base, producing action plans that incorporate force and visual feedback and enabling adaptation to changing conditions. We tested ELLMER on a robot tasked with coffee making and plate decoration; these tasks consist of a sequence of sub-tasks from drawer opening to pouring, each benefiting from distinct feedback types and methods. We show that the ELLMER framework allows the robot to complete the tasks. This demonstration marks progress towards scalable, efficient and ‘intelligent robots’ able to complete complex tasks in uncertain environments.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1038/s42256-025-01005-x

Cites

59 of 59 references verified0 citation errors · 0 not checkable

Statistics

Figures

Fig. 1 Schematic of the system framework. The schematic illustrates the system framework, showing the high-level (above the blue dashed horizontal line) and low-level (below the blue dashed horizontal line) system architecture. User queries are fed into a transformer via voice recognition software. The transformer (GPT-4) takes this input and integrates it with (i) an image (C) of the environment (via an azure Kinect depth camera); (ii) knowledge base of code examples, including various functions stored in a database. The transformer can decompose the higher-order abstracted task into actionable high-level sub-tasks, retrieve relevant code examples from the knowledge base, adapt them and write Python (v.3.8) code tailored to these tasks. The resultant code is then sent to the robot controller (A). The controller processes the code and sends control signals (λ) to the robot. The actions (a) are controlled with force (F) and vision (V) feedback. The model uses vision to identify the properties of different objects (for example, pose X of a coffee cup), so it can grasp objects accurately. The robot uses force (f) and torque (τ) feedback (available via an ATI force transducer) to manipulate objects skilfully (for example, determine how much water to pour). Feedback is necessary due to noise within the vision signal (ηvision), the robot joint angles (ηangle) and the force transducer signal (ηforce). The feedback updates the motion in the ROS to achieve the desired goal through velocity commands of both linear (vxyz) and angular (vrpy) velocities. These commands generate trajectories based on appropriate forces and spatiotemporal patterns to achieve the sub-goals. The use of feedback loops, including 40-Hz updates of the end-effector position (p) and orientation (q), allow the robot to respond to disturbance (for example, the robot tracking a cup to determine its new position after it is moved by the user).
Fig. 1 Schematic of the system framework. The schematic illustrates the system framework, showing the high-level (above the blue dashed horizontal line) and low-level (below the blue dashed horizontal line) system architecture. User queries are fed into a transformer via voice recognition software. The transformer (GPT-4) takes this input and integrates it with (i) an image (C) of the environment (via an azure Kinect depth camera); (ii) knowledge base of code examples, including various functions stored in a database. The transformer can decompose the higher-order abstracted task into actionable high-level sub-tasks, retrieve relevant code examples from the knowledge base, adapt them and write Python (v.3.8) code tailored to these tasks. The resultant code is then sent to the robot controller (A). The controller processes the code and sends control signals (λ) to the robot. The actions (a) are controlled with force (F) and vision (V) feedback. The model uses vision to identify the properties of different objects (for example, pose X of a coffee cup), so it can grasp objects accurately. The robot uses force (f) and torque (τ) feedback (available via an ATI force transducer) to manipulate objects skilfully (for example, determine how much water to pour). Feedback is necessary due to noise within the vision signal (ηvision), the robot joint angles (ηangle) and the force transducer signal (ηforce). The feedback updates the motion in the ROS to achieve the desired goal through velocity commands of both linear (vxyz) and angular (vrpy) velocities. These commands generate trajectories based on appropriate forces and spatiotemporal patterns to achieve the sub-goals. The use of feedback loops, including 40-Hz updates of the end-effector position (p) and orientation (q), allow the robot to respond to disturbance (for example, the robot tracking a cup to determine its new position after it is moved by the user).CC BY · PubMed Central
Fig. 2 Kinova robot in action. a–f, Action shots of the Kinova Gen3 robot preparing coffee (a–e) and decorating a plate (f).
Fig. 2 Kinova robot in action. a–f, Action shots of the Kinova Gen3 robot preparing coffee (a–e) and decorating a plate (f).CC BY · PubMed Central
Fig. 3 Vision detection module. Illustration of the zero-shot vision detection module identifying a hand, white mug and black kettle, as well as extracting target poses for robotic grasping.
Fig. 3 Vision detection module. Illustration of the zero-shot vision detection module identifying a hand, white mug and black kettle, as well as extracting target poses for robotic grasping.CC BY · PubMed Central
Fig. 4 Force, velocity and position feedback. Force (N), velocity (m s–1) and position (m) plots during the robot’s coffee preparation, illustrating the diverse force feedback across the different motions. The drawing component has been omitted for clarity.
Fig. 4 Force, velocity and position feedback. Force (N), velocity (m s–1) and position (m) plots during the robot’s coffee preparation, illustrating the diverse force feedback across the different motions. The drawing component has been omitted for clarity.CC BY · PubMed Central
Fig. 5 Drawing process visualization. Illustration of the drawing process across different queries. The top row shows the generated image, contour plot and drawing produced when instructed to create a ‘random animal’. The second row displays the corresponding outputs for a ‘random food’ and the third row illustrates the results for a ‘random plant’.
Fig. 5 Drawing process visualization. Illustration of the drawing process across different queries. The top row shows the generated image, contour plot and drawing produced when instructed to create a ‘random animal’. The second row displays the corresponding outputs for a ‘random food’ and the third row illustrates the results for a ‘random plant’.CC BY · PubMed Central
Fig. 6 Coffee and plate decoration. Kinova Gen3 robot having prepared the coffee and decorated a plate.
Fig. 6 Coffee and plate decoration. Kinova Gen3 robot having prepared the coffee and decorated a plate.CC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 58 references

Research works printed but not linked: 8

  1. Hussein, A., Gaber, M. M., Elyan, E. & Jayne, C. Imitation learning: a survey of learning methods. ACM Comput. Surv. 50, 21:1–21:35 (2017).
  2. Shridhar, M., Manuelli, L. & Fox, D. Perceiver-Actor: a multi-task transformer for robotic manipulation. In Proc. 6th Conference on Robot Learning 785–799 (PMLR, 2023).
  3. Huang, W., Abbeel, P., Pathak, D. & Mordatch, I. Language models as zero-shot planners: extracting actionable knowledge for embodied agents. In Proc. 39th International Conference on Machine Learning 9118–9147 (PMLR, 2022).
  4. Belkhale, S., Cui, Y. & Sadigh, D. Data quality in imitation learning. In Advances in Neural Information Processing Systems (NeurIPS) 80375–80395 (Curran Associates, 2024).
  5. Chang, J., Uehara, M., Sreenivas, D., Kidambi, R. & Sun, W. Mitigating covariate shift in imitation learning via offline data with partial coverage. In Advances in Neural Information Processing Systems 965–979 (Curran Associates, 2021).
  6. Nair, S., Rajeswaran, A., Kumar, V., Finn, C. & Gupta, A. R3M: a universal visual representation for robot manipulation. In Proc. 6th Conference on Robot Learning Vol. 205, 892–909 (PMLR, 2022).
  7. Hong, J., Levine, S. & Dragan, A. Learning to influence human behavior with offline reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS) 36094–36105 (Curran Associates, 2024).
  8. OpenAI. GPT-4 technical report. Preprint at http://arxiv.org/abs/2303.08774 (2024). · arXiv

Cited by

Sorted by citations

Related

See all

Metadata

Authors
Mon-Williams, R., Li, G., Long, R., Du, W., & Lucas, C. G.
Affiliations
  • Princeton University (US)
  • University of Edinburgh (GB)
  • The Alan Turing Institute (GB)
Corresponding author
  • Mon-Williams, Ruaridh
  • Li, Gen
ORCID
Venue
Nature Machine Intelligence
Publisher
Springer Science and Business Media LLC
Published
2025-03-19
Volume
7
Issue
4
Pages
592-601
ISSN
2522-5839 (ISSN-L)
DOI
10.1038/s42256-025-01005-x
PMC
PMC12088599
OpenAlex
W4408555470
Cited by
132
References
58
Citations by year
2026: 94 · 2025: 38
FWCI
48.49
Type
journal article
Language
en
Open access
Licence
Locations
  • Nature Machine Intelligence (publishedVersion, cc-by)
  • Research Square (acceptedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • Edinburgh Research Explorer (University of Edinburgh) (submittedVersion, cc-by)
Funders
RCUK | Engineering and Physical Sciences Research Council (EP/L016834/1, EP/S023208/1)
Topics
  • Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
Keywords
unpredictable environments, GPT-4, retrieval-augmented generation, sensorimotor feedback, force feedback, visual feedback, action planning, long-horizon tasks, intelligent robots

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review