Dialfred: Dialogue-enabled agents for embodied instruction following
Open accessAbstract
Language-guided Embodied AI benchmarks requiring an agent to navigate an environment and manipulate objects typically allow one-way communication: the human user gives a natural language command to the agent, and the agent can only follow the command passively. We presentDialFRED, a dialogue-enabled embodied instruction following benchmark based on the ALFRED benchmark. DialFRED allows an agent to actively ask questions to the human user; the additional information in the user's response is used by the agent to better complete its task. We release a human-annotated dataset with 53 K task-relevant questions and answers and an oracle to answer questions. To tackle DialFRED, we propose a questioner-performer framework wherein the questioner is pre-trained with the human-annotated data and fine-tuned with reinforcement learning. Experimental results show that asking the right questions leads to significantly improved task performance. We make DialFRED publicly available and encourage researchers to propose and evaluate their solutions to building dialog-enabled embodied agents:https://github.com/xfgao/DialFRED.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Integrating visual input, language processing, and spatial reasoning lets robots follow natural language instructions, navigate to visual goals, or perform goal-directed tasks with greater flexibility and adaptability.
“By integrating visual input, language processing, and spatial reasoning, robots can now follow natural language instructions, navigate toward visual goals, or perform goal-directed navigation tasks with greater flexibility and adaptability”
From Multimodal fusion and vision–language models: A survey for robot vision (2026) · cites this paper as [128]
The Baselines (p. 6) describe an agent using language instructions and visual observations to predict actions, and the Related Work (p. 2) identifies tasks requiring navigation and manipulation. The Results (p. 7) report improved task performance when question-answer dialogue is added. The paper does not establish the broader claim that integrating spatial reasoning enables navigation toward visual goals or makes robots generally more flexible and adaptable. (LLM, conf=0.84)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
The citation lists haven’t been fetched yet
References 37 · Cited by 55
Research works printed but not linked: 3
- [3][3] V. Blukis, C. Paxton, D. Fox, A. Garg, and Y. Artzi, 'A persistent spatial semantic representation for high-level natural language instruction execution,' in Conference on Robot Learning , 2021, pp. 706-717. -
- [4][4] S. Y. Min, D. S. Chaplot, P. K. Ravikumar, Y. Bisk, and R. Salakhutdinov, 'FILM: Following instructions in language with modular methods,' in International Conference on Learning Representations , 2022. -
- [10][10] J. Thomason, M. Murray, M. Cakmak, and L. Zettlemoyer, 'Visionand-dialog navigation,' in Conference on Robot Learning , 2020, pp. 394-406. -
- Authors
- Gao, X., Gao, Q., Gong, R., Lin, K., Thattai, G., & Sukhatme, G. S.
- Affiliations
- University of California, Los Angeles (US)
- Amazon (United States) (US)
- University of Southern California (US)
- Venue
- IEEE Robotics and Automation Letters
- Published
- 2022-10
- Volume
- 7
- Issue
- 4
- Pages
- 10049-10056
- ISSN
- 2377-3766 (ISSN-L)
- 2377-3774
- OpenAlex
- W4225675121
- Cited by
- 55
- References
- 37
- Citations by year
- 2026: 14 · 2025: 16 · 2024: 6 · 2023: 17 · 2022: 2
- FWCI
- 4.13
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE Robotics and Automation Letters (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Amazon Alexa AI
- Topics
- Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Keywords
- questioning, reinforcement learning, human-robot dialogue, object manipulation, navigation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
