CoChat
Sign Up
Cited by 105Speech and dialogue systemsSocial Robot Interaction and HRIRobot Manipulation and Learning

TEACh: Task-Driven Embodied Agents That Chat

Open access
Padmakumar, Aishwarya, Thomason, Jesse, Shrivastava, Ayush, Lange, Patrick, Narayan-Chen, Anjali, Gella, Spandana, Piramuthu, Robinson, Tur, Gokhan +1 more·2022·Proceedings of the AAAI Conference on Artificial Intelligence
DOI 10.1609/aaai.v36i2.20097
Open this paper in CoChat — free account

Abstract

Robots operating in human spaces must be able to engage in natural language interaction, both understanding and executing instructions, and using conversation to resolve ambiguity and correct mistakes. To study this, we introduce TEACh, a dataset of over 3,000 human-human, interactive dialogues to complete household tasks in simulation. A Commander with access to oracle information about a task communicates in natural language with a Follower. The Follower navigates through and interacts with the environment to complete tasks varying in complexity from "Make Coffee" to "Prepare Breakfast", asking questions and getting additional information from the Commander. We propose three benchmarks using TEACh to study embodied intelligence challenges, and we evaluate initial models' abilities in dialogue understanding, language grounding, and task execution.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1609/aaai.v36i2.20097

Cites

2 of 2 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • HM3D and Gibson contain abundant, traversable, simulated indoor 3D scans collected from real-world buildings, which support learning of various visual navigation problems.

    “Both datasets contain abundant, traversable, and simulated indoor 3D scans collected from real-world buildings, which support the learning of various visual navigation problems [12, 44, 55, 70].”

    From Scaling Data Generation in Vision-and-Language Navigation (2023) · cites this paper as [55]

    The Related Work section (p. 2) discusses photorealistic indoor environments for vision-and-language navigation, but does not identify HM3D or Gibson. Searches for those dataset names and for real-world building scans found no passage in our copy addressing their contents or use for navigation learning. (LLM, conf=0.72)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 34

  1. Abramson, J.; Ahuja, A.; Brussee, A.; Carnevale, F.; Cassin, M.; Clark, S.; Dudzik, A.; Georgiev, P.; Guy, A.; Harley, T.; Hill, F.; Hung, A.; Kenton, Z.; Landon, J.; Lillicrap, T.; Mathewson, K.; Muldal, A.; Santoro, A.; Savinov, N.; Varma, V.; Wayne, G.; Wong, N.; Yan, C.; and Zhu, R. 2020. Imitating Interactive Intelligence. arXiv .
  2. Anderson, P.; Wu, Q.; Teney, D.; Bruce, J.; Johnson, M.; S¨ underhauf, N.; Reid, I.; Gould, S.; and van den Hengel, A. 2018. Vision-and-Language Navigation: Interpreting Visually-Grounded Navigation Instructions in Real Environments. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) .
  3. Bisk, Y.; Holtzman, A.; Thomason, J.; Andreas, J.; Bengio, Y.; Chai, J.; Lapata, M.; Lazaridou, A.; May, J.; Nisnevich, A.; Pinto, N.; and Turian, J. 2020. Experience Grounds Language. In Empirical Methods in Natural Language Processing (EMNLP) .
  4. Bisk, Y.; Shih, K.; Choi, Y.; and Marcu, D. 2018. Learning Interpretable Spatial Operations in a Rich 3D Blocks World. In Proceedings of the Thirty Second AAAI Conference on Artificial Intelligence (AAAI) , volume 32.
  5. Blukis, V.; Misra, D.; Knepper, R. A.; and Artzi, Y. 2018. Mapping Navigation Instructions to Continuous Control Actions with Position Visitation Prediction. In Proceedings of the Conference on Robot Learning (CoRL) .
  6. Blukis, V.; Paxton, C.; Fox, D.; Garg, A.; and Artzi, Y . 2021. APersistent Spatial Semantic Representation for High-level Natural Language Instruction Execution. arXiv preprint arXiv:2107.05612 . · arXiv
  7. Chen, D.; and Mooney, R. 2011. Learning to Interpret Natural Language Navigation Instructions from Observations. In Proceedings of the Twenty Fifth AAAI Conference on Artificial Intelligence (AAAI) , volume 25.
  8. Chen, H.; Suhr, A.; Misra, D.; Snavely, N.; and Artzi, Y. 2019. Touchdown: Natural Language Navigation and Spatial Reasoning in Visual Street Environments. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 12538-12547.
  9. Chi, T.-C.; Shen, M.; Eric, M.; Kim, S.; and Hakkani-T¨ ur, D. 2020. Just Ask: An Interactive Learning Framework for Vision and Language Navigation. In Proceedings of the AAAI Conference on Artificial Intelligence .
  10. Harnad, S. 1990. The Symbol Grounding Problem. Physica D: Nonlinear Phenomena , 42(1-3): 335-346.
  11. He, K.; Gkioxari, G.; Doll´ ar, P.; and Girshick, R. B. 2017. Mask R-CNN. International Conference on Computer Vision (ICCV) .
  12. Kim, B.; Bhambri, S.; Singh, K. P.; Mottaghi, R.; and Choi, J. 2021. Agent with the Big Picture: Perceiving Surroundings for Interactive Instruction Following. In Embodied AI Workshop CVPR .
  13. Kim, H.; Zala, A.; Burri, G.; Tan, H.; and Bansal, M. 2020. ArraMon: A Joint Navigation-Assembly Instruction Interpretation Task in Dynamic Environments. In Findings of the Association for Computational Linguistics: EMNLP 2020 .
  14. Kollar, T.; Tellex, S.; Walter, M. R.; Huang, A.; Bachrach, A.; Hemachandra, S.; Brunskill, E.; Banerjee, A.; Roy, D.; Teller, S.; et al. 2013. Generalized Grounding Graphs: A Probabilistic Framework for Understanding Grounded Language. Journal of Artificial Intelligence Research .
  15. MacMahon, M.; Stankiewicz, B.; and Kuipers, B. 2006. Walk the Talk: Connecting Language, Knowledge, and Action in Route Instructions. In Proceedings of the Twentieth AAAI Conference on Artifial Intelligence (AAAI) , volume 20.
  16. Matuszek, C.; Herbst, E.; Zettlemoyer, L.; and Fox, D. 2013. Learning to Parse Natural Language Commands to a Robot Control System. In Experimental Robotics , 403-415. Springer.
  17. Mei, H.; Bansal, M.; and Walter, M. 2016. Listen, attend, and walk: Neural mapping of navigational instructions to action sequences. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence (AAAI) , volume 30.
  18. Misra, D. K.; Bennett, A.; Blukis, V.; Niklasson, E.; Shatkhin, M.; and Artzi, Y. 2018. Mapping Instructions to Actions in 3D Environments with Visual Goal Prediction. In Riloff, E.; Chiang, D.; Hockenmaier, J.; and Tsujii, J., eds., Empirical Methods in Natural (EMNLP) .
  19. Narayan-Chen, A.; Jayannavar, P.; and Hockenmaier, J. 2019. Collaborative Dialogue in Minecraft. In Association for Computational Linguistics (ACL) .
  20. Nguyen, K.; and Daum´ e III, H. 2019. Help, Anna! Visual Navigation with Natural Multimodal Assistance via Retrospective Curiosity-Encouraging Imitation Learning. In Conference on Empirical Methods in Natural Language Processing and International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) .
  21. Pashevich, A.; Schmid, C.; and Sun, C. 2021. Episodic Transformer for Vision-and-Language Navigation. arXiv preprint arXiv:2105.06453 . · arXiv
  22. Puig, X.; Ra, K.; Boben, M.; Li, J.; Wang, T.; Fidler, S.; and Torralba, A. 2018. VirtualHome: Simulating Household Activities via Programs. In Computer Vision and Pattern Recognition (CVPR) .
  23. Roman, H. R.; Bisk, Y.; Thomason, J.; Celikyilmaz, A.; and Gao, J. 2020. RMM: A Recursive Mental Model for Dialog Navigation. In Findings of Empirical Methods in Natural Language Processing (EMNLP Findings) .
  24. Shah, R.; Wild, C.; Wang, S. H.; Alex, N.; Houghton, B.; Guss, W.; Mohanty, S.; Kanervisto, A.; Milani, S.; Topin, N.; et al. 2021. The MineRL BASALT Competition on Learning from Human Feedback. arXiv preprint arXiv:2107.01969 . · arXiv
  25. Shridhar, M.; Thomason, J.; Gordon, D.; Bisk, Y.; Han, W.; Mottaghi, R.; Zettlemoyer, L.; and Fox, D. 2020. ALFRED: ABenchmark for Interpreting Grounded Instructions for Everyday Tasks. In Computer Vision and Pattern Recognition (CVPR) .
  26. Shrivastava, A.; Gopalakrishnan, K.; Liu, Y.; Piramuthu, R.; T¨ ur, G.; Parikh, D.; and Hakkani-T¨ ur, D. 2021. VISITRON: Visual Semantics-Aligned Interactively Trained Object-Navigator. arXiv preprint arXiv:2105.11589 . · arXiv
  27. Suglia, A.; Gao, Q.; Thomason, J.; Thattai, G.; and Sukhatme, G. 2021. Embodied BERT: A Transformer Model for Embodied, Language-guided Visual Task Completion. arXiv preprint arXiv:2108.04927 . · arXiv
  28. Suhr, A.; Yan, C.; Schluger, J.; Yu, S.; Khader, H.; Mouallem, M.; Zhang, I.; and Artzi, Y. 2019. Executing Instructions in Situated Collaborative Interactions. In Empirical Methods in Natural Language Processing and International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) .
  29. Tellex, S.; Knepper, R. A.; Li, A.; Roy, N.; and Rus, D. 2016. Asking for Help Using Inverse Semantics. In Robotics: Science and Systems Conference (RSS) .
  30. Thomason, J.; Gordon, D.; and Bisk, Y. 2018. Shifting the Baseline: Single Modality Performance on Visual Navigation & QA. arXiv preprint arXiv:1811.00613 . · arXiv
  31. Thomason, J.; Murray, M.; Cakmak, M.; and Zettlemoyer, L. 2019. Vision-and-Dialog Navigation. In Conference on Robot Learning (CoRL) .
  32. Thomason, J.; Padmakumar, A.; Sinapov, J.; Walker, N.; Jiang, Y.; Yedidsion, H.; Hart, J.; Stone, P.; and Mooney, R. 2020. Jointly improving parsing and perception for natural language commands through human-robot dialog. Journal of Artificial Intelligence Research , 67: 327-374.
  33. Zhang, Y.; and Chai, J. 2021. Hierarchical Task Learning from Language Instructions with Unified Transformers and Self-Monitoring. arXiv preprint arXiv:2106.03427 . · arXiv
  34. Zhu, W.; Hu, H.; Chen, J.; Deng, Z.; Jain, V.; Ie, E.; and Sha, F. 2020. BabyWalk: Going Farther in Vision-and-Language Navigation by Taking Baby Steps. In Association for Computational Linguistics (ACL) .

Cited by

Sorted by citations

Authors
Padmakumar, A., Thomason, J., Shrivastava, A., Lange, P., Narayan-Chen, A., Gella, S., Piramuthu, R., Tur, G., & Hakkani-Tur, D.
Affiliations
  • Amazon (United States) (US)
  • University of Southern California (US)
  • University of Michigan (US)
ORCID
Venue
Proceedings of the AAAI Conference on Artificial Intelligence
Publisher
Association for the Advancement of Artificial Intelligence (AAAI)
Published
2022-06-28
Volume
36
Issue
2
Pages
2017-2025
ISSN
  • 2159-5399 (ISSN-L)
  • 2374-3468
DOI
10.1609/aaai.v36i2.20097
OpenAlex
W3203663566
Cited by
105
References
2
Citations by year
2026: 18 · 2025: 19 · 2024: 16 · 2023: 32 · 2022: 20
FWCI
12.03
Type
conference paper
Language
en
Open access
Locations
  • Proceedings of the AAAI Conference on Artificial Intelligence (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
  • Social Sciences › Psychology › Social Psychology › Social Robot Interaction and HRI
  • Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
Keywords
embodied agents, human-robot dialogue, dialogue dataset, natural language interaction, language grounding, dialogue understanding, task execution, benchmarking

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review