CoChat
Sign Up
Cited by 115Multimodal Machine Learning ApplicationsSpeech and dialogue systemsReinforcement Learning in Robotics

Help, Anna! Visual Navigation with Natural Multimodal Assistance via Retrospective Curiosity-Encouraging Imitation Learning

Open access
Nguyen, Khanh, Daumé III, Hal·2019·Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)
DOI 10.18653/v1/d19-1063
Open this paper in CoChat — free accountOpen PDF

Abstract

Khanh Nguyen, Hal Daumé III. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.18653/v1/d19-1063

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Integrating visual input, language processing, and spatial reasoning lets robots follow natural language instructions, navigate to visual goals, or perform goal-directed tasks with greater flexibility and adaptability.

    “By integrating visual input, language processing, and spatial reasoning, robots can now follow natural language instructions, navigate toward visual goals, or perform goal-directed navigation tasks with greater flexibility and adaptability”

    From Multimodal fusion and vision–language models: A survey for robot vision (2026) · cites this paper as [126]

    The task description in §2 describes a simulated navigation agent using camera images, natural-language instructions, and destination images to reach a requested object. The results report that language-based assistance improves task completion in novel environments. The paper does not establish the broader claim about physical robots or identify spatial reasoning as a distinct capability responsible for greater flexibility and adaptability; page numbers are unavailable in our copy. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

The citation lists haven’t been fetched yet

References 53 · Cited by 115

Research works printed but not linked: 41

  1. Peter Anderson, Angel Chang, Devendra Singh Chap- lot, Alexey Dosovitskiy, Saurabh Gupta, Vladlen Koltun, Jana Kosecka, Jitendra Malik, Roozbeh Mottaghi, Manolis Savva, et al. 2018a. On evalu- ation of embodied navigation agents. arXiv preprint arXiv:1807.06757. · arXiv
  2. Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton van den Hengel. 2018b. Vision- and-language navigation: Interpreting visually- grounded navigation instructions in real environ- ments. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3674-3683.
  3. Marcin Andrychowicz, Bowen Baker, Maciek Chociej, Rafal Jozefowicz, Bob McGrew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, et al. 2018. Learning dexterous in-hand manipulation. arXiv preprint arXiv:1808.00177. · arXiv
  4. Joyce Y Chai, Rui Fang, Changsong Liu, and Lanbo She. 2016. Collaborative language grounding to- ward situated human-robot dialogue. AI Magazine, 37(4):32-45.
  5. Angel Chang, Angela Dai, Thomas Funkhouser, Ma- ciej Halber, Matthias Niessner, Manolis Savva, Shu- ran Song, Andy Zeng, and Yinda Zhang. 2017. Mat- terport3D: Learning from RGB-D data in indoor en- vironments. International Conference on 3D Vision (3DV).
  6. Kai-Wei Chang, Akshay Krishnamurthy, Alekh Agar- wal, Hal Daume III, and John Langford. 2015. Learning to search better than your teacher. In Pro- ceedings of the International Conference of Machine Learning.
  7. Howard Chen, Alane Shur, Dipendra Misra, Noah Snavely, Ian Artzi, Yoav, Stephen Gould, and An- ton van den Hengel. 2019. Touchdown: Natural language navigation and spatial reasoning in visual street environments. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recog- nition (CVPR).
  8. Maxime Chevalier-Boisvert, Dzmitry Bahdanau, Salem Lahlou, Lucas Willems, Chitwan Saharia, Thien Huu Nguyen, and Yoshua Bengio. 2019. Babyai: A platform to study the sample efficiency of grounded language learning. In Proceedings of the International Conference on Learning Representations.
  9. Marc-Alexandre Côté, Ákos Kádár, Xingdi Yuan, Ben Kybartas, Tavian Barnes, Emery Fine, James Moore, Matthew Hausknecht, Layla El Asri, Mah- moud Adada, Wendy Tay, and Adam Trischler. 2018. Textworld: A learning environment for text- based games. In Computer Games Workshop at ICML/IJCAI.
  10. Abhishek Das, Samyak Datta, Georgia Gkioxari, Ste- fan Lee, Devi Parikh, and Dhruv Batra. 2018. Em- bodied question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.
  11. Hal Daumé III, John Langford, and Daniel Marcu. 2009. Search-based structured prediction. In Machine learning, volume 75, pages 297-325. Springer.
  12. He He, Jason Eisner, and Hal Daumé III. 2012. Imi- tation learning by coaching. In Proceedings of Ad- vances in Neural Information Processing Systems, pages 3149-3157.
  13. Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Ju- lian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, et al. 2018. Qt-opt: Scalable deep rein- forcement learning for vision-based robotic manip- ulation. In Proceedings of the Conference on Robot Learning.
  14. Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. 2017. Progressive growing of gans for improved quality, stability, and variation. In Pro- ceedings of the International Conference on Learn- ing Representations.
  15. Janne Karttunen, Anssi Kanervisto, Ville Hautamäki, and Ville Kyrki. 2019. From video game to real robot: The transfer between action spaces. arXiv preprint arXiv:1905.00741. · arXiv
  16. Michał Kempka, Marek Wydmuch, Grzegorz Runc, Jakub Toczek, and Wojciech Jaśkowski. 2016. Viz- doom: A doom-based ai research platform for visual reinforcement learning. In Computational Intelli- gence and Games (CIG), 2016 IEEE Conference on, pages 1-8. IEEE.
  17. Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hin- ton. 2016. Layer normalization. arXiv preprint arXiv:1607.06450. · arXiv
  18. Changsong Liu, Shaohua Yang, Sari Saba-Sadiya, Nishant Shukla, Yunzhong He, Song-Chun Zhu, and Joyce Chai. 2016. Jointly learning grounded task structures from language instruction and visual demonstration. In Proceedings of Emperical Meth- ods in Natural Language Processing, pages 1482- 1492.
  19. Chih-Yao Ma, Zuxuan Wu, Ghassan AlRegib, Caim- ing Xiong, and Zsolt Kira. 2019b. The regretful agent: Heuristic-aided navigation through progress estimation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6732-6740.
  20. Dipendra Misra, Andrew Bennett, Valts Blukis, Eyvind Niklasson, Max Shatkhin, and Yoav Artzi. 2018. Mapping instructions to actions in 3d environments with visual goal prediction. In Proceedings of Em- perical Methods in Natural Language Processing, pages 2667-2678. Association for Computational Linguistics.
  21. Dipendra Misra, John Langford, and Yoav Artzi. 2017. Mapping instructions and visual observations to ac- tions with reinforcement learning. Proceedings of Emperical Methods in Natural Language Process- ing.
  22. Khanh Nguyen, Debadeepta Dey, Chris Brockett, and Bill Dolan. 2019. Vision-based navigation with language-based assistance via imitation learning with indirect intervention. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 12527-12537.
  23. Yuankai Qi, Qi Wu, Peter Anderson, Marco Liu, Chunhua Shen, and Anton van den Hengel. 2019. Rerere: Remote embodied referring expressions in real indoor environments. arXiv preprint arXiv:1904.10151. · arXiv
  24. Stephane Ross and J Andrew Bagnell. 2014. Rein- forcement and imitation learning via interactive no- regret learning. arXiv preprint arXiv:1406.5979. · arXiv
  25. Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. 2011. A reduction of imitation learning and struc- tured prediction to no-regret online learning. In Proceedings of Artificial Intelligence and Statistics, pages 627-635.
  26. Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. 2015. Imagenet large scale visual recognition chal- lenge. International Journal of Computer Vision, 115(3):211-252.
  27. Paul E Rybski, Kevin Yoon, Jeremy Stolarz, and Manuela M Veloso. 2007. Interactive robot task training through dialog and demonstration. In Pro- ceedings of the ACM/IEEE international conference on Human-robot interaction, pages 49-56. ACM.
  28. Amr Sharaf and Hal Daumé III. 2017. Structured pre- diction via learning to search under bandit feedback. In Proceedings of the 2nd Workshop on Structured Prediction for Natural Language Processing, pages 17-26.
  29. Lanbo She, Shaohua Yang, Yu Cheng, Yunyi Jia, Joyce Chai, and Ning Xi. 2014. Back to the blocks world: Learning new actions through situated human-robot dialogue. In Proceedings of the 15th Annual Meet- ing of the Special Interest Group on Discourse and Dialogue (SIGDIAL), pages 89-97.
  30. Wen Sun, Arun Venkatraman, Geoffrey J Gordon, By- ron Boots, and J Andrew Bagnell. 2017. Deeply aggrevated: Differentiable imitation learning for se- quential prediction. In Proceedings of the Interna- tional Conference of Machine Learning.
  31. Jesse Thomason, Daniel Gordan, and Yonatan Bisk. 2019a. Shifting the baseline: Single modality per- formance on visual navigation & qa. In Conference of the North American Chapter of the Association for Computational Linguistics.
  32. Jesse Thomason, Michael Murray, Maya Cakmak, and Luke Zettlemoyer. 2019b. Vision-and-dialog navi- gation. In Proceedings of the Conference on Robot Learning.
  33. Emanuel Todorov, Tom Erez, and Yuval Tassa. 2012. Mujoco: A physics engine for model-based con- trol. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026-5033. IEEE.
  34. Jack Urbanek, Angela Fan, Siddharth Karamcheti, Saachi Jain, Samuel Humeau, Emily Dinan, Tim Rocktäschel, Douwe Kiela, Arthur Szlam, and Ja- son Weston. 2019. Learning to speak and act in a fantasy text adventure game. arXiv preprint arXiv:1903.03094. · arXiv
  35. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information pro- cessing systems, pages 5998-6008.
  36. Oriol Vinyals, Timo Ewalds, Sergey Bartunov, Petko Georgiev, Alexander Sasha Vezhnevets, Michelle Yeo, Alireza Makhzani, Heinrich Küttler, John Aga- piou, Julian Schrittwieser, et al. 2017. Starcraft ii: A new challenge for reinforcement learning. arXiv preprint arXiv:1708.04782. · arXiv
  37. Harm de Vries, Kurt Shuster, Dhruv Batra, Devi Parikh, Jason Weston, and Douwe Kiela. 2018. Talk the walk: Navigating new york city through grounded dialogue. arXiv preprint arXiv:1807.03367. · arXiv
  38. Xin Wang, Qiuyuan Huang, Asli Celikyilmaz, Jian- feng Gao, Dinghan Shen, Yuan-Fang Wang, William Yang Wang, and Lei Zhang. 2019. Re- inforced cross-modal matching and self-supervised imitation learning for vision-language navigation. In Proceedings of the IEEE Conference on Com- puter Vision and Pattern Recognition.
  39. Xin Wang, Wenhan Xiong, Hongmin Wang, and William Yang Wang. 2018. Look before you leap: Bridging model-free and model-based rein- forcement learning for planned-ahead vision-and- language navigation. In Proceedings of the Euro- pean Conference on Computer Vision.
  40. Sean Welleck, Ilia Kulikov, Stephen Roller, Emily Dinan, Kyunghyun Cho, and Jason Weston. 2019. Neural text generation with unlikelihood training. arXiv preprint arXiv:1908.04319. · arXiv
  41. Erik Wijmans, Samyak Datta, Oleksandr Maksymets, Abhishek Das, Georgia Gkioxari, Stefan Lee, Irfan Essa, Devi Parikh, and Dhruv Batra. 2019. Em- bodied question answering in photorealistic environ- ments with point cloud perception. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6659-6668.

Authors
Nguyen, K., & Daumé III, H.
Affiliations
Microsoft Research New York City (United States) (US)
Corresponding author
Nguyen, Khanh
ORCID
Nguyen, Khanh
Venue
Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)
Publisher
Association for Computational Linguistics
Published
2019
Pages
684-695
DOI
10.18653/v1/d19-1063
OpenAlex
W2970340522
Cited by
115
References
53
Citations by year
2026: 9 · 2025: 10 · 2024: 12 · 2023: 19 · 2022: 31 · 2021: 19 · 2020: 15
FWCI
5.30
Type
conference paper
Language
en
Open access
Licence
cc-by
Locations
Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
Keywords
visual navigation, human-robot interaction, assistive robots

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review