CoChat
Sign Up
Cited by 54Multimodal Machine Learning ApplicationsReinforcement Learning in RoboticsSpatial Cognition and Navigation

Environment-Agnostic Multitask Learning for Natural Language Grounded Navigation

Open access
Wang, Xin Eric, Jain, Vihan, Ie, Eugene, Wang, William Yang, Kozareva, Zornitsa, Ravi, Sujith·2020·Lecture Notes in Computer Science
DOI 10.1007/978-3-030-58586-0_25
Open this paper in CoChat — free accountOpen PDF

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1007/978-3-030-58586-0_25

Cites

6 of 6 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The process toward large-scale training involves key sub-problems such as building navigation graphs, recovering corrupted rendered images, and generating navigational instructions, which significantly influence the quality of collected data.

    “the process towards such large-scale training involves solving a series of key sub-problems such as how to build navigation graphs [10, 30, 37], how to recover corrupted rendered images [8, 43], and how to generate navigational instructions [23, 25, 78, 83], which significantly influence the quality of collected data”

    From Scaling Data Generation in Vision-and-Language Navigation (2023) · cites this paper as [78]

    Split into 2 atoms — 0 confirmed, 0 partially supported, 0 disputed, 0 misattributed, 2 unverifiable. Expand the claim to see per-atom verdicts.

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 46

  1. [1]1. Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S., Irving, G., Isard, M., Kudlur, M., Levenberg, J., Monga, R., Moore, S., Murray, D.G., Steiner, B., Tucker, P., Vasudevan, V., Warden, P., Wicke, M., Yu, Y., Zheng, X.: Tensorflow: A system for large-scale machine learning. In: 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI 16). pp. 265-283 (2016), https://www.usenix.org/system/files/conference/ osdi16/osdi16-abadi.pdf
  2. [2]2. Anderson, P., Chang, A., Chaplot, D.S., Dosovitskiy, A., Gupta, S., Koltun, V., Kosecka, J., Malik, J., Mottaghi, R., Savva, M., Zamir, A.R.: On evaluation of embodied navigation agents. arXiv (2018), arXiv:1807.06757 [cs.AI] · arXiv
  3. [3]3. Anderson, P., Wu, Q., Teney, D., Bruce, J., Johnson, M., S¨ underhauf, N., Reid, I., Gould, S., van den Hengel, A.: Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 3674-3683 (2018)
  4. [4]4. Bain, M., Sammut, C.: A framework for behavioural cloning. In: Machine Intelligence 15, Intelligent Agents [St. Catherine's College, Oxford, July 1995]. pp. 103129. Oxford University, Oxford, UK, UK (1999), http://dl.acm.org/citation. cfm?id=647636.733043
  5. [5]5. Caruana, R.: Multitask learning: A knowledge-based source of inductive bias. In: Proceedings of the Tenth International Conference on Machine Learning. pp. 4148. Morgan Kaufmann (1993)
  6. [7]7. Chen, H., Suhr, A., Misra, D., Snavely, N., Artzi, Y.: Touchdown: Natural language navigation and spatial reasoning in visual street environments. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 1253812547 (2019)
  7. [8]8. Collobert, R., Weston, J.: A unified architecture for natural language processing: Deep neural networks with multitask learning. In: Proceedings of the 25th International Conference on Machine Learning. pp. 160-167. ICML '08, ACM, New York, NY, USA (2008). https://doi.org/10.1145/1390156.1390177, http: //doi.acm.org/10.1145/1390156.1390177 · DOI · Paper page
  8. [9]9. Das, A., Datta, S., Gkioxari, G., Lee, S., Parikh, D., Batra, D.: Embodied question answering. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. pp. 2054-2063 (2018)
  9. [11]11. Deng, L., Hinton, G., Kingsbury, B.: New types of deep neural network learning for speech recognition and related applications: an overview. In: 2013 IEEE International Conference on Acoustics, Speech and Signal Processing. pp. 8599-8603 (May 2013). https://doi.org/10.1109/ICASSP.2013.6639344 · DOI
  10. [12]12. Espeholt, L., Soyer, H., Munos, R., Simonyan, K., Mnih, V., Ward, T., Doron, Y., Firoiu, V., Harley, T., Dunning, I., Legg, S., Kavukcuoglu, K.: IMPALA: Scalable distributed deep-RL with importance weighted actor-learner architectures. In: Dy, J., Krause, A. (eds.) Proceedings of the 35th International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 80, pp. 1407-1416. PMLR, Stockholmsmssan, Stockholm Sweden (10-15 Jul 2018), http://proceedings.mlr.press/v80/espeholt18a.html
  11. [13]13. Finn, C., Abbeel, P., Levine, S.: Model-agnostic meta-learning for fast adaptation of deep networks. In: Proceedings of the 34th International Conference on Machine Learning-Volume 70. pp. 1126-1135. JMLR. org (2017)
  12. [15]15. Fu, T.J., Wang, X., Peterson, M., Grafton, S., Eckstein, M., Wang, W.Y.: Counterfactual vision-and-language navigation via adversarial path sampling. arXiv preprint arXiv:1911.07308 (2019) · arXiv
  13. [16]16. Ganin, Y., Lempitsky, V.: Unsupervised domain adaptation by backpropagation. In: Proceedings of the 32Nd International Conference on International Conference on Machine Learning - Volume 37. pp. 1180-1189. ICML'15, JMLR.org (2015), http://dl.acm.org/citation.cfm?id=3045118.3045244
  14. [17]17. Girshick, R.: Fast r-cnn. In: 2015 IEEE International Conference on Computer Vision (ICCV). pp. 1440-1448 (Dec 2015). https://doi.org/10.1109/ICCV.2015.169 · DOI · Paper page
  15. [18]18. Hao, W., Li, C., Li, X., Carin, L., Gao, J.: Towards learning a generic agent for vision-and-language navigation via pre-training. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (June 2020)
  16. [19]19. Hemachandra, S., Duvallet, F., Howard, T.M., Roy, N., Stentz, A., Walter, M.R.: Learning models for following natural language directions in unknown environments. In: 2015 IEEE International Conference on Robotics and Automation (ICRA). pp. 5608-5615. IEEE (2015)
  17. [20]20. Huang, H., Jain, V., Mehta, H., Baldridge, J., Ie, E.: Multi-modal discriminative model for vision-and-language navigation. In: Proceedings of the Combined Workshop on Spatial Language Understanding (SpLU) and Grounded Communication for Robotics (RoboNLP). pp. 40-49. Association for Computational Linguistics, Minneapolis, Minnesota (2019). https://doi.org/10.18653/v1/W19-1605, https://www.aclweb.org/anthology/W19-1605 · DOI
  18. [22]22. Jain, V., Magalh˜ aes, G., Ku, A., Vaswani, A., Ie, E., Baldridge, J.: Stay on the path: Instruction fidelity in vision-and-language navigation. In: ACL (2019)
  19. [23]23. Ke, L., Li, X., Bisk, Y., Holtzman, A., Gan, Z., Liu, J., Gao, J., Choi, Y., Srinivasa, S.: Tactical rewind: Self-correction via backtracking in vision-and-language navigation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 6741-6749 (2019)
  20. [25]25. Lansing, L., Jain, V., Mehta, H., Huang, H., Ie, E.: Valan: Vision and language agent navigation. ArXiv abs/1912.03241 (2019) · arXiv
  21. [26]26. Li, J., Wang, X., Tang, S., Shi, H., Wu, F., Zhuang, Y., Wang, W.Y.: Unsupervised reinforcement learning of transferable meta-skills for embodied navigation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 12123-12132 (2020)
  22. [27]27. Li, X., Li, C., Xia, Q., Bisk, Y., C ¸elikyilmaz, A., Gao, J., Smith, N.A., Choi, Y.: Robust navigation with language pretraining and stochastic sampling. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLPIJCNLP 2019, Hong Kong, China, November 3-7, 2019. pp. 1494-1499. Association for Computational Linguistics (2019). https://doi.org/10.18653/v1/D19-1159, https://doi.org/10.18653/v1/D19-1159 · DOI
  23. [28]28. Li, Y., Baldwin, T., Cohn, T.: What's in a Domain? Learning Domain-Robust Text Representations using Adversarial Training. In: NAACL-HLT (2018)
  24. [29]29. Liu, A.H., Liu, Y.C., Yeh, Y.Y., Wang, Y.C.F.: A unified feature disentangler for multi-domain image translation and manipulation. In: Advances in Neural Information Processing Systems. pp. 2590-2599 (2018)
  25. [30]30. Ma, C.Y., Lu, J., Wu, Z., AlRegib, G., Kira, Z., Socher, R., Xiong, C.: Selfmonitoring navigation agent via auxiliary progress estimation. arXiv preprint arXiv:1901.03035 (2019) · arXiv
  26. [31]31. Ma, C.Y., Wu, Z., AlRegib, G., Xiong, C., Kira, Z.: The regretful agent: Heuristicaided navigation through progress estimation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 6732-6740 (2019)
  27. [32]32. Manolis Savva*, Abhishek Kadian*, Oleksandr Maksymets*, Zhao, Y., Wijmans, E., Jain, B., Straub, J., Liu, J., Koltun, V., Malik, J., Parikh, D., Batra, D.: Habitat: A Platform for Embodied AI Research. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) (2019)
  28. [33]33. Mirowski, P., Grimes, M., Malinowski, M., Hermann, K.M., Anderson, K., Teplyashin, D., Simonyan, K., Kavukcuoglu, K., Zisserman, A., Hadsell, R.: Learning to navigate in cities without a map. In: Bengio, S., Wallach, H., Larochelle, H., Grauman, K., Cesa-Bianchi, N., Garnett, R. (eds.) Advances in Neural Information Processing Systems 31, pp. 2419-2430. Curran Associates, Inc. (2018), http://papers.nips.cc/paper/ 7509-learning-to-navigate-in-cities-without-a-map.pdf
  29. [34]34. Mirowski, P.W., Pascanu, R., Viola, F., Soyer, H., Ballard, A., Banino, A., Denil, M., Goroshin, R., Sifre, L., Kavukcuoglu, K., Kumaran, D., Hadsell, R.: Learning to navigate in complex environments. ArXiv abs/1611.03673 (2016) · arXiv
  30. [35]35. Nguyen, K., Daum´ e III, H.: Help, anna! visual navigation with natural multimodal assistance via retrospective curiosity-encouraging imitation learning. arXiv preprint arXiv:1909.01871 (2019) · arXiv
  31. [36]36. Nguyen, K., Dey, D., Brockett, C., Dolan, B.: Vision-based navigation with language-based assistance via imitation learning with indirect intervention. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 12527-12537 (2019)
  32. [37]37. Peng, X., Huang, Z., Sun, X., Saenko, K.: Domain agnostic learning with disentangled representations. In: Proceedings of the 36th International Conference on - Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA. pp. 5102-5112 (2019)
  33. [38]38. Qi, Y., Wu, Q., Anderson, P., Wang, X., Wang, W.Y., Shen, C., Hengel, A.v.d.: Reverie: Remote embodied visual referring expression in real indoor environments. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 9982-9991 (2020)
  34. [39]39. Ramsundar, B., Kearnes, S.M., Riley, P., Webster, D., Konerding, D.E., Pande, V.S.: Massively multitask networks for drug discovery. ArXiv abs/1502.02072 (2015) · arXiv
  35. [40]40. Romijnders, R., Meletis, P., Dubbelman, G.: A domain agnostic normalization layer for unsupervised adversarial domain adaptation. In: 2019 IEEE Winter Conference on Applications of Computer Vision (WACV). pp. 1866-1875. IEEE (2019)
  36. [41]41. Schuster, M., Paliwal, K.K.: Bidirectional recurrent neural networks. IEEE Trans. Signal Processing 45 , 2673-2681 (1997)
  37. [43]43. Teh, Y., Bapst, V., Czarnecki, W.M., Quan, J., Kirkpatrick, J., Hadsell, R., Heess, N., Pascanu, R.: Distral: Robust multitask reinforcement learning. In: Advances in Neural Information Processing Systems. pp. 4496-4506 (2017)
  38. [44]44. Thomason, J., Murray, M., Cakmak, M., Zettlemoyer, L.: Vision-and-dialog navigation. In: Conference on Robot Learning (CoRL) (2019)
  39. [45]45. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, glyph[suppress] L., Polosukhin, I.: Attention is all you need. In: Advances in neural information processing systems. pp. 5998-6008 (2017)
  40. [46]46. de Vries, H., Shuster, K., Batra, D., Parikh, D., Weston, J., Kiela, D.: Talk the walk: Navigating new york city through grounded dialogue. arXiv preprint arXiv:1807.03367 (2018) · arXiv
  41. [47]47. Wang, X., Huang, Q., Celikyilmaz, A., Gao, J., Shen, D., Wang, Y.F., Wang, W.Y., Zhang, L.: Reinforced cross-modal matching and self-supervised imitation learning for vision-language navigation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 6629-6638 (2019)
  42. [48]48. Wang, X., Xiong, W., Wang, H., Yang Wang, W.: Look before you leap: Bridging model-free and model-based reinforcement learning for planned-ahead vision-andlanguage navigation. In: Proceedings of the European Conference on Computer Vision (ECCV). pp. 37-53 (2018)
  43. [49]49. Williams, R.J.: Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning 8 (3), 229-256 (May 1992). https://doi.org/10.1007/BF00992696, https://doi.org/10.1007/BF00992696 · DOI
  44. [50]50. Wu, Y., Wu, Y., Gkioxari, G., Tian, Y.: Building generalizable agents with a realistic and rich 3d environment. arXiv preprint arXiv:1801.02209 (2018) · arXiv
  45. [51]51. Wu, Y., Wu, Y., Tamar, A., Russell, S., Gkioxari, G., Tian, Y.: Learning and planning with a semantic model. arXiv preprint arXiv:1809.10842 (2018) · arXiv
  46. [52]52. Xia, F., R. Zamir, A., He, Z.Y., Sax, A., Malik, J., Savarese, S.: Gibson env: realworld perception for embodied agents. In: Computer Vision and Pattern Recognition (CVPR), 2018 IEEE Conference on. IEEE (2018)

Cited by

Sorted by citations

Authors
Wang, X. E., Jain, V., Ie, E., Wang, W. Y., Kozareva, Z., & Ravi, S.
Affiliations
  • University of California, Santa Barbara (US)
  • Google (United States) (US)
Corresponding author
Wang, Xin Eric
ORCID
Venue
Lecture Notes in Computer Science
Publisher
Springer International Publishing
Published
2020
Pages
413-430
ISSN
  • 0302-9743 (ISSN-L)
  • 1611-3349
DOI
10.1007/978-3-030-58586-0_25
OpenAlex
W3007299332
Cited by
54
References
6
Citations by year
2026: 4 · 2025: 4 · 2024: 6 · 2023: 11 · 2022: 12 · 2021: 10 · 2020: 6 · 2019: 1
FWCI
2.96
Type
conference paper
Language
en
Open access
Licence
Locations
  • Lecture notes in computer science (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Engineering › Automotive Engineering › Spatial Cognition and Navigation
Keywords
vision-and-language navigation, multi-task learning, domain generalization, unknown environments, navigation policy, cross-task knowledge transfer

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review