Learning Semantic Maps from Natural Language Descriptions
Open accessThe DOI matches its Crossref record; 5 of the 5 references it cites were found in a registry or an index.
Abstract
This paper proposes an algorithm that enables robots to efficiently learn human-centric models of their environment from natural language descriptions.Typical semantic mapping approaches augment metric maps with higher-level properties of the robot's surroundings (e.g., place type, object locations), but do not use this information to improve the metric map.The novelty of our algorithm lies in fusing high-level knowledge, conveyed by speech, with metric information from the robot's low-level sensor streams.Our method jointly estimates a hybrid metric, topological, and semantic representation of the environment.This semantic graph provides a common framework in which we integrate concepts from natural language descriptions (e.g., labels and spatial relations) with metric observations from low-level sensors.Our algorithm efficiently maintains a factored distribution over semantic graphs based upon the stream of natural language and low-level sensor information.We evaluate the algorithm's performance and demonstrate that the incorporation of information from natural language increases the metric, topological and semantic accuracy of the recovered environment model.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
FAB-MAP: Probabilistic Localization and Mapping in the Space of Appearance
Cummins, M., & Newman, P. (2008) · The International Journal of Robotics Research · doi:10.1177/0278364908090961 · Cited by 1,472 · cited 1×
“A number of solutions exist that identify loop closures based upon visual appearance [19, 20] and local metric structure [21], among others.”
BackgroundiSAM: Incremental Smoothing and Mapping
Kaess, M., Ranganathan, A., & Dellaert, F. (2008) · IEEE Transactions on Robotics · doi:10.1109/tro.2008.2006706 · Cited by 1,076
Incremental mapping of large cyclic environments
Gutmann, J.-S., & Konolige, K. (2003) · Proceedings 1999 IEEE International Symposium on Computational Intelligence in Robotics and Automation. CIRA'99 (Cat. No.99EX375) · doi:10.1109/cira.1999.810068 · Cited by 568
Large-scale semantic mapping and reasoning with heterogeneous modalities
Pronobis, A., & Jensfelt, P. (2012) · 2012 IEEE International Conference on Robotics and Automation · doi:10.1109/icra.2012.6224637 · Cited by 248
A Joint Model of Language and Perception for Grounded Attribute Learning
Matuszek, C., FitzGerald, N., Zettlemoyer, L., Bo, L., & Fox, D. (2012) · Maryland Shared Open Access Repository · doi:10.13016/m2ehu6-1m90 · Cited by 248
Cited by
Visual Place Recognition: A Survey
Lowry, S., Sünderhauf, N., Newman, P., Leonard, J. J., Cox, D., Corke, P., & Milford, M. J. (2016) · IEEE Transactions on Robotics · doi:10.1109/tro.2015.2496823 · Cited by 1,079
“The addition of semantic data to maps can improve planning and navigation tasks [65] and requires place recognition to be”
BackgroundBisk, Y., Holtzman, A., Thomason, J., Andreas, J., Bengio, Y., Chai, J., Lapata, M., Lazaridou, A., May, J., Nisnevich, A., et al. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.703 · Cited by 274
End-to-End Driving Via Conditional Imitation Learning
Codevilla, F., Muller, M., Lopez, A., Koltun, V., & Dosovitskiy, A. (2018) · 2018 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2018.8460487 · Cited by 129
Tell Me Dave: Context-Sensitive Grounding of Natural Language to Manipulation Instructions
Kumar Misra, D., Sung, J., Lee, K., & Saxena, A. (2014) · Robotics: Science and Systems X · doi:10.15607/rss.2014.x.005 · Cited by 59
Grounding Language to Entities and Dynamics for Generalization in Reinforcement Learning
Hanjie, A. W., Zhong, V., & Narasimhan, K. (2021) · arXiv · doi:10.48550/arxiv.2101.07393 · Cited by 15
Metadata
- Authors
- Walter, M., Hemachandra, S., Homberg, B., Tellex, S., & Teller, S.
- Venue
- Robotics: Science and Systems IX
- Published
- 2013-06-23
- OpenAlex
- W2296721894
- Cited by
- 84
- References
- 5
- FWCI
- 18.88
- Type
- conference paper
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.