CLIP-Fields: Weakly Supervised Semantic Fields for Robotic Memory
Open accessThe DOI matches its Crossref record; 12 of the 12 references it cites were found in a registry or an index.
Abstract
We propose CLIP-Fields, an implicit scene model that can be used for a variety of tasks, such as segmentation, instance identification, semantic search over space, and view localization.CLIP-Fields learns a mapping from spatial locations to semantic embedding vectors.Importantly, we show that this mapping can be trained with supervision coming only from webimage and web-text trained models such as CLIP, Detic, and Sentence-BERT; and thus uses no direct human supervision.When compared to baselines like Mask-RCNN, our method outperforms on few-shot instance identification or semantic segmentation on the HM3D dataset with only a fraction of the examples.Finally, we show that using CLIP-Fields as a scene memory, robots can perform semantic navigation in real-world environments.Our code and demonstration videos are available here: https://mahis.life/clip-fields
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Learning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293
Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
Reimers, N., & Gurevych, I. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1410 · Cited by 12,778
Detecting Twenty-Thousand Classes Using Image-Level Supervision
Zhou, X., Girdhar, R., Joulin, A., Krähenbühl, P., & Misra, I. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-20077-9_21 · Cited by 564
ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks
Shridhar, M., Thomason, J., Gordon, D., Bisk, Y., Han, W., Mottaghi, R., Zettlemoyer, L., & Fox, D. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01075 · Cited by 540
Visual Language Maps for Robot Navigation
Huang, C., Mees, O., Zeng, A., & Burgard, W. (2023) · 2023 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra48891.2023.10160969 · Cited by 376
Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Grounding
Ku, A., Anderson, P., Patel, R., Ie, E., & Baldridge, J. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.356 · Cited by 274
CLIPort: What and Where Pathways for Robotic Manipulation
Shridhar, M., Manuelli, L., & Fox, D. (2021) · arXiv · doi:10.48550/arxiv.2109.12098 · Cited by 223
ScanQA: 3D Question Answering for Spatial Scene Understanding
Azuma, D., Miyanishi, T., Kurita, S., & Kawanabe, M. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01854 · Cited by 197
Navigating to objects in the real world
Gervet, T., Chintala, S., Batra, D., Malik, J., & Chaplot, D. S. (2023) · Science Robotics · doi:10.1126/scirobotics.adf6991 · Cited by 125 · cited 2×
“In order to perform a variety of complex tasks in human environments, robots often rely on a spatial semantic memory [2, 19, 11].”
BackgroundSelf-citationStructured Scene Memory for Vision-Language Navigation
Wang, H., Wang, W., Liang, W., Xiong, C., & Shen, J. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00835 · Cited by 123 · cited 2×
“Much recent progress on vision-language navigation problems such as ALFRED [25] or RXR [16] has used spatial representations or structured memory as a key component to solving the problem [19, 2, 33, 10].”
Background
Cited by
ChatGPT for Robotics: Design Principles and Model Abilities
Vemprala, S. H., Bonatti, R., Bucker, A., & Kapoor, A. (2024) · IEEE Access · doi:10.1109/access.2024.3387941 · Cited by 570
LERF: Language Embedded Radiance Fields
Kerr, J., Kim, C. M., Goldberg, K., Kanazawa, A., & Tancik, M. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01807 · Cited by 370
LangSplat: 3D Language Gaussian Splatting
Qin, M., Li, W., Zhou, J., Wang, H., & Pfister, H. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.01895 · Cited by 268
ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planning
Gu, Q., Kuwajerwala, A., Morin, S., Jatavallabhula, K. M., Sen, B., Agarwal, A., Rivera, C., Paul, W., Ellis, K., Chellappa, R., et al. (2024) · 2024 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra57147.2024.10610243 · Cited by 406
“…the 2D representations produced by image and language foundation models to the 3D world and show impressive results on open-vocabulary tasks, including language-guided object grounding [17], [18], [24], [26], [36], 3D reasoning [37], [38], robot manipulation [39]–[41] and navigation [42], [43].”
BackgroundConceptFusion: Open-set Multimodal 3D Mapping
Jatavallabhula, K. M., Kuwajerwala, A., Gu, Q., Omama, M., Chen, T., Maalouf, A., Li, S., Iyer, G., Saryazdi, S., Keetha, N., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.07241 · Cited by 228
ConceptFusion: Open-set multimodal 3D mapping
Jatavallabhula, K., Kuwajerwala, A., Gu, Q., Omama, M., Iyer, G., Saryazdi, S., Chen, T., Maalouf, A., Li, S., Keetha, N., et al. (2023) · Robotics: Science and Systems XIX · doi:10.15607/rss.2023.xix.066 · Cited by 228
Hierarchical Open-Vocabulary 3D Scene Graphs for Language-Grounded Robot Navigation
Werby, A., Huang, C., Büchner, M., Valada, A., & Burgard, W. (2024) · Robotics: Science and Systems XX · doi:10.15607/rss.2024.xx.077 · Cited by 124
“…CLIP [21] and their fine-tuned counterparts, a number of works proposed map representations that integrate visual-language features into geometric maps, enabling open-vocabulary indexing of objects [11, 16, 10, 13, 35, 17, 12], audio data [16, 13] and images [26, 16] in an unstructured environment.”
BackgroundInfluential
Metadata
- Authors
- (Mahi)Shafiullah, N., Paxton, C., Pinto, L., Chintala, S., & Szlam, A.
- Venue
- Robotics: Science and Systems XIX
- Published
- 2023-07-10
- OpenAlex
- W4385430678
- Cited by
- 82
- References
- 12
- FWCI
- 15.50
- Type
- conference paper
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.