Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 87 of the 87 references it cites were found in a registry or an index; 13 of the 17 sources its claims cite were checked for support.

ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planning

Open access
Gu, Qiao, Kuwajerwala, Ali, Morin, Sacha, Jatavallabhula, Krishna Murthy, Sen, Bipasha, Agarwal, Aditya, Rivera, Corban, Paul, William, Ellis, Kirsty, Chellappa, Rama, Gan, Chuang, de Melo, Celso Miguel, Tenenbaum, Joshua B., Torralba, Antonio, Shkurti, Florian, Paull, Liam · 2024 · 2024 IEEE International Conference on Robotics and Automation (ICRA)

Abstract

For robots to perform a wide variety of tasks, they require a 3D representation of the world that is semantically rich, yet compact and efficient for task-driven perception and planning. Recent approaches have attempted to leverage features from large vision-language models to encode semantics in 3D representations. However, these approaches tend to produce maps with per-point feature vectors, which do not scale well in larger environments, nor do they contain semantic spatial relationships between entities in the environment, which are useful for downstream planning. In this work, we propose ConceptGraphs, an open-vocabulary graph-structured representation for 3D scenes. ConceptGraphs is built by leveraging 2D foundation models and fusing their output to 3D by multi-view association. The resulting representations generalize to novel semantic classes, without the need to collect large 3D datasets or finetune models. We demonstrate the utility of this representation through a number of downstream planning tasks that are specified through abstract (language) prompts and require complex reasoning over spatial and semantic concepts. To explore the full scope of our experiments and results, we encourage readers to visit our project webpage.

Overview

CoChat Summary

TL;DR

ConceptGraphs is an open-vocabulary, object-centric 3D scene graph built by fusing class-agnostic 2D segmentation and vision-language features into 3D objects, then using LVLMs/LLMs to caption nodes and infer edges. It supports language-queryable robot perception and planning across manipulation, navigation, localization, and map updates without 3D finetuning.

Key findings

  • On Replica scene graph construction, ConceptGraphs achieved average node precision 0.71 and edge precision 0.88; the detector variant CG-D achieved 0.61 node precision and 0.91 edge precision.
  • Open-vocabulary 3D semantic segmentation on Replica: ConceptGraphs reached 40.63 mAcc and 35.95 F-mIoU, compared with ConceptFusion+SAM at 31.53 mAcc and 38.70 F-mIoU.
  • For text-query object retrieval, LLM-based retrieval handled complex queries better than CLIP: on Replica negation queries, CLIP R@1 was 0.26 vs LLM 0.80; on REAL Lab affordance queries, CLIP R@1 was 0.40 vs LLM 1.00; on REAL Lab negation queries, CLIP R@1 was 0.00 vs LLM 1.00.
  • Real-robot demonstrations included a Clearpath Jackal UGV for navigation/object search/traversability and a Boston Dynamics Spot Arm for open-vocabulary pick-and-place, e.g., responding to “cuddly quacker” by grabbing a duck plush toy.

Methods

  • Given posed RGB-D frames, ConceptGraphs runs class-agnostic segmentation (SAM in experiments), extracts visual descriptors (CLIP), projects masks to 3D, denoises with DBSCAN, and associates/fuses detections across views using geometric and semantic similarity.
  • Each object node is captioned by an LVLM (LLaVA) from up to 10 best views, then refined by an LLM (GPT-4); scene graph edges are generated by computing 3D bounding-box IoU, pruning with a minimum spanning tree, and prompting an LLM to label spatial relationships.
  • Evaluation used Replica and a real-world REAL Lab scan, with Amazon Mechanical Turk human evaluators for node/edge accuracy, plus semantic segmentation, text-query retrieval, and real-robot tasks on wheeled and legged platforms.

Limitations

  • The paper states node captioning errors from LVLMs like LLaVA, occasional missed small/thin objects, duplicate detections, and significant computational/economic costs from multiple LVLM and proprietary LLM inferences.
  • The appendix adds that LLaVA-7B misclassified a non-negligible number of small objects as toothbrushes or pairs of scissors, and that the work does not explicitly focus on improving LLM-based planning over 3D scene graphs.
  • Evident from the text: open-vocabulary node/edge quality was evaluated via human evaluators rather than automated metrics, and LLM retrieval performance is limited by node caption accuracy.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1109/icra57147.2024.10610243

Cites

87 of 87 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

4 Valid · 7 Not valid · 2 Not checked · 30 Own results and interpretations

9 more claims

Create a free account to see all 43 claims and to verify the 2 references still open.

See all claims — free account
  • It may state two separate claims as one. 21
  • Probably not a substantial claim. 10
  • The quote only partly states this claim. 1

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

2 more claims about this paper

Create a free account to see every claim other papers make about this one.

See all claims — free account

Citations

Cites

Show all 80 references

Research works printed but not linked: 1

  1. Shen, Distilled feature fields enable few-shot manipulation

Cited by

See all 406

Metadata

Authors
Gu, Q., Kuwajerwala, A., Morin, S., Jatavallabhula, K. M., Sen, B., Agarwal, A., Rivera, C., Paul, W., Ellis, K., Chellappa, R., Gan, C., de Melo, C. M., Tenenbaum, J. B., Torralba, A., Shkurti, F., & Paull, L.
Affiliations
  • University of Toronto (CA)
  • Université de Montréal
  • Université de Montréal (CA)
  • MIT
  • Johns Hopkins University Applied Physics Laboratory (US)
  • Johns Hopkins University (US)
  • UMass
  • DEVCOM Army Research Laboratory (US)
  • Massachusetts Institute of Technology (US)
Venue
2024 IEEE International Conference on Robotics and Automation (ICRA)
Publisher
IEEE
Published
2024-05-13
Pages
5021-5028
ISSN
1050-4729 (ISSN-L)
OpenAlex
W4401414715
Cited by
406
References
80
Citations by year
2026: 210 · 2025: 131 · 2024: 62
FWCI
34.97
Type
conference paper
Language
en
Open access
Locations
  • Proceedings - IEEE International Conference on Robotics and Automation/Proceedings (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
Army Research Laboratory
Topics
  • Physical Sciences › Computer Science › Computer Graphics and Computer-Aided Design › 3D Shape Modeling and Analysis
  • Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
Keywords
3D scene graphs, robot perception, robot planning, vision-language models, 3D scene understanding, spatial reasoning, language prompts

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-11-01, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-11-01 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planning. https://app.cochat.ai/papers/10.1109/icra57147.2024.10610243
The paper
Gu, Q., Kuwajerwala, A., Morin, S., Jatavallabhula, K. M., Sen, B., Agarwal, A., Rivera, C., Paul, W., Ellis, K., Chellappa, R., Gan, C., de Melo, C. M., Tenenbaum, J. B., Torralba, A., Shkurti, F., & Paull, L. (2024). ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planning. 2024 IEEE International Conference on Robotics and Automation (ICRA). https://doi.org/10.1109/icra57147.2024.10610243

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.