CoChat
Sign Up
Cited by 34Advanced Neural Network Applications3D Shape Modeling and AnalysisMultimodal Machine Learning Applications

Lowis3D: Language-Driven Open-World Instance-Level 3D Scene Understanding

Open access
Ding, Runyu, Yang, Jihan, Xue, Chuhui, Zhang, Wenqing, Bai, Song, Qi, Xiaojuan·2024·IEEE Transactions on Pattern Analysis and Machine Intelligence
DOI 10.1109/tpami.2024.3410324
Open this paper in CoChat — free accountOpen PDF

Abstract

Open-world instance-level scene understanding aims to locate and recognize unseen object categories that are not present in the annotated dataset. This task is challenging because the model needs to both localize novel 3D objects and infer their semantic categories. A key factor for the recent progress in 2D open-world perception is the availability of large-scale image-text pairs from the Internet, which cover a wide range of vocabulary concepts. However, this success is hard to replicate in 3D scenarios due to the scarcity of 3D-text pairs. To address this challenge, we propose to harness pre-trained vision-language (VL) foundation models that encode extensive knowledge from image-text pairs to generate captions for multi-view images of 3D scenes. This allows us to establish explicit associations between 3D shapes and semantic-rich captions. Moreover, to enhance the fine-grained visual-semantic representation learning from captions for object-level categorization, we design hierarchical point-caption association methods to learn semantic-aware embeddings that exploit the 3D geometry between 3D points and multi-view images. In addition, to tackle the localization challenge for novel classes in the open-world setting, we develop debiased instance localization, which involves training object grouping modules on unlabeled data using instance-level pseudo supervision. This significantly improves the generalization capabilities of instance grouping and, thus, the ability to accurately locate novel objects. We conduct extensive experiments on 3D semantic, instance, and panoptic segmentation tasks, covering indoor and outdoor scenes across three datasets. Our method outperforms baseline methods by a significant margin in semantic segmentation (e.g. 34.5% ∼ 65.3%), instance segmentation (e.g. 21.8% ∼ 54.0%), and panoptic segmentation (e.g. 14.7% ∼ 43.3%). Code will be available.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/tpami.2024.3410324

Cites

10 of 10 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Recent approaches that encode semantics in 3D representations using large vision-language model features tend to produce maps with per-point feature vectors, which do not scale well in larger environments.

    “However, these approaches tend to produce maps with per-point feature vectors, which do not scale well in larger environments”

    From ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planning (2024) · cites this paper as [19]

    The Related Work section (page not available in our copy) says recent 3D approaches align points with image regions to distill dense semantic embeddings and may require heavy disk storage. It does not say these approaches produce maps with per-point feature vectors or establish that they fail to scale in larger environments. (LLM, conf=0.83)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 51

  1. A. Dai, A. X. Chang, M. Savva, M. Halber, T. Funkhouser, and M. Nießner, "Scannet: Richly-annotated 3d reconstructions of indoor scenes," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 5828-5839.
  2. B. Graham, M. Engelcke, and L. Van Der Maaten, "3d semantic segmen- tation with submanifold sparse convolutional networks," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 9224-9232.
  3. T. Vu, K. Kim, T. M. Luu, X. T. Nguyen, and C. D. Yoo, "Softgroup for 3d instance segmentation on 3d point clouds," in CVPR, 2022.
  4. I. Misra, R. Girdhar, and A. Joulin, "An End-to-End Transformer Model for 3D Object Detection," in ICCV, 2021.
  5. A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark et al., "Learning transferable visual models from natural language supervision," in International Con- ference on Machine Learning. PMLR, 2021, pp. 8748-8763.
  6. L. Yuan, D. Chen, Y. Chen, N. Codella, X. Dai, J. Gao, H. Hu, X. Huang, B. Li, C. Li, C. Liu, M. Liu, Z. Liu, Y. Lu, Y. Shi, L. Wang, J. Wang, B. Xiao, Z. Xiao, J. Yang, M. Zeng, L. Zhou, and P. Zhang, "Florence: A new foundation model for computer vision," CoRR, vol. abs/2111.11432, 2021. [Online]. Available: https://arxiv.org/abs/2111.11432 · arXiv
  7. P. Sharma, N. Ding, S. Goodman, and R. Soricut, "Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image cap- tioning," in Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2018, pp. 2556- 2565.
  8. X. Gu, T.-Y. Lin, W. Kuo, and Y. Cui, "Open-vocabulary object de- tection via vision and language knowledge distillation," arXiv preprint arXiv:2104.13921, 2021. · arXiv
  9. H. Rasheed, M. Maaz, M. U. Khattak, S. Khan, and F. S. Khan, "Bridging the gap between object and image-level representations for open-vocabulary detection," in 36th Conference on Neural Information Processing Systems (NIPS), 2022.
  10. M. Xu, Z. Zhang, F. Wei, Y. Lin, Y. Cao, H. Hu, and X. Bai, "A simple baseline for zero-shot semantic segmentation with pre-trained vision- language model," arXiv preprint arXiv:2112.14757, 2021. · arXiv
  11. B. Li, K. Q. Weinberger, S. Belongie, V. Koltun, and R. Ranftl, "Language-driven semantic segmentation," in International Conference on Learning Representations, 2022. [Online]. Available: https: //openreview.net/forum?id=RriDjddCLN
  12. C. Zhou, C. C. Loy, and B. Dai, "Extract free dense labels from clip," in European Conference on Computer Vision (ECCV), 2022.
  13. J. Xu, S. Liu, A. Vahdat, W. Byeon, X. Wang, and S. De Mello, "Open- vocabulary panoptic segmentation with text-to-image diffusion models," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 2955-2966.
  14. R. Zhang, Z. Guo, W. Zhang, K. Li, X. Miao, B. Cui, Y. Qiao, P. Gao, and H. Li, "Pointclip: Point cloud understanding by clip," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 8552-8562.
  15. T. Huang, B. Dong, Y. Yang, X. Huang, R. W. Lau, W. Ouyang, and W. Zuo, "Clip2point: Transfer clip to point cloud classification with image-depth pre-training," arXiv preprint arXiv:2210.01055, 2022. · arXiv
  16. P. Wang, A. Yang, R. Men, J. Lin, S. Bai, Z. Li, J. Ma, C. Zhou, J. Zhou, and H. Yang, "Ofa: Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework," CoRR, vol. abs/2202.03052, 2022. · arXiv
  17. A. Yu, V. Ye, M. Tancik, and A. Kanazawa, "pixelnerf: Neural radiance fields from one or few images," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 4578-4587.
  18. S. Peng, K. Genova, C. M. Jiang, A. Tagliasacchi, M. Pollefeys, and T. Funkhouser, "Openscene: 3d scene understanding with open vocabu- laries," in CVPR, 2023.
  19. R. Chen, Y. Liu, L. Kong, X. Zhu, Y. Ma, Y. Li, Y. Hou, Y. Qiao, and W. Wang, "Clip2scene: Towards label-efficient 3d scene understanding by clip," arXiv preprint arXiv:2301.04926, 2023. · arXiv
  20. I. Armeni, O. Sener, A. R. Zamir, H. Jiang, I. Brilakis, M. Fischer, and S. Savarese, "3d semantic parsing of large-scale indoor spaces," in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 1534-1543.
  21. W. K. Fong, R. Mohan, J. V. Hurtado, L. Zhou, H. Caesar, O. Beijbom, and A. Valada, "Panoptic nuscenes: A large-scale benchmark for lidar panoptic segmentation and tracking," IEEE Robotics and Automation Letters, vol. 7, no. 2, pp. 3795-3802, 2022.
  22. R. Ding, J. Yang, C. Xue, W. Zhang, S. Bai, and X. Qi, "Pla: Language- driven open-vocabulary 3d scene understanding," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
  23. C. R. Qi, L. Yi, H. Su, and L. J. Guibas, "Pointnet++: Deep hierar- chical feature learning on point sets in a metric space," arXiv preprint arXiv:1706.02413, 2017. · arXiv
  24. Q. Huang, W. Wang, and U. Neumann, "Recurrent slice networks for 3d segmentation of point clouds," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 2626-2635.
  25. H. Thomas, C. R. Qi, J.-E. Deschaud, B. Marcotegui, F. Goulette, and L. J. Guibas, "Kpconv: Flexible and deformable convolution for point clouds," Proceedings of the IEEE International Conference on Computer Vision, 2019.
  26. M. Xu, R. Ding, H. Zhao, and X. Qi, "Paconv: Position adaptive convolu- tion with dynamic kernel assembling on point clouds," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 3173-3182.
  27. X. Lai, J. Liu, L. Jiang, L. Wang, H. Zhao, S. Liu, X. Qi, and J. Jia, "Stratified transformer for 3d point cloud segmentation," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, 2022, pp. 8500-8509.
  28. C. Choy, J. Gwak, and S. Savarese, "4d spatio-temporal convnets: Minkowski convolutional neural networks," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 3075-3084.
  29. B. Graham and L. van der Maaten, "Submanifold sparse convolutional networks," arXiv preprint arXiv:1706.01307, 2017. · arXiv
  30. L. Yi, W. Zhao, H. Wang, M. Sung, and L. J. Guibas, "Gspn: Generative shape proposal network for 3d instance segmentation in point cloud," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 3947-3956.
  31. B. Yang, J. Wang, R. Clark, Q. Hu, S. Wang, A. Markham, and N. Trigoni, "Learning object bounding boxes for 3d instance segmen- tation on point clouds," in Advances in Neural Information Processing Systems, 2019, pp. 6737-6746.
  32. L. Jiang, H. Zhao, S. Shi, S. Liu, C.-W. Fu, and J. Jia, "Pointgroup: Dual- set point grouping for 3d instance segmentation," Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  33. J. V. Hurtado, R. Mohan, W. Burgard, and A. Valada, "Mopt: Multi-object panoptic tracking," arXiv preprint arXiv:2004.08189, 2020. · arXiv
  34. F. Hong, H. Zhou, X. Zhu, H. Li, and Z. Liu, "Lidar-based panoptic segmentation via dynamic shifting network," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 13 090-13 099.
  35. M. Bucher, T.-H. Vu, M. Cord, and P. Pérez, "Zero-shot semantic segmentation," Advances in Neural Information Processing Systems, vol. 32, 2019.
  36. Z. Gu, S. Zhou, L. Niu, Z. Zhao, and L. Zhang, "Context-aware feature generation for zero-shot semantic segmentation," in Proceedings of the 28th ACM International Conference on Multimedia, 2020, pp. 1921- 1929.
  37. Y. Xian, S. Choudhury, Y. He, B. Schiele, and Z. Akata, "Semantic projection network for zero-and few-label semantic segmentation," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 8256-8265.
  38. D. Baek, Y. Oh, and B. Ham, "Exploiting a joint embedding space for generalized zero-shot semantic segmentation," in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 9536-9545.
  39. B. Michele, A. Boulch, G. Puy, M. Bucher, and R. Marlet, "Generative zero-shot learning for semantic segmentation of 3d point clouds," in 2021 International Conference on 3D Vision (3DV). IEEE, 2021, pp. 992- 1002.
  40. A. Zareian, K. D. Rosa, D. H. Hu, and S.-F. Chang, "Open-vocabulary object detection using captions," in Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, 2021, pp. 14 393- 14 402.
  41. J. Yang, R. Ding, Z. Wang, and X. Qi, "Regionplc: Regional point- language contrastive learning for open-world 3d scene understanding," arXiv preprint arXiv:2304.00962, 2023. · arXiv
  42. R. Mokady, A. Hertz, and A. H. Bermano, "Clipcap: Clip prefix for image captioning," arXiv preprint arXiv:2111.09734, 2021. · arXiv
  43. M. Z. Hossain, F. Sohel, M. F. Shiratuddin, and H. Laga, ACM Computing Surveys (CsUR), vol. 51, no. 6, pp. 1-36, 2019.
  44. A. Dai, M. Nießner, M. Zollhöfer, S. Izadi, and C. Theobalt, "Bundle- fusion: Real-time globally consistent 3d reconstruction using on-the-fly surface reintegration," ACM Transactions on Graphics (ToG), vol. 36, no. 4, p. 1, 2017.
  45. Q.-Y. Zhou, J. Park, and V. Koltun, "Open3D: A modern library for 3D data processing," arXiv:1801.09847, 2018. · arXiv
  46. G. Ghiasi, X. Gu, Y. Cui, and T.-Y. Lin, "Scaling open-vocabulary image segmentation with image-level labels," in Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXVI. Springer, 2022, pp. 540-557.
  47. J. Zhang, R. Dong, and K. Ma, "Clip-fo3d: Learning free open- world 3d scene representations from 2d dense clip," arXiv preprint arXiv:2303.04748, 2023. · arXiv
  48. Y. Zeng, C. Jiang, J. Mao, J. Han, C. Ye, Q. Huang, D.-Y. Yeung, Z. Yang, X. Liang, and H. Xu, "Clip2: Contrastive language-image- point pretraining from real-world point cloud data," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 15 244-15 253.
  49. A. Cheraghian, S. Rahman, D. Campbell, and L. Petersson, "Transductive zero-shot learning for 3d point cloud classification," in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2020, pp. 923-933.
  50. S. Ioffe and C. Szegedy, "Batch normalization: Accelerating deep net- work training by reducing internal covariate shift," in International conference on machine learning. PMLR, 2015, pp. 448-456.
  51. I. Loshchilov and F. Hutter, "Decoupled weight decay regularization," arXiv preprint arXiv:1711.05101, 2017. · arXiv

Cited by

Sorted by citations

Authors
Ding, R., Yang, J., Xue, C., Zhang, W., Bai, S., & Qi, X.
Affiliations
  • University of Hong Kong (HK)
  • ByteDance (China) (CN)
ORCID
Venue
IEEE Transactions on Pattern Analysis and Machine Intelligence
Publisher
Institute of Electrical and Electronics Engineers (IEEE)
Published
2024-12
Volume
46
Issue
12
Pages
8517-8533
ISSN
  • 0162-8828 (ISSN-L)
  • 2160-9292
  • 1939-3539
DOI
10.1109/tpami.2024.3410324
OpenAlex
W4399410938
Cited by
34
References
10
Citations by year
2026: 13 · 2025: 13 · 2024: 8
FWCI
4.34
Type
journal article
Language
en
Open access
Licence
Locations
  • IEEE Transactions on Pattern Analysis and Machine Intelligence (publishedVersion)
  • PubMed (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • The HKU Scholars Hub (University of Hong Kong) (submittedVersion, cc-by-nc-nd)
  • arXiv (Cornell University)
Funders
  • Hong Kong Research Grant Council (27209621)
  • General Research Fund Scheme (17202422)
  • RGC Theme-based research (T45-701/22-R)
  • RGC Matching Fund Scheme
  • JC STEM Lab of Robotics for Soft Materials
  • The Hong Kong Jockey Club Charities Trust
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Engineering › Computational Mechanics › 3D Shape Modeling and Analysis
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
Keywords
vision-language foundation models, pseudo-supervision, instance segmentation, panoptic segmentation, 3D semantic segmentation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review