CoChat
Sign Up
Cited by 62Advanced Neural Network ApplicationsDomain Adaptation and Few-Shot Learning3D Shape Modeling and Analysis

The DOI matches its Crossref record; 7 of the 7 references it cites were found in a registry or an index.

CLIP-FO3D: Learning Free Open-world 3D Scene Representations from 2D Dense CLIP

Open access
Zhang, Junbo, Dong, Runpei, Ma, Kaisheng·2023·2023 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW)
DOI 10.1109/iccvw60793.2023.00219
Open this paper in CoChat — free accountOpen PDF

Abstract

Training a 3D scene understanding model requires complicated human annotations, which are laborious to collect and result in a model only encoding close-set object semantics. In contrast, vision-language pre-training models (e.g., CLIP) have shown remarkable open-world reasoning properties. To this end, we propose directly transferring CLIP’s vision feature space to 3D scene understanding model without any form of supervision. We first modify CLIP’s input and forwarding process so that it can be adapted to extract dense pixel features for 3D scene contents. We then project multi-view image features to the point cloud and train a 3D scene understanding model with feature distillation. Without any annotations or additional training, our model achieves promising annotation-free semantic segmentation results on open-vocabulary semantics and long-tailed concepts. Besides, serving as a cross-modal pre-training framework, our method can be used to improve data efficiency during fine-tuning. Our model outperforms previous SOTA methods in various zero-shot and data-efficient learning benchmarks. Most importantly, our model successfully inherits CLIP’s rich-structured knowledge, allowing 3D scene understanding models to recognize not only object concepts but also open-world semantics.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/iccvw60793.2023.00219

Cites

7 of 7 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 7 references

Research works printed but not linked: 85

  1. 10.1109/TPAMI.2012.120 · DOI
  2. 10.1109/CVPR52688.2022.00967 · DOI
  3. 10.1109/CVPR.2016.170 · DOI
  4. Bucher, 2019, Zero-shot semantic segmentation, Adv. Neural Inform. Process. Syst. (NeurIPS)
  5. 10.1109/CVPR52729.2023.00678 · DOI
  6. Chen, 2022, Zero-shot point cloud segmentation by transferring geometric primitives
  7. 10.1007/978-3-031-19824-3_32 · DOI
  8. 10.1007/978-3-031-19824-3_32 · DOI
  9. Cheraghian, Mitigating the hubness problem for zero-shot learning of 3d objects
  10. 10.1109/WACV45572.2020.9093545 · DOI
  11. 10.1007/s11263-022-01650-4 · DOI
  12. 10.23919/MVA.2019.8758063 · DOI
  13. 10.1109/CVPR.2019.00319 · DOI
  14. 10.1109/CVPR.2017.261 · DOI · Paper page
  15. 10.1007/978-3-030-01249-6_28 · DOI
  16. Delitzas, 2023, Multi-clip: Contrastive vision-language pre-training for question answering tasks in 3d scenes
  17. 10.1109/cvpr52729.2023.00677 · DOI · Paper page
  18. Ding, Open-vocabulary universal image segmentation with maskclip
  19. Dong, Int. Conf. Learn. Represent. (ICLR)
  20. Dosovitskiy, An image is worth 16x16 words: Transformers for image recognition at scale
  21. 10.1109/CVPR52688.2022.01369 · DOI
  22. 10.1007/978-3-031-20077-9_41 · DOI
  23. Gadre, 2022, Clip on wheels: Zeroshot object navigation as object localization and exploration
  24. Ghiasi, Openvocabulary image segmentation
  25. Gu, Open-vocabulary object detection via vision and language knowledge distillation
  26. Ha, Semantic abstraction: Openworld 3d scene understanding from 2d vision-language models
  27. Hess, 2022, Lidarclip or: How i learned to talk to point clouds
  28. 10.1109/CVPR46437.2021.01533 · DOI
  29. Huang, 2022, Ponder: Point cloud pre-training via neural rendering
  30. 10.1016/j.patcog.2023.110086 · DOI
  31. Huang, 2022, Clip2point: Transfer clip to point cloud classification with image-depth pre-training
  32. Kerr, 2023, Lerf: Language embedded radiance fields
  33. Kobayashi, 2022, Decomposing nerf for editing via feature field distillation, Adv. Neural Inform. Process. Syst. (NeurIPS)
  34. Li, Language-driven semantic segmentation
  35. 10.1609/aaai.v36i2.20040 · DOI
  36. 10.1109/cvpr52729.2023.00682 · DOI
  37. Liu, 2023, Openshape: Scaling up 3d shape representation towards open-world understanding
  38. 10.1007/978-3-031-20044-1_16 · DOI
  39. 10.1109/ICCV48922.2021.00079 · DOI
  40. Liu, 2021, Learning from 2d: Pixel-to-point knowledge transfer for 3d pretraining
  41. Lu, 2022, Open-vocabulary 3d detection via image-level class and debiased cross-modal contrastive learning
  42. 10.1109/CVPR52688.2022.00695 · DOI
  43. 10.1109/3DV53792.2021.00107 · DOI
  44. 10.1109/cvpr52729.2023.01860 · DOI
  45. 10.1109/CVPR52729.2023.00083 · DOI
  46. 10.1109/SICE.2002.1195629 · DOI
  47. 10.1007/978-3-031-20086-1_35 · DOI · Paper page
  48. 10.1109/ISMAR.2008.4637336 · DOI
  49. 10.1109/CVPR52729.2023.00085 · DOI · Paper page
  50. Qi, 2023, Contrast with reconstruct: Contrastive 3d representation learning guided by generative pretraining
  51. Qian, 2022, Pix4point: Image pretrained transformers for 3d point cloud understanding
  52. Radford, Learning transferable visual models from natural language supervision
  53. Rasheed, 2022, Bridging the gap between object and image-level representations for open-vocabulary detection, Adv. Neural Inform. Process. Syst. (NeurIPS)
  54. 10.1109/CVPR52688.2022.00549 · DOI
  55. 10.1007/978-3-031-19827-4_8 · DOI
  56. Rozenberszki, 2023, Unscene3d: Unsupervised 3d instance segmentation for indoor scenes
  57. 10.1109/CVPR52688.2022.00966 · DOI
  58. Takmaz, 2023, Openmask3d: Open-vocabulary 3d instance segmentation
  59. Tsagkas, 2023, Vl-fields: Towards language-grounded neural implicit spatial representations
  60. Wang, 2022, P2p: Tuning pre-trained image models for point cloud analysis with point-to-pixel prompting, Adv. Neural Inform. Process. Syst. (NeurIPS)
  61. 10.1109/TPAMI.2018.2857768 · DOI · Paper page
  62. 10.1007/978-3-030-58580-8_34 · DOI
  63. 10.1007/978-3-031-19836-6_36 · DOI
  64. 10.1109/CVPR52729.2023.00289 · DOI
  65. 10.1109/CVPR52688.2022.01760 · DOI
  66. 10.1109/CVPR52729.2023.00120 · DOI
  67. Yang, 2023, Regionplc: Regional point-language contrastive learning for open-world 3d scene understanding
  68. Yao, 2022, 3d point cloud pretraining with knowledge distillation from 2d images
  69. 10.1109/CVPR46437.2021.01511 · DOI
  70. 10.1109/CVPR46437.2021.01161 · DOI
  71. 10.1007/978-3-031-19818-2_11 · DOI
  72. 10.1109/CVPR52688.2022.01871 · DOI
  73. Zabari, 2021, Semantic segmentation inthe-wild without seeing any segmentation examples
  74. 10.1007/978-3-031-20077-9_7 · DOI
  75. 10.1109/CVPR46437.2021.01416 · DOI
  76. Zeid, 2023, Point2vec for self-supervised representation learning on point clouds
  77. 10.1109/cvpr52729.2023.01463 · DOI
  78. 10.1609/aaai.v37i3.25453 · DOI
  79. Zhang, 2022, Point-m2ae: Multi-scale masked autoencoders for hierarchical point cloud pre-training, Adv. Neural Inform. Process. Syst. (NeurIPS)
  80. 10.1109/CVPR52688.2022.00836 · DOI
  81. 10.1109/CVPR52729.2023.02085 · DOI
  82. 10.1109/ICCV48922.2021.01009 · DOI
  83. 10.1109/CVPR52729.2023.01690 · DOI
  84. 10.1109/CVPR52688.2022.01629 · DOI
  85. 10.1007/978-3-031-19815-1_40 · DOI · Paper page

Cited by

Sorted by citations

See all 62

Metadata

Authors
Zhang, J., Dong, R., & Ma, K.
Affiliations
  • Tsinghua University (CN)
  • Xi'an Jiaotong University (CN)
ORCID
Venue
2023 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW)
Publisher
IEEE
Published
2023-10-02
Pages
2040-2051
ISSN
2473-9936 (ISSN-L)
DOI
10.1109/iccvw60793.2023.00219
OpenAlex
W4390190234
Cited by
62
References
7
Citations by year
2026: 13 · 2025: 22 · 2024: 25 · 2023: 2
FWCI
6.31
Type
conference paper
Language
en
Open access
Licence
Locations
  • IEEE International Conference on Computer Vision Workshops (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • Research and Development
  • Tsinghua University
  • National Natural Science Foundation of China
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Computer Science › Computer Graphics and Computer-Aided Design › 3D Shape Modeling and Analysis
Keywords
CLIP, 3D scene understanding, feature distillation, unsupervised transfer learning, point cloud semantic segmentation, open-vocabulary semantic segmentation, zero-shot learning

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: CLIP-FO3D: Learning Free Open-world 3D Scene Representations from 2D Dense CLIP. https://app.cochat.ai/papers/10.1109/iccvw60793.2023.00219
The paper
Zhang, J., Dong, R., & Ma, K. (2023). CLIP-FO3D: Learning Free Open-world 3D Scene Representations from 2D Dense CLIP. 2023 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW). https://doi.org/10.1109/iccvw60793.2023.00219