ViewRefer: Grasp the Multi-view Knowledge for 3D Visual Grounding
Abstract
Understanding 3D scenes from multi-view inputs has been proven to alleviate the view discrepancy issue in 3D visual grounding. However, existing methods normally neglect the view cues embedded in the text modality and fail to weigh the relative importance of different views. In this paper, we propose ViewRefer, a multi-view framework for 3D visual grounding exploring how to grasp the view knowledge from both text and 3D modalities. For the text branch, ViewRefer leverages the diverse linguistic knowledge of large-scale language models, e.g., GPT, to expand a single grounding text to multiple geometry-consistent descriptions. Meanwhile, in the 3D modality, a transformer fusion module with inter-view attention is introduced to boost the interaction of objects across views. On top of that, we further present a set of learnable multi-view prototypes, which memorize scene-agnostic knowledge for different views, and enhance the framework from two perspectives: a view-guided attention module for more robust text features, and a view-guided scoring strategy during the final prediction. With our designed paradigm, ViewRefer achieves superior performance on three benchmarks and surpasses the second-best by +2.8%, +1.5%, and +1.35% on Sr3D, Nr3D, and ScanRefer.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Transfer learning through pre-trained networks streamlines training by allowing efficient fine-tuning for specific applications, enhancing adaptability and performance across diverse tasks, particularly when acquiring extensive labeled data is impractical.
“Transfer learning, through the use of pre-trained networks, streamlines the training process by allowing models to be efficiently fine-tuned for specific applications, thereby enhancing adaptability and performance across diverse tasks, and proving particularly beneficial in scenarios where acquiring extensive labeled data is impractical or unfeasible [188], [189].”
From From Google Gemini to OpenAI Q* (Q-Star): A Survey on Reshaping the Generative Artificial Intelligence (AI) Research Landscape (2025) · cites this paper as [188]
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · see citing works
Proceedings of IEEE Conference on Computer Vision and Pattern Recognition
(1993) · doi:10.1109/cvpr.1993 · Cited by 17,989 · see citing works
Kaugeranna, R., Kaugeranna, E., & Aion, (. S. 4. 6. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.giscience.2023.43 · Cited by 13,695 · see citing works
Masked Autoencoders Are Scalable Vision Learners
He, K., Chen, X., Xie, S., Li, Y., Dollar, P., & Girshick, R. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01553 · Cited by 8,489 · see citing works
ScanNet: Richly-Annotated 3D Reconstructions of Indoor Scenes
Dai, A., Chang, A. X., Savva, M., Halber, M., Funkhouser, T., & Niessner, M. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.261 · Cited by 4,368 · see citing works
Multi-view Convolutional Neural Networks for 3D Shape Recognition
Su, H., Maji, S., Kalogerakis, E., & Miller, E. G. (2015) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.114 · Cited by 3,515 · see citing works
Learning to Prompt for Vision-Language Models
Zhou, K., Yang, J., Loy, C. C., & Liu, Z. (2022) · International Journal of Computer Vision · doi:10.1007/s11263-022-01653-1 · Cited by 3,304 · cited 1× · see citing works
“Recently, multiple multi-modality networks in 2D vision [32, 10, 55, 58, 59, 13, 25, 54, 31, 46, 49, 44, 37, 62] show effective performance.”
BackgroundPointRCNN: 3D Object Proposal Generation and Detection From Point Cloud
Shi, S., Wang, X., & Li, H. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00086 · Cited by 3,093 · see citing works
LXMERT: Learning Cross-Modality Encoder Representations from Transformers
Tan, H., & Bansal, M. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1514 · Cited by 2,319 · see citing works
Conditional Prompt Learning for Vision-Language Models
Zhou, K., Yang, J., Loy, C. C., & Liu, Z. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01631 · Cited by 1,918 · see citing works
Kirillov, A., He, K., Girshick, R., Rother, C., & Dollár, P. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00963 · Cited by 1,659 · see citing works
CLIP-Adapter: Better Vision-Language Models with Feature Adapters
Gao, P., Geng, S., Zhang, R., Ma, T., Fang, R., Zhang, Y., Li, H., & Qiao, Y. (2023) · International Journal of Computer Vision · doi:10.1007/s11263-023-01891-x · Cited by 1,341 · see citing works
Self-citationHabitat: A Platform for Embodied AI Research
Savva, M., Kadian, A., Maksymets, O., Zhao, Y., Wijmans, E., Jain, B., Straub, J., Liu, J., Koltun, V., Malik, J., et al. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00943 · Cited by 1,237 · see citing works
MDETR - Modulated Detection for End-to-End Multi-Modal Understanding
Kamath, A., Singh, M., LeCun, Y., Synnaeve, G., Misra, I., & Carion, N. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00180 · Cited by 761 · see citing works
PETR: Position Embedding Transformation for Multi-view 3D Object Detection
Liu, Y., Wang, T., Zhang, X., & Sun, J. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19812-0_31 · Cited by 664 · see citing works
M3D-RPN: Monocular 3D Region Proposal Network for Object Detection
Brazil, G., & Liu, X. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00938 · Cited by 591 · see citing works
Wang, X., Huang, Q., Celikyilmaz, A., Gao, J., Shen, D., Wang, Y.-F., Wang, W. Y., & Zhang, L. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00679 · Cited by 522 · see citing works
PointCLIP: Point Cloud Understanding by CLIP
Zhang, R., Guo, Z., Zhang, W., Li, K., Miao, X., Cui, B., Qiao, Y., Gao, P., & Li, H. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00836 · Cited by 483 · see citing works
Self-citationPointGroup: Dual-Set Point Grouping for 3D Instance Segmentation
Jiang, L., Zhao, H., Shi, S., Liu, S., Fu, C., & Jia, J. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00492 · Cited by 467 · see citing works
Tip-Adapter: Training-Free Adaption of CLIP for Few-Shot Classification
Zhang, R., Zhang, W., Fang, R., Gao, P., Li, K., Dai, J., Qiao, Y., & Li, H. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19833-5_29 · Cited by 464 · see citing works
Self-citationPatch-NetVLAD: Multi-Scale Fusion of Locally-Global Descriptors for Place Recognition
Hausler, S., Garg, S., Xu, M., Milford, M. J., & Fischer, T. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01392 · Cited by 461 · see citing works
A Fast and Accurate One-Stage Approach to Visual Grounding
Yang, Z., Gong, B., Wang, L., Huang, W., Yu, D., & Luo, J. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00478 · Cited by 431 · see citing works
A Survey of Embodied AI: From Simulators to Research Tasks
Duan, J., Yu, S., Tan, H. L., Zhu, H., & Tan, C. (2022) · IEEE Transactions on Emerging Topics in Computational Intelligence · doi:10.1109/tetci.2022.3141105 · Cited by 420 · see citing works
ScanRefer: 3D Object Localization in RGB-D Scans Using Natural Language
Chen, D. Z., Chang, A. X., & Nießner, M. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58565-5_13 · Cited by 315 · see citing works
CrossPoint: Self-Supervised Cross-Modal Contrastive Learning for 3D Point Cloud Understanding
Afham, M., Dissanayake, I., Dissanayake, D., Dharmasiri, A., Thilakarathna, K., & Rodrigo, R. P. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00967 · Cited by 306 · see citing works
Learning to Assemble Neural Module Tree Networks for Visual Grounding
Liu, D., Zhang, H., Zha, Z.-J., & Wu, F. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00477 · Cited by 304 · see citing works
Self-Supervised Pretraining of 3D Features on any Point-Cloud
Zhang, Z., Girdhar, R., Joulin, A., & Misra, I. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01009 · Cited by 285 · see citing works
ReferIt3D: Neural Listeners for Fine-Grained 3D Object Identification in Real-World Scenes
Achlioptas, P., Abdelreheem, A., Xia, F., Elhoseiny, M., & Guibas, L. J. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58452-8_25 · Cited by 255 · see citing works
Vector-quantized Image Modeling with Improved VQGANPreprint
Yu, J., Li, X., Koh, J. Y., Zhang, H., Pang, R., Qin, J., Ku, A., Xu, Y., Baldridge, J., & Wu, Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.04627 · Cited by 227 · see citing works
Self-citationVision-language navigation with self-supervised auxiliary reasoning tasks
Zhu, F., Zhu, Y., Chang, X., & Liang, X. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01003 · Cited by 223 · see citing works
Frozen CLIP Models are Efficient Video Learners
Lin, Z., Geng, S., Zhang, R., Gao, P., de Melo, G., Wang, X., Dai, J., Qiao, Y., & Li, H. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-19833-5_23 · Cited by 213 · see citing works
Self-citationPointCLIP V2: Prompting CLIP and GPT for Powerful 3D Open-world Learning
Zhu, X., Zhang, R., He, B., Guo, Z., Zeng, Z., Qin, Z., Zhang, S., & Gao, P. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00249 · Cited by 211 · see citing works
Self-citationPrompt, Generate, Then Cache: Cascade of Foundation Models Makes Strong Few-Shot Learners
Zhang, R., Hu, X., Li, B., Huang, S., Deng, H., Qiao, Y., Gao, P., & Li, H. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01460 · Cited by 177 · see citing works
Self-citation3DVG-Transformer: Relation Modeling for Visual Grounding on Point Clouds
Zhao, L., Cai, D., Sheng, L., & Xu, D. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00292 · Cited by 153 · see citing works
Learning 3D Representations from 2D Pre-Trained Models via Image-to-Point Masked Autoencoders
Zhang, R., Wang, L., Qiao, Y., Gao, P., & Li, H. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.02085 · Cited by 132 · see citing works
Self-citationCALIP: Zero-Shot Enhancement of CLIP with Parameter-Free Attention
Guo, Z., Zhang, R., Qiu, L., Ma, X., Miao, X., He, X., & Cui, B. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i1.25152 · Cited by 128 · see citing works
Self-citationText-Guided Graph Neural Networks for Referring 3D Instance Segmentation
Huang, P.-H., Lee, H.-H., Chen, H.-T., & Liu, T.-L. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i2.16253 · Cited by 125 · see citing works
Yuan, Z., Yan, X., Liao, Y., Zhang, R., Wang, S., Li, Z., & Cui, S. R. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00181 · Cited by 119 · see citing works
Self-citationPoint-M2AE: Multi-scale Masked Autoencoders for Hierarchical Point Cloud Pre-trainingPreprint
Zhang, R., Guo, Z., Fang, R., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2205.14401 · Cited by 117 · see citing works
Self-citationMulti-View Transformer for 3D Visual Grounding
Huang, S., Chen, Y., Jia, J., & Wang, L. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01508 · Cited by 116 · see citing works
SAT: 2D Semantics Assisted Training for 3D Visual Grounding
Yang, Z., Zhang, S., Wang, L., & Luo, J. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00187 · Cited by 104 · see citing works
Cross-Modality Knowledge Distillation Network for Monocular 3D Object Detection
Yu, H., Dai, H., & Ding, Y. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-20080-9_6 · Cited by 95 · see citing works
3D-SPS: Single-Stage 3D Visual Grounding via Referred Point Progressive Selection
Luo, J., Fu, J., Kong, X., et al. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01596 · Cited by 86 · see citing works
PiMAE: Point Cloud and Image Interactive Masked Autoencoders for 3D Object Detection
Chen, A., Zhang, K., Zhang, R., Wang, Z., Lu, Y., Guo, Y., & Zhang, S. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00512 · Cited by 85 · see citing works
Self-citationTransRefer3D: Entity-and-Relation Aware Transformer for Fine-Grained 3D Visual Grounding
He, D., Zhao, Y., Luo, J., Hui, T., Huang, S., Zhang, A., & Liu, S. (2021) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3474085.3475397 · Cited by 81 · see citing works
Can Language Understand Depth?
Zhang, R., Zeng, Z., Guo, Z., & Li, Y. (2022) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3503161.3549201 · Cited by 69 · see citing works
Self-citationStarting from Non-Parametric Networks for 3D Point Cloud Analysis
Zhang, R., Wang, L., Wang, Y., Gao, P., Li, H., & Shi, J.-B. (2023) · Computer Vision and Pattern Recognition · doi:10.1109/cvpr52729.2023.00517 · Cited by 65 · see citing works
Self-citationJoint-MAE: 2D-3D Joint Masked Autoencoders for 3D Point Cloud Pre-training
Guo, Z., Zhang, R., Qiu, L., Li, X., & Heng, P. (2023) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2023/88 · Cited by 56 · see citing works
Self-citationImage2Point: 3D Point-Cloud Understanding with 2D Image Pretrained Models
Xu, C., Yang, S., Galanti, T., Wu, B., Yue, X., Zhai, B., Zhan, W., Vajda, P., Keutzer, K., & Tomizuka, M. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19836-6_36 · Cited by 54 · see citing works
3D Concept Learning and Reasoning from Multi-View Images
Hong, Y., Lin, C., Du, Y., Chen, Z., Tenenbaum, J. B., & Gan, C. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00888 · Cited by 42 · see citing works
VT-CLIP: Enhancing Vision-Language Models with Visual-guided TextsPreprint
Qiu, L., Zhang, R., Guo, Z., Zeng, Z., Guo, Z., Li, Y., & Zhang, G. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2112.02399 · Cited by 39 · see citing works
Self-citationDSPoint: Dual-scale Point Cloud Recognition with High-frequency FusionPreprint
Zhang, R., Zeng, Z., Guo, Z., Gao, X., Fu, K., & Shi, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.10332 · Cited by 19 · see citing works
Self-citationNearest Neighbors Meet Deep Neural Networks for Point Cloud Analysis
Zhang, R., Wang, L., Guo, Z., & Shi, J. (2023) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv56688.2023.00130 · Cited by 17 · see citing works
Self-citationTiG-BEV: Multi-view BEV 3D Object Detection via Target Inner-Geometry LearningPreprint
Huang, P., Liu, L., Zhang, R., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2212.13979 · Cited by 16 · see citing works
Self-citation3D Concept Grounding on Neural FieldsPreprint
Hong, Y., Du, Y., Lin, C., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2207.06403 · Cited by 5 · see citing works
Collaboration of Pre-trained Models Makes Better Few-shot LearnerPreprint
Zhang, R., Li, B., Zhāng, W., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2209.12255 · Cited by 3 · see citing works
Self-citationBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805
Flagged first, then by citations
Cited by
Sorted by citations
McIntosh, T. R., Susnjak, T., Liu, T., Watters, P., Xu, D., Liu, D., & Halgamuge, M. N. (2025) · Technologies · doi:10.3390/technologies13020051 · Cited by 102 · see citing works
“These models facilitate accurate and data-efficient analysis by employing multi-view pipelines and cross-attention blocks [158], [159].”
Method
- Authors
- Guo, Z., Tang, Y., Zhang, R., Wang, D., Wang, Z., Zhao, B., & Li, X.
- Affiliations
- Chinese University of Hong Kong (HK)
- Shanghai Artificial Intelligence Laboratory (CN)
- Northwestern Polytechnical University (CN)
- Venue
- 2023 IEEE/CVF International Conference on Computer Vision (ICCV)
- Publisher
- IEEE
- Published
- 2023-10-01
- Pages
- 15326-15337
- OpenAlex
- W4390871962
- Cited by
- 44
- References
- 57
- Citations by year
- 2026: 9 · 2025: 16 · 2024: 19
- FWCI
- 4.70
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- IEEE/CVF International Conference on Computer Vision (ICCV) (publishedVersion)
- Funders
- National Natural Science Foundation of China
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
- Physical Sciences › Engineering › Computational Mechanics › 3D Shape Modeling and Analysis
- Keywords
- 3D visual grounding, large language models, 3D scene understanding
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
