Bridging the Gap between 2D and 3D Visual Question Answering: A Fusion Approach for 3D VQA
Open accessAbstract
In 3D Visual Question Answering (3D VQA), the scarcity of fully annotated data and limited visual content diversity hampers the generalization to novel scenes and 3D concepts (e.g., only around 800 scenes are utilized in ScanQA and SQA dataset). Current approaches resort supplement 3D reasoning with 2D information. However, these methods face challenges: either they use top-down 2D views that introduce overly complex and sometimes question-irrelevant visual clues, or they rely on globally aggregated scene/image-level representations from 2D VLMs, losing the fine-grained vision-language correlations. To overcome these limitations, our approach utilizes question-conditional 2D view selection procedure, pinpointing semantically relevant 2D inputs for crucial visual clues. We then integrate this 2D knowledge into the 3D-VQA system via a two-branch Transformer structure. This structure, featuring a Twin-Transformer design, compactly combines 2D and 3D modalities and captures fine-grained correlations between modalities, allowing them mutually augmenting each other. Integrating proposed mechanisms above, we present BridgeQA, that offers a fresh perspective on multi-modal transformer-based architectures for 3D-VQA. Experiments validate that BridgeQA achieves state-of-the-art on 3D-VQA datasets and significantly outperforms existing solutions. Code is available at https://github.com/matthewdm0816/BridgeQA.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In multispectral image fusion, noise can be removed by jointly optimising different modal data through a variational model.
“e.g., in multispectral image fusion tasks, noise is removed by jointly optimising different modal data through a variational model [298].”
From Multimodal fusion and vision–language models: A survey for robot vision (2026) · cites this paper as [298]
The 2D-3D VQA Model and Question Answering sections describe fusing 2D images with 3D scene features and jointly optimizing answer predictions, not removing noise from multispectral images. Searches for multispectral fusion, denoising, and variational optimization found no passage addressing the claim; page numbers were not available in our copy. (LLM, conf=0.76)
Experimental — accuracy not yet measuredExisting denoising methods mainly follow modality-specific denoising and cross-modality denoising.
“Existing denoising methods mainly have two directions: modality-specific denoising [297,298] and cross-modality denoising [299,300,301].”
From Multimodal fusion and vision–language models: A survey for robot vision (2026) · cites this paper as [298]
The 2D-3D VQA Model section describes fusion and cross-attention between visual modalities, not denoising. Searches for “modality-specific denoising,” “cross-modality denoising,” and noise-related terms found no passage in our copy classifying existing denoising methods into those two directions. The available passages have no page numbers. (LLM, conf=0.77)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Learning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293 · see citing works
Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering
Anderson, P., He, X., Buehler, C., Teney, D., Johnson, M., Gould, S. J., & Zhang, L. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00636 · Cited by 5,117 · see citing works
LXMERT: Learning Cross-Modality Encoder Representations from Transformers
Tan, H., & Bansal, M. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1514 · Cited by 2,319 · see citing works
UNITER: UNiversal Image-TExt Representation Learning
Chen, Y.-C., Li, L., Yu, L., El Kholy, A., Ahmed, F., Gan, Z., Cheng, Y., & Liu, J. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58577-8_7 · Cited by 2,055 · see citing works
Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning
Sharma, P., Ding, N., Goodman, S., & Soricut, R. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1238 · Cited by 1,879 · see citing works
Li, J., Li, D., Xiong, C., & Hoi, S. (2022) · arXiv · doi:10.48550/arxiv.2201.12086 · Cited by 1,741 · see citing works
Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks
Li, X., Yin, X. P., Li, C., Zhang, P., Hu, X., Zhang, L., Wang, L., Hu, H., Li, D., Wei, F., et al. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58577-8_8 · Cited by 1,736 · see citing works
Deep Modular Co-Attention Networks for Visual Question Answering
Zhou, Y., Yu, J., Cui, Y., Tao, D., & Tian, Q. C. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00644 · Cited by 1,013 · see citing works
VinVL: Revisiting Visual Representations in Vision-Language Models
Zhang, P., Li, X., Hu, X., Yang, J., Zhang, L., Wang, L., Choi, Y., & Gao, J. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00553 · Cited by 965 · see citing works
SimVLM: Simple Visual Language Model Pretraining with Weak SupervisionPreprint
Wang, Z., Yu, J., Yu, A. W., Dai, Z., Tsvetkov, Y., & Cao, Y. J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.10904 · Cited by 517 · see citing works
LXMERT: Learning Cross-Modality Encoder Representations from TransformersPreprint
Tan, H. X., & Bansal, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1908.07490 · Cited by 433 · see citing works
ScanQA: 3D Question Answering for Spatial Scene Understanding
Azuma, D., Miyanishi, T., Kurita, S., & Kawanabe, M. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01854 · Cited by 197 · see citing works
UNITER: UNiversal Image-TExt Representation LearningPreprint
Chen, Y.-C., Li, L., Yu, L., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.11740 · Cited by 184 · see citing works
3D-VisTA: Pre-trained Transformer for 3D Vision and Text Alignment
Zhu, Z., Ma, X., Chen, Y., Deng, Z., Huang, S., & Li, Q. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00272 · Cited by 129 · see citing works
VoteNet: A Deep Learning Label Fusion Method for Multi-atlas Segmentation
Ding, Z., Han, X., & Niethammer, M. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-32248-9_23 · Cited by 114 · see citing works
Deep Modular Co-Attention Networks for Visual Question AnsweringPreprint
Zhou, Y., Yu, J., Cui, Y., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.10770 · Cited by 98 · see citing works
VQA-LOL: Visual Question Answering Under the Lens of Logic
Gokhale, T., Banerjee, P., Baral, C. R., & Yang, Y. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58589-1_23 · Cited by 80 · see citing works
VinVL: Revisiting Visual Representations in Vision-Language ModelsPreprint
Zhang, P., Li, X., Hu, X., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.00529 · Cited by 60 · see citing works
Context-aware Alignment and Mutual Masking for 3D-Language Pre-training
Jin, Z., Hayat, M., Yang, Y., Guo, Y., & Lei, Y. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01057 · Cited by 56 · see citing works
CLIP-Guided Vision-Language Pre-training for Question Answering in 3D Scenes
Parelli, M., Delitzas, A., Hars, N., Vlassis, G., Anagnostidis, S., Bachmann, G., & Hofmann, T. (2023) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw59228.2023.00593 · Cited by 48 · see citing works
Vision-Language Pre-training with Object Contrastive Learning for 3D Scene Understanding
Zhang, T., He, S., Dai, T., Wang, Z., Chen, B., & Xia, S. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i7.28559 · Cited by 29 · see citing works
SQA3D: Situated Question Answering in 3D ScenesPreprint
Ma, X., Yong, S., Zheng, Z., Li, Q., Liang, Y., Zhu, S. C., & Huang, S. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2210.07474 · Cited by 16 · see citing works
VQA-LOL: Visual Question Answering under the Lens of LogicPreprint
Gokhale, T., Banerjee, P., Baral, C. R., & Yang, Y. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.08325 · Cited by 9 · see citing works
VoteNet: A Deep Learning Label Fusion Method for Multi-Atlas Segmentation
Ding, Z., Han, X., & Niethammer, M. (2019) · arXiv · doi:10.48550/arxiv.1904.08963 · Cited by 7 · see citing works
Multi-CLIP: Contrastive Vision-Language Pre-training for Question Answering tasks in 3D ScenesPreprint
Delitzas, A., Parelli, M., Hars, N., Vlassis, G., Anagnostidis, S., Bachmann, G., & Hofmann, T. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2306.02329 · Cited by 7 · see citing works
Towards Explainable 3D Grounded Visual Question Answering: A New Benchmark and Strong BaselinePreprint
Zhao, L.-C., Cai, D., Zhang, J., Sheng, L., Xu, D., Zheng, R., Zhao, Y., Wang, L., & Fan, X. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2209.12028 · Cited by 3 · see citing works
Vision-Language Pre-training with Object Contrastive Learning for 3D Scene UnderstandingPreprint
Zhang, T., He, S., Tao, D., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.10714 · Cited by 3 · see citing works
CLIP-Guided Vision-Language Pre-training for Question Answering in 3D ScenesPreprint
Parelli, M., Delitzas, A., Hars, N., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2304.06061 · Cited by 1 · see citing works
3D-VisTA: Pre-trained Transformer for 3D Vision and Text AlignmentPreprint
Zhu, Z., Ma, X., Chen, Y., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2308.04352 · Cited by 1 · see citing works
OpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.08774
Flagged first, then by citations
Cited by
Sorted by citations
Multimodal fusion and vision–language models: A survey for robot vision
Han, X., Chen, S., Fu, Z., Feng, Z., Fan, L., An, D., Wang, C., Guo, L., Meng, W., Zhang, X., et al. (2026) · Information Fusion · doi:10.1016/j.inffus.2025.103652 · Cited by 41 · see citing works
Training-Free Video Temporal Grounding Using Large-Scale Pre-trained Models
Zheng, M., Cai, X., Chen, Q., Peng, Y., & Liu, Y. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73007-8_2 · Cited by 19 · see citing works
Self-citationEmbodied intelligence for 3D understanding: A survey on 3D Scene question answering
Li, Z., Yu, H., Ding, Y., Li, Y., He, Y., & Akhtar, N. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103624 · Cited by 8 · see citing works
ResVG: Enhancing Relation and Semantic Understanding in Multiple Instances for Visual Grounding
Zheng, M., Zhang, J., Chen, Q., Peng, Y., & Liu, Y. (2024) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3664647.3681660 · Cited by 6 · see citing works
Self-citationOpen-Vocabulary Hoi Detection With Interaction-Aware Prompt and Concept Calibration
Lei, T., Yin, S., Chen, Q., Peng, Y., & Liu, Y. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.02220 · Cited by 6 · see citing works
Self-citationXu, Z., Lei, T., Li, Z., Wang, G., Chen, Q., Peng, Y., & Liu, Y. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.01467 · Cited by 3 · see citing works
Self-citationMo, W., Chen, Q., Peng, Y., Huang, S., & Liu, Y. (2025) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3746027.3758244 · Cited by 2 · see citing works
Self-citationLearn 3D VQA Better with Active Selection and Reannotation
Zhou, S., Liu, Y., & Zheng, F. (2025) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3746027.3755515 · Cited by 2 · see citing works
Self-citationCascade Transformer for Hierarchical Semantic Reasoning in Text-Based Visual Question Answering
Gao, Y., Feng, D., Yang, L. T., Yang, J., Jiang, X., & Yang, J. (2025) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3729242 · Cited by 1 · see citing works
Human-Inspired Situated Question Answering with Large Language Models
Zhao, X., Xu, W., Cao, J., Fu, T., Ren, R., & Zhang, X. (2025) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme59968.2025.11209566 · Cited by 1 · see citing works
VizArQA: A Foundation for Visual Question Answering in Architectural Simulation
Mehta, M. D., Ezemba, J., Tucker, C., & McComb, C. (2026) · ASME Journal of Engineering for Sustainable Buildings and Cities · doi:10.1115/1.4071865 · Cited by 1 · see citing works
3D Visual Question Answering via 2D-Enhanced Multi-Modal Fusion Framework
Guo, S., Mao, X.-L., & Huang, H. (2025) · International Conference on Computer Supported Cooperative Work in Design (CSCWD) · doi:10.1109/cscwd64889.2025.11033521
Scene-guided Attention Network for Spatial Understanding in 3D Scenes
Jiang, Y.-Q., Zhang, J.-W., Lin, C., Yu, Y., & Yang, Z. (2025) · International Conference on Multimedia Retrieval · doi:10.1145/3731715.3733426 · Cited by 1 · see citing works
Robust 3D Visual Question Answering via Bias Learning
Mao, A., Wen, S., Chen, F., Yi, R., & Liu, Y. (2025) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2025.3590456
Self-citationHCNQA: Enhancing 3D VQA with Hierarchical Concentration Narrowing Supervision
Zhou, S., Zhu, J., Huang, Q., Wang, F., Zhang, Y., & Zheng, F. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-04546-1_3
Investigating Domain Gaps for Indoor 3D Object Detection
Zhao, Z., Xu, Z., Chen, Q., Peng, Y., & Liu, Y. (2025) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3746027.3758275
Self-citationBridging the Gap between 2D and 3D Visual Question Answering: A Fusion Approach for 3D VQA
Association for Artificial Intelligence 2024, Liu, Y., & Mo, W. (2024) · Underline Science Inc. · doi:10.48448/mm3b-bf47
Self-citationVision-Language Models for 3D Scene Understanding: Applications and Developments
An, J., Huang, K., Li, M., & Zhang, M. (2025) · International Conference on Mechatronics, Robotics, and Artificial Intelligence (MRAI) · doi:10.1109/mrai65197.2025.11135710
Language Generation-based 3D Scene Understanding for Spatial Media
Kim, Y., Yang, J., & Kang, J. (2026) · Journal of Broadcast Engineering · doi:10.5909/jbe.2026.31.1.41
Multi-View Images Suffice 3D Reasoning Through Chain-of-Thought Selection and Question-Guided Fusion
Xu, B., Wu, J., Zhang, W., Su, C., Liang, J., Sun, Z. X., & Lei, Z. (2026) · IEEE Transactions on Image Processing · doi:10.1109/tip.2026.3683282
- Authors
- Mo, W., & Liu, Y.
- Affiliations
- Peking University (CN)
- ORCID
- Liu, Yang
- Venue
- Proceedings of the AAAI Conference on Artificial Intelligence
- Published
- 2024-03-24
- Volume
- 38
- Issue
- 5
- Pages
- 4261-4268
- ISSN
- 2159-5399 (ISSN-L)
- 2374-3468
- OpenAlex
- W4393148632
- Cited by
- 20
- References
- 23
- Citations by year
- 2026: 3 · 2025: 14 · 2024: 3
- FWCI
- 4.14
- Type
- conference paper
- Language
- en
- Open access
- diamond
- doi.org
- Locations
- Proceedings of the AAAI Conference on Artificial Intelligence (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Natural Science Foundation of China (62372014)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
- Physical Sciences › Engineering › Computational Mechanics › 3D Shape Modeling and Analysis
- Keywords
- 2D-3D fusion, multimodal transformer, 3D point clouds, vision-language models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
