The DOI matches its DataCite record; 61 of the 69 references it cites were found in a registry or an index.
SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities
Open accessAbstract
Understanding and reasoning about spatial relationships is a fundamental capability for Visual Question Answering (VQA) and robotics. While Vision Language Models (VLM) have demonstrated remarkable performance in certain VQA benchmarks, they still lack capabilities in 3D spatial reasoning, such as recognizing quantitative relationships of physical objects like distances or size differences. We hypothesize that VLMs' limited spatial reasoning capability is due to the lack of 3D spatial knowledge in training data and aim to solve this problem by training VLMs with Internet-scale spatial reasoning data. To this end, we present a system to facilitate this approach. We first develop an automatic 3D spatial VQA data generation framework that scales up to 2 billion VQA examples on 10 million real-world images. We then investigate various factors in the training recipe, including data quality, training pipeline, and VLM architecture. Our work features the first internet-scale 3D spatial reasoning dataset in metric space. By training a VLM on such data, we significantly enhance its ability on both qualitative and quantitative spatial VQA. Finally, we demonstrate that this VLM unlocks novel downstream applications in chain-of-thought spatial reasoning and robotics due to its quantitative estimation capability. Project website: https://spatial-vlm.github.io/
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Our results (Table 2, Fig. 5) show that our instruction-tuned model achieves much higher precision than baselines using in-context learning [17, 22].”
From RoboPoint: A Vision-Language Model for Spatial Affordance Prediction for Robotics (2024) · cites this paper as [22]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“SpatialVLM [22] took a step forward to predict spatial relations in metric space, but we show that ROBOPOINT can achieve better performance on real-world spatial reasoning tasks by locating affordances as points.”
From RoboPoint: A Vision-Language Model for Spatial Affordance Prediction for Robotics (2024) · cites this paper as [22]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460
Chen, L.-C., Papandreou, G., Kokkinos, I., Murphy, K., & Yuille, A. L. (2018) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2017.2699184 · Cited by 22,392
Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A. C., Lo, W.-Y., et al. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00371 · Cited by 11,953
Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations
Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L.-J., Shamma, D. A., et al. (2017) · International Journal of Computer Vision · doi:10.1007/s11263-016-0981-7 · Cited by 5,107
Chain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505
Self-citationSimultaneous localization and mapping: part I
Durrant-Whyte, H., & Bailey, T. (2006) · IEEE Robotics & Automation Magazine · doi:10.1109/mra.2006.1638022 · Cited by 4,199
Past, Present, and Future of Simultaneous Localization and Mapping: Toward the Robust-Perception Age
Cadena, C., Carlone, L., Carrillo, H., Latif, Y., Scaramuzza, D., Neira, J., Reid, I., & Leonard, J. J. (2016) · IEEE Transactions on Robotics · doi:10.1109/tro.2016.2624754 · Cited by 3,596
PaLM: Scaling Language Modeling with Pathways
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.02311 · Cited by 2,590
Liu, H., Li, C., Wu, Q., & Lee, Y. J. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-1516 · Cited by 2,428
Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering
Goyal, Y., Khot, T., Summers-Stay, D., Batra, D., & Parikh, D. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.670 · Cited by 2,287
Research works printed but not linked: 8
- Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. Palm 2 technical report. arXiv preprint arXiv:2305.10403, 2023. · arXiv
- Anonymous. Flexcap: Generating rich, localized, and flexible captions in images. In Submitted to The Twelfth International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=7Phicg0WAg. under review.
- Liangyu Chen, Bo Li, Sheng Shen, Jingkang Yang, Chunyuan Li, Kurt Keutzer, Trevor Darrell, and Ziwei Liu. Language models are visual reasoning coordinators. In ICLR 2023 Workshop on Mathematical and Empirical Understanding of Foundation Models, 2023.
- Martin Ester, Hans-Peter Kriegel, Jörg Sander, Xiaowei Xu, et al. A density-based algorithm for discovering clusters in large spatial databases with noise. In kdd, volume 96, pages 226-231, 1996.
- Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, pages 2961-2969, 2017.
- Parsa Mahmoudieh, Deepak Pathak, and Trevor Darrell. Zero-shot reward specification via grounded natural language. In International Conference on Machine Learning, pages 14743-14752. PMLR, 2022.
- Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PMLR, 2021.
- Jesse Thomason, Mohit Shridhar, Yonatan Bisk, Chris Paxton, and Luke Zettlemoyer. Language grounding with 3d objects. In Conference on Robot Learning, pages 1691-1701. PMLR, 2022.
Cited by
Sorted by citations
Items 401–454 of 454
Peng, S., Lu, L., Shi, B., & Chu, D. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37556-8_6
VoxAnchor: Explicit Voxel-Semantic Grounding for Spatial Understanding in Videos
Li, X., Long, T., Zhou, J., Zeng, C., Chen, J., Yang, J., & Xie, J. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37255-0_19
Chen, W., Wang, J., Jiang, Y., Dai, K., Sharbati, R., & Wasse, A. D. (2026) · Developments in the Built Environment · doi:10.1016/j.dibe.2026.101048
Improving Multimodal Spatial Understanding via Feedback Conditioned Policy Optimization
Zhu, H., Wang, Y., & Miao, R. (2026) · doi:10.1145/3838457.3838568
Context-Interactive Reasoning for Group Activity Detection
Ai, X., Ren, W., Han, S., Xu, H., Dong, Q., Su, P., Wang, Z., Lin, S., Wang, Z., & Liu, H. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37032-7_3
Li, M., Cheng, Z., Zhang, H., Li, D., Zhang, B., Jia, X., Yu, H. L., & Wendin, S. N. (2026) · IEEE/ASME Transactions on Mechatronics · doi:10.1109/tmech.2026.3727238
Uni-LaDiR: Latent Diffusion Unifies Multimodal ReasoningPreprint
Kang, H., Zhang, Y., Kuang, N. L., Gu, J., Ma, Y., & Qin, L. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.19878
Performance Optimization of Mllm-Based 3D Spatial Perception Tasks Via Sam
Jia, Z., Zhao, Y., Li, T., Li, H., Zhou, M., Su, J., & Li, C. (2026) · IEEE International Conference on Pattern Recognition and Machine Learning (PRML) · doi:10.1109/prml71097.2026.11687251
SurgGraph: Quantitative Laparoscopic Video Understanding via Geometry-Grounded Scene GraphsPreprint
Wang, J., Natalie, R., Singleterry, M. D., Bellos, F., George, B., Sandhu, G., Corso, J. J., Guo, A., Popov, V., & Wang, X. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.25651
MVVBench: Benchmarking 4D Reasoning in Vision-Language ModelsPreprint
Chung, H., Park, B., Lee, J., Kim, H., Choi, J., & Kim, B.-H. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.30952
Spatial-OPSD: Self-Improving Spatial Reasoning via Label-Free Self-DistillationPreprint
Liu, Z., Chen, Z., Chen, K., Sun, M., An, X., Jing, H., & Huang, R. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.37055
SemGraph-Route for Safe Zero-Shot UAV Navigation in GPS-Denied Environments
Javaid, S., Saeed, N., & He, B. (2026) · IEEE Open Journal of Vehicular Technology · doi:10.1109/ojvt.2026.3739067
STACK: 3D Spatial Task Analysis in Collaborative Construction Keyframes
Jung, C., Mannan, S., Fitzgerald, J., Seefried, E., Venkatesha, V., Anindho, S., & Blanchard, N. (2026) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3776574.3831227
From Reasoning Failures to Composable Video Spatial IntelligencePreprint
Sun, P., Xiao, J., Rajaraman, R., Kao, S.-H., & Yao, A. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.01999
SpatialAgent: Spatial Question Answering with LLM Agent and Perception Models
Huang, H.-W., Lu, J., Kim, P., Shangguan, J., Cheng, J.-H., Chen, K.-M., Yang, C.-Y., Alattar, B., Lin, Y.-R., Kim, S., et al. (2026) · IEEE International Conference on Multimedia Information Processing and Retrieval (MIPR) · doi:10.1109/mipr70517.2026.00013
Majeed, S., & Ali, R. (2024) · Frontiers in Computational Spatial Intelligence
Gutiérrez, R., Huber, M., Regal, G., & Tscheligi, M. (2026) · ACM Symposium on Spatial User Interaction (SUI) · doi:10.1145/3822518.3830064
Finetuning Vision-Language Models for Grid-Based Navigation
Olkov, I., & Gavrilov, A. (2026) · International Russian Automation Conference (RusAutoCon) · doi:10.1109/rusautocon69876.2026.11707460
Yun, W.-H., Shin, J., Seo, B.-S., Kim, J., & Han, B. (2026) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra57385.2026.11695842
SpatialDreamer: Incentivizing Spatial Reasoning via Active Mental Imagery
Cao, M., Li, X., Liu, X., Reid, I. D., & Liang, X. (2026) · doi:10.1109/cvprf71909.2026.00689
Lian, S., Wu, C., Yang, L. T., Yuan, H., Yu, B., Zhang, L., & Chen, K. (2026) · doi:10.1109/cvprf71909.2026.00940
PURESPACE: A Benchmark for Abstract Spatial Reasoning in Vision-Language Models
Li, J., Zhang, Z., Fan, L., & Wang, W. (2026) · doi:10.1109/cvprf71909.2026.00194
UnrealSpace: Analyzing Spatial Understanding and Reasoning in Controllable Simulation
Ma, W., Cen, S., Shen, J., Lee, R., Begiristain, L., Zhuang, Y., Peng, J., Yu, Z., Song, T., Qi, X., et al. (2026) · doi:10.1109/cvprf71909.2026.00930
BadVLM: Towards Efficient and Resilient Backdoor Attacks on Large Vision-Language Models
Dang, B. L., Truong, V. T., & Le, L. B. (2026) · doi:10.1109/cvprf71909.2026.00074
Wu, X., Li, Z., Jin, J., Gouthaman, K. V., Raj, V., Sinha, N., Chen, J., Du, F., & Manocha, D. (2026) · doi:10.1109/cvprf71909.2026.00904
Navigation Explainability for Object Avoidance with LLMs for Assistive Robots
Young, C., Lian, Y., Kim, J. T., & Ha, S. (2026) · IEEE RO-MAN · doi:10.1109/ro-man72286.2026.11715000
Closed‐Loop Cross‐Level Joint Reasoning for Task‐Oriented Grasping in Cluttered Environments
Xu, S., Zhang, Z., Zha, J., Zhao, S., Shang, Z., & Han, R. (2026) · Electronics Letters · doi:10.1049/ell2.70606
MemAct: Streaming Spatial Understanding with Memory and Active ExplorationPreprint
Wei, Y., Huang, W., Chen, Q., Hou, L., & Qi, X. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.17640
Geospatial Vector Geometry as a Native Spatial Modality for Vision-Language ModelsPreprint
Fei, H., Chen, B., Xue, H., Yue, Y., Xu, Y., & Zhou, X. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.16769
Self-citationABot-ER: Unified Embodied Reasoning through Agentic Data-Model Co-EvolutionPreprint
Ding, Q., Xu, H.-D., Xia, C., Shen, J., Guan, Z., Yuan, Z., Zhang, J., Yang, D., Tian, J., Xu, Y., et al. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.18355
HumanPhys: A Human-Centric Physical-World Perception Benchmark for Real-World Visual ReasoningPreprint
Zhang, Y., Zhou, E., Lin, J., Zhou, D., Wang, W., Huang, Y., & Liu, J. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.14679
DynThinker: Unifying Static 3D and Dynamic 4D Spatial ReasoningPreprint
Cao, Q., Li, H., Guo, Z., Gao, S., Gao, Z., Zhang, H., He, F., Han, J., Xu, H., Liang, X., et al. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.20598
DK-VLM: Geometry as Position for Spatial Reasoning in Vision-Language ModelsPreprint
Su, H., Tian, M., Huang, T., Gao, M., Wei, L., Li, D., Han, J., Xu, H., Wu, X., & Zuo, W. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.22356
Physical AI: A Survey of Models, Learning, Systems, and Open ChallengesPreprint
Gao, B., Gautam, A., Bagchi, A., Liang, Y., Li, W., Li, R., Zhou, Z., Mao, Y., Yan, X., He, X., et al. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.21171
CMDiff: Cross Modality Guidance for Guided Thermal Super-Resolution with Diffusion Models
Xu, J., Li, H., Cheng, C., Shen, Z., Luo, X., Gulimila, K., Song, X., & Wu, X.-J. (2026) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn69854.2026.11728971
Huang, X., Zhao, J., Liu, X., Xu, J., Zhang, Z., Luo, Z., & Chen, X. (2026) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn69854.2026.11729404
Can the Parts Fool the Test? Counterfactual Pairing Cycles for Relational OODPreprint
Zhou, Y., & Xiang, Y. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.23248
ACE-Brain-0.6: Scaling Spatial Grounding for Physical Foundation ModelsPreprint
Wang, Y., Tao, T., He, Z., Luo, S., Mao, W., Mu, Y., Pan, F., Deng, H., Lu, L., Hou, Z., et al. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.27040
SpatialMind: Spatially Aware On-Device Embodied AI via Viewpoint Integration
Wang, H., Xue, Q., Liu, W., Yang, B., Yin, X., & Gao, W. (2026) · ACM Annual International Conference on Mobile Computing and Networking (MobiCom) · doi:10.1145/3795866.3844467
SpatialBrain: Spatial Intelligence Growing from ExperiencePreprint
Zhou, Y., Gong, Z., Ye, X., Liu, Y., Li, D., Zhou, X., Zhong, Z., & Yang, X. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.29317
Zheng, Y., Pan, S., Zhu, Z., Liu, Y., Ge, Y., Zhang, Y., Chen, Z., Xu, C., Yang, X., Jin, Y., et al. (2026) · International Journal of Computer Vision · doi:10.1007/s11263-026-03056-y
Ou, Y., Hu, B., Wu, Y., & He, R. (2026) · Signal Image and Video Processing · doi:10.1007/s11760-026-05805-5
BEHAVIOR-Spatial Benchmark: Evaluating Spatial Understanding for RoboticsPreprint
Chen, S., Huang, H., Wang, Q., Huang, W., Li, M., Zhang, R., Fei-Fei, L., & Wu, J. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.34051
EKT-Router: Expert-Evidence Alignment for Embodied Vision-Language ModelsPreprint
Ruan, W., Xu, Y., Wei, Y., Zhou, B., Li, S., Chen, H. H., Zhou, K., Li, Y., Chen, Y.-C., & Cheng, J. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.33044
Learning and Reasoning with Geometry: How Generation Helps Spatial UnderstandingPreprint
Yang, R., Li, Y., Chen, S., Huang, J., Lin, Y., & Zhao, H. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.36022
Oh, J., & Kim, S. (2026) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3830398.3830697
MolmoSpaces-CommonSense: Evaluating Commonsense in Robotic Manipulation PoliciesPreprint
Guru, A. S., Argus, M. J., Park, J. S., Duan, J., Farhadi, A., & Krishna, R. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.41377
SketchMate: Agentic Geometric Reasoning for Zero-Shot Precision InsertionPreprint
Wang, J. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.41844
Wu, J., Wang, Q., Huang, X., Peng, Q., Li, M., Guan, J., Liu, Q., Wu, S., Wang, L., Bai, S., et al. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2611.02026
Neuro Inspired Robotics: What Robot Learning Can Borrow from the Sensorimotor BrainPreprint
Zhang, R., & Deng, D. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2611.02510
E3CoverNet: Progressive E(3)-Aware 3D Backbone for Geometry-Sensitive Vision-Language Alignment
Guo, Y., Wu, J., Liu, H., Sun, H., & Li, F. (2026) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3767308.3836014
From Pixels to Anatomy: Spatially-Grounded Medical Vision-Language Reasoning
Cao, C., & Xu, M. (2026) · doi:10.1145/3841452.3841491
谢凤宁, Lu, Q., Chen, S., & Rao, H. (2026) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3767308.3835883
VRT-3D-VLM: Visual Reference Tokens with Explicit 3D Geometric Bias for Vision-Language Driving
Tan, Q., Zhou, Z., Xia, Y., & Meng, L. (2026) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3767308.3836544
Metadata
- Authors
- Chen, B., Xu, Z., Kirmani, S., Ichter, B., Driess, D., Florence, P., Sadigh, D., Guibas, L., & Xia, F.
- Affiliations
- Google DeepMind (United Kingdom) (GB)
- Google (United States) (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2024
- Pages
- 14455-14465
- ISSN
- 1063-6919 (ISSN-L)
- arXiv
- 2401.12168
- OpenAlex
- W4402716288
- Cited by
- 269
- References
- 69
- Citations by year
- 2026: 122 · 2025: 127 · 2024: 20
- FWCI
- 55.53
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
- Keywords
- vision-language models, spatial reasoning, 3D spatial reasoning, visual question answering, qualitative spatial reasoning, automatic data generation, metric spaces, chain-of-thought reasoning, robotics, distance estimation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
