The DOI matches its DataCite record; 61 of the 69 references it cites were found in a registry or an index.
SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities
Open accessAbstract
Understanding and reasoning about spatial relationships is a fundamental capability for Visual Question Answering (VQA) and robotics. While Vision Language Models (VLM) have demonstrated remarkable performance in certain VQA benchmarks, they still lack capabilities in 3D spatial reasoning, such as recognizing quantitative relationships of physical objects like distances or size differences. We hypothesize that VLMs' limited spatial reasoning capability is due to the lack of 3D spatial knowledge in training data and aim to solve this problem by training VLMs with Internet-scale spatial reasoning data. To this end, we present a system to facilitate this approach. We first develop an automatic 3D spatial VQA data generation framework that scales up to 2 billion VQA examples on 10 million real-world images. We then investigate various factors in the training recipe, including data quality, training pipeline, and VLM architecture. Our work features the first internet-scale 3D spatial reasoning dataset in metric space. By training a VLM on such data, we significantly enhance its ability on both qualitative and quantitative spatial VQA. Finally, we demonstrate that this VLM unlocks novel downstream applications in chain-of-thought spatial reasoning and robotics due to its quantitative estimation capability. Project website: https://spatial-vlm.github.io/
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Our results (Table 2, Fig. 5) show that our instruction-tuned model achieves much higher precision than baselines using in-context learning [17, 22].”
From RoboPoint: A Vision-Language Model for Spatial Affordance Prediction for Robotics (2024) · cites this paper as [22]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“SpatialVLM [22] took a step forward to predict spatial relations in metric space, but we show that ROBOPOINT can achieve better performance on real-world spatial reasoning tasks by locating affordances as points.”
From RoboPoint: A Vision-Language Model for Spatial Affordance Prediction for Robotics (2024) · cites this paper as [22]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460
Chen, L.-C., Papandreou, G., Kokkinos, I., Murphy, K., & Yuille, A. L. (2018) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2017.2699184 · Cited by 22,392
Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A. C., Lo, W.-Y., et al. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00371 · Cited by 11,953
Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations
Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L.-J., Shamma, D. A., et al. (2017) · International Journal of Computer Vision · doi:10.1007/s11263-016-0981-7 · Cited by 5,107
Chain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505
Self-citationSimultaneous localization and mapping: part I
Durrant-Whyte, H., & Bailey, T. (2006) · IEEE Robotics & Automation Magazine · doi:10.1109/mra.2006.1638022 · Cited by 4,199
Past, Present, and Future of Simultaneous Localization and Mapping: Toward the Robust-Perception Age
Cadena, C., Carlone, L., Carrillo, H., Latif, Y., Scaramuzza, D., Neira, J., Reid, I., & Leonard, J. J. (2016) · IEEE Transactions on Robotics · doi:10.1109/tro.2016.2624754 · Cited by 3,596
PaLM: Scaling Language Modeling with Pathways
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.02311 · Cited by 2,590
Liu, H., Li, C., Wu, Q., & Lee, Y. J. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-1516 · Cited by 2,428
Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering
Goyal, Y., Khot, T., Summers-Stay, D., Batra, D., & Parikh, D. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.670 · Cited by 2,287
Deep Ordinal Regression Network for Monocular Depth Estimation
Fu, H., Gong, M., Wang, C., Batmanghelich, K., & Tao, D. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00214 · Cited by 1,991
Playing for Data: Ground Truth from Computer Games
Richter, S. R., Vineet, V., Roth, S., & Koltun, V. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46475-6_7 · Cited by 1,981
CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning
Johnson, J., Hariharan, B., van der Maaten, L., Fei-Fei, L., Zitnick, C. L., & Girshick, R. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.215 · Cited by 1,781
Microsoft COCO Captions: Data Collection and Evaluation Server
Chen, X., Fang, H., Lin, T.-Y., Vedantam, R., Gupta, S., Dollar, P., & Zitnick, C. L. (2015) · arXiv · doi:10.48550/arxiv.1504.00325 · Cited by 1,614
Flamingo: a Visual Language Model for Few-Shot Learning
Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.14198 · Cited by 1,555
InstructBLIP: Towards General-purpose Vision-Language Models with Instruction TuningPreprint
Dai, W., Li, J., Li, D., Tiong, A. M. H., Zhao, J., Wang, W., Li, B., Fung, P., & Hoi, S. C. H. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.06500 · Cited by 1,372
Habitat: A Platform for Embodied AI Research
Savva, M., Kadian, A., Maksymets, O., Zhao, Y., Wijmans, E., Jain, B., Straub, J., Liu, J., Koltun, V., Malik, J., et al. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00943 · Cited by 1,237
Finetuned Language Models Are Zero-Shot Learners
Wei, J., Bosma, M., Zhao, V. Y., Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M., & Le, Q. V. (2021) · arXiv · doi:10.48550/arxiv.2109.01652 · Cited by 1,101
OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge
Marino, K., Rastegari, M., Farhadi, A., & Mottaghi, R. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00331 · Cited by 845
Gibson Env: Real-World Perception for Embodied Agents
Xia, F., Zamir, A. R., He, Z., Sax, A., Malik, J., & Savarese, S. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00945 · Cited by 763
Self-citationLiu, H., Li, C., Wu, Q., & Lee, Y. J. (2023) · arXiv · doi:10.48550/arxiv.2304.08485 · Cited by 682
Do As I Can, Not As I Say: Grounding Language in Robotic Affordances
Ahn, M., Brohan, A., Brown, N., Chebotar, Y., Cortes, O., David, B., Finn, C., Fu, C., Gopalakrishnan, K., Hausman, K., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.01691 · Cited by 624
Self-citationPaLM-E: An Embodied Multimodal Language Model
Driess, D., Xia, F., Sajjadi, M. S. M., Lynch, C., Chowdhery, A., Ichter, B., Wahid, A., Tompson, J., Vuong, Q., Yu, T., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.03378 · Cited by 591
Self-citationAI2-THOR: An Interactive 3D Environment for Visual AI
Kolve, E., Mottaghi, R., Han, W., VanderBilt, E., Weihs, L., Herrasti, A., Deitke, M., Ehsani, K., Gordon, D., Zhu, Y., et al. (2017) · arXiv · doi:10.48550/arxiv.1712.05474 · Cited by 578
ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks
Shridhar, M., Thomason, J., Gordon, D., Bisk, Y., Han, W., Mottaghi, R., Zettlemoyer, L., & Fox, D. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01075 · Cited by 540
IQA: Visual Question Answering in Interactive Environments
Gordon, D., Kembhavi, A., Rastegari, M., Redmon, J., Fox, D., & Farhadi, A. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00430 · Cited by 377
Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents
Huang, W., Abbeel, P., Pathak, D., & Mordatch, I. (2022) · arXiv · doi:10.48550/arxiv.2201.07207 · Cited by 304
Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality
Thrush, T., Jiang, R., Bartolo, M., Singh, A., Williams, A., Kiela, D., & Ross, C. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00517 · Cited by 296
RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control
Brohan, A., Brown, N., Carbajal, J., Chebotar, Y., Chen, X., Choromanski, K., Ding, T., Driess, D., Dubey, A., Finn, C., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.15818 · Cited by 276
Self-citationLVLM-EHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Models
Xu, P., Shao, W., Zhang, K., Gao, P., Liu, S., Lei, M., Meng, F., Huang, S., Qiao, Y., & Luo, P. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3507000 · Cited by 273
Pix2seq: A Language Modeling Framework for Object DetectionPreprint
Chen, T., Saxena, S., Li, L., Fleet, D. J., & Hinton, G. E. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.10852 · Cited by 272
Scaling Up Vision-Language Pretraining for Image Captioning
Hu, X., Gan, Z., Wang, J., Yang, Z., Liu, Z., Lu, Y., & Wang, L. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01745 · Cited by 261
Learning 3D Semantic Scene Graphs From 3D Indoor Reconstructions
Wald, J., Dhamo, H., Navab, N., & Tombari, F. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00402 · Cited by 246
ZoeDepth: Zero-shot Transfer by Combining Relative and Metric Depth
Bhat, S. F., Birkl, R., Wofk, D., Wonka, P., & Müller, M. (2023) · arXiv · doi:10.48550/arxiv.2302.12288 · Cited by 242
PaLI: A Jointly-Scaled Multilingual Language-Image ModelPreprint
Chen, X., Wang, X., Changpinyo, S., Piergiovanni, A., Padlewski, P., Salz, D., Goodman, S., Grycner, A., Mustafa, B., Beyer, L., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2209.06794 · Cited by 235
CLIPort: What and Where Pathways for Robotic Manipulation
Shridhar, M., Manuelli, L., & Fox, D. (2021) · arXiv · doi:10.48550/arxiv.2109.12098 · Cited by 223
Liu, F., Emerson, G., & Collier, N. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00566 · Cited by 212
Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language
Zeng, A., Attarian, M., Ichter, B., Choromanski, K., Wong, A., Welker, S., Tombari, F., Purohit, A., Ryoo, M., Sindhwani, V., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.00598 · Cited by 205
Self-citationPhysically Grounded Vision-Language Models for Robotic Manipulation
Gao, J., Sarkar, B., Xia, F., Xiao, T. G., Wu, J., Ichter, B., Majumdar, A., & Sadigh, D. (2024) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra57147.2024.10610090 · Cited by 185
Self-citationVision-Language Pre-Training: Basics, Recent Advances, and Future Trends
Gan, Z., Li, L., Li, C., Wang, L., Liu, Z., & Gao, J. (2022) · Foundations and Trends® in Computer Graphics and Vision · doi:10.1561/0600000105 · Cited by 155
ShapeMask: Learning to Segment Novel Objects by Refining Shape Priors
Kuo, W., Angelova, A., Malik, J., & Lin, T.-Y. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00930 · Cited by 126
Navigating to objects in the real world
Gervet, T., Chintala, S., Batra, D., Malik, J., & Chaplot, D. S. (2023) · Science Robotics · doi:10.1126/scirobotics.adf6991 · Cited by 125
Vision-Language Models as Success DetectorsPreprint
Du, Y., Konyushkova, K., Denil, M., Raju, A., Landon, J., Hill, F., de Freitas, N., & Cabi, S. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2303.07280 · Cited by 117
RoboVQA: Multimodal Long-Horizon Reasoning for Robotics
Sermanet, P., Ding, T., Zhao, J. W., Xia, F., Dwibedi, D., Gopalakrishnan, K., Chan, C., Dulac-Arnold, G., Maddineni, S., Joshi, N. J., et al. (2024) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra57147.2024.10610216 · Cited by 108
Self-citationCan Foundation Models Perform Zero-Shot Task Specification For Robot Manipulation?Preprint
Cui, Y.-C., Niekum, S., Gupta, A., Kumar, V., & Rajeswaran, A. (2022) · Conference on Learning for Dynamics & Control · doi:10.48550/arxiv.2204.11134 · Cited by 108
Robotic Skill Acquisition via Instruction Augmentation with Vision-Language Models
Xiao, T. G., Chan, H., Sermanet, P., Wahid, A., Brohan, A., Hausman, K., Levine, S., & Tompson, J. (2023) · Robotics: Science and Systems · doi:10.15607/rss.2023.xix.029 · Cited by 84
Scene Graph Reasoning for Visual Question AnsweringPreprint
Hildebrandt, M. E., Li, H., Koner, R., Volker Tresp, C., & Günnemann, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.01072 · Cited by 78
MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge
Fan, L., Wang, G., Jiang, Y., Mandlekar, A., Yang, Y., Zhu, H., Tang, A., Huang, D.-A., Zhu, Y., & Anandkumar, A. (2022) · arXiv · doi:10.48550/arxiv.2206.08853 · Cited by 72
PIGLeT: Language Grounding Through Neuro-Symbolic Interaction in a 3D World
Zellers, R., Holtzman, A., Peters, M. E., Mottaghi, R., Kembhavi, A., Farhadi, A., & Choi, Y. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2021.acl-long.159 · Cited by 69
Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced AnnotationPreprint
Nair, S., Mitchell, E., Chen, K., Ichter, B., Savarese, S., & Finn, C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.01115 · Cited by 60
Self-citationHemachandra, S. M., Walter, M. R., Tellex, S., & Teller, S. J. (2014) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2014.6907235 · Cited by 52
Prismer: A Vision-Language Model with Multi-Task ExpertsPreprint
Liu, S., Fan, L., Johns, E., Yu, Z., Xiao, C., & Anandkumar, A. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2303.02506 · Cited by 47
Toward Grounded Commonsense Reasoning
Kwon, M., Hu, H., Myers, V., Karamcheti, S., Dragan, A. D., & Sadigh, D. (2024) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra57147.2024.10611218 · Cited by 41
Self-citationLanguage Models Meet World Models: Embodied Experiences Enhance Language Models
Xiang, J., Tao, T., Gu, Y., Shu, T., Wang, Z., Yang, Z., & Hu, Z. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-3295 · Cited by 34
Mind's Eye: Grounded Language Model Reasoning through Simulation
Liu, R., Wei, J., Gu, S. S., Wu, T.-Y., Vosoughi, S., Cui, C., Zhou, D., & Dai, A. M. (2022) · arXiv · doi:10.48550/arxiv.2210.05359 · Cited by 24
Grounding Language Attributes to Objects using Bayesian Eigenobjects
Cohen, V., Burchfiel, B., Nguyen, T. N., Gopalan, N., Tellex, S., & Konidaris, G. (2019) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros40897.2019.8968603 · Cited by 20
Towards In-context Scene Understanding
Balažević, I., Steiner, D., Parthasarathy, N., Arandjelović, R., & Hénaff, O. J. (2023) · neural information processing systems · doi:10.52202/075280-2785 · Cited by 17
Vision-Language Models are Zero-Shot Reward Models for Reinforcement LearningPreprint
Rocamonde, J., Montesinos, V., Nava, E., Perez, E., & Lindner, D. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.12921 · Cited by 14
Grounding Natural Language Instructions: Can Large Language Models Capture Spatial Information?Preprint
Rozanova, J., Ferreira, D. M., Dubba, K. S. R., Cheng, W., Zhang, D., & Freitas, A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.08634 · Cited by 8
Practical Imitation Learning in the Real World via Task Consistency Loss
Khansari, M., Ho, D., Du, Y., Fuentes, A., Bennice, M., Sievers, N., Kirmani, S., Bai, Y., & Jang, E. (2022) · arXiv · doi:10.48550/arxiv.2202.01862 · Cited by 3
Self-citationFlexCap: Describe Anything in Images in Controllable Detail
Dwibedi, D., Jain, V., Tompson, J., Zisserman, A., & Aytar, Y. (2024) · Advances in Neural Information Processing Systems 37 · doi:10.52202/079017-3530 · Cited by 1
Research works printed but not linked: 8
- Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. Palm 2 technical report. arXiv preprint arXiv:2305.10403, 2023. · arXiv
- Anonymous. Flexcap: Generating rich, localized, and flexible captions in images. In Submitted to The Twelfth International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=7Phicg0WAg. under review.
- Liangyu Chen, Bo Li, Sheng Shen, Jingkang Yang, Chunyuan Li, Kurt Keutzer, Trevor Darrell, and Ziwei Liu. Language models are visual reasoning coordinators. In ICLR 2023 Workshop on Mathematical and Empirical Understanding of Foundation Models, 2023.
- Martin Ester, Hans-Peter Kriegel, Jörg Sander, Xiaowei Xu, et al. A density-based algorithm for discovering clusters in large spatial databases with noise. In kdd, volume 96, pages 226-231, 1996.
- Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, pages 2961-2969, 2017.
- Parsa Mahmoudieh, Deepak Pathak, and Trevor Darrell. Zero-shot reward specification via grounded natural language. In International Conference on Machine Learning, pages 14743-14752. PMLR, 2022.
- Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PMLR, 2021.
- Jesse Thomason, Mohit Shridhar, Yonatan Bisk, Chris Paxton, and Luke Zettlemoyer. Language grounding with 3d objects. In Conference on Robot Learning, pages 1691-1701. PMLR, 2022.
Cited by
Sorted by citations
Foundation models in robotics: Applications, challenges, and the future
Firoozi, R., Tucker, J., Tian, S., Majumdar, A., Sun, J., Liu, W., Zhu, Y., Song, S., Kapoor, A., Hausman, K., et al. (2025) · The International Journal of Robotics Research · doi:10.1177/02783649241281508 · Cited by 238
Self-citationMultimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook.
AlSaad, R., Abd-alrazaq, A., Boughorbel, S., Ahmed, A., Renault, M.-A., Damseh, R., & Sheikh, J. (2024) · Journal of medical Internet research · doi:10.2196/59505 · Cited by 349
“…has been made in the field of general-domain image-text M-LLMs through the development of models such as GLaMM [71], Qwen-VL [72], SpatialVLM [73], InternVL [74], Osprey [75], Vary [76], ShareGPT4V [77], OtterHD [78], LION [79], SPHINX [80], BLIVA [81], SVIT [82], LLaVA [46], and CoVLM [83].”
CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Models
Zhao, Q., Lu, Y., Kim, M. J., Fu, Z., Zhang, Z., Wu, Y., Li, Z., Ma, Q., Han, S., Finn, C., et al. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.00166 · Cited by 231
A survey on integration of large language models with intelligent robots
Kim, Y., Kim, D., Choi, J., Park, J., Oh, N., & Park, D. (2024) · Intelligent Service Robotics · doi:10.1007/s11370-024-00550-5 · Cited by 179
Causal Inference Meets Deep Learning: A Comprehensive Survey
Jiao, L., Wang, Y., Liu, X., Li, L., Liu, F., Ma, W., Guo, Y., Chen, P., Yang, S., & Hou, B. (2024) · Research · doi:10.34133/research.0467 · Cited by 141
Metadata
- Authors
- Chen, B., Xu, Z., Kirmani, S., Ichter, B., Driess, D., Florence, P., Sadigh, D., Guibas, L., & Xia, F.
- Affiliations
- Google DeepMind (United Kingdom) (GB)
- Google (United States) (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2024
- Pages
- 14455-14465
- ISSN
- 1063-6919 (ISSN-L)
- arXiv
- 2401.12168
- OpenAlex
- W4402716288
- Cited by
- 269
- References
- 69
- Citations by year
- 2026: 122 · 2025: 127 · 2024: 20
- FWCI
- 55.53
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
- Keywords
- vision-language models, spatial reasoning, 3D spatial reasoning, visual question answering, qualitative spatial reasoning, automatic data generation, metric spaces, chain-of-thought reasoning, robotics, distance estimation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
