CoChat
Sign Up
Cited by 12Multimodal Machine Learning ApplicationsLogic, Reasoning, and KnowledgeSpatial Cognition and Navigation

The DOI matches its Crossref record; 2 of the 2 references it cites were found in a registry or an index.

TopViewRS: Vision-Language Models as Top-View Spatial Reasoners

Open access
Li, Chengzu, Zhang, Caiqi, Zhou, Han, Collier, Nigel, Korhonen, Anna, Vulić, Ivan·2024·Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing
DOI 10.18653/v1/2024.emnlp-main.106
Open this paper in CoChat — free accountOpen PDF

Abstract

Top-view perspective denotes a typical way in which humans read and reason over different types of maps, and it is vital for localization and navigation of humans as well as of 'non-human' agents, such as the ones backed by large Vision-Language Models (VLMs).Nonetheless, spatial reasoning capabilities of modern VLMs in this setup remain unattested and underexplored.In this work, we study their capability to understand and reason over spatial relations from the top view.The focus on top view also enables controlled evaluations at different granularity of spatial reasoning; we clearly disentangle different abilities (e.g., recognizing particular objects versus understanding their relative positions).We introduce the TOPVIEWRS (Top-View Reasoning in Space) dataset, consisting of 11,384 multiple-choice questions with either realistic or semantic top-view map as visual input.We then use it to study and evaluate VLMs across 4 perception and reasoning tasks with different levels of complexity.Evaluation of 10 representative open-and closedsource VLMs reveals the gap of more than 50% compared to average human performance, and it is even lower than the random baseline in some cases.Although additional experiments show that Chain-of-Thought reasoning can boost model capabilities by 5.82% on average, the overall performance of VLMs remains limited.Our findings underscore the critical need for enhanced model capability in top-view spatial reasoning and set a foundation for further research towards human-level proficiency of VLMs in real-world multimodal tasks.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.18653/v1/2024.emnlp-main.106

Cites

2 of 2 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Flagged first, then by citations

Research works printed but not linked: 64

  1. Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023. Qwen-vl: A versatile vision- language model for understanding, localization, text reading, and beyond. Preprint, arXiv:2308.12966. · arXiv
  2. Folco Bertini Baldassini, Mustafa Shukor, Matthieu Cord, Laure Soulier, and Benjamin Piwowarski. 2024. What makes multimodal in-context learning work? Preprint, arXiv:2404.15736. · arXiv
  3. Pratyay Banerjee, Tejas Gokhale, Yezhou Yang, and Chitta Baral. 2021. Weakly supervised relative spa- tial reasoning for visual question answering. In Pro- ceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 1908-1918.
  4. Cesar Cadena, Luca Carlone, Henry Carrillo, Yasir Latif, Davide Scaramuzza, Jose Neira, Ian Reid, and John J. Leonard. 2016. Past, present, and future of simultaneous localization and mapping: Toward the robust-perception age. IEEE Transactions on Robotics, 32(6):1309-1332. · DOI · Paper page
  5. Khyathi Raghavi Chandu, Yonatan Bisk, and Alan W Black. 2021. Grounding 'grounding' in NLP. In Findings of the Association for Computational Lin- guistics: ACL-IJCNLP 2021, pages 4283-4305, On- line. Association for Computational Linguistics. · DOI
  6. Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Niessner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. 2017. Mat- terport3d: Learning from rgb-d data in indoor envi- ronments. International Conference on 3D Vision (3DV).
  7. Howard Chen, Alane Suhr, Dipendra Misra, Noah Snavely, and Yoav Artzi. 2019. Touchdown: Natural language navigation and spatial reasoning in visual street environments. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE. · DOI
  8. Junmo Cho, Jaesik Yoon, and Sungjin Ahn. 2024. Spatially-aware transformers for embodied agents. In The Twelfth International Conference on Learning Representations.
  9. Maurice Diesendruck, Jianzhe Lin, Shima Imani, Gay- athri Mahalingam, Mingyang Xu, and Jie Zhao. 2024. Learning how to ask: Cycle-consistency refines prompts in multimodal foundation models. Preprint, arXiv:2402.08756. · arXiv
  10. Tianai Dong, Alberto Testoni, Luciana Benotti, and Raf- faella Bernardi. 2021. Visually grounded follow-up questions: a dataset of spatial questions which re- quire dialogue history. In Proceedings of Second International Combined Workshop on Spatial Lan- guage Understanding and Grounded Communication for Robotics, pages 22-31, Online. Association for Computational Linguistics. · DOI
  11. Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Xilin Wei, Songyang Zhang, Haodong Duan, Maosong Cao, Wenwei Zhang, Yining Li, Hang Yan, Yang Gao, Xinyue Zhang, Wei Li, Jingwen Li, Kai Chen, Conghui He, Xingcheng Zhang, Yu Qiao, Dahua Lin, and Jiaqi Wang. 2024. Internlm-xcomposer2: Master- ing free-form text-image composition and compre- hension in vision-language large model. Preprint, arXiv:2401.16420. · arXiv
  12. Russell Epstein, E Z Patai, Joshua Julian, and Hugo Spiers. 2017. The cognitive map in humans: Spa- tial navigation and beyond. Nature Neuroscience, 20:1504-1513. · DOI · Paper page
  13. Christian Freksa. 1991. Qualitative Spatial Reasoning, pages 361-372. Springer Netherlands, Dordrecht. · DOI
  14. Gemini Team Google. 2024. Gemini: A family of highly capable multimodal models. Preprint, arXiv:2312.11805. · arXiv
  15. Meera Hahn, Jacob Krantz, Dhruv Batra, Devi Parikh, James Rehg, Stefan Lee, and Peter Anderson. 2020. Where are you? localization from embodied dialog. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 806-822, Online. Association for Computa- tional Linguistics. · DOI
  16. Meera Hahn and James M. Rehg. 2022. Transformer- based localization from embodied dialog with large- scale pre-training. In Proceedings of the 2nd Confer- ence of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th Interna- tional Joint Conference on Natural Language Pro- cessing (Volume 2: Short Papers), pages 295-301, Online only. Association for Computational Linguis- tics.
  17. Yining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Gan. 2023. 3d-llm: Injecting the 3d world into large lan- guage models. Preprint, arXiv:2307.12981. · arXiv
  18. Amita Kamath, Jack Hessel, and Kai-Wei Chang. 2023. What's "up" with vision-language models? investi- gating their struggle with spatial reasoning. In Pro- ceedings of the 2023 Conference on Empirical Meth- ods in Natural Language Processing, pages 9161- 9175, Singapore. Association for Computational Lin- guistics. · DOI
  19. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. Preprint, arXiv:2001.08361. · arXiv
  20. Sebastian Koch, Narunas Vaskevicius, Mirco Colosi, Pedro Hermosilla, and Timo Ropinski. 2024. Open3dsg: Open-vocabulary 3d scene graphs from point clouds with queryable objects and open-set re- lationships. Preprint, arXiv:2402.12259. · arXiv
  21. J. Richard Landis and Gary G. Koch. 1977. The mea- surement of observer agreement for categorical data. Biometrics, 33(1):159-174.
  22. Hugo Laurençon, Lucile Saulnier, Léo Tronchon, Stas Bekman, Amanpreet Singh, Anton Lozhkov, Thomas Wang, Siddharth Karamcheti, Alexander M. Rush, Douwe Kiela, Matthieu Cord, and Victor Sanh. 2023. Obelics: An open web-scale filtered dataset of interleaved image-text documents. Preprint, arXiv:2306.16527. · arXiv
  23. Hugo Laurençon, Léo Tronchon, Matthieu Cord, and Victor Sanh. 2024. What matters when building vision-language models? Preprint, arXiv:2405.02246. · arXiv
  24. Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yix- iao Ge, and Ying Shan. 2023a. Seed-bench: Bench- marking multimodal llms with generative compre- hension. Preprint, arXiv:2307.16125. · arXiv
  25. Chengzu Li, Chao Zhang, Simone Teufel, Rama Sanand Doddipatla, and Svetlana Stoyanchev. 2024a. Seman- tic map-based generation of navigation instructions. Preprint, arXiv:2403.19603. · arXiv
  26. Hao Li, Jinfa Huang, Peng Jin, Guoli Song, Qi Wu, and Jie Chen. 2023b. Weakly-supervised 3d spa- tial reasoning for text-based visual question an- swering. IEEE Transactions on Image Processing, 32:3367-3382. · DOI
  27. Hongyang Li, Chonghao Sima, Jifeng Dai, Wenhai Wang, Lewei Lu, Huijie Wang, Jia Zeng, Zhiqi Li, Jiazhi Yang, Hanming Deng, Hao Tian, Enze Xie, Jiangwei Xie, Li Chen, Tianyu Li, Yang Li, Yulu Gao, Xiaosong Jia, Si Liu, Jianping Shi, Dahua Lin, and Yu Qiao. 2024b. Delving into the devils of bird's- eye-view perception: A review, evaluation and recipe. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(4):2151-2170. · DOI
  28. Hongyang Li, Chonghao Sima, Jifeng Dai, Wenhai Wang, Lewei Lu, Huijie Wang, Jia Zeng, Zhiqi Li, Jiazhi Yang, Hanming Deng, Hao Tian, Enze Xie, Jiangwei Xie, Li Chen, Tianyu Li, Yang Li, Yulu Gao, Xiaosong Jia, Si Liu, Jianping Shi, Dahua Lin, and Yu Qiao. 2024c. Delving into the devils of bird's- eye-view perception: A review, evaluation and recipe. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(4):2151-2170. · DOI
  29. Liunian Harold Li, Jack Hessel, Youngjae Yu, Xiang Ren, Kai-Wei Chang, and Yejin Choi. 2023c. Sym- bolic chain-of-thought distillation: Small models can also "think" step-by-step. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2665- 2679, Toronto, Canada. Association for Computa- tional Linguistics. · DOI
  30. Yunxin Li, Longyue Wang, Baotian Hu, Xinyu Chen, Wanqi Zhong, Chenyang Lyu, Wei Wang, and Min Zhang. 2023d. A comprehensive evaluation of gpt-4v on knowledge-intensive visual question answering. Preprint, arXiv:2311.07536. · arXiv
  31. Fangyu Liu, Guy Emerson, and Nigel Collier. 2023a. Visual spatial reasoning. Transactions of the Associ- ation for Computational Linguistics, 11:635-651. · DOI
  32. Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. 2023c. Mmbench: Is your multi-modal model an all-around player? Preprint, arXiv:2307.06281. · arXiv
  33. Manolis Savva*, Abhishek Kadian*, Oleksandr Maksymets*, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Ma- lik, Devi Parikh, and Dhruv Batra. 2019. Habitat: A Platform for Embodied AI Research. In Proceed- ings of the IEEE/CVF International Conference on Computer Vision (ICCV).
  34. Roshanak Mirzaee and Parisa Kordjamshidi. 2022. Transfer learning with synthetic corpora for spatial role labeling and reasoning. In Proceedings of the 2022 Conference on Empirical Methods in Natu- ral Language Processing, pages 6148-6165, Abu Dhabi, United Arab Emirates. Association for Com- putational Linguistics. · DOI
  35. Roshanak Mirzaee, Hossein Rajaby Faghihi, Qiang Ning, and Parisa Kordjamshidi. 2021. SPARTQA: A textual question answering benchmark for spatial reasoning. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, pages 4582-4598, Online. Association for Computational Linguistics. · DOI
  36. Ida Momennejad, Hosein Hasanbeig, Felipe Vieira Fru- jeri, Hiteshi Sharma, Nebojsa Jojic, Hamid Palangi, Robert Ness, and Jonathan Larson. 2023. Evaluating cognitive maps and planning in large language mod- els with cogeval. In Advances in Neural Information Processing Systems, volume 36, pages 69736-69751.
  37. Laksh Nanwani, Anmol Agarwal, Kanishk Jain, Raghav Prabhakar, Aaron Monis, Aditya Mathur, Kr- ishna Murthy Jatavallabhula, A. H. Abdul Hafez, Vineet Gandhi, and K. Madhava Krishna. 2023. Instance-level semantic maps for vision language navigation. In 2023 32nd IEEE International Confer- ence on Robot and Human Interactive Communica- tion (RO-MAN). IEEE. OpenAI. 2022. Chatgpt blog post. OpenAI. 2023. GPT-4V(ision) Technical Work and Authors. · DOI
  38. Priyam Parashar, Vidhi Jain, Xiaohan Zhang, Jay Vakil, Sam Powers, Yonatan Bisk, and Chris Paxton. 2023. Slap: Spatial-language attention policies. In Pro- ceedings of The 7th Conference on Robot Learning, volume 229 of Proceedings of Machine Learning Research, pages 3571-3596. PMLR.
  39. Lang Peng, Zhirong Chen, Zhangjie Fu, Pengpeng Liang, and Erkang Cheng. 2023. Bevsegformer: Bird's eye view semantic segmentation from arbitrary camera rigs. In 2023 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). IEEE. · DOI
  40. Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang. 2023a. Is ChatGPT a general-purpose natural language pro- cessing task solver? In Proceedings of the 2023 Con- ference on Empirical Methods in Natural Language Processing, pages 1339-1384, Singapore. Associa- tion for Computational Linguistics. · DOI
  41. Zequn Qin, Jingyu Chen, Chao Chen, Xiaozhi Chen, and Xi Li. 2023b. Unifusion: Unified multi-view fusion transformer for spatial-temporal represen- tation in bird's-eye-view. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 8690-8699.
  42. Navid Rajabi and Jana Kosecka. 2023. Towards grounded visual spatial reasoning in multi-modal vi- sion language models. Preprint, arXiv:2308.09778. · arXiv
  43. Julia Rozanova, Deborah Ferreira, Krishna Dubba, Weiwei Cheng, Dell Zhang, and Andre Freitas. 2021. Grounding natural language instructions: Can large language models capture spatial information? Preprint, arXiv:2109.08634. · arXiv
  44. Baifeng Shi, Ziyang Wu, Maolin Mao, Xin Wang, and Trevor Darrell. 2024. When do we not need larger vision models? Preprint, arXiv:2403.13043. · arXiv
  45. Zhengxiang Shi, Qiang Zhang, and Aldo Lipani. 2022. Stepgame: A new benchmark for robust multi-hop spatial reasoning in texts. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 11321-11329. · DOI
  46. Aleksandar Shtedritski, Christian Rupprecht, and An- drea Vedaldi. 2023. What does clip know about a red circle? visual prompt engineering for vlms. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE. · DOI
  47. Andrew Szot, Alex Clegg, Eric Undersander, Erik Wijmans, Yili Zhao, John Turner, Noah Maestre, Mustafa Mukadam, Devendra Chaplot, Oleksandr Maksymets, Aaron Gokaslan, Vladimir Vondrus, Sameer Dharur, Franziska Meier, Wojciech Galuba, Angel Chang, Zsolt Kira, Vladlen Koltun, Jitendra Malik, Manolis Savva, and Dhruv Batra. 2021. Habi- tat 2.0: Training home assistants to rearrange their habitat. In Advances in Neural Information Process- ing Systems (NeurIPS).
  48. Damien Teney, Lingqiao Liu, and Anton Van Den Hen- gel. 2017. Graph-structured representations for vi- sual question answering. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3233-3241. · DOI
  49. Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng, and Liwei Wang. 2024. Visual autoregressive modeling: Scalable image generation via next-scale prediction. Preprint, arXiv:2404.02905. · arXiv
  50. Hugo Touvron, Louis Martin, Kevin Stone, Peter Al- bert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, An- thony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Di- ana Liskovich, Yinghai Lu, Yuning Mao, Xavier Mar- tinet, Todor Mihaylov, Pushkar Mishra, Igor Moly- bog, Yixin Nie, Andrew Poulton, Jeremy Reizen- stein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subrama- nian, Xiaoqing Ellen Tan, Binh Tang, Ross Tay- lor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Ro- driguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023. Llama 2: Open foundation and fine- tuned chat models. Preprint, arXiv:2307.09288. · arXiv
  51. David Unger, Nikhil Gosala, Varun Ravi Kumar, Shubhankar Borse, Abhinav Valada, and Senthil Yogamani. 2023. Multi-camera bird's eye view perception for autonomous driving. Preprint, arXiv:2309.09080. · arXiv
  52. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. 2022. Chain-of-thought prompt- ing elicits reasoning in large language models. In Advances in Neural Information Processing Systems, volume 35, pages 24824-24837.
  53. Jason Weston, Antoine Bordes, Sumit Chopra, and Tomás Mikolov. 2016. Towards ai-complete question answering: A set of prerequisite toy tasks. In 4th In- ternational Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings.
  54. Jiayang Wu, Wensheng Gan, Zefeng Chen, Shicheng Wan, and Philip S. Yu. 2023. Multimodal large language models: A survey. Preprint, arXiv:2311.13165. · arXiv
  55. Wenshan Wu, Shaoguang Mao, Yadong Zhang, Yan Xia, Li Dong, Lei Cui, and Furu Wei. 2024. Visualization- of-thought elicits spatial reasoning in large language models. Preprint, arXiv:2404.03622. · arXiv
  56. Yutaro Yamada, Yihan Bao, Andrew Kyle Lampinen, Jungo Kasai, and Ilker Yildirim. 2024. Evaluat- ing spatial understanding of large language models. Transactions on Machine Learning Research.
  57. Cheng Yan, Guansong Pang, Lei Wang, Jile Jiao, Xue- tao Feng, Chunhua Shen, and Jingjing Li. 2021. Bv- person: A large-scale dataset for bird-view person re-identification. In Proceedings of the IEEE/CVF In- ternational Conference on Computer Vision (ICCV), pages 10943-10952.
  58. Zhun Yang, Adam Ishay, and Joohyung Lee. 2023. Cou- pling large language models with logic programming for robust and general reasoning from text. In Find- ings of the Association for Computational Linguis- tics: ACL 2023, pages 5186-5219, Toronto, Canada. Association for Computational Linguistics. · DOI · Paper page
  59. Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. 2023. A sur- vey on multimodal large language models. Preprint, arXiv:2306.13549. · arXiv
  60. Chao Zhang, Mohan Li, Ignas Budvytis, and Stephan Liwicki. 2024a. Dialoc: An iterative ap- proach to embodied dialog localization. Preprint, arXiv:2403.06846. · arXiv
  61. Xingxuan Zhang, Jiansheng Li, Wenjing Chu, Junjia Hai, Renzhe Xu, Yuqing Yang, Shikai Guan, Ji- azheng Xu, and Peng Cui. 2024b. On the out-of- distribution generalization of multimodal large lan- guage models. Preprint, arXiv:2402.06599. · arXiv
  62. Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Be- ichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian-Yun Nie, and Ji-Rong Wen. 2023. A survey of large language models. Preprint, arXiv:2303.18223. · arXiv
  63. Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. 2024. Gpt-4v(ision) is a generalist web agent, if grounded. Preprint, arXiv:2401.01614. · arXiv
  64. Ruiqi Zhong, Dhruba Ghosh, Dan Klein, and Jacob Steinhardt. 2021. Are larger pretrained language models uniformly better? comparing performance at the instance level. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 3813-3827, Online. Association for Computa- tional Linguistics. · DOI

Cited by

Sorted by citations

Metadata

Authors
Li, C., Zhang, C., Zhou, H., Collier, N., Korhonen, A., & Vulić, I.
Affiliations
University of Cambridge (GB)
ORCID
Venue
Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing
Publisher
Association for Computational Linguistics
Published
2024
Pages
1786-1807
DOI
10.18653/v1/2024.emnlp-main.106
OpenAlex
W4404782629
Cited by
12
References
2
Citations by year
2026: 6 · 2025: 6
FWCI
2.24
Type
conference paper
Language
en
Open access
Licence
cc-by
Locations
  • Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • Apollo (University of Cambridge) (submittedVersion, cc-by)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Logic, Reasoning, and Knowledge
  • Social Sciences › Psychology › Experimental and Cognitive Psychology › Spatial Cognition and Navigation
Keywords
vision-language models, spatial relations, chain-of-thought reasoning, multimodal reasoning

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review