CoChat
Sign Up
PreprintCited by 4Spatial Cognition and NavigationMultimodal Machine Learning ApplicationsAdvanced Neural Network Applications

Defining and Evaluating Visual Language Models' Basic Spatial Abilities: A Perspective from Psychometrics

Open access
Xu, Wenrui, Lyu, Dalin, Wang, Weihang, Feng, Jie, Gao, Chen, Li, Yong·2025·arXiv
DOI 10.48550/arxiv.2502.11859arXiv 2502.11859

Preprint — published as 10.18653/v1/2025.acl-long.567

Open this paper in CoChat — free account

Abstract

The Theory of Multiple Intelligences underscores the hierarchical nature of cognitive capabilities. To advance Spatial Artificial Intelligence, we pioneer a psychometric framework defining five Basic Spatial Abilities (BSAs) in Visual Language Models (VLMs): Spatial Perception, Spatial Relation, Spatial Orientation, Mental Rotation, and Spatial Visualization. Benchmarking 13 mainstream VLMs through nine validated psychometric experiments reveals significant gaps versus humans (average score 24.95 vs. 68.38), with three key findings: 1) VLMs mirror human hierarchies (strongest in 2D orientation, weakest in 3D rotation) with independent BSAs (Pearson's r<0.4); 2) Smaller models such as Qwen2-VL-7B surpass larger counterparts, with Qwen leading (30.82) and InternVL2 lagging (19.6); 3) Interventions like chain-of-thought (0.100 accuracy gain) and 5-shot training (0.259 improvement) show limits from architectural constraints. Identified barriers include weak geometry encoding and missing dynamic simulation. By linking psychometric BSAs to VLM capabilities, we provide a diagnostic toolkit for spatial intelligence evaluation, methodological foundations for embodied AI development, and a cognitive science-informed roadmap for achieving human-like spatial intelligence.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2502.11859

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Xu et al. translate psychometric testing into AI evaluation by defining five Basic Spatial Abilities and diagnosing geometry and rotation weaknesses across 13 VLMs.

    “Xu et al. translate psychometric testing into AI evaluation by defining five Basic Spatial Abilities and diagnosing geometry and rotation weaknesses across 13 VLMs [227].”

    From Understanding World or Predicting Future? A Comprehensive Survey of World Models (2026) · cites this paper as [227]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 37

  1. George K. Bennett, Harold G. Seashore, and Alexander G. Wesman. 2012. The differential aptitude test : A review and critique.
  2. Marc H. Bornstein and Howard Gardner. 1986. Frames of mind: The theory of multiple intelligences. Journal of Aesthetic Education , 20(2):120.
  3. John B. Carroll. 1993. Human Cognitive Abilities: A Survey of Factor-Analytic Studies . Cambridge University Press, Cambridge.
  4. Raymond B. Cattell. 1963. Theory of fluid and crystallized intelligence: A critical experiment. Journal of Educational Psychology , 54(1):1-22.
  5. Anthony G. Cohn and Jose Hernandez-Orallo. 2023. Dialectical language model evaluation: An initial appraisal of the commonsense spatial reasoning abilities of LLMs. https://doi.org/10.48550/arXiv.2304.11164 · DOI
  6. Jiafei Duan, Samson Yu, Hui Li Tan, Hongyuan Zhu, and Cheston Tan. 2022. A survey of embodied AI: From simulators to research tasks. IEEE Transactions on Emerging Topics in Computational Intelligence , 6(2):230-244.
  7. Benedict C. O. F. Fehringer. 2023. R-Cube-SR Test: A New Test for Spatial Relations Distinguishable From Visualization. European Journal of Psychological Assessment , 39(1):37-48.
  8. Alinda Friedman, Bernd Kohler, Peri Gunalp, Alexander P. Boone, and Mary Hegarty. 2020. A computerized spatial orientation test. Behavior Research Methods , 52(2):799-812.
  9. Rao Fu, Jingyu Liu, Xilun Chen, Yixin Nie, and Wenhan Xiong. 2024. Scene-LLM: Extending Language Model for 3D Visual Understanding and Reasoning. https://doi.org/10.48550/arXiv.2403.11401 · DOI · Paper page
  10. David C. Geary. 2022. Spatial ability as a distinct domain of human cognition: An evolutionary perspective. Intelligence , 90:101616. https://doi.org/10.1016/j.intell.2021.101616 · DOI
  11. David C. Geary, Mary K. Hoard, Lara Nugent, and John E. Scofield. 2021. In-class attention, spatial ability, and mathematics anxiety predict acrossgrade gains in adolescents' mathematics achievement. Journal of Educational Psychology , 113(4):754-769.
  12. Mary Hegarty, Madeleine Keehner, Peter Khooshabeh, and Daniel R. Montello. 2009. How spatial abilities enhance, and are enhanced by, dental education. Learning and Individual Differences , 19(1):61-70.
  13. Mary Hegarty, Daniel R. Montello, Anthony E. Richardson, Toru Ishikawa, and Kristin Lovelace. 2006. Spatial abilities at different scales: Individual differences in aptitude-test performance and spatial-layout learning. Intelligence , 34(2):151-176.
  14. John L. Horn. 1968. Organization of abilities and the development of intelligence. Psychological Review , 75(3):242-259.
  15. WJohnson and T Bouchardjr. 2005. The structure of human intelligence: It is verbal, perceptual, and image rotation (VPR), not fluid and crystallized. Intelligence , 33(4):393-416. https://doi.org/10.1016/j.intell.2004.12.002 · DOI
  16. Wendy Johnson and Thomas J. Bouchard. 2007. Sex differences in mental abilities: g masks the dimensions on which they lie. Intelligence , 35(1):23-39.
  17. Rex E. Jung and Richard J. Haier. 2007. The parietofrontal integration theory (P-FIT) of intelligence: Converging neuroimaging evidence. Behavioral and Brain Sciences , 30(2):135-154.
  18. Michael J. Kane and Randall W. Engle. 2002. The role of prefrontal cortex in working-memory capacity, executive attention, and general fluid intelligence: An individual-differences perspective. Psychonomic Bulletin &amp; Review , 9(4):637-671. https://doi.org/10.3758/BF03196323 · DOI · Paper page
  19. Petros Katsioloudis, Vukica Jovanovic, and Mildred Jones. 2014. A comparative analysis of spatial visualization ability and drafting models for industrial and technology education students. Journal of Technology Education , 26:88-104.
  20. Marcia C. Linn and Anne C. Petersen. 1985. Emergence and characterization of sex differences in spatial ability: A meta-analysis. Child Development , 56(6):1479-1498.
  21. Eleanor E. Maccoby and Carol N. Jacklin. 1974. The Psychology of Sex Differences . The Psychology of Sex Differences. Stanford University Press.
  22. Yukiko Maeda, So Yoon Yoon, K. Kim-Kang, and P.K. Imbrie. 2013. Psychometric properties of the revised PSVT: R for measuring first year engineering students' spatial ability. International Journal of Engineering Education , 29.
  23. Peter Herbert Maier. 1996. Spatial geometry and spatial ability-How to make solid geometry solid. In Selected Papers from the Annual Conference of Didactics of Mathematics , Osnabrueck, Germany: Gesellschaft für Didaktik der Mathematik (GDM).
  24. Kevin S. McGrew. 2009. CHC theory and the human cognitive abilities project: Standing on the shoulders of the giants of psychometric intelligence research. Intelligence , 37(1):1-10. https://doi.org/10.1016/j.intell.2008.08.004 · DOI · Paper page
  25. William B. Michael, J.P. Guilford, Benjamin Fruchter, and Wayne S. Zimmerman. 1957. The description of spatial-visualization abilities. Educational and Psychological Measurement , 17(2):185-199.
  26. Ida Momennejad, Hosein Hasanbeig, Felipe Vieira, Hiteshi Sharma, Robert Osazuwa Ness, Nebojsa Jojic, Hamid Palangi, and Jonathan Larson. 2023. Evaluating cognitive maps and planning in large language models with CogEval. https://arxiv.org/abs/2309.15129v1. · arXiv
  27. Anita Pawlak-Jakubowska and Ewa Terczy´ nska. 2023. Evaluation of STEM students' spatial abilities based on a novel net cube imagination test. Scientific Reports , 13(1):17296.
  28. James W. Pellegrino, David L. Alderton, and Valerie J. Shute. 1984. Understanding spatial ability. Educational Psychologist , 19(4):239-253.
  29. M. Peters, B. Laeng, K. Latham, M. Jackson, R. Zaiyouna, and C. Richardson. 1995. A Redrawn Vandenberg and Kuse Mental Rotations Test - Different Versions and Factors That Affect Performance. Brain and Cognition , 28(1):39-58.
  30. Ronen Porat and Ciprian Ceobanu. 2023. Spatial ability: Understanding the past, looking into the future. European Proceedings of Educational Sciences , Education, Reflection, Development - ERD 2022.
  31. Manasi Sharma. 2023. Exploring and improving the spatial reasoning abilities of large language models. https://doi.org/10.48550/arXiv.2312.01054 · DOI
  32. C. Spearman. 1904. "general intelligence," objectively determined and measured. American Journal of Psychology , 15(2):201.
  33. L. L. Thurstone. 1950. Some primary abilities in visual thinking. Proceedings of The American Philosophical Society , 94(6):517-521.
  34. Steven G. Vandenberg and Allan R. Kuse. 1978. Mental Rotations, a Group Test of Three-Dimensional Spatial Visualization. Perceptual and Motor Skills , 47(2):599-604.
  35. Philip E. Vernon. 1965. Ability factors and environmental influences. American Psychologist , 20(9):723733.
  36. Guy Vingerhoets, Engelien Lannoo, and Sabien Bauwens. 1996. Analysis of the money road-map test performance in normal and brain-damaged subjects. Archives of Clinical Neuropsychology , 11(1):1-9.
  37. Daniel Voyer, Susan Voyer, and M. Philip Bryden. 1995. Magnitude of sex differences in spatial abilities: A meta-analysis and consideration of critical variables. Psychological Bulletin , 117(2):250-270. https://doi.org/10.1037/0033-2909.117.2.250 · DOI · Paper page

Cited by

Sorted by citations

Authors
Xu, W., Lyu, D., Wang, W., Feng, J., Gao, C., & Li, Y.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2025
Pages
11571-11590
ISSN
0736-587X (ISSN-L)
DOI
10.48550/arxiv.2502.11859
arXiv
2502.11859
OpenAlex
W4412945400
Cited by
4
References
0
Citations by year
2026: 3 · 2025: 1
FWCI
6.35
Type
conference paper
Language
en
Open access
Licence
Locations
  • Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Engineering › Automotive Engineering › Spatial Cognition and Navigation
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
Keywords
vision-language models, spatial ability, spatial reasoning, psychometrics, spatial cognition

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review