The DOI matches its Crossref record; 73 of the 73 references it cites were found in a registry or an index.
Do Vision and Language Models Share Concepts? A Vector Space Alignment Study
Open accessAbstract
Large-scale pretrained language models (LMs) are said to “lack the ability to connect utterances to the world” (Bender and Koller, 2020), because they do not have “mental models of the world” (Mitchell and Krakauer, 2023). If so, one would expect LM representations to be unrelated to representations induced by vision models. We present an empirical evaluation across four families of LMs (BERT, GPT-2, OPT, and LLaMA-2) and three vision model architectures (ResNet, SegFormer, and MAE). Our experiments show that LMs partially converge towards representations isomorphic to those of vision models, subject to dispersion, polysemy, and frequency. This has important implications for both multi-modal processing and the LM understanding debate (Mitchell and Krakauer, 2023).1
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Also, Li et al. [111] demonstrate that language models partially converge towards representations isomorphic to those of vision models.”
From Understanding World or Predicting Future? A Comprehensive Survey of World Models (2026) · cites this paper as [111]
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 1×
“We dial back from function and study whether the concept representations converge toward structural similarity (isomorphism).”
BackgroundBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 1×
“More related to our work, Huh et al. (2024) proposes a similar hypothesis, although studying it from a different perspective, and our findings corroborate theirs.”
ResultImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380 · cited 4×
“We build bimodal dictionaries of image-text pairs based on the ImageNet-21K dataset (Russakovsky et al., 2015) and the CLDI (cross-lingual dictionary induction) dataset (Hartmann and Søgaard, 2018).”
MethodInfluentialPyTorch: An Imperative Style, High-Performance Deep Learning Library
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. (2019) · arXiv · doi:10.48550/arxiv.1912.01703 · Cited by 31,473
Advances in Neural Information Processing Systems 19
(2007) · The MIT Press eBooks · doi:10.52202/085713 · Cited by 15,539
Learning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293 · cited 7×
“CLIP Results.”
MethodInfluentialTransformers: State-of-the-Art Natural Language Processing
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/2020.emnlp-demos.6 · Cited by 8,741
Masked Autoencoders Are Scalable Vision Learners
He, K., Chen, X., Xie, S., Li, Y., Dollar, P., & Girshick, R. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01553 · Cited by 8,489 · cited 1×
“We dial back from function and study whether the concept representations converge toward structural similarity (isomorphism).”
BackgroundSearle, J. R. (1980) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00005756 · Cited by 6,954 · cited 3×
“Some researchers have argued that LMs are ‘‘all syntax, no semantics’’, i.e., that they learn form, but not meaning (Searle, 1980; Bender and Koller, 2020; Marcus et al., 2023).”
BackgroundEmerging Properties in Self-Supervised Vision Transformers
Caron, M., Touvron, H., Misra, I., Jegou, H., Mairal, J., Bojanowski, P., & Joulin, A. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00951 · Cited by 5,638
SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers
Xie, E., Wang, W., Yu, Z., Anandkumar, A., Alvarez, J. M., & Luo, P. (2021) · arXiv · doi:10.48550/arxiv.2105.15203 · Cited by 5,090 · cited 7×
“We see that ResNet models score highest overall, followed by SegFormers, while MAE models rank third.”
MethodInfluentialScene Parsing through ADE20K Dataset
Zhou, B., Zhao, H., Puig, X., Fidler, S., Barriuso, A., & Torralba, A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.544 · Cited by 3,326 · cited 1×
“The SegFormer models we use are pretrained with ImageNet-1K (Russakovsky et al., 2015) and finetuned with ADE20K (Zhou et al., 2017).”
MethodLlama 2: Open Foundation and Fine-Tuned Chat Models
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.09288 · Cited by 2,604 · cited 3×
“CLIP results are presented in Appendix C. (Zhang et al., 2022), and LLaMA-2 (Touvron et al., 2023).”
ResultZhu, Y., Kiros, R., Zemel, R., Salakhutdinov, R., Urtasun, R., Torralba, A., & Fidler, S. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.11 · Cited by 2,264
A Generalized Solution of the Orthogonal Procrustes Problem
Schönemann, P. H. (1966) · Psychometrika · doi:10.1007/bf02289451 · Cited by 2,040 · cited 2×
“6 Following Conneau et al. (2018), we use Procrustes analysis (Sch¨onemann, 1966) to align the representations of VMs to those of LMs, given a bimodal dictionary ( § 4.1).”
MethodResultInfluentialNavigli, R., & Ponzetto, S. P. (2012) · Artificial Intelligence · doi:10.1016/j.artint.2012.07.001 · Cited by 1,465 · cited 2×
“Since many of the aliases in our bimodal dictionary are not in BabelNet, e.g., multi-word expressions, we also consider ‘language dispersion’ in Wikipedia.”
MethodEmergent Abilities of Large Language Models
Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Metzler, D., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.07682 · Cited by 1,264
Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data
Bender, E. M., & Koller, A. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.463 · Cited by 1,082 · cited 6×
“Most researchers agree, however, that LMs ‘‘lack the ability to connect utterances to the world’’ (Bender and Koller, 2020), because they do not have ‘‘mental models of the world’’ (Mitchell and Krakauer, 2023).”
BackgroundResultInfluentialRepresentation in Cognitive Science
Shea, N. (2018) · doi:10.1093/oso/9780198812883.001.0001 · Cited by 676 · cited 1×
“Many now use the term representation to mean a state of a cognitive system, i.e., neural network, that responds selectively to certain bodily and environmental conditions (Shea, 2018).”
BackgroundA robust self-learning method for fully unsupervised cross-lingual mappings of word embeddings
Artetxe, M., Labaka, G., & Agirre, E. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1073 · Cited by 555 · cited 1×
“We experimented with algorithms for un-supervised embedding alignment (Conneau et al., 2018; Artetxe et al., 2018; Hoshen and Wolf, 2018) and found that all suffer from the degenerate solu-tion problem described in Hartmann et al. (2018), i.e., that no algorithm is currently available that can…”
MethodThe neural architecture of language: Integrative modeling converges on predictive processing
Schrimpf, M., Blank, I. A., Tuckute, G., Kauf, C., Hosseini, E. A., Kanwisher, N., Tenenbaum, J. B., & Fedorenko, E. (2021) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2105646118 · Cited by 517
Brain-Score: Which Artificial Neural Network for Object Recognition is most Brain-Like?
Schrimpf, M., Kubilius, J., Hong, H., Majaj, N. J., Rajalingham, R., Issa, E. B., Kar, K., Bashivan, P., Prescott-Roy, J., Geiger, F., et al. (2018) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/407007 · Cited by 458 · cited 1×
“Cognitive scientists have also noted how the objectives of supervised language and vision models bear resemblances to predictive processing (Schrimpf et al., 2018; Goldstein et al., 2021; Caucheteux et al., 2022; Li et al., 2023) (but see Antonello and Huth (2022) for a critical discussion of such…”
BackgroundOpt: Open pre-trained transformer language models
Zhang, S. (2025) · TIB Data Manager · doi:10.57702/dn0pmcbs · Cited by 405
Well-Read Students Learn Better: On the Importance of Pre-training Compact Models
Turc, I., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1908.08962 · Cited by 398
Fellbaum, C. (2010) · Theory and Applications of Ontology: Computer Applications · doi:10.1007/978-90-481-8847-5_10 · Cited by 359
The debate over understanding in AI’s large language models
Mitchell, M., & Krakauer, D. C. (2023) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2215907120 · Cited by 340
Emergent linguistic structure in artificial neural networks trained by self-supervision
Manning, C. D., Clark, K., Hewitt, J., Khandelwal, U., & Levy, O. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1907367117 · Cited by 339 · cited 1×
“The literature on large-scale, pretrained models has reported seemingly emergent properties (Søgaard et al., 2018; Manning et al., 2020; Garneau et al., 2021; Teehan et al., 2022; Wei et al., 2022), many of which relate to induction of world knowledge.”
BackgroundBrains and algorithms partially converge in natural language processing
Caucheteux, C., & King, J.-R. (2022) · Communications Biology · doi:10.1038/s42003-022-03036-1 · Cited by 318
On the Limitations of Unsupervised Bilingual Dictionary Induction
Søgaard, A., Ruder, S., & Vulić, I. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1072 · Cited by 254 · cited 4×
“(Wei et al., 2022) while others have speculated if this is an effect of compression dynamics (Søgaard et al., 2018; Garneau et al., 2021).”
InfluentialSelf-citationMeaning without reference in large language modelsPreprint
Piantadosi, S. T., & Hill, F. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2208.02957 · Cited by 249
Learning Image Embeddings using Convolutional Neural Networks for Improved Multi-Modal Semantics
Kiela, D., & Bottou, L. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1005 · Cited by 224 · cited 2×
“We have discussed various implications, but not all: In the past, researchers have speculated if image representations could act as an interlingua for cross-lingual knowledge transfer (Bergsma and Van Durme, 2011; Kiela and Bottou, 2014; Vuli´c et al., 2016; Hartmann and Søgaard, 2018).”
BackgroundWord Translation Without Parallel Data
Conneau, A., Lample, G., Ranzato, M., Denoyer, L., & Jeǵou, H. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1710.04087 · Cited by 219 · cited 3×
“We experimented with algorithms for un-supervised embedding alignment (Conneau et al., 2018; Artetxe et al., 2018; Hoshen and Wolf, 2018) and found that all suffer from the degenerate solu-tion problem described in Hartmann et al. (2018), i.e., that no algorithm is currently available that can…”
MethodInfluentialPredictive Processing and the Representation Wars
Williams, D. (2017) · Minds and Machines · doi:10.1007/s11023-017-9441-6 · Cited by 181 · cited 1×
“The ‘‘representation wars’’ (Williams, 2018) refers to a controversy around the role of mental representations in cognitive processes.”
BackgroundToneva, M., & Wehbe, L. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.11833 · Cited by 178 · cited 1×
“We use PCA and Procrustes Analysis or ridge regression (Toneva and Wehbe, 2019) to map into the length-frequency space and report the best of those as a second, stronger baseline.”
MethodMaking AI Intelligible: Philosophical Foundations
Cappelen, H., & Dever, J. (2021) · doi:10.1093/oso/9780192894724.001.0001 · Cited by 162 · cited 2×
“…Sahlgren and Carlsson, 2021; Piantadosi and Hill, 2022), 3 whereas some have posited that a form of externalist referential semantics is possible, at least for chatbots engaged in direct conversation (Cappelen and Dever, 2021; Butlin, 2021; Mollo and Millière, 2023; Mandelk-ern and Linzen, 2023).”
BackgroundNon-Adversarial Unsupervised Word Translation
Hoshen, Y., & Wolf, L. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1043 · Cited by 159 · cited 1×
“Our alignment precision is nevertheless very high, highlighting the persistence of the structural similarities between LMs and VMs.”
BackgroundSelf-supervised learning through the eyes of a childPreprint
Orhan, A. E., Gupta, V. V., & Lake, B. M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.16189 · Cited by 141 · cited 1×
“As such, the field has shown a growing interest in computational representations induced with self-supervised learning (Orhan et al., 2020; Halvagal and Zenke, 2022).”
BackgroundIs this a wampimuk? Cross-modal mapping between distributional semantics and the visual world
Lazaridou, A., Bruni, E., & Baroni, M. (2014) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.3115/v1/p14-1132 · Cited by 114 · cited 1×
“In these studies, the partial alignability of brain and model representations is interpreted as evidence that brain and models might process language in the same way (Caucheteux and King, 2022).”
BackgroundHalvagal, M. S., & Zenke, F. (2023) · Nature Neuroscience · doi:10.1038/s41593-023-01460-y · Cited by 81 · cited 1×
“As such, the field has shown a growing interest in computational representations induced with self-supervised learning (Orhan et al., 2020; Halvagal and Zenke, 2022).”
BackgroundThinking ahead: spontaneous prediction in context as a keystone of language in humans and machinesPreprint
Goldstein, A., Zada, Z., Buchnik, E., Schain, M., Price, A. R., Aubrey, B., Nastase, S. A., Feder, A., Emanuel, D., Cohen, A., et al. (2020) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2020.12.02.403477 · Cited by 77
Do Language Models’ Words Refer?
Mandelkern, M., & Linzen, T. (2024) · Computational Linguistics · doi:10.1162/coli_a_00522 · Cited by 77
Can Language Models Encode Perceptual Structure Without Grounding? A Case Study in Color
Abdou, M., Kulmizev, A., Hershcovich, D., Frank, S., Pavlick, E., & Søgaard, A. (2021) · arXiv · doi:10.48550/arxiv.2109.06129 · Cited by 70 · cited 1×
“We therefore represent words by embedding English Wikipedia sentences, using the token representations that form the concept, decontextualizing these representations by averaging across different sentences (Abdou et al., 2021).”
MethodSelf-citationRepresentation Engineering: A Top-Down Approach to AI Transparency
Zou, A., Phan, L., Chen, S., Campbell, J., Guo, P., Ren, R., Pan, A., Yin, X., Mazeika, M., Dombrowski, A.-K., et al. (2023) · arXiv · doi:10.48550/arxiv.2310.01405 · Cited by 65 · cited 1×
“More related to our work, Huh et al. (2024) proposes a similar hypothesis, although studying it from a different perspective, and our findings corroborate theirs.”
ResultCan Language Models Encode Perceptual Structure Without Grounding? A Case Study in Color
Abdou, M., Kulmizev, A., Hershcovich, D., Frank, S., Pavlick, E., & Søgaard, A. (2021) · Proceedings of the 25th Conference on Computational Natural Language Learning · doi:10.18653/v1/2021.conll-1.9 · Cited by 64
Self-citationIntuition Pumps and the Proper Use of Thought Experiments
Brendel, E. (2004) · dialectica · doi:10.1111/j.1746-8361.2004.tb00293.x · Cited by 63 · cited 1×
“Thought experiments are, in general, fallible intuition pumps (Brendel, 2004), and we believe our results strongly suggest that the second thought experiment of Bender and Koller (2020) is misleading.”
ResultNORMA: Neighborhood Sensitive Maps for Multilingual Word Embeddings
Nakashole, N. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1047 · Cited by 61
Learning bilingual lexicons using the visual similarity of labeled web images
Bergsma, S., & Durme, B. V. (2011) · doi:10.5591/978-1-57735-516-8/ijcai11-296 · Cited by 52
Coelho Mollo, D., & Millière, R. (2026) · Philosophy and the Mind Sciences · doi:10.33735/phimisci.2026.12307 · Cited by 48
Mollo, D. C., & Millière, R. (2023) · arXiv · doi:10.48550/arxiv.2304.01481 · Cited by 47
Visual Bilingual Lexicon Induction with Transferred ConvNet Features
Kiela, D., Vulić, I., & Clark, S. C. (2015) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d15-1015 · Cited by 46 · cited 1×
“…(1) to evaluate whether our approach is robust across parts of speech we use concepts of all POS (nouns, adjectives, and verbs) as training and evaluation data; and (2) to evaluate whether the mapping learned by nouns generalizes to other POS, we use only nouns as training data and evaluate…”
MethodAntonello, R., & Huth, A. (2023) · Neurobiology of Language · doi:10.1162/nol_a_00087 · Cited by 45 · cited 1×
“…scientists have also noted how the objectives of supervised language and vision models bear resemblances to predictive processing (Schrimpf et al., 2018; Goldstein et al., 2021; Caucheteux et al., 2022; Li et al., 2023) (but see Antonello and Huth (2022) for a critical discussion of such work).”
BackgroundSharing Our Concepts with Machines
Butlin, P. (2021) · Erkenntnis · doi:10.1007/s10670-021-00491-w · Cited by 44 · cited 2×
“…Sahlgren and Carlsson, 2021; Piantadosi and Hill, 2022), 3 whereas some have posited that a form of externalist referential semantics is possible, at least for chatbots engaged in direct conversation (Cappelen and Dever, 2021; Butlin, 2021; Mollo and Milli`ere, 2023; Mandelkern and Linzen, 2023).”
BackgroundNon-Linear Instance-Based Cross-Lingual Mapping for Non-Isomorphic Embedding Spaces
Glavaš, G., & Vulić, I. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.675 · Cited by 42
Holism, Conceptual-Role Semantics, and Syntactic Semantics
Rapaport, W. J. (2002) · Minds and Machines · doi:10.1023/a:1013765011735 · Cited by 39 · cited 1×
“…who receives text messages in this language and follows Others have argued that LMs have inferential semantics, but not referential semantics (Rapaport, 2002; Sahlgren and Carlsson, 2021; Piantadosi and Hill, 2022), 3 whereas some have posited that a form of externalist referential semantics…”
BackgroundThe Platonic Representation Hypothesis
Huh, M., Cheung, B., Wang, T., & Isola, P. (2024) · arXiv · doi:10.48550/arxiv.2405.07987 · Cited by 35
The Singleton Fallacy: Why Current Critiques of Language Models Miss the Point
Sahlgren, M., & Carlsson, F. (2021) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2021.682578 · Cited by 34 · cited 1×
“…no prior knowledge of a foreign language, who receives text messages in this language and follows Others have argued that LMs have inferential semantics, but not referential semantics (Rapaport, 2002; Sahlgren and Carlsson, 2021; Piantadosi and Hill, 2022), 3 whereas some have posited that a…”
BackgroundLinearly Mapping from Image to Text Space
Merullo, J., Castricato, L., Eickhoff, C., & Pavlick, E. (2022) · arXiv · doi:10.48550/arxiv.2209.15162 · Cited by 34
Walsh, T. (2011) · Cited by 32
Leivada, E., Marcus, G., Günther, F., & Murphy, E. (2026) · Philosophical Transactions of the Royal Society A Mathematical Physical and Engineering Sciences · doi:10.1098/rsta.2025.0008 · Cited by 32
Traces of Meaning Itself: Encoding Distributional Word Vectors in Brain Activity
Sassenhagen, J., & Fiebach, C. J. (2019) · Neurobiology of Language · doi:10.1162/nol_a_00003 · Cited by 27 · cited 1×
“Studies have looked at the alignability of neural language representations and human brain activations, with more promising results as language models grow better at modeling language (Sassenhagen and Fiebach, 2020; Schrimpf et al., 2021).”
BackgroundStepping Back Inside Leibniz’s Mill
Lodge, P., & Bobro, M. (1998) · The Monist · doi:10.5840/monist199881427 · Cited by 24 · cited 1×
“The Mill Argument states that mental states cannot be reduced to physical states, so if the capacity to understand language requires mental states, this capacity cannot be instantiated, merely imitated, by machines.”
BackgroundLong-range and hierarchical language predictions in brains and algorithmsPreprint
Caucheteux, C., Gramfort, A., & King, J.-R. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.14232 · Cited by 24 · cited 2×
“…scientists have also noted how the objectives of supervised language and vision models bear resemblances to predictive processing (Schrimpf et al., 2018; Goldstein et al., 2021; Caucheteux et al., 2022; Li et al., 2023) (but see Antonello and Huth (2022) for a critical discussion of such work).”
BackgroundMulti-Modal Representations for Improved Bilingual Lexicon Learning
Vulić, I., Kiela, D., Clark, S. R. L., & Moens, M. (2016) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p16-2031 · Cited by 21 · cited 2×
“We have discussed various implications, but not all: In the past, researchers have speculated if image representations could act as an interlingua for cross-lingual knowledge transfer (Bergsma and Van Durme, 2011; Kiela and Bottou, 2014; Vuli´c et al., 2016; Hartmann and Søgaard, 2018).”
BackgroundFrom Brain Space to Distributional Space: The Perilous Journeys of fMRI Decoding
Minnema, G., & Herbelot, A. (2019) · Annual Meeting of the Association for Computational Linguistics: Student Research Workshop (ACL SRW) · doi:10.18653/v1/p19-2021 · Cited by 15 · cited 1×
“10 Note that this performance metric is much more conservative than other metrics used for similar problems, including pairwise matching accuracy, percentile rank, and Pearson correlation (Minnema and Herbelot, 2019).”
MethodA Sentence is Worth a Thousand Pictures: Can Large Language Models Understand Human Language?Preprint
Marcus, G., Leivada, E., & Murphy, E. (2023) · arXiv.org · doi:10.48550/arxiv.2308.00109 · Cited by 15 · cited 1×
“Some researchers have argued that LMs are ‘‘all syntax, no semantics’’, i.e., that they learn form, but not meaning (Searle, 1980; Bender and Koller, 2020; Marcus et al., 2023).”
BackgroundHalvagal, M. S., & Zenke, F. (2022) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2022.03.17.484712 · Cited by 12
Analogy Training Multilingual Encoders
Garneau, N., Hartmann, M., Sandholm, A., Ruder, S., Vulić, I., & Søgaard, A. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i14.17524 · Cited by 11 · cited 2×
“(Wei et al., 2022) while others have speculated if this is an effect of compression dynamics (Søgaard et al., 2018; Garneau et al., 2021).”
BackgroundSelf-citationStructural Similarities Between Language Models and Neural Response MeasurementsPreprint
Li, J., Karamolegkou, A., Kementchedjhieva, Y., Abdou, M., Lehmann, S., & Søgaard, A. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2306.01930 · Cited by 11 · cited 1×
“…scientists have also noted how the objectives of supervised language and vision models bear resemblances to predictive processing (Schrimpf et al., 2018; Goldstein et al., 2021; Caucheteux et al., 2022; Li et al., 2023) (but see Antonello and Huth (2022) for a critical discussion of such work).”
BackgroundSelf-citationWhy is unsupervised alignment of English embeddings from different algorithms so hard?
Hartmann, M., Kementchedjhieva, Y., & Søgaard, A. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1056 · Cited by 10 · cited 5×
“We have discussed various implications, but not all: In the past, researchers have speculated if image representations could act as an interlingua for cross-lingual knowledge transfer (Bergsma and Van Durme, 2011; Kiela and Bottou, 2014; Vuli´c et al., 2016; Hartmann and Søgaard, 2018).”
BackgroundMethodInfluentialSelf-citationLimitations of Cross-Lingual Learning from Image Search
Hartmann, M., & Soegaard, A. (2018) · Workshop on Representation Learning for NLP (RepL4NLP) · doi:10.18653/v1/w18-3021 · Cited by 9
Emergent Structures and Training Dynamics in Large Language Models
Teehan, R., Clinciu, M., Serikov, O., Szczechla, E., Seelam, N., Mirkin, S., & Gokaslan, A. (2022) · Proceedings of BigScience Episode #5 -- Workshop on Challenges & Perspectives in Creating Large Language Models · doi:10.18653/v1/2022.bigscience-1.11 · Cited by 7
Non-Linearity in Mapping Based Cross-Lingual Word Embeddings
Zhao, J., & Gilman, A. (2020) · Language Resources and Evaluation · Cited by 2
OPT: Open Pre-trained Transformer Language Models
Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V., et al. (2022) · arXiv · doi:10.48550/arxiv.2205.01068
Research works printed but not linked: 3
- Bird, 2009, Natural language processing with Python: Analyzing text with the natural language toolkit
- Dosovitskiy, 2021, An image is worth 16x16 words: Transformers for image recognition at scale
- Marconi, 1997, Lexical Competence
Cited by
Understanding World or Predicting Future? A Comprehensive Survey of World Models
Ding, J., Zhang, Y., Shang, Y., Zhang, Y., Zong, Z., Feng, J., Yuan, Y., Su, H., Li, N., Sukiennik, N., et al. (2026) · ACM Computing Surveys · doi:10.1145/3746449 · Cited by 169
“Also, Li et al.[143] demonstrate that language models partially converge towards representations isomorphic to those of vision models.”
Why concepts are (probably) vectors
Piantadosi, S. T., Muller, D. C. Y., Rule, J., Kaushik, K., Gorenstein, M. I., Leib, E. R., & Sanford, E. M. (2024) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2024.06.011 · Cited by 87
Takata, R., Masumori, A., & Ikegami, T. (2024) · Entropy · doi:10.3390/e26121092 · Cited by 37
What Do Language Models Hear? Probing for Auditory Representations in Language Models
Ngo, J., & Kim, Y. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.297 · Cited by 5
Cao, X., Li, J., Ying, T., & Liu, Z. (2026) · Information Processing & Management · doi:10.1016/j.ipm.2026.105028 · Cited by 3
Self-citation
Metadata
- Authors
- Li, J., Kementchedjhieva, Y., Fierro, C., & Søgaard, A.
- Affiliations
- University of Copenhagen (DK)
- Mohamed bin Zayed University of Artificial Intelligence (AE)
- Venue
- Transactions of the Association for Computational Linguistics
- Publisher
- MIT Press
- Published
- 2024
- Volume
- 12
- Pages
- 1232-1249
- ISSN
- 2307-387X (ISSN-L)
- OpenAlex
- W4403051422
- Cited by
- 6
- References
- 73
- Citations by year
- 2026: 2 · 2025: 2 · 2024: 2
- FWCI
- 0.84
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Locations
- Transactions of the Association for Computational Linguistics (publishedVersion, cc-by)
- Research at the University of Copenhagen (University of Copenhagen) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, cc-by)
- Research at the University of Copenhagen (University of Copenhagen) (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Funders
- Danmarks Grundforskningsfond
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- language models, vision models, shared representations, conceptual representation, pre-trained language models, vision transformer, multimodal representation, polysemy, word frequency, language comprehension, BERT, GPT-2, OPT, Llama 2, ResNet, SegFormer, MAE
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.