CoChat
Sign Up
Cited by 7Multimodal Machine Learning ApplicationsDomain Adaptation and Few-Shot LearningImage Retrieval and Classification Techniques

Do Vision and Language Encoders Represent the World Similarly?

Open access
Maniparambil, Mayug, Akshulakov, Raiymbek, Dahou Djilali, Yasser Abdelaziz, Seddik, Mohamed EI Amine, Narayan, Sanath, Mangalam, Karttikeya, O'Connor, Noel E.·2024·2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
DOI 10.1109/cvpr52733.2024.01359
Open this paper in CoChat — free accountOpen PDF

Abstract

Aligned text-image encoders such as CLIP have become the de-facto model for vision-language tasks. Further-more, modality-specific encoders achieve impressive per-formances in their respective domains. This raises a cen-tral question: does an alignment exist between uni-modal vision and language encoders since they fundamentally rep-resent the same physical world? Analyzing the latent spaces structure of vision and language models on image-caption benchmarks using the Centered Kernel Alignment (CKA), we find that the representation spaces of unaligned and aligned encoders are semantically similar. In the absence of statistical similarity in aligned encoders like CLIP, we show that a possible matching of unaligned encoders exists with-out any training. We frame this as a seeded graph-matching problem exploiting the semantic similarity between graphs and propose two methods - a Fast Quadratic Assignment Problem optimization, and a novel localized CKA metric-based matching/retrieval. We demonstrate the effectiveness of this on several downstream tasks including cross-lingual, cross-domain caption matching and image classification. Code available at github.com/mayug/0-shot-llm-vision.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/cvpr52733.2024.01359

Cites

6 of 6 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • After calibration, the previously reported convergence in global metrics largely disappears.

    “We find that, after calibration, the previously reported convergence in global metrics (Huh et al., 2024;Maniparambil et al., 2024;Tjandrasuwita et al., 2025) largely disappears, suggesting it was driven primarily by width and depth confounders, whereas local neighborhood-based metrics retain significant cross-modal alignment (Figure 2c).”

    From Revisiting the Platonic Representation Hypothesis: An Aristotelian View (2026) · cites this paper as Maniparambil et al., 2024

  • Well-trained vision encoders on large datasets exhibit high semantic similarity with language encoders regardless of the training paradigm.

    “In work concurrent to ours, Maniparambil et al. (2024) show well-trained vision encoders on large datasets exhibit high semantic similarity with language encoders regardless of the training paradigm (supervised, self-supervised, or language-supervised).”

    From The Platonic Representation Hypothesis (2024) · cites this paper as Maniparambil et al. (2024)

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 41

  1. Pranav Aggarwal and Ajinkya Kale. Towards zero- shot cross-lingual image retrieval. arXiv preprint arXiv:2012.05107, 2020. · arXiv
  2. Harsh Agrawal, Karan Desai, Yufei Wang, Xinlei Chen, Rishabh Jain, Mark Johnson, Dhruv Batra, Devi Parikh, Ste- fan Lee, and Peter Anderson. Nocaps: Novel object caption- ing at scale. In Proceedings of the IEEE/CVF international conference on computer vision, pages 8948-8957, 2019. 2, 5, 16
  3. Richard Antonello, Javier S Turek, Vy Vo, and Alexander Huth. Low-dimensional structure in the space of language representations is reflected in brain responses. Advances in neural information processing systems, 2021. 2
  4. Yamini Bansal, Preetum Nakkiran, and Boaz Barak. Revis- iting model stitching to compare neural representations. Ad- vances in neural information processing systems, 2021. 2
  5. Serguei Barannikov, Ilya Trofimov, Nikita Balabin, and Evgeny Burnaev. Representation topology divergence: A method for comparing neural network representations. arXiv preprint arXiv:2201.00058, 2021. 2 · arXiv
  6. Lisa Bonheme and Marek Grzes. How do variational autoen- coders learn? insights from representational similarity. arXiv preprint arXiv:2205.08399, 2022. 2 · arXiv
  7. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Sub- biah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakan- tan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Lan- guage models are few-shot learners. Advances in neural in- formation processing systems, 33:1877-1901, 2020. 4
  8. Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre- training to recognize long-tail visual concepts. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3558-3568, 2021. 1
  9. Ting Chen, Simon Kornblith, Mohammad Norouzi, and Ge- offrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on ma- chine learning, pages 1597-1607. PMLR, 2020. 1
  10. Alexis Conneau, Guillaume Lample, Marc'Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. Word translation with- out parallel data. arXiv preprint arXiv:1710.04087, 2017. 2 · arXiv
  11. Corinna Cortes, Mehryar Mohri, and Afshin Rostamizadeh. Algorithms for learning kernels based on centered alignment. The Journal of Machine Learning Research, 13(1):795-828, 2012. 1
  12. David F Crouse. On implementing 2d rectangular assign- ment algorithms. IEEE Transactions on Aerospace and Elec- tronic Systems, 52(4):1679-1696, 2016. 7
  13. Adrián Csiszárik, Péter Korösi-Szabó, Akos K Matszan- gosz, Gergely Papp, and Dániel Varga. Similarity and matching of neural network representations. arXiv preprint arXiv:2110.14633, 2021. 2 · arXiv
  14. J. Deng, W. Dong, R. Socher, L. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2009. 2
  15. Arthur Gretton, Olivier Bousquet, Alex Smola, and Bernhard Schölkopf. Measuring statistical dependence with hilbert- schmidt norms. In International conference on algorithmic learning theory, pages 63-77. Springer, 2005. 2
  16. Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Proceedings of the Thirteenth Inter- national Conference on Artificial Intelligence and Statistics, pages 297-304. JMLR Workshop and Conference Proceed- ings, 2010. 1
  17. Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representa- tion learning with noisy text supervision. In International conference on machine learning, pages 4904-4916. PMLR, 2021. 1
  18. Diederik P Kingma and Max Welling. Auto-encoding varia- tional bayes. ICLR, 2014. 2
  19. Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network represen- tations revisited. In International conference on machine learning, pages 3519-3529. PMLR, 2019. 1, 2
  20. Harold W Kuhn. The hungarian method for the assignment problem. Naval research logistics quarterly, 2(1-2):83-97, 1955. 4
  21. Karel Lenc and Andrea Vedaldi. Understanding image repre- sentations by measuring their equivariance and equivalence. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 991-999, 2015. 2
  22. Yixuan Li, Jason Yosinski, Jeff Clune, Hod Lipson, and John Hopcroft. Convergent learning: Do different neural networks learn the same representations? arXiv preprint arXiv:1511.07543, 2015. 2 · arXiv
  23. Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014. 1, 3, 5, 16
  24. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettle- moyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019. 1, 3, 4 · arXiv
  25. Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feicht- enhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF conference on com- puter vision and pattern recognition, pages 11976-11986, 2022. 4
  26. Wan-Duo Kurt Ma, JP Lewis, and W Bastiaan Kleijn. The hsic bottleneck: Deep learning without back-propagation. In Proceedings of the AAAI conference on artificial intelli- gence, pages 5085-5092, 2020. 2
  27. Tomas Mikolov, Quoc V Le, and Ilya Sutskever. Exploiting similarities among languages for machine translation (2013). arXiv preprint arXiv:1309.4168, 2022. 2 · arXiv
  28. Ari Morcos, Maithra Raghu, and Samy Bengio. Insights on representational similarity in neural networks with canoni- cal correlation. Advances in neural information processing systems, 31, 2018. 2
  29. Luca Moschella, Valentino Maiorca, Marco Fumero, An- tonio Norelli, Francesco Locatello, and Emanuele Rodolà. Relative representations enable zero-shot latent space com- munication. In The Eleventh International Conference on Learning Representations, 2022. 1, 2, 4, 5, 6, 7, 13, 14, 16
  30. Antonio Norelli, Marco Fumero, Valentino Maiorca, Luca Moschella, Emanuele Rodola, and Francesco Locatello. Asif: Coupled data turns unimodal models to multimodal without training. arXiv preprint arXiv:2210.01738, 2022. 1, 2 · arXiv
  31. Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Repre- sentation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748, 2018. 1 · arXiv
  32. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An im- perative style, high-performance deep learning library. Ad- vances in neural information processing systems, 32, 2019. 6
  33. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsu- pervised multitask learners. OpenAI blog, 1(8):9, 2019. 4
  34. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, pages 8748-8763. PMLR, 2021. 1, 4, 5, 16
  35. Maithra Raghu, Thomas Unterthiner, Simon Kornblith, Chiyuan Zhang, and Alexey Dosovitskiy. Do vision trans- formers see like convolutional neural networks? Advances in Neural Information Processing Systems, 34:12116-12128, 2021. 1
  36. Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Lan- guage Processing. Association for Computational Linguis- tics, 2019. 4
  37. Anton Tsitsulin, Marina Munkhoeva, Davide Mottin, Pana- giotis Karras, Alex Bronstein, Ivan Oseledets, and Em- manuel Müller. The shape of data: Intrinsic distance for data distributions. arXiv preprint arXiv:1905.11141, 2019. 2 · arXiv
  38. Joshua T Vogelstein, John M Conroy, Vince Lyzinski, Louis J Podrazik, Steven G Kratzer, Eric T Harley, Don- niell E Fishkind, R Jacob Vogelstein, and Carey E Priebe. Fast approximate quadratic programming for graph match- ing. PLOS one, 10(4):e0121002, 2015. 3
  39. Ivan Vulić, Sebastian Ruder, and Anders Søgaard. Are all good word vector spaces isomorphic? arXiv preprint arXiv:2004.04070, 2020. 2 · arXiv
  40. Sanghyun Woo, Shoubhik Debnath, Ronghang Hu, Xinlei Chen, Zhuang Liu, In So Kweon, and Saining Xie. Con- vnext v2: Co-designing and scaling convnets with masked autoencoders. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16133- 16142, 2023. 1, 5, 16
  41. John M Wu, Yonatan Belinkov, Hassan Sajjad, Nadir Dur- rani, Fahim Dalvi, and James Glass. Similarity analysis of contextual word representation models. arXiv preprint arXiv:2005.01172, 2020. · arXiv

Cited by

Sorted by citations

Authors
Maniparambil, M., Akshulakov, R., Dahou Djilali, Y. A., Seddik, M. E. A., Narayan, S., Mangalam, K., & O'Connor, N. E.
Affiliations
  • Dublin City University (IE)
  • University of California, Berkeley (US)
  • Technological Innovation Institute
  • Technology Innovation Institute (AE)
ORCID
Venue
2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
Publisher
IEEE
Published
2024-06-16
Pages
14334-14343
ISSN
1063-6919 (ISSN-L)
DOI
10.1109/cvpr52733.2024.01359
OpenAlex
W4402716370
Cited by
7
References
6
Citations by year
2026: 4 · 2025: 3
FWCI
1.44
Type
conference paper
Language
en
Open access
Licence
Locations
  • IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by-sa)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image Retrieval and Classification Techniques
Keywords
vision-language alignment, vision encoder, latent representations, centered kernel alignment, graph matching, quadratic assignment problem, image classification, CLIP

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review