CoChat
Sign Up
PreprintCited by 3Multimodal Machine Learning ApplicationsDomain Adaptation and Few-Shot LearningMultisensory perception and integration

Understanding the Emergence of Multimodal Representation Alignment

Open access
Tjandrasuwita, Megan, Ekbote, Chanakya, Ziyin, Liu, Liang, Paul Pu·2025·arXiv
DOI 10.48550/arxiv.2502.16282arXiv 2502.16282
Open this paper in CoChat — free accountOpen PDF

Abstract

Multimodal representation learning is fundamentally about transforming incomparable modalities into comparable representations. While prior research primarily focused on explicitly aligning these representations through targeted learning objectives and model architectures, a recent line of work has found that independently trained unimodal models of increasing scale and performance can become implicitly aligned with each other. These findings raise fundamental questions regarding the emergence of aligned representations in multimodal learning. Specifically: (1) when and why does alignment emerge implicitly? and (2) is alignment a reliable indicator of performance? Through a comprehensive empirical investigation, we demonstrate that both the emergence of alignment and its relationship with task performance depend on several critical data characteristics. These include, but are not necessarily limited to, the degree of similarity between the modalities and the balance between redundant and unique information they provide for the task. Our findings suggest that alignment may not be universally beneficial; rather, its impact on performance varies depending on the dataset and task. These insights can help practitioners determine whether increasing alignment between modalities is advantageous or, in some cases, detrimental to achieving optimal performance. Code is released at https://github.com/MeganTj/multimodal_alignment.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2502.16282

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • As neural networks scale in data volume, model capacity, and task diversity, their internal representation spaces exhibit increasing alignment, converging toward a shared statistical model of reality.

    “as neural networks scale in data volume, model capacity, and task diversity, their internal representation spaces exhibit increasing alignment, converging toward a shared statistical model of reality”

    From Platonic Representation Hypothesis on World Models (2026) · cites this paper as Tjandrasuwita et al., 2025

    The Representation Alignment section (p. 3) reports increasing alignment as vision and language models scale in capacity and performance, and describes convergence toward modality-agnostic representations as an implication. It does not establish the claimed effects of scaling data volume or task diversity, or convergence toward a shared statistical model of reality. The paper's own results (RQ1, p. 5) further qualify the trend: greater capacity does not reliably increase alignment when modalities contain substantial unique information. (LLM, conf=0.90)

    Experimental — accuracy not yet measured

  • After calibration, the previously reported convergence in global metrics largely disappears.

    “We find that, after calibration, the previously reported convergence in global metrics (Huh et al., 2024;Maniparambil et al., 2024;Tjandrasuwita et al., 2025) largely disappears, suggesting it was driven primarily by width and depth confounders, whereas local neighborhood-based metrics retain significant cross-modal alignment (Figure 2c).”

    From Revisiting the Platonic Representation Hypothesis: An Aristotelian View (2026) · cites this paper as Tjandrasuwita et al., 2025

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 58

  1. Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., and Hasson, Y. Flamingo: a visual language model for few-shot learning. arXiv preprint arXiv:2204.14198 , 2022. · arXiv
  2. Andrew, G., Arora, R., Bilmes, J., and Livescu, K. Deep canonical correlation analysis. In ICML , 2013.
  3. Arevalo, J., Solorio, T., Montes-y Gómez, M., and González, F. A. Gated Multimodal Units for Information Fusion, 2017. URL http://arxiv.org/abs/1702.01992 . · arXiv
  4. Bagher Zadeh, A., Liang, P. P., Poria, S., Cambria, E., and Morency, L.-P. Multimodal Language Analysis in the Wild: CMU-MOSEI Dataset and Interpretable Dynamic Fusion Graph. In Gurevych, I. and Miyao, Y. (eds.), Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pp. 2236-2246, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10.18653/v1/P18-1208. URL https://aclanthology. org/P18-1208/ . · DOI · Paper page
  5. Baltrušaitis, T., Ahuja, C., and Morency, L.-P. Multimodal machine learning: A survey and taxonomy. IEEE TPAMI , 41(2):423-443, 2018.
  6. Bansal, Y., Nakkiran, P., and Barak, B. Revisiting Model Stitching to Compare Neural Representations, June 2021. URL http: //arxiv.org/abs/2106.07682 . arXiv:2106.07682. · arXiv
  7. Barannikov, S., Trofimov, I., Balabin, N., and Burnaev, E. Representation Topology Divergence: A Method for Comparing Neural Network Representations. In Proceedings of the 39th International Conference on Machine Learning , pp. 1607-1626. PMLR, June 2022. URL https://proceedings.mlr. press/v162/barannikov22a.html . ISSN: 2640-3498.
  8. Bertinetto, L., Valmadre, J., Henriques, J. F., Vedaldi, A., and Torr, P. H. Fully-convolutional siamese networks for object tracking. In Computer Vision-ECCV 2016 Workshops: Amsterdam, The Netherlands, October 8-10 and 15-16, 2016, Proceedings, Part II 14 , pp. 850-865. Springer, 2016.
  9. Bonheme, L. and Grzes, M. How do Variational Autoencoders Learn? Insights from Representational Similarity, September 2022. URL http://arxiv.org/abs/2205.08399 . arXiv:2205.08399 [cs]. · arXiv
  10. Cai, Q., Wang, H., Li, Z., and Liu, X. A survey on multimodal datadriven smart healthcare systems: approaches and applications. IEEE Access , 7:133583-133599, 2019.
  11. Castro, S., Hazarika, D., Pérez-Rosas, V., Zimmermann, R., Mihalcea, R., and Poria, S. Towards Multimodal Sarcasm Detection (An \_obviously\_ Perfect Paper). In Korhonen, A., Traum, D., and Màrquez, L. (eds.), Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pp. 4619-4629, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/P19-1455. URL https://aclanthology.org/P19-1455/ . · DOI · Paper page
  12. Csiszárik, A., K\Horösi-Szabó, P., Matszangosz, A., Papp, G., and Varga, D. Similarity and Matching of Neural Network Representations. In Advances in Neural Information Processing Systems , volume 34, pp. 56565668. Curran Associates, Inc., 2021. URL https: //proceedings.neurips.cc/paper/2021/hash/ 2cb274e6ce940f47beb8011d8ecb1462-Abstract. html .
  13. Dufumier, B., Castillo-Navarro, J., Tuia, D., and Thiran, J.-P. What to align in multimodal contrastive learning?, September 2024. URL http://arxiv.org/abs/2409.07402 . arXiv:2409.07402 [cs]. · arXiv
  14. Frome, A., Corrado, G. S., Shlens, J., Bengio, S., Dean, J., Ranzato, M., and Mikolov, T. Devise: A deep visual-semantic embedding model. In Advances in neural information processing systems , pp. 2121-2129, 2013.
  15. Golub, G. H. and Zha, H. The canonical correlations of matrix pairs and their numerical computation . Springer, 1995.
  16. Gretton, A., Bousquet, O., Smola, A., and Schölkopf, B. Measuring statistical dependence with hilbert-schmidt norms. In International conference on algorithmic learning theory , pp. 63-77. Springer, 2005.
  17. Hasan, M. K., Rahman, W., Bagher Zadeh, A., Zhong, J., Tanveer, M. I., Morency, L.-P., and Hoque, M. E. UR-FUNNY: A Multimodal Language Dataset for Understanding Humor. In Inui, K., Jiang, J., Ng, V ., and Wan, X. (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) , pp. 20462056, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v1/D19-1211. URL https://aclanthology.org/D19-1211/ . · DOI · Paper page
  18. He, K., Chen, X., Xie, S., Li, Y., Dollár, P., and Girshick, R. Masked Autoencoders Are Scalable Vision Learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 16000-16009, 2022.
  19. Hu, D., Nie, F., and Li, X. Deep multimodal clustering for unsupervised audiovisual learning. In CVPR , pp. 9248-9257, 2019.
  20. Huh, M., Cheung, B., Wang, T., and Isola, P. The Platonic Representation Hypothesis, July 2024. URL http://arxiv. org/abs/2405.07987 . arXiv:2405.07987. · arXiv
  21. Jia, C., Yang, Y., Xia, Y., Chen, Y.-T., Parekh, Z., and Pham, H. Scaling up visual and vision-language representation learning with noisy text supervision. In ICML , pp. 4904-4916. PMLR, 2021.
  22. Kirchner, E. A., Fairclough, S. H., and Kirchner, F. Embedded multimodal interfaces in robotics: applications, future trends, and societal implications , pp. 523-576. Association for Computing Machinery and Morgan & Claypool, 2019. ISBN 9781970001754. URL https://doi.org/ 10.1145/3233795.3233810 . · DOI
  23. Klabunde, M., Schumacher, T., Strohmaier, M., and Lemmerich, F. Similarity of Neural Network Models: A Survey of Functional and Representational Measures, August 2024. URL http:// arxiv.org/abs/2305.06329 . arXiv:2305.06329 [cs]. · arXiv
  24. Kornblith, S., Norouzi, M., Lee, H., and Hinton, G. Similarity of Neural Network Representations Revisited, July 2019. URL http://arxiv.org/abs/1905.00414 . arXiv:1905.00414. · arXiv
  25. Lai, P. L. and Fyfe, C. Kernel and nonlinear canonical correlation analysis. International journal of neural systems , 10(05):365377, 2000.
  26. Lecun, Y., Bottou, L., Bengio, Y., and Haffner, P. Gradientbased learning applied to document recognition. Proceedings of the IEEE , 86(11):2278-2324, November 1998. ISSN 15582256. doi: 10.1109/5.726791. URL https://ieeexplore. ieee.org/document/726791 . Conference Name: Proceedings of the IEEE. · DOI · Paper page
  27. Lee, M. A., Zhu, Y., Srinivasan, K., Shah, P., and Savarese, S. Making sense of vision and touch: Self-supervised learning of multimodal representations for contact-rich tasks. In ICRA , pp. 8943-8950. IEEE, 2019.
  28. Lenc, K. and Vedaldi, A. Understanding Image Representations by Measuring Their Equivariance and Equivalence. Int. J. Comput. Vision , 127(5):456-476, May 2019. ISSN 0920-5691. doi: 10.1007/s11263-018-1098-y. URL https://doi.org/10. 1007/s11263-018-1098-y . · DOI · Paper page
  29. Li, J., Selvaraju, R., Gotmare, A., Joty, S., Xiong, C., and Hoi, S. C. H. Align before fuse: Vision and language representation learning with momentum distillation. Advances in neural information processing systems , 34:9694-9705, 2021.
  30. Li, Y., Yosinski, J., Clune, J., Lipson, H., and Hopcroft, J. Convergent Learning: Do different neural networks learn the same representations? In Proceedings of the 1st International Workshop on Feature Extraction: Modern Questions and Challenges at NIPS 2015 , pp. 196-212. PMLR, December 2015. URL https://proceedings.mlr.press/v44/ li15convergent.html . ISSN: 1938-7228.
  31. Liang, P. P., Lyu, Y., Fan, X., Wu, Z., Cheng, Y., Wu, J., and Chen, L. Y. Multibench: Multiscale benchmarks for multimodal representation learning. In NeurIPS Datasets and Benchmarks Track , 2021.
  32. Liang, P. P., Cheng, Y., Fan, X., Ling, C. K., Nie, S., Chen, R. J., and Deng, Z. Quantifying & modeling multimodal interactions: An information decomposition framework. In NeurIPS , 2023a.
  33. Liang, P. P., Deng, Z., Ma, M., Zou, J., Morency, L.-P., and Salakhutdinov, R. Factorized contrastive learning: Going beyond multi-view redundancy. In NeurIPS , 2023b.
  34. Liang, P. P., Zadeh, A., and Morency, L.-P. Foundations & trends in multimodal machine learning: Principles, challenges, and open questions. ACM Computing Surveys , 56(10):1-42, 2024.
  35. McFee, B., Raffel, C., Liang, D., Ellis, D. P., McVicar, M., Battenberg, E., and Nieto, O. librosa: Audio and music signal analysis in python. In Proceedings of the 14th python in science conference , volume 8, 2015.
  36. Mekhaldi, D. Multimodal document alignment: towards a fullyindexed multimedia archive. In Proceedings of the Multimedia Information Retrieval Workshop, SIGIR, Amsterdam, the Netherlands , 2007.
  37. Morcos, A. S., Raghu, M., and Bengio, S. Insights on representational similarity in neural networks with canonical correlation, October 2018. URL http://arxiv.org/abs/ 1806.05759 . arXiv:1806.05759 [stat]. · arXiv
  38. Oquab, M., Darcet, T., Moutakanni, T., Vo, H. V., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El-Nouby, A., Assran, M., Ballas, N., Galuba, W., Howes, R., Huang, P.-Y., Li, S.-W., Misra, I., Rabbat, M., Sharma, V., Synnaeve, G., Xu, H., Jegou, H., Mairal, J., Labatut, P., Joulin, A., and Bojanowski, P. DINOv2: Learning Robust Visual Features without Supervision. Transactions on Machine Learning Research , July 2023. ISSN 2835-8856. URL https://openreview.net/forum? id=a68SUt6zFt .
  39. Peng, H., Long, F., and Ding, C. Feature selection based on mutual information criteria of max-dependency, max-relevance, and min-redundancy. IEEE Transactions on pattern analysis and machine intelligence , 27(8):1226-1238, 2005.
  40. Pérez-Rúa, J.-M., Vielzeuf, V., Pateux, S., Baccouche, M., and Jurie, F. Mfas: Multimodal fusion architecture search. In CVPR , pp. 6966-6975, 2019.
  41. Pérez-Rúa, J.-M., Vielzeuf, V., Pateux, S., Baccouche, M., and Jurie, F. MFAS: Multimodal Fusion Architecture Search, March 2019. URL http://arxiv.org/abs/1903.06496 . arXiv:1903.06496 [cs]. · arXiv
  42. Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., and Agarwal, S. Learning transferable visual models from natural language supervision. In ICML , pp. 8748-8763. PMLR, 2021a.
  43. Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, I. Learning Transferable Visual Models From Natural Language Supervision. In Proceedings of the 38th International Conference on Machine Learning , pp. 8748-8763. PMLR, July 2021b. URL https://proceedings.mlr. press/v139/radford21a.html . ISSN: 2640-3498.
  44. Raghu, M., Gilmer, J., Yosinski, J., and Sohl-Dickstein, J. SVCCA: Singular Vector Canonical Correlation Analysis for Deep Learning Dynamics and Interpretability, November 2017. URL http://arxiv.org/abs/1706.05806 . arXiv:1706.05806 [stat]. · arXiv
  45. Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Radford, A., Chen, M., and Sutskever, I. Zero-shot text-to-image generation. In ICML , pp. 8821-8831. PMLR, 2021.
  46. Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A. C., and Fei-Fei, L. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision , 115(3): 211-252, December 2015. ISSN 1573-1405. doi: 10.1007/ s11263-015-0816-y. URL https://doi.org/10.1007/ s11263-015-0816-y .
  47. Schrimpf, M., Kubilius, J., Hong, H., Majaj, N. J., Rajalingham, R., Issa, E. B., Kar, K., Bashivan, P., Prescott-Roy, J., Schmidt, K., Yamins, D. L. K., and DiCarlo, J. J. Brain-Score: Which Artificial Neural Network for Object Recognition is most Brain-Like?, September 2018. URL https://www. biorxiv.org/content/10.1101/407007v1 . Pages: 407007 Section: New Results. · DOI
  48. Singer, U., Polyak, A., Hayes, T., Yin, X., An, J., Zhang, S., and Hu, Q. Make-a-video: Text-to-video generation without text-video data. arXiv preprint arXiv:2209.14792 , 2022. · arXiv
  49. Srinivasan, K., Raman, K., Chen, J., Bendersky, M., and Najork, M. WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learning. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval , SIGIR '21, pp. 2443-2449, New York, NY, USA, July 2021. Association for Computing Machinery. ISBN 978-1-4503-8037-9. doi: 10.1145/3404835.3463257. URL https://dl.acm.org/ doi/10.1145/3404835.3463257 . · DOI · Paper page
  50. Sun, C., Myers, A., Vondrick, C., Murphy, K., and Schmid, C. Videobert: A joint model for video and language representation learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pp. 7464-7473, 2019.
  51. Thompson, B. Canonical Correlation Analysis. In Encyclopedia of Statistics in Behavioral Science . John Wiley & Sons, Ltd, 2005. ISBN 978-0-470-01319-9. doi: 10.1002/0470013192.bsa068. · DOI
  52. Tian, Y., Sun, C., Poole, B., Krishnan, D., Schmid, C., and Isola, P. What makes for good views for contrastive learning? NeurIPS , 33:6827-6839, 2020.
  53. Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D., Blecher, L., Ferrer, C. C., Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu, W., Fuller, B., Gao, C., Goswami, V., Goyal, N., et al. Llama 2: Open Foundation and Fine-Tuned Chat Models, July 2023. URL http://arxiv. org/abs/2307.09288 . arXiv:2307.09288 [cs]. · arXiv
  54. Wang, W., Arora, R., Livescu, K., and Bilmes, J. On deep multiview representation learning. In ICML , pp. 1083-1092. PMLR, 2015.
  55. Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., Davison, J., Shleifer, S., Platen, P. v., Ma, C., Jernite, Y., Plu, J., Xu, C., Scao, T. L., Gugger, S., Drame, M., Lhoest, Q., and Rush, A. M. Transformers: State-of-the-Art Natural Language Processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations , pp. 38-45, Online, October 2020. Association for Computational Linguistics. URL https://www.aclweb.org/ anthology/2020.emnlp-demos.6 .
  56. Workshop, B., Scao, T. L., Fan, A., Akiki, C., Pavlick, E., Ili´ c, S., Hesslow, D., Castagné, R., Luccioni, A. S., Yvon, F., Gallé, M., Tow, J., Rush, A. M., Biderman, S., Webson, A., Ammanamanchi, P. S., Wang, T., Sagot, B., Muennighoff, N., Moral, A. V. d., Ruwase, O., Bawden, R., Bekman, S., McMillanMajor, A., Beltagy, I., Nguyen, H., Saulnier, L., Tan, S., et al. BLOOM: A 176B-Parameter Open-Access Multilingual Language Model, June 2023. URL http://arxiv.org/abs/ 2211.05100 . arXiv:2211.05100 [cs]. · arXiv
  57. Xiao, Y., Codevilla, F., Gurram, A., Urfalioglu, O., and López, A. M. Multimodal end-to-end autonomous driving. IEEE Transactions on Intelligent Transportation Systems , 23(1):537547, 2020.
  58. Zadeh, A., Zellers, R., Pincus, E., and Morency, L.-P. MOSI: Multimodal Corpus of Sentiment Intensity and Subjectivity Analysis in Online Opinion Videos, August 2016. URL http:// arxiv.org/abs/1606.06259 . arXiv:1606.06259 [cs]. · arXiv

Cited by

Sorted by citations

Authors
Tjandrasuwita, M., Ekbote, C., Ziyin, L., & Liang, P. P.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2025
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2502.16282
arXiv
2502.16282
OpenAlex
W4415067645
Cited by
0
References
0
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Social Sciences › Psychology › Experimental and Cognitive Psychology › Multisensory perception and integration
Keywords
multimodal representation learning, representation alignment, task performance, data characteristics, cross-modal learning

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review