Linearly Mapping from Image to Text Space
Open accessAbstract
The extent to which text-only language models (LMs) learn to represent features of the non-linguistic world is an open question. Prior work has shown that pretrained LMs can be taught to caption images when a vision model's parameters are optimized to encode images in the language space. We test a stronger hypothesis: that the conceptual representations learned by frozen text-only models and vision-only models are similar enough that this can be achieved with a linear map. We show that the image representations from vision models can be transferred as continuous prompts to frozen LMs by training only a single linear projection. Using these to prompt the LM achieves competitive performance on captioning and visual question answering tasks compared to models that tune both the image encoder and text decoder (such as the MAGMA model). We compare three image encoders with increasing amounts of linguistic supervision seen during pretraining: BEIT (no linguistic information), NF-ResNET (lexical category information), and CLIP (full natural language descriptions). We find that all three encoders perform equally well at transferring visual property information to the language model (e.g., whether an animal is large or small), but that image encoders pretrained with linguistic supervision more saliently encode category information (e.g., distinguishing hippo vs. elephant) and thus perform significantly better on benchmark language-and-vision tasks. Our results indicate that LMs encode conceptual information structurally similarly to vision-based models, even those that are solely trained on images. Code is available here: https://github.com/jmerullo/limber
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Recent studies have found a striking representational alignment between a wide variety of ANNs and the human brain, despite their having different architectures, network initializations, learning objectives, and even different modalities such as language and vision.
“Furthermore, recent studies have found a striking representational alignment between a wide variety of ANNs and the human brain, despite their having different architectures, network initializations, learning objectives, and even different modalities such as language and vision [12,15–22]”
From The Umwelt Representation Hypothesis: rethinking Universality (2026) · cites this paper as [19]
The Conclusion (p. 9) reports conceptual similarity between language-model and vision-model representations, supporting the cross-modal part of the claim. But the paper does not establish alignment between ANNs and the human brain. Its Discussion (p. 9) also says performance varies with the image encoder’s pretraining, rather than showing alignment despite different learning objectives. (LLM, conf=0.84)
Experimental — accuracy not yet measured[88] and [89] investigate whether it is possible to project entire images into an LLM's representation space, for example, to generate a description, or to ask or answer questions about the image.
“[88] and more recently [89] investigate whether it is possible to project entire images into an LLM’s representation space, for example, to generate a description, or to ask or answer questions about the image.”
From Symbols and grounding in large language models (2023) · cites this paper as [89]
Both [88] and [89] report positive results for projecting images into an LLM's representation space.
“Despite the simplicity, both papers report positive results.”
From Symbols and grounding in large language models (2023) · cites this paper as [89]
In [89], an LLM trained only on text could generate captions with a surprising level of detail if the image was encoded by a multimodal system with access to both language and vision during training.
“For example, in [89], we found that an LLM which was trained only on text could generate captions with a surprising level of detail if the image was encoded by a multimodal system (i.e. one with access to both language and vision during training).”
From Symbols and grounding in large language models (2023) · cites this paper as [89]
[88-91] are notable for using only a simple linear projection to perform the mapping from images into an LLM's representation space.
“While other work has done this using various deep learning architectures [88–91] are notable for using only a simple linear projection in order to perform this mapping.”
From Symbols and grounding in large language models (2023) · cites this paper as [89]
Even when given an image encoded by a system with no linguistic knowledge whatsoever, the LLM was able to generate a caption with coarse-grained conceptual awareness.
“Perhaps more impressively, even when given an image encoded by a system with no linguistic knowledge whatsoever, the LLM was able to generate a caption with course-grained conceptual awareness (e.g. mislabelling a tennis racket as a baseball bat) (figure 2).”
From Symbols and grounding in large language models (2023) · cites this paper as [89]
A single linear projection is sufficient to stitch a vision model to an LLM and achieve good performance on visual question answering and image captioning.
“Merullo et al. (2022) extended model stitching to the crossmodal setting, finding that a single linear projection is sufficient to stitch a vision model to an LLM and achieve good performance on visual question answering and image captioning.”
From The Platonic Representation Hypothesis (2024) · cites this paper as Merullo et al. (2022)
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 37
- Harsh Agrawal, Karan Desai, Yufei Wang, Xinlei Chen, Rishabh Jain, Mark Johnson, Dhruv Batra, Devi Parikh, Stefan Lee, and Peter Anderson. Nocaps: Novel object captioning at scale. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pp. 8948-8957, 2019.
- Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. arXiv preprint arXiv:2204.14198 , 2022. · arXiv
- Yamini Bansal, Preetum Nakkiran, and Boaz Barak. Revisiting model stitching to compare neural representations. Advances in Neural Information Processing Systems , 34:225-236, 2021.
- Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: Bert pre-training of image transformers. In International Conference on Learning Representations , 2021.
- Andy Brock, Soham De, Samuel L Smith, and Karen Simonyan. High-performance large-scale image recognition without normalization. In International Conference on Machine Learning , pp. 1059-1071. PMLR, 2021.
- Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877-1901, 2020.
- Karan Desai and Justin Johnson. Virtex: Learning visual representations from textual annotations. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pp. 11162-11173, 2021.
- Jacob Devlin, Hao Cheng, Hao Fang, Saurabh Gupta, Li Deng, Xiaodong He, Geoffrey Zweig, and Margaret Mitchell. Language models for image captioning: The quirks and what works. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers) , pp. 100-105, 2015.
- Constantin Eichenberg, Sid Black, Samuel Weinbach, Letitia Parcalabescu, and Anette Frank. Magma - multimodal augmentation of generative models through adapter-based finetuning. ArXiv , abs/2112.05253, 2021. · arXiv
- Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the V in VQA matter: Elevating the role of image understanding in Visual Question Answering. In Conference on Computer Vision and Pattern Recognition (CVPR) , 2017.
- Liangke Gui, Qiuyuan Huang, Alex Hauptmann, Yonatan Bisk, and Jianfeng Gao. Training visionlanguage transformers from captions alone. arXiv preprint arXiv:2205.09256 , 2022. · arXiv
- Yaru Hao, Haoyu Song, Li Dong, Shaohan Huang, Zewen Chi, Wenhui Wang, Shuming Ma, and Furu Wei. Language models are general-purpose interfaces. arXiv preprint arXiv:2206.06336 , 2022. · arXiv
- Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. CLIPScore: a reference-free evaluation metric for image captioning. In EMNLP , 2021.
- Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning , volume 97 of Proceedings of Machine Learning Research , pp. 3519-3529. PMLR, 09-15 Jun 2019. URL https: //proceedings.mlr.press/v97/kornblith19a.html .
- Nikolaus Kriegeskorte, Marieke Mur, and Peter A Bandettini. Representational similarity analysisconnecting the branches of systems neuroscience. Frontiers in systems neuroscience , pp. 4, 2008.
- Karel Lenc and Andrea Vedaldi. Understanding image representations by measuring their equivariance and equivalence. In Proceedings of the IEEE conference on computer vision and pattern recognition , pp. 991-999, 2015.
- Belinda Z Li, Maxwell Nye, and Jacob Andreas. Implicit representations of meaning in neural language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , pp. 1813-1827, 2021.
- Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ ar, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision , pp. 740-755. Springer, 2014.
- Xudong Lin, Gedas Bertasius, Jue Wang, Shih-Fu Chang, Devi Parikh, and Lorenzo Torresani. Vx2text: End-to-end learning of video-based text generation from multimodal inputs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 7005-7015, 2021.
- Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations , 2018.
- Kevin Lu, Aditya Grover, Pieter Abbeel, and Igor Mordatch. Pretrained transformers as universal computation engines. CoRR , abs/2103.05247, 2021. URL https://arxiv.org/abs/ 2103.05247 . · arXiv
- Ziyang Luo, Yadong Xi, Rongsheng Zhang, and Jing Ma. A frustratingly simple approach for endto-end image captioning, 2022. URL https://arxiv.org/abs/2201.12723 . · arXiv
- George A Miller. Wordnet: a lexical database for english. Communications of the ACM , 38(11): 39-41, 1995.
- Ron Mokady, Amir Hertz, and Amit H Bermano. Clipcap: Clip prefix for image captioning. arXiv preprint arXiv:2111.09734 , 2021. · arXiv
- Cory Paik, St´ ephane Aroca-Ouellette, Alessandro Roncone, and Katharina Kann. The world of an octopus: How reporting bias influences a language model's perception of color. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pp. 823-835, 2021.
- Roma Patel and Ellie Pavlick. Mapping language models to grounded conceptual spaces. In International Conference on Learning Representations , 2022. URL https://openreview.net/ forum?id=gJcEM8sxHK .
- Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning , pp. 8748-8763. PMLR, 2021.
- S Rajbhandari, J Rasley, O Ruwase, and Y He. Zero: memory optimization towards training a trillion parameter models. arxiv e-prints arxiv: 11910.02054 (2019), 2019.
- Thomas Scialom, Patrick Bordes, Paul-Alexis Dray, Jacopo Staiano, and Patrick Gallinari. What bert sees: Cross-modal transfer for visual question generation. In Proceedings of the 13th International Conference on Natural Language Generation , pp. 327-337, 2020.
- Sheng Shen, Liunian Harold Li, Hao Tan, Mohit Bansal, Anna Rohrbach, Kai-Wei Chang, Zhewei Yao, and Kurt Keutzer. How much can clip benefit vision-and-language tasks? In International Conference on Learning Representations , 2021.
- Yi-Lin Sung, Jaemin Cho, and Mohit Bansal. Vl-adapter: Parameter-efficient transfer learning for vision-and-language tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 5227-5237, 2022.
- Maria Tsimpoukelli, Jacob Menick, Serkan Cabi, S. M. Ali Eslami, Oriol Vinyals, and Felix Hill. Multimodal few-shot learning with frozen language models. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems , 2021. URL https://openreview.net/forum?id=WtmMyno9Tq2 .
- Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus-based image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition , pp. 4566-4575, 2015.
- Zhenhailong Wang, Manling Li, Ruochen Xu, Luowei Zhou, Jie Lei, Xudong Lin, Shuohang Wang, Ziyi Yang, Chenguang Zhu, Derek Hoiem, et al. Language models with image descriptors are strong few-shot video-language learners. arXiv preprint arXiv:2205.10747 , 2022. · arXiv
- Zhibiao Wu and Martha Palmer. Verbs semantics and lexical selection. In Proceedings of the 32nd annual meeting on Association for Computational Linguistics , pp. 133-138, 1994.
- Yujia Xie, Luowei Zhou, Xiyang Dai, Lu Yuan, Nguyen Bach, Ce Liu, and Michael Zeng. Visual clues: Bridging vision and language foundations for image paragraph captioning. arXiv preprint arXiv:2206.01843 , 2022. · arXiv
- Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer. Lit: Zero-shot transfer with locked-image text tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 18123-18133, 2022.
Cited by
Sorted by citations
Muse: Text-To-Image Generation via Masked Generative Transformers
Chang, H., Zhang, H., Barber, J., Maschinot, A., Lezama, J., Jiang, L., Yang, M.-H., Murphy, K., Freeman, W. T., Rubinstein, M., et al. (2023) · arXiv · doi:10.48550/arxiv.2301.00704 · Cited by 157 · see citing works
“…Muse model learns to map these rich visual and semantic concepts in the LLM embeddings to the generated images; it has been shown in recent work (Merullo et al., 2022) that the conceptual representations learned by LLM’s are roughly linearly mappable to those learned by models trained on vision…”
MethodSelf-Supervised Multimodal Learning: A Survey
Zong, Y., Aodha, O. M., & Hospedales, T. M. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3429301 · Cited by 122 · see citing works
Symbols and grounding in large language models
Pavlick, E. (2023) · Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences · doi:10.1098/rsta.2022.0041 · Cited by 95 · see citing works
Self-citationUnified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Action
Lu, J., Clark, C., Lee, S.-H., Zhang, Z., Khosla, S., Marten, R., Hoiem, D., & Kembhavi, A. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02497 · Cited by 98 · see citing works
Generating meaning: active inference and the scope and limits of passive AI
Pezzulo, G., Parr, T., Cisek, P., Clark, A., & Friston, K. (2024) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2023.10.002 · Cited by 80 · see citing works
Generative AI in the context of assistive technologies: Trends, limitations and future directions
Fu, B., Hadid, A., & Damer, N. (2024) · Image and Vision Computing · doi:10.1016/j.imavis.2024.105347 · Cited by 52 · see citing works
MolCA: Molecular Graph-Language Modeling with Cross-Modal Projector and Uni-Modal Adapter
Liu, Z., Li, S., Luo, Y., Fei, H., Cao, Y., Kawaguchi, K., Wang, X., & Chua, T. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.966 · Cited by 76 · see citing works
LMEye: An Interactive Perception Network for Large Language Models
Li, Y., Hu, B., Chen, X., Ma, L., Xu, Y. D., & Zhang, M. (2024) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2024.3428317 · Cited by 43 · see citing works
Why concepts are (probably) vectors
Piantadosi, S. T., Muller, D. C. Y., Rule, J., Kaushik, K., Gorenstein, M. I., Leib, E. R., & Sanford, E. M. (2024) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2024.06.011 · Cited by 37 · see citing works
Emergent Linear Representations in World Models of Self-Supervised Sequence Models
Nanda, N., Lee, A., & Wattenberg, M. (2023) · Proceedings of the 6th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP · doi:10.18653/v1/2023.blackboxnlp-1.2 · Cited by 55 · see citing works
The Platonic Representation Hypothesis
Huh, M., Cheung, B., Wang, T., & Isola, P. (2024) · arXiv · doi:10.48550/arxiv.2405.07987 · Cited by 35 · see citing works
“Neural networks also show substantial alignment with biological representations in the brain (Yamins et al., 2014).”
Residual-based Language Models are Free Boosters for Biomedical Imaging Tasks
Lai, Z., Wu, J., Chen, S., Zhou, Y., & Hovakimyan, N. (2024) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw63382.2024.00515 · Cited by 34 · see citing works
Mañas, O., Rodríguez, P., Ahmadi, S., Nematzadeh, A., Goyal, Y., & Agrawal, A. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.185 · Cited by 29 · see citing works
eP-ALM: Efficient Perceptual Augmentation of Language Models
Shukor, M., Dancette, C., & Cord, M. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.02016 · Cited by 16 · see citing works
Situational Awareness Matters in 3D Vision Language Reasoning
Man, Y., Gui, L.-Y., & Wang, Y.-X. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01298 · Cited by 17 · see citing works
Using Large Language Models to Detect Self-Regulated Learning in Think-Aloud ProtocolsPreprint
Zhang, J., Borchers, C., Aleven, V., & Baker, R. S. (2024) · doi:10.35542/osf.io/hrtz6 · Cited by 16 · see citing works
A Vision Check-up for Language Models
Sharma, P., Shaham, T. R., Baradad, M., Rodriíuez-Muñoz, A., Duggal, S., Isola, P., Torralba, A., & Fu, S. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.01366 · Cited by 15 · see citing works
Multimodal Neurons in Pretrained Text-Only Transformers
Schwettmann, S., Chowdhury, N., Klein, S., Bau, D., & Torralba, A. B. (2023) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw60793.2023.00308 · Cited by 13 · see citing works
Pragmatics in Language Grounding: Phenomena, Tasks, and Modeling Approaches
Fried, D., Tomlin, N., Hu, J. C. W., Patel, R., & Nematzadeh, A. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.840 · Cited by 20 · see citing works
Object Recognition as Next Token Prediction
Yue, K., Chen, B.-C., Geiping, J., Li, H., Goldstein, T., & Lim, S.-N. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01575 · Cited by 10 · see citing works
LLMDiff: Diffusion Model Using Frozen LLM Transformers for Precipitation Nowcasting
She, L., Zhang, C., Man, X., & Shao, J. (2024) · Sensors · doi:10.3390/s24186049 · Cited by 9 · see citing works
Generating Meaning: Active Inference and the Scope and Limits of Passive AI
Pezzulo, G., Parr, T., Cisek, P., Clark, A., & Friston, K. (2023) · doi:10.31234/osf.io/8xgzv · Cited by 8 · see citing works
EXCALIBUR: Encouraging and Evaluating Embodied Exploration
Zhu, H., Kapoor, R., Min, S. Y., Han, W., Li, J., Geng, K., Neubig, G., Bisk, Y., Kembhavi, A., & Weihs, L. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01434 · Cited by 9 · see citing works
VIXEN: Visual Text Comparison Network for Image Difference Captioning
Black, A., Shi, J., Fan, Y., Bui, T., & Collomosse, J. P. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i2.27843 · Cited by 9 · see citing works
Do Vision and Language Encoders Represent the World Similarly?
Maniparambil, M., Akshulakov, R., Dahou Djilali, Y. A., Seddik, M. E. A., Narayan, S., Mangalam, K., & O'Connor, N. E. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.01359 · Cited by 7 · see citing works
Do Vision and Language Models Share Concepts? A Vector Space Alignment Study
Li, J., Kementchedjhieva, Y., Fierro, C., & Søgaard, A. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00698 · Cited by 6 · see citing works
Querying as Prompt: Parameter-Efficient Learning for Multimodal Language Model
Liang, T., Huang, J., Kong, M. C., Chen, L., & Zhu, Q. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02536 · Cited by 6 · see citing works
Ma, X., Xie, H., & Qin, S. J. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103419 · Cited by 6 · see citing works
Text2Concept: Concept Activation Vectors Directly from Text
Moayeri, M., Rezaei, K., Sanjabi, M., & Feizi, S. (2023) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw59228.2023.00386 · Cited by 5 · see citing works
Dumb Meaning: Machine Learning and Artificial Semantics
Bajohr, H. (2024) · Digitale Gesellschaft · doi:10.1515/9783839469224-003 · Cited by 12 · see citing works
Shades of zero: Distinguishing impossibility from inconceivability
Hu, J., Sosa, F., & Ullman, T. D. (2025) · Journal of Memory and Language · doi:10.1016/j.jml.2025.104640 · Cited by 5 · see citing works
Hakimov, S., & Schlangen, D. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.894 · Cited by 3 · see citing works
Artificial Intelligence - Intelligent Art?
Voigts, E., Auer, R. M., Elflein, D., Kunas, S., Röhnert, J., & Seelinger, C. (2024) · transcript Verlag eBooks · doi:10.1515/9783839469224 · Cited by 3 · see citing works
Artificial Intelligence - Intelligent Art?
Voigts, E., Auer, R. M., Elflein, D., Kunas, S., Röhnert, J., & Seelinger, C. (2024) · Digitale Gesellschaft · doi:10.14361/9783839469224 · Cited by 3 · see citing works
Shipard, J., Wiliem, A., Thanh, K. N., Xiang, W., & Fookes, C. (2025) · Pattern Recognition · doi:10.1016/j.patcog.2025.112323 · Cited by 4 · see citing works
Unifying Modern AI with Robotics: Survey on MDPs with Diffusion and Foundation ModelsPreprint
Zhang, Z., Chen, R., Yang, Z., Xie, S., Chen, H., & Xiong, H. (2025) · doi:10.36227/techrxiv.174494994.46313754/v1 · Cited by 1 · see citing works
Back into Plato's Cave: Examining Cross-modal Representational Convergence at Scale
Koepke, A. S., Zverev, D., Ginosar, S., & Efros, A. A. (2026) · arXiv · doi:10.48550/arxiv.2604.18572
“Merullo et al. [67] showed that a simple learned linear transformation could map between frozen vision encoders and LLMs. Moschella et al. [70] use similarities to an anchor set.”
From Drop-Off to Recovery: A Mechanistic Analysis of Segmentation in MLLMs
Wu, B., Kim, S., & Akata, Z. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37314-4_18
- Authors
- Merullo, J., Castricato, L., Eickhoff, C., & Pavlick, E.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2022
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2209.15162
- OpenAlex
- W4300978570
- Cited by
- 34
- References
- 0
- Citations by year
- 2026: 1 · 2025: 4 · 2024: 18 · 2023: 11
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- image captioning, visual question answering, linear projection, frozen language models, vision encoder, CLIP, BEiT, cross-modal representation learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
