The DOI matches its DataCite record; 69 of the 69 references it cites were found in a registry or an index.
Language Is Not All You Need: Aligning Perception with Language Models
Open accessAbstract
A big convergence of language, multimodal perception, action, and world modeling is a key step toward artificial general intelligence. In this work, we introduce Kosmos-1, a Multimodal Large Language Model (MLLM) that can perceive general modalities, learn in context (i.e., few-shot), and follow instructions (i.e., zero-shot). Specifically, we train Kosmos-1 from scratch on web-scale multimodal corpora, including arbitrarily interleaved text and images, image-caption pairs, and text data. We evaluate various settings, including zero-shot, few-shot, and multimodal chain-of-thought prompting, on a wide range of tasks without any gradient updates or finetuning. Experimental results show that Kosmos-1 achieves impressive performance on (i) language understanding, generation, and even OCR-free NLP (directly fed with document images), (ii) perception-language tasks, including multimodal dialogue, image captioning, visual question answering, and (iii) vision tasks, such as image recognition with descriptions (specifying classification via text instructions). We also show that MLLMs can benefit from cross-modal transfer, i.e., transfer knowledge from language to multimodal, and from multimodal to language. In addition, we introduce a dataset of Raven IQ test, which diagnoses the nonverbal reasoning capability of MLLMs.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Furthermore, developing more efficient transformer architectures tailored for multimodal data, such as Kosmos-1 [162], Muse [163], and PixArt-α [164], presents a viable solution.”
From Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook. (2024) · cites this paper as [162]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Huang et al. [77] introduced KOSMOS-1, which is capable of understanding general patterns, following instructions, and learning based on context.”
From A Survey on Evaluation of Large Language Models (2024) · cites this paper as [77]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · cited 3×
“6 Zero-Shot Image Classification We report the zero-shot image classification performance on ImageNet [DDS09].”
BackgroundMethodMicrosoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460 · cited 2×
“Image Captioning Table 2a shows the captioning performance on COCO [39] Karpathy test split and Flickr30k [40] test set.”
BackgroundLanguage Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515 · cited 1×
“We preprocess the data in the “full-sentence” format [LOG19], which packs each input sequence with full sentences that are sampled continuously from one or more documents.”
MethodLearning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293 · cited 3×
“In recent years, vision-language learning and representation models has garnered significant attention [2, 3, 4, 34, 50, 51, 52, 53, 54].”
BackgroundMethodRecursive Deep Models for Semantic Compositionality Over a Sentiment Treebank
Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A., & Potts, C. (2013) · Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d13-1170 · Cited by 6,878
CIDEr: Consensus-based image description evaluation
Vedantam, R., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7299087 · Cited by 4,845 · cited 3×
“Specifically, our model achieves a CIDEr score of 67.1 on the Flickr30k dataset, compared to 60.6 and 61.5 for the Flamingo-3B and Flamingo-9B models, respectively.”
MethodInfluentialChain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505 · cited 2×
“LLMs also serve as basic reasoners [10], which is complementary to perception modules on complex tasks.”
BackgroundKudo, T., & Richardson, J. (2018) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-2012 · Cited by 4,454 · cited 2×
“We adopt a learning rate warm-up and decay schedule, and use SentencePiece for tokenization.”
MethodChain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263
Research works printed but not linked: 5
- Elia Bruni, Gemma Boleda, Marco Baroni, and Nam Khanh Tran. Distributional semantics in technicolor. In ACL, 2012.
- Patricia A Carpenter, Marcel A Just, and Peter Shell. What one intelligence test measures: a theoretical account of the processing in the raven progressive matrices test. Psychological review, 97(3):404, 1990.
- Hector Levesque, Ernest Davis, and Leora Morgenstern. The winograd schema chal- lenge. In Thirteenth International Conference on the Principles of Knowledge Repre- sentation and Reasoning, 2012.
- Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S. Gordon. Choice of plau- sible alternatives: An evaluation of commonsense causal reasoning. In AAAI Spring Symposium, 2011.
- Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. WinoGrande: An adversarial winograd schema challenge at scale. In AAAI, pages 8732-8740, 2020.
Cited by
Sorted by citations
A Survey on Evaluation of Large Language Models
Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., et al. (2024) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3641289 · Cited by 4,254
Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B., et al. (2025) · ACM Transactions on Information Systems · doi:10.1145/3703155 · Cited by 4,688
The rise and potential of large language model based agents: a survey
Xi, Z., Chen, W., Guo, X., He, W., Ding, Y., Hong, B., Zhang, M., Wang, J., Jin, S., Zhou, E., et al. (2025) · Science China Information Sciences · doi:10.1007/s11432-024-4222-0 · Cited by 970
“The second approach to endowing an LLM-based agent with visual understanding capabilities is modality fusion, where a visual encoder is integrated with an LLM at the embedding level [209,210].”
Self-citationVideo-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding
Zhang, H., Li, X. S., & Bing, L. (2023) · Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP) · doi:10.18653/v1/2023.emnlp-demo.49 · Cited by 744
A Survey on Large Language Models: Applications, Challenges, Limitations, and Practical UsagePreprint
Hadi, M. U., Al-Tashi, Q., Qureshi, R., Shah, A., Muneer, A., Irfan, M., Zafar, A., Shaikh, M. B., Akhtar, N., Hassan, S. Z., et al. (2023) · doi:10.36227/techrxiv.23589741.v1 · Cited by 475
A review of deep learning techniques for speech processing
Mehrish, A., Majumder, N., Bharadwaj, R., Mihalcea, R., & Poria, S. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101869 · Cited by 389
Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook.
AlSaad, R., Abd-alrazaq, A., Boughorbel, S., Ahmed, A., Renault, M.-A., Damseh, R., & Sheikh, J. (2024) · Journal of medical Internet research · doi:10.2196/59505 · Cited by 349
“Furthermore, developing more efficient transformer architectures tailored for multimodal data, such as Kosmos-1 [162], Muse [163], and PixArt-α [164], presents a viable solution.”
RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control
Brohan, A., Brown, N., Carbajal, J., Chebotar, Y., Chen, X., Choromanski, K., Ding, T., Driess, D., Dubey, A., Finn, C., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.15818 · Cited by 276
“Finally, we describe how to make these models practical for robot tasks, addressing challenges with model size and inference speed to enable real-time control.”
BackgroundMethodThe Rise and Potential of Large Language Model Based Agents: A Survey
Xi, Z., Chen, W., Guo, X., He, W., Ding, Y., Hong, B., Zhang, M., Wang, J., Jin, S., Zhou, E., et al. (2023) · arXiv · doi:10.48550/arxiv.2309.07864 · Cited by 972
Self-citationMVBench: A Comprehensive Multi-modal Video Understanding Benchmark
Li, K., Wang, Y., He, Y., Li, Y., Wang, Y., Liu, Y., Wang, Z., Xu, J., Chen, G., Lou, P., et al. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.02095 · Cited by 300
“In the past few years, Multi-modal Large Language Models (MLLMs) [1, 16, 26, 39, 42, 47, 59, 104] have gradually driven the advance in vision-language learning, by plugging visual encoders within various pretrained LLMs [10, 15, 58, Figure 1.”
BackgroundMultimodal Large Language Models: A Survey
Wu, J., Gan, W., Chen, Z., Wan, S., & Yu, P. S. (2023) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata59044.2023.10386743 · Cited by 369
“In 2023, KOSMOS-1 was released by Microsoft [25].”
MethodLarge AI Models in Health Informatics: Applications, Challenges, and the Future
Qiu, J., Li, L., Sun, J., Peng, J., Shi, P., Zhang, R., Dong, Y., Lam, K., Lo, F. P.-W., Xiao, B. J., et al. (2023) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2023.3316750 · Cited by 267
“Forrepresentationlearning,thefeaturesfrom differentmodalitiesarethenalignedthroughcontrastivelearn-ing[54],[55],[56]orfusedintoaunifiedrepresentationthrough another encoder on the top of all extracted features [58], [59], [60], [61], [62].”
BackgroundMM-LLMs: Recent Advances in MultiModal Large Language Models
Zhang, D., Yu, Y., Dong, J., Li, C., Su, D., Chu, C., & Yu, D. (2024) · Findings of the Association for Computational Linguistics ACL 2024 · doi:10.18653/v1/2024.findings-acl.738 · Cited by 260
ViperGPT: Visual Inference via Python Execution for Reasoning
Surís, D., Menon, S., & Vondrick, C. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01092 · Cited by 273
“Due to the practical difficulty of using these methods, the field has primarily moved towards end-to-end all-in-one models [1, 22, 23, 30].”
MethodSpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities
Zhang, D., Li, S., Zhang, X., Zhan, J., Wang, P., Zhou, Y., & Qiu, X. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.1055 · Cited by 549
mPLUG-OwI2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration
Ye, Q., Xu, H., Ye, J., Yan, M., Hu, A., Liu, H., Qian, Q., Zhang, J., & Huang, F. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01239 · Cited by 431
“Instruction Tuning with MLLMs.”
Zhang, W., Cai, M., Zhang, T., Zhuang, Y., & Mao, X. (2024) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2024.3409624 · Cited by 265
“Models like VisualGPT [4], BLIP [5], Flamingo [7], and Kosmos-1 [26] showcase strong multimodal reasoning potential after aligning LLMs with image modalities.”
MethodGLaMM: Pixel Grounding Large Multimodal Model
Rasheed, H., Maaz, M., Shaji, S., Shaker, A., Khan, S., Cholakkal, H., Anwer, R. M., Xing, E. P., Yang, M., & Khan, F. S. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01236 · Cited by 302
Vision-Language Models in Remote Sensing: Current progress and future trends
Xiang, L., Wen, C., Hu, Y., Yuan, Z., & Zhu, X. X. (2024) · IEEE Geoscience and Remote Sensing Magazine · doi:10.1109/mgrs.2024.3383473 · Cited by 252
“In particular, to deal with the limited training data, an auxiliary decoder, trained for multilabel scene classification, has been used to assist the encoder in the training process by leveraging its conceptual similarity to image captioning and its ability to highlight semantic classes.”
MethodLet’s Have a Chat! A Conversation with ChatGPT: Technology, Applications, and Limitations
Shahriar, S., & Hayawi, K. (2023) · Artificial Intelligence and Applications · doi:10.47852/bonviewaia3202939 · Cited by 167
Let's have a chat! A Conversation with ChatGPT: Technology, Applications, and Limitations
Shahriar, S., & Hayawi, K. (2023) · arXiv · doi:10.48550/arxiv.2302.13817 · Cited by 167
Evaluating Large Language Models in Class-Level Code Generation
Du, X., Liu, M., Wang, K., Wang, H., Jun-wei, L., Chen, Y., Feng, J., Sha, C., Peng, X., & Lou, Y. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3597503.3639219 · Cited by 271
Reasoning with Language Model Prompting: A Survey
Qiao, S., Ou, Y., Zhang, N., Chen, X., Yao, Y., Deng, S., Tan, C., Huang, F., & Chen, H. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.294 · Cited by 346
Kuang, W., Qian, B., Li, Z., Chen, D., Gao, D., Pan, X., Xie, Y., Li, Y., Ding, B., & Zhou, J. (2024) · Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining · doi:10.1145/3637528.3671573 · Cited by 142
On the Opportunities and Challenges of Foundation Models for GeoAI (Vision Paper)
Mai, G., Huang, W., Sun, J., Song, S., Mishra, D. R., Liu, N., Gao, S., Liu, T., Cong, G., Hu, Y., et al. (2024) · ACM Transactions on Spatial Algorithms and Systems · doi:10.1145/3653070 · Cited by 384
“Furthermore, we think the urban perception task, as a classic task in urban geography, is more challenging than current visual question-answering tasks commonly used in VLFM research [64, 150] partly due to their partially subjective nature and the rarity of annotated datasets.”
BackgroundMethodInfluentialWang, D., Feng, L., Ye, J., Zou, J., & Zheng, Y. (2023) · MedComm – Future Medicine · doi:10.1002/mef2.43 · Cited by 127
Liu, F., Emerson, G., & Collier, N. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00566 · Cited by 212
Foundation models in smart agriculture: Basics, opportunities, and challenges
Li, J., Xu, M., Xiang, L., Chen, D., Zhuang, W., Yin, X., & Li, Z. (2024) · Computers and Electronics in Agriculture · doi:10.1016/j.compag.2024.109032 · Cited by 159
Yu, T., Yao, Y., Zhang, H., He, T., Han, Y., Cui, G., Hu, J., Liu, Z., Zheng, H.-T., & Sun, M. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01310 · Cited by 267
“Models are pretrained on large-scale image-text pairs [4, 11, 19, 42, 47] or interleaved data [2, 3, 16] to learn the semantic mapping between visual and text signals.”
BackgroundHadi, M. U., Al-Tashi, Q., Qureshi, R., Shah, A., Muneer, A., Irfan, M., Zafar, A., Shaikh, M. B., Akhtar, N., Hassan, S. Z., et al. (2023) · doi:10.36227/techrxiv.23589741.v4 · Cited by 193
Generative Multimodal Models are In-Context Learners
Sun, Q., Cui, Y., Zhang, X., Zhang, F., Yu, Q., Wang, Y., Rao, Y., Liu, J., Huang, T., & Wang, X. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.01365 · Cited by 147
“Recent years have witnessed the rapid growth of large multimodal models [5, 19, 31, 71].”
BackgroundIshmam, M. F., Shovon, M. S. H., Mridha, M. F., & Dey, N. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102270 · Cited by 107
Preventing Zero-Shot Transfer Degradation in Continual Learning of Vision-Language Models
Zheng, Z., Ma, M., Wang, K., Qin, Z., Yue, X., & You, Y. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01752 · Cited by 122
“The deep learning community tends to build large models with a huge dataset [4, 8], including vision-language models [54, 24].”
BackgroundAlpha-CLIP: A CLIP Model Focusing on Wherever you Want
Sun, Z., Fang, Y., Wu, T., Zhang, P., Zang, Y., Kong, S., Xiong, Y., Lin, D., & Wang, J. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01237 · Cited by 151
“By using this pseudo-labeling base-line, Kosmos-2 releases the GRIT dataset and equips multi-modal model [15] with local perception capabilities.”
A Survey of Generative AI Applications
Gozalo-Brizuela, R., & Merchan, E. E. G. (2024) · Journal of Computer Science · doi:10.3844/jcssp.2024.801.818 · Cited by 101
GIT-Mol: A multi-modal large language model for molecular science with graph, image, and text
Liu, P., Ren, Y., Tao, J., & Ren, Z. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.108073 · Cited by 114
“• Modality-Adaptive Learning models like BEIT-3 [7] and KOSMOS-1 [22] use the Mixture of Experts (MoE) [23] and the MAGNETO [24] Transformer.”
MethodWhen brain-inspired AI meets AGI
Zhao, L., Zhang, L., Wu, Z., Chen, Y., Dai, H., Yu, X., Liu, Z., Zhang, T., Hu, X., Jiang, X., et al. (2023) · Meta-Radiology · doi:10.1016/j.metrad.2023.100005 · Cited by 116
The Revolution of Multimodal Large Language Models: A Survey
Caffagni, D., Cocchi, F., Barsellotti, L., Moratelli, N., Sarto, S., Baraldi, L., Baraldi, L., Cornia, M., & Cucchiara, R. (2024) · Findings of the Association for Computational Linguistics ACL 2024 · doi:10.18653/v1/2024.findings-acl.807 · Cited by 98
Hadi, M. U., Al-Tashi, Q., Qureshi, R., Shah, A., Muneer, A., Irfan, M., Zafar, A., Shaikh, M. B., Akhtar, N., Wu, J., et al. (2023) · doi:10.36227/techrxiv.23589741 · Cited by 111
Hallucination Augmented Contrastive Learning for Multimodal Large Language Model
Jiang, C., Xu, H., Dong, M., Chen, J., Ye, W., Yan, M., Ye, Q., Zhang, J., Huang, F., & Zhang, S. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02553 · Cited by 140
Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Action
Lu, J., Clark, C., Lee, S.-H., Zhang, Z., Khosla, S., Marten, R., Hoiem, D., & Kembhavi, A. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02497 · Cited by 195
MMA: Multi-Modal Adapter for Vision-Language Models
Yang, L., Zhang, R., Wang, Y., & Xie, X. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02249 · Cited by 108
“Vision-Language Models (VLMs) [1, 28, 30, 50, 68, 70, 71, 74] have recently been introduced as a class of foundation models.”
BackgroundMethodGenerating meaning: active inference and the scope and limits of passive AI
Pezzulo, G., Parr, T., Cisek, P., Clark, A., & Friston, K. (2024) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2023.10.002 · Cited by 80
“The lack of anchoring on embodied reality motivates novel foundation models for embodied intelligence that include multiple modalities [115] iv or that mimic the visual cortex, rather than starting from language [116].”
ChatGPT: Systematic Review, Applications, and Agenda for Multidisciplinary Research
Singh, H., & Singh, A. (2023) · Journal of Chinese Economic and Business Studies · doi:10.1080/14765284.2023.2210482 · Cited by 82
“The former is mainly used for recognizing the context in classification tasks and language (Huang et al. 2023).”
BackgroundYe, J., Hu, A., Xu, H., Ye, Q., Yan, M., Xu, G., Li, C., Tian, J., Qian, Q., Zhang, J., et al. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.187 · Cited by 97
Retrieving Multimodal Information for Augmented Generation: A Survey
Zhao, R., Chen, H., Wang, W., Jiao, F., Long, X., Qin, C., Ding, B., Guo, X., Li, M., Li, X., et al. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.314 · Cited by 75
Self-citationInstructDiffusion: A Generalist Modeling Interface for Vision Tasks
Geng, Z., Yang, B., Hang, T., Li, C., Gu, S., Zhang, T., Bao, J., Zhang, Z., Li, H., Hu, H., et al. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01208 · Cited by 133
“On the other hand, the success of large language models such as GPT series [7, 53, 56, 57], PaLM [4, 12], and LLaMA [68], has been attracting a lot of research interest [24, 38, 64, 69, 73, 74, 80] in augmenting the large language models with visual capabilities.”
BackgroundGPT-4V(ision) is a Human-Aligned Evaluator for Text-to-3D Generation
Wu, T., Yang, G., Li, Z., Zhang, K., Liu, Z., Guibas, L. J., Lin, D., & Wetzstein, G. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02098 · Cited by 111
“Initial efforts of LMM involve combining computer vision with LLMs by fine-tuning visual encoders to align with language embed-dings [2, 4, 17, 27, 33, 34, 64] or converting visual information to text [26, 58, 66, 71].”
MethodMM1: Methods, Analysis and Insights from Multimodal LLM Pre-training
McKinzie, B., Gan, Z., Fauconnier, J.-P., Dodge, S., Zhang, B., Dufter, P., Shah, D., Du, X., Peng, F., Belyi, A., et al. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73397-0_18 · Cited by 309
Hadi, M. U., Al-Tashi, Q., Qureshi, R., Shah, A., Muneer, A., Irfan, M., Zafar, A., Shaikh, M. B., Akhtar, N., Hassan, S. Z., et al. (2023) · doi:10.36227/techrxiv.23589741.v3 · Cited by 71
The Limitations of Large Language Models for Understanding Human Language and Cognition
Cuskley, C. F., Woods, R. J. H., & Flaherty, M. (2024) · Open Mind · doi:10.1162/opmi_a_00160 · Cited by 88
“…multimodal models that can deal not only with text, but also with data from other modalities, including images and video (Bubeck et al., 2023; Huang, Dong, et al., 2023), and all manner of other modalities that even humans generally cannot readily interpret (e.g., thermal data; Girdhar et…”
BackgroundMethodInstruct-ReID: A Multi-Purpose Person Re-Identification Task with Instructions
He, W., Deng, Y., Tang, S., Chen, Q., Xie, Q., Wang, Y., Bai, L., Zhu, F., Zhao, R., Ouyang, W., et al. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01659 · Cited by 75
“† denotes that the test mode is VIS-to-IR and ‡ denotes IR-to-VIS mode on LLCM. 14.9 63.9 [1] 65.8 [17] 62.9 [17] 40.8 [3] 45.9 [5] 71.8 [4] 93.0 [19] 71.8 [19] 77.7 [11] DeiT [12”
InfluentialUrban attractiveness according to ChatGPT: Contrasting AI and human insights
Malekzadeh, M., Willberg, E. S., Torkko, J., & Toivonen, T. (2024) · Computers Environment and Urban Systems · doi:10.1016/j.compenvurbsys.2024.102243 · Cited by 63
Ghosh, A., Acharya, A., Saha, S., Jain, V., & Chadha, A. R. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4783140 · Cited by 169
On Scaling Up a Multilingual Vision and Language Model
Chen, X., Djolonga, J., Padlewski, P., Mustafa, B., Changpinyo, S., Wu, J., Riquelme, C., Goodman, S., Wang, X., Tay, Y., et al. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01368 · Cited by 337
“[Right] The Pareto frontier between few-shot and fine-tuned performance, comparing PaLI-X with PaLI [5], Flamingo [10], and Kosmos-1 [11]. form the visual input (See Appendix A).”
RegionGPT: Towards Region Understanding Vision Language Model
Guo, Q., Mello, S. D., Yin, H., Byeon, W., Cheung, K. C., Yu, Y., Luo, P., & Liu, S. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01309 · Cited by 74
“Some other interesting works have made remarkable progress by extending LLM to audio [7, 20], medical VQA [31, 57] and control systems [16, 32].”
BackgroundHadi, M. U., Al-Tashi, Q., Qureshi, R., Shah, A., Muneer, A., Irfan, M., Zafar, A., Shaikh, M. B., Akhtar, N., Hassan, S. Z., et al. (2023) · doi:10.36227/techrxiv.23589741.v2 · Cited by 54
Generative AI and Its Educational Implications
Łodzikowski, K., Foltz, P. W., & Behrens, J. T. (2024) · Postdigital Science and Education · doi:10.1007/978-3-031-64487-0_2 · Cited by 45
Zhang, Z., Yao, Y., Zhang, A., Tang, X., Ma, X., He, Z., Wang, Y., Gerstein, M., Wang, R., Liu, G., et al. (2025) · ACM Computing Surveys · doi:10.1145/3719341 · Cited by 51
“More recent studies have also explored modeling the inputs of different modalities into the same vector space, thus resulting in any-to-any representation learning [46, 88, 160] and interleaved multimodal representation learning [57, 190].”
InfluentialHuang, X., Liu, Z. Y., Zhang, M., 桂, 韬., 韩, 先., 户, 保., 黄, 民., 黄, 萱., 刘, 康., 刘, 挺., et al. (2023) · Scientia Sinica Informationis · doi:10.1360/ssi-2023-0113 · Cited by 42
KAM-CoT: Knowledge Augmented Multimodal Chain-of-Thoughts Reasoning
Mondal, D., Modi, S., Panda, S., Singh, R., & Rao, G. S. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i17.29844 · Cited by 67
neuroGPT-X: toward a clinic-ready large language model
Guo, E., Gupta, M., Sinha, S., Rössler, K., Tatagiba, M. S., Akagami, R., Al‐Mefty, O., Sugiyama, T., Stieg, P. E., Pickett, G. E., et al. (2023) · Journal of neurosurgery · doi:10.3171/2023.7.jns23573 · Cited by 50
Breaking Common Sense: WHOOPS! A Vision-and-Language Benchmark of Synthetic and Compositional Images
Bitton-Guetta, N., Bitton, Y., Hessel, J., Schmidt, L., Elovici, Y., Stanovsky, G., & Schwartz, R. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00247 · Cited by 37
UPRISE: Universal Prompt Retrieval for Improving Zero-Shot Evaluation
Cheng, D., Huang, S., Bi, J., Zhan, Y., Liu, J., Wang, Y., Sun, H., Wei, F., Deng, W., & Zhang, Q. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.758 · Cited by 32
Self-citationModel-Driven Prompt Engineering
Clarisó, R., & Cabot, J. (2023) · ACM/IEEE International Conference on Model Driven Engineering Languages and Systems (MODELS) · doi:10.1109/models58315.2023.00020 · Cited by 41
SAM4MLLM: Enhance Multi-Modal Large Language Model for Referring Expression Segmentation
Chen, Y.-C., Li, W., Sun, C., Wang, Y.-C. F., & Chen, C. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73004-7_19 · Cited by 50
Faithful AI in Medicine: A Systematic Review with Large Language Models and Beyond
Xie, Q., Schenck, E. J., Yang, H. S., Chen, Y., Peng, Y., & Wang, F. (2023) · doi:10.21203/rs.3.rs-3661764/v1 · Cited by 57
VCoder: Versatile Vision Encoders for Multimodal Large Language Models
Jain, J., Yang, J., & Shi, H. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02644 · Cited by 53
“Among various open-source MLLMs [5, 23, 33, 34, 73], we chose LLaVA-1.”
BackgroundMethodGenerative language reconstruction from brain recordings
Ye, Z., Ai, Q., Liu, Y., de Rijke, M., Zhang, M., Lioma, C., & Ruotsalo, T. (2025) · Communications Biology · doi:10.1038/s42003-025-07731-7 · Cited by 24
CapsFusion: Rethinking Image-Text Data at Scale
Yu, Q., Sun, Q., Zhang, X., Cui, Y., Zhang, F., Cao, Y. Q., Wang, X., & Liu, J. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01330 · Cited by 58
“Avoid simply concatenating the sentences.”
Generative AI: Overview, Economic Impact, and Applications in Asset ManagementPreprint
Luk, M. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4574814 · Cited by 24
Jaeger, J. (2024) · Neurons Behavior Data analysis and Theory · doi:10.51628/001c.94404 · Cited by 30
“One commonspeculation is that the extremely high-dimensional nature of the weightmatrix (and thus the config-uration space) of an LLM allowsfor the emergence of “higher-level” abilities, such as the detectionof meaning without external referents (see, for example, Piantadosi and Hill (2022)).”
BackgroundKaul, P., Li, Z., Yang, H., Dukler, Y., Swaminathan, A. D., Taylor, C. J., & Soatto, S. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02571 · Cited by 37
“[1, 6, 7, 17, 24, 25, 30, 36, 44, 48, 54] LVLMs take input text and images and generate text responses to enable multi-modal perception and com* prehension.”
Urrutia, F., & Araya, R. (2023) · Journal of Educational Computing Research · doi:10.1177/07356331231191174 · Cited by 24
Pretrained Language Models as Visual Planners for Human Assistance
Patel, D., Eghbal-zadeh, H., Kamra, N., Iuzzolino, M. L., Jain, U., & Desai, R. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01404 · Cited by 22
“For instance, [47, 59, 2, 48, 29, 18] train large transformer based multi-modal sequence prediction models.”
BackgroundA Simple Recipe for Contrastively Pre-Training Video-First Encoders Beyond 16 Frames
Papalampidi, P., Koppula, S., Pathak, S., Chiu, J., Heyward, J., Pătrăucean, V., Shen, J., Miech, A., Zisserman, A., & Nematzdeh, A. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01364 · Cited by 29
“Similar work at smaller scale has additionally included captioning losses [32, 76], more contrastive losses [10, 38, 43, 66], masking/masked autoen-coding [15, 16, 18, 19, 33, 35, 40, 55], and combinations thereof [13, 23, 54, 58, 61, 72, 73, 75, 82].”
BackgroundFashion-GPT: Integrating LLMs with Fashion Retrieval System
Chen, Q., Zhang, T., Nie, M., Wang, Z., Xu, S., Shi, W., & Cao, Z. (2023) · doi:10.1145/3607827.3616844 · Cited by 16
Potential merits and flaws of large language models in epilepsy care: A critical review
Diessen, E. G. A. L. V., van Amerongen, R. A., Zijlmans, M., & Otte, W. M. (2024) · Epilepsia · doi:10.1111/epi.17907 · Cited by 23
“26”
Power-Llava: Large Language and Vision Assistant for Power Transmission Line Inspection
Wang, J., Li, M., Luo, H., Zhu, J., Yang, A., Rong, M., & Wang, X. (2024) · Proceedings - International Conference on Image Processing · doi:10.1109/icip51287.2024.10648271 · Cited by 22
“Targeting at feature alignment in the first stage, we collect 608K brief image-text pairs, consisting of 558K from the public dataset released by LLaVA [12] and 50K generated by VL-models [6, 9, 12]functioning as captioners.”
MethodMiniMedGPT: Efficient Large Vision–Language Model for medical Visual Question Answering
Alsabbagh, A. R., Mansour, T., Al-Kharabsheh, M., Ebdah, A. S., Al-Emaryeen, R., Al-Nahhas, S., Mahafza, W. S., & Al-Kadi, O. S. (2025) · Pattern Recognition Letters · doi:10.1016/j.patrec.2025.01.001 · Cited by 17
A Survey of Reasoning with Foundation ModelsPreprint
Sun, J., Zheng, C., Xie, E., Liu, Z., Chu, R., Liu, J., Xu, J., Ding, M., Li, H., Geng, M., et al. (2023) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/ac4sp · Cited by 143
Self-citationLow-Rank Approximation for Sparse Attention in Multi-Modal LLMs
Lin, S., Chen, Y., Yang, S., Ding, X., Ge, Y., Chen, Y.-C., & Shan, Y. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01306 · Cited by 20
“Additionally, in the multi-modality domain, frameworks like BLIP-2 [26], LLaVA [30, 31], KOSMOS-1 [18], and PaLM-E [10], have advanced multi-modal comprehension by effectively fine-tuning foundational models such as LLaMA [42] and Vi-cuna [7].”
VideoGrounding-DINO: Towards Open-Vocabulary Spatio- Temporal Video Grounding
Wasim, S. T., Naseer, M., Khan, S. A., Yang, M., & Khan, F. S. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01789 · Cited by 20
“We take inspiration from recent foundation models for spatial grounding [4, 8, 10, 12, 14, 29].”
BackgroundMethodInfluentialVisual Instruction Tuning with Polite Flamingo
Chen, D., Liu, J., Dai, W., & Wang, B. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i16.29727 · Cited by 27
Panagopoulou, A., Xue, L., Yu, N., Li, J., Li, D., Joty, S., Xu, R., Savarese, S., Xiong, C., & Niebles, J. C. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72995-9_11 · Cited by 92
Self-citationSignatures of cross-modal alignment in children's early conceptsPreprint
Aho, K., Roads, B. D., & Love, B. C. (2023) · doi:10.31234/osf.io/v6fnq · Cited by 12
ChatNT: A Multimodal Conversational Agent for DNA, RNA and Protein TasksPreprint
Richard, G., de Almeida, B. P., Dalla-Torre, H., Blum, C. F., Hexemer, L., Pandey, P., Laurent, S., Lopez, M., Laterre, A., Lang, M., et al. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.04.30.591835 · Cited by 14
Large Search Model: Redefining Search Stack in the Era of LLMs
Wang, L., Yang, N., Huang, X., Yang, L., Majumder, R., & Wei, F. (2023) · ACM SIGIR Forum · doi:10.1145/3642979.3643006 · Cited by 19
“Techniques like multi-query attention [Shazeer, 2019] and grouped-query attention [Ainslie et al., 2023] are commonly employed to reduce the inference memory footprint at the cost of model quality.”
BackgroundMethodSelf-citationKhanmohammadi, S., Golafshani, E. M., Bai, Y., Li, H., Bazli, M., & Arashpour, M. (2023) · International Journal of Disaster Risk Reduction · doi:10.1016/j.ijdrr.2023.103972 · Cited by 13
Mani, G., & Namomsa, G. B. (2023) · IEEE AFRICON · doi:10.1109/africon55910.2023.10293675 · Cited by 12
“Recent progress (post-ChatGPT) is focused on similar areas: e.g., see offerings from Google [26] and Microsoft [27, 28].”
BackgroundPath to Medical AGI: Unify Domain-specific Medical LLMs with the Lowest CostPreprint
Zhou, J., Chen, X., & Gao, X. (2023) · medRxiv · doi:10.1101/2023.06.23.23291802 · Cited by 21
“To solve the above two challenges, a number of opensource multimodal LLMs were proposed [16], [17], [18], [19], [20], [20], [21], [22], [23], [24], [25], [26], [27], [28], [29], [30], [31], [32], [33], [34], [35], [36], [37].”
The evolution of language models: From N-Grams to LLMs, and beyond
Raeini, M. (2025) · Natural Language Processing Journal · doi:10.1016/j.nlp.2025.100168 · Cited by 26
Bouyamourn, A. Z. A. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.192 · Cited by 13
Towards More Unified In-Context Visual Understanding
Sheng, D., Chen, D., Tan, Z., Liu, Q., Chu, Q., Bao, J., Gong, T., Liu, B., Xu, S., & Yu, N. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01269 · Cited by 14
“Inspired by the success of ChatGPT as well as GPT-4 [32, 33], recent works primarily concentrate on aligning visual features with the pre-trained LLMS for multimodal comprehension tasks [18,24,26,27,44,45,53,57].”
MethodSelf-citationWang, S., Xie, B., Ding, L., Gao, X., Chen, J., & Xiang, Y. (2024) · ACM Conference on Recommender Systems (RecSys) · doi:10.1145/3640457.3688124 · Cited by 10
“SeCor leverages the advanced representing and transfer capabilities of LLMs[17, 18] to combine collaborative and linguistic information, forming efficient hybrid representations.”
Decoding the Language of Chickens - An Innovative NLP Approach to Enhance Poultry WelfarePreprint
Neethirajan, S. R. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.04.29.591707 · Cited by 10
The limitations of large language models for understanding human language and cognitionPreprint
Cuskley, C. F., Woods, R. J. H., & Flaherty, M. (2024) · doi:10.31234/osf.io/hbwrg · Cited by 10
Generating Meaning: Active Inference and the Scope and Limits of Passive AI
Pezzulo, G., Parr, T., Cisek, P., Clark, A., & Friston, K. (2023) · doi:10.31234/osf.io/8xgzv · Cited by 8
Violet: A Vision-Language Model for Arabic Image Captioning with Gemini Decoder
Mohamed, A., Alwajih, F., Nagoudi, E. M. B., Inciarte, A. A., & Abdul-Mageed, M. (2023) · Arabic Natural Language Processing Conference (ArabicNLP) · doi:10.18653/v1/2023.arabicnlp-1.1 · Cited by 14
Learning Visual Abstract Reasoning through Dual-Stream Networks
Zhao, K., Xu, C., & Si, B. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i15.29641 · Cited by 10
“Models combined with language models can exhibit zero-shot or few-shot learning capabilities in such non-verbal reasoning tasks (Huang et al. 2023), and our framework can flexibly integrate multimodal information in the future.”
Background
Metadata
- Authors
- Huang, S., Dong, L., Wang, W., Hao, Y., Singhal, S., Ma, S., Lv, T., Cui, L., Mohammed, O. K., Patra, B., Liu, Q., Aggarwal, K., Chi, Z., Bjorck, J., Chaudhary, V., Som, S., Song, X., & Wei, F.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- Pages
- 72096-72109
- ISSN
- 1049-5258 (ISSN-L)
- arXiv
- 2302.14045
- OpenAlex
- W7133185515
- Cited by
- 235
- References
- 65
- Citations by year
- 2026: 37 · 2025: 47 · 2024: 95 · 2023: 55
- FWCI
- 86.06
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- neural information processing systems (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Categorization, perception, and language
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- language models, perception, multimodal alignment, vision-language models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
