Language Is Not All You Need: Aligning Perception with Language Models
Open accessAbstract
A big convergence of language, multimodal perception, action, and world modeling is a key step toward artificial general intelligence. In this work, we introduce Kosmos-1, a Multimodal Large Language Model (MLLM) that can perceive general modalities, learn in context (i.e., few-shot), and follow instructions (i.e., zero-shot). Specifically, we train Kosmos-1 from scratch on web-scale multimodal corpora, including arbitrarily interleaved text and images, image-caption pairs, and text data. We evaluate various settings, including zero-shot, few-shot, and multimodal chain-of-thought prompting, on a wide range of tasks without any gradient updates or finetuning. Experimental results show that Kosmos-1 achieves impressive performance on (i) language understanding, generation, and even OCR-free NLP (directly fed with document images), (ii) perception-language tasks, including multimodal dialogue, image captioning, visual question answering, and (iii) vision tasks, such as image recognition with descriptions (specifying classification via text instructions). We also show that MLLMs can benefit from cross-modal transfer, i.e., transfer knowledge from language to multimodal, and from multimodal to language. In addition, we introduce a dataset of Raven IQ test, which diagnoses the nonverbal reasoning capability of MLLMs.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Developing more efficient transformer architectures tailored for multimodal data, such as Kosmos-1, Muse, and PixArt-α, presents a viable solution.
“Furthermore, developing more efficient transformer architectures tailored for multimodal data, such as Kosmos-1 [162], Muse [163], and PixArt-α [164], presents a viable solution.”
From Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook. (2024) · cites this paper as [162]
Section 2 identifies KOSMOS-1 as a Transformer-based model for multimodal input, supporting its use as an example. Sections 2.1 and 2.2 describe ways to reduce image embeddings and improve optimization, but the paper does not establish that KOSMOS-1 is a more efficient architecture or that developing such architectures is a viable solution. This verdict concerns only the Kosmos-1 citation, not Muse or PixArt-α. (LLM, conf=0.84)
Experimental — accuracy not yet measuredKOSMOS-1 is capable of understanding general patterns, following instructions, and learning based on context.
“Huang et al. [77] introduced KOSMOS-1, which is capable of understanding general patterns, following instructions, and learning based on context.”
From A Survey on Evaluation of Large Language Models (2024) · cites this paper as [77]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · cited 3× · see citing works
“6 Zero-Shot Image Classification We report the zero-shot image classification performance on ImageNet [DDS09].”
BackgroundMethodMicrosoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460 · cited 2× · see citing works
“Image Captioning Table 2a shows the captioning performance on COCO [39] Karpathy test split and Flickr30k [40] test set.”
BackgroundLanguage Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · see citing works
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515 · cited 1× · see citing works
“We preprocess the data in the “full-sentence” format [LOG19], which packs each input sequence with full sentences that are sampled continuously from one or more documents.”
MethodLearning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293 · cited 3× · see citing works
“In recent years, vision-language learning and representation models has garnered significant attention [2, 3, 4, 34, 50, 51, 52, 53, 54].”
BackgroundMethodRecursive Deep Models for Semantic Compositionality Over a Sentiment Treebank
Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A., & Potts, C. (2013) · Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d13-1170 · Cited by 6,878 · see citing works
CIDEr: Consensus-based image description evaluation
Vedantam, R., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7299087 · Cited by 4,845 · cited 3× · see citing works
“Specifically, our model achieves a CIDEr score of 67.1 on the Flickr30k dataset, compared to 60.6 and 61.5 for the Flamingo-3B and Flamingo-9B models, respectively.”
MethodInfluentialChain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505 · cited 2× · see citing works
“LLMs also serve as basic reasoners [10], which is complementary to perception modules on complex tasks.”
BackgroundKudo, T., & Richardson, J. (2018) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-2012 · Cited by 4,454 · cited 2× · see citing works
“We adopt a learning rate warm-up and decay schedule, and use SentencePiece for tokenization.”
MethodChain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263 · see citing works
InstructBLIP: Towards General-purpose Vision-Language Models with Instruction TuningPreprint
Dai, W., Li, J., Li, D., Tiong, A., Zhao, J.-Q., Wang, W., Li, B. A., Fung, P., & Hoi, S. (2023) · Neural Information Processing Systems · doi:10.48550/arxiv.2305.06500 · Cited by 4,191 · cited 1× · see citing works
“Meanwhile, some work [57, 58, 59] introduce visual instruction tuning to enhance instruction following capabilities.”
BackgroundSelf-citationIEEE Transactions on Pattern Analysis and Machine Intelligence
Fu, K.-S. (2004) · doi:10.1002/0471667196.ess1206 · Cited by 3,552 · cited 1×
“We use the test set of COCO Karpathy split [KFF17], which re-partitions the train2014 and val2014 images [LMB14] into 113,287, 5,000, and 5,000 for the training set, validation set, and test set, respectively.”
MethodGaussian Error Linear Units (GELUs)
Hendrycks, D., & Gimpel, K. (2016) · arXiv · doi:10.48550/arxiv.1606.08415 · Cited by 3,537 · see citing works
PaLM: Scaling Language Modeling with Pathways
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.02311 · Cited by 2,590 · cited 1× · see citing works
“Apart from the capabilities found in previous LLMs [1, 11], MLLMs enable new usages and possibilities.”
BackgroundYoung, P., Lai, A., Hodosh, M., & Hockenmaier, J. (2014) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00166 · Cited by 2,439 · cited 2× · see citing works
“Image Captioning Table 2a shows the captioning performance on COCO [39] Karpathy test split and Flickr30k [40] test set.”
BackgroundSPICE: Semantic Propositional Image Caption Evaluation
Anderson, P., Fernando, B., Johnson, M., & Gould, S. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46454-1_24 · Cited by 1,986 · cited 1× · see citing works
“We use COCOEvalCap4 to compute CIDEr [VLZP15] and SPICE [AFJG16] scores as the evaluation metrics.”
MethodConceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning
Sharma, P., Ding, N., Goodman, S., & Soricut, R. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1238 · Cited by 1,879 · cited 1× · see citing works
“The image-caption pairs are constructed from several datasets, including English LAION-2B [19], LAION-400M [20], COYO-700M [21], and Conceptual Captions [22, 23].”
MethodSuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems
Wang, A., Pruksachatkun, Y., Nangia, N., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. R. (2019) · arXiv · doi:10.48550/arxiv.1905.00537 · Cited by 1,051 · cited 1× · see citing works
“e, PIQA), and three datasets BoolQ, CB, and COPA from the SuperGLUE benchmark [WPN19].”
MethodDeep Visual-Semantic Alignments for Generating Image Descriptions
Karpathy, A., & Fei-Fei, L. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2598339 · Cited by 1,048 · cited 2× · see citing works
“Metric Zero-shot Few-shot Language tasks StoryCloze [26] Commonsense reasoning Accuracy ✓ ✓ HellaSwag [27] Commonsense NLI Accuracy ✓ ✓ Winograd [28] Word ambiguity Accuracy ✓ ✓ Winogrande [29] Word ambiguity Accuracy ✓ ✓ PIQA [30] Physical commonsense Accuracy ✓ ✓ BoolQ [31] Question answering Accuracy ✓ ✓ CB [32] Te…”
BackgroundThe Flan Collection: Designing Data and Methods for Effective Instruction TuningPreprint
Longpre, S., Hou, L., Vu, T., Webson, A., Chung, H. W., Tay, Y., Zhou, D., Le, Q. V., Zoph, B., Wei, J., et al. (2023) · International Conference on Machine Learning · doi:10.48550/arxiv.2301.13688 · Cited by 1,029 · see citing works
LAION-5B: An open large-scale dataset for training next generation image-text models
Schuhmann, C., Beaumont, R., Vencu, R., Gordon, C., Wightman, R., Cherti, M., Coombes, T., Katta, A., Mullis, C., Wortsman, M., et al. (2022) · arXiv · doi:10.48550/arxiv.2210.08402 · Cited by 1,022 · cited 1× · see citing works
“The image-caption pairs are constructed from several datasets, including English LAION-2B [19], LAION-400M [20], COYO-700M [21], and Conceptual Captions [22, 23].”
MethodFlamingo: a visual language model for few-shot learning
Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1723 · Cited by 966 · see citing works
Li, J., Li, D., Savarese, S., & Hoi, S. (2023) · arXiv · doi:10.48550/arxiv.2301.12597 · Cited by 908 · cited 2× · see citing works
“More importantly, unlocking multimodal input [2, 3, 4, 5, 6, 7] greatly widens the applications of language models to more high-value areas, such as multimodal machine learning, document intelligence, and robotics.”
BackgroundSelf-citationMegatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
Shoeybi, M., Patwary, M., Puri, R., LeGresley, P., Casper, J., & Catanzaro, B. (2019) · arXiv · doi:10.48550/arxiv.1909.08053 · Cited by 893 · cited 1× · see citing works
“We also include the Common Crawl snapshots (2020-50 and 2021-04) datasets, CC-Stories, and RealNews datasets [17, 18].”
MethodImageBind One Embedding Space to Bind Them All
Girdhar, R., El-Nouby, A., Liu, Z., Singh, M., Alwala, K. V., Joulin, A., & Misra, I. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01457 · Cited by 863 · cited 1× · see citing works
“In recent years, vision-language learning and representation models has garnered significant attention [2, 3, 4, 34, 50, 51, 52, 53, 54].”
BackgroundHellaSwag: Can a Machine Really Finish Your Sentence?
Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., & Choi, Y. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1472 · Cited by 830 · cited 1× · see citing works
“Metric Zero-shot Few-shot Language tasks StoryCloze [26] Commonsense reasoning Accuracy ✓ ✓ HellaSwag [27] Commonsense NLI Accuracy ✓ ✓ Winograd [28] Word ambiguity Accuracy ✓ ✓ Winogrande [29] Word ambiguity Accuracy ✓ ✓ PIQA [30] Physical commonsense Accuracy ✓ ✓ BoolQ [31] Question answering…”
BackgroundGIT: A Generative Image-to-text Transformer for Vision and LanguagePreprint
Wang, J., Yang, Z.-Y., Hu, X., Li, L., Lin, K., Gan, Z., Liu, Z.-C., Liu, C., & Wang, L.-J. (2022) · Trans. Mach. Learn. Res. · doi:10.48550/arxiv.2205.14100 · Cited by 817 · cited 2× · see citing works
“KOSMOS-1 significantly outperforms GIT [WYH22] by 4.”
MethodLiu, H., Li, C., Wu, Q., & Lee, Y. J. (2023) · arXiv · doi:10.48550/arxiv.2304.08485 · Cited by 682 · cited 1× · see citing works
“Meanwhile, some work [57, 58, 59] introduce visual instruction tuning to enhance instruction following capabilities.”
BackgroundRaven, J. (2003) · Springer eBooks · doi:10.1007/978-1-4615-0153-4_11 · Cited by 637 · see citing works
Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts
Changpinyo, S., Sharma, P. K., Ding, N., & Soricut, R. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00356 · Cited by 626 · see citing works
VizWiz Grand Challenge: Answering Visual Questions from Blind People
Gurari, D., Li, Q., Stangl, A. J., Guo, A., Lin, C., Grauman, K., Luo, J., & Bigham, J. P. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00380 · Cited by 621 · cited 5× · see citing works
“Visual Question Answering Table 2b reports the visual question answering results on VQAv2 [41] and VizWiz [42].”
BackgroundMethodInfluentialPaLM-E: An Embodied Multimodal Language Model
Driess, D., Xia, F., Sajjadi, M. S. M., Lynch, C., Chowdhery, A., Ichter, B., Wahid, A., Tompson, J., Vuong, Q., Yu, T., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.03378 · Cited by 591 · cited 1× · see citing works
“Another category of research focus on learning multimodality abilities based on LLMs [7, 55, 56].”
BackgroundUnnatural Instructions: Tuning Language Models with (Almost) No Human LaborPreprint
Honovich, O., Scialom, T., Levy, O., & Schick, T. (2022) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2212.09689 · Cited by 516 · cited 1× · see citing works
“To improve instruction-following capabilities, we perform language-only instruction tuning using Unnatural Instructions [24] and FLANv2 [25] datasets.”
MethodThe Pile: An 800GB Dataset of Diverse Text for Language Modeling
Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., et al. (2021) · arXiv · doi:10.48550/arxiv.2101.00027 · Cited by 492 · cited 1× · see citing works
“Text Corpora We train our model with The Pile [16] and Common Crawl (CC).”
MethodMaking the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering
Goyal, Y., Khot, T., Agrawal, A., Summers-Stay, D., Batra, D., & Parikh, D. (2019) · International Journal of Computer Vision · doi:10.1007/s11263-018-1116-0 · Cited by 487 · cited 2× · see citing works
“Visual Question Answering Table 2b reports the visual question answering results on VQAv2 [41] and VizWiz [42].”
MethodThe CommitmentBank: Investigating projection in naturally occurring discourse
de Marneffe, M.-C., Simons, M., & Tonhauser, J. (2019) · Digital Access to Libraries · doi:10.18148/sub/2019.v23i2.601 · Cited by 465 · cited 1× · see citing works
“…Accuracy ✓ ✓ Winogrande [29] Word ambiguity Accuracy ✓ ✓ PIQA [30] Physical commonsense Accuracy ✓ ✓ BoolQ [31] Question answering Accuracy ✓ ✓ CB [32] Textual entailment Accuracy ✓ ✓ COPA [33] Causal reasoning Accuracy ✓ ✓ Rendered SST-2 [34] OCR-free sentiment classification Accuracy ✓…”
BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions
Clark, C., Lee, K., Chang, M.-W., Kwiatkowski, T., Collins, M., & Toutanova, K. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1300 · Cited by 391 · cited 1× · see citing works
“…✓ ✓ Winograd [28] Word ambiguity Accuracy ✓ ✓ Winogrande [29] Word ambiguity Accuracy ✓ ✓ PIQA [30] Physical commonsense Accuracy ✓ ✓ BoolQ [31] Question answering Accuracy ✓ ✓ CB [32] Textual entailment Accuracy ✓ ✓ COPA [33] Causal reasoning Accuracy ✓ ✓ Rendered SST-2 [34] OCR-free…”
LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs
Schuhmann, C., Vencu, R., Beaumont, R., Kaczmarczyk, R., Mullis, C., Katta, A., Coombes, T., Jitsev, J., & Komatsuzaki, A. (2021) · arXiv · doi:10.48550/arxiv.2111.02114 · Cited by 358 · cited 3× · see citing works
“Image-Caption Pairs The image-caption pairs are constructed from several datasets, including English LAION-2B [SBV22], LAION-400M [SVB21], COYO-700M [BPK22], and Conceptual Captions [SDGS18, CSDS21].”
MethodMultimodal Few-Shot Learning with Frozen Language Models
Tsimpoukelli, M., Menick, J., Cabi, S., Eslami, S. M. A., Vinyals, O., & Hill, F. (2021) · arXiv · doi:10.48550/arxiv.2106.13884 · Cited by 355 · cited 2× · see citing works
“In recent years, vision-language learning and representation models has garnered significant attention [2, 3, 4, 34, 50, 51, 52, 53, 54].”
BackgroundThe hateful memes challenge: Detecting hate speech in multimodal memes
Kiela, D., Firooz, H., Mohan, A., Goswami, V., Singh, A., Ringshia, P., & Testuggine, D. (2020) · arXiv · doi:10.48550/arxiv.2005.04790 · Cited by 352 · cited 1× · see citing works
“…entailment Accuracy ✓ ✓ COPA [33] Causal reasoning Accuracy ✓ ✓ Rendered SST-2 [34] OCR-free sentiment classification Accuracy ✓ HatefulMemes [35] OCR-free meme classification ROC AUC ✓ Cross-modal transfer RelativeSize [36] Commonsense reasoning (object size) Accuracy ✓ MemoryColor [37]…”
BackgroundSmith, S., Patwary, M., Norick, B., LeGresley, P., Rajbhandari, S., Casper, J., Liu, Z., Prabhumoye, S., Zerveas, G., Korthikanti, V., et al. (2022) · arXiv · doi:10.48550/arxiv.2201.11990 · Cited by 286 · see citing works
Self-citationRosenbloom, L. (2015) · Choice Reviews Online · doi:10.5860/choice.189890 · Cited by 273 · see citing works
Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers
Wang, C., Chen, S., Wu, Y., Zhang, Z., Zhou, L., Liu, S., Chen, Z., Liu, Y., Wang, H., Li, J., et al. (2023) · arXiv · doi:10.48550/arxiv.2301.02111 · Cited by 250 · see citing works
Self-citationPaLI: A Jointly-Scaled Multilingual Language-Image ModelPreprint
Chen, X., Wang, X., Changpinyo, S., Piergiovanni, A., Padlewski, P., Salz, D., Goodman, S., Grycner, A., Mustafa, B., Beyer, L., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2209.06794 · Cited by 235 · cited 1× · see citing works
“In recent years, vision-language learning and representation models has garnered significant attention [2, 3, 4, 34, 50, 51, 52, 53, 54].”
BackgroundOn the Representation Collapse of Sparse Mixture of Experts
Chi, Z., Dong, L., Huang, S.-H., et al. (2022) · Neural Information Processing Systems · doi:10.48550/arxiv.2204.09179 · Cited by 223 · cited 1×
“In the future, we would like to scale up K OSMOS -1 in terms of model size [13, 14, 60], and integrate the speech [12] capability into K OSMOS -1.”
MethodSelf-citationCompressive Transformers for Long-Range Sequence Modelling
Rae, J. W., Potapenko, A., Jayakumar, S. M., & Lillicrap, T. P. (2019) · arXiv · doi:10.48550/arxiv.1911.05507 · Cited by 205 · see citing works
A Length-Extrapolatable TransformerPreprint
Sun, Y., Dong, L., Patra, B., Ma, S.-M., Huang, S.-H., Benhaim, A., Chaudhary, V., Song, X., & Wei, F. (2022) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2212.10554 · Cited by 192 · cited 1× · see citing works
“Compared with the standard Transformer architecture, we include the following modifications: We use M AGNETO [14], a Transformer variant, as the backbone architecture and X P OS [15] relative position encoding for better long-context modeling.”
MethodSelf-citationZeroCap: Zero-Shot Image-to-Text Generation for Visual-Semantic Arithmetic
Tewel, Y., Shalev, Y., Schwartz, I., & Wolf, L. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01739 · Cited by 166 · see citing works
Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language TasksPreprint
Wang, W., Bao, H., Li, D., Björck, J., Peng, Z., Liu, Q., Aggarwal, K., Mohammed, O. K., Singhal, S., Som, S., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2208.10442 · Cited by 163 · cited 3× · see citing works
“In recent years, vision-language learning and representation models has garnered significant attention [2, 3, 4, 34, 50, 51, 52, 53, 54].”
BackgroundSelf-citationJin, W., Cheng, Y., Shen, Y., Chen, W., & Ren, X. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.197 · Cited by 144 · see citing works
LSDSem 2017 Shared Task: The Story Cloze Test
Mostafazadeh, N., Roth, M., Louis, A. P., Chambers, N., & Allen, J. F. (2017) · LSDSem@EACL · doi:10.18653/v1/w17-0906 · Cited by 129 · cited 1× · see citing works
“Metric Zero-shot Few-shot Language tasks StoryCloze [26] Commonsense reasoning Accuracy ✓ ✓ HellaSwag [27] Commonsense NLI Accuracy ✓ ✓ Winograd [28] Word ambiguity Accuracy ✓ ✓ Winogrande [29] Word ambiguity Accuracy ✓ ✓ PIQA [30] Physical commonsense Accuracy ✓ ✓ BoolQ [31] Question answering…”
BackgroundNeural Codec Language Models are Zero-Shot Text to Speech Synthesizers
Chen, S., Wang, C., Wu, Y., Zhang, Z., Zhou, L., Liu, S., Chen, Z., Liu, T., Wang, H., Li, J., et al. (2025) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2025.3530270 · Cited by 128 · cited 2× · see citing works
“In the future, we would like to scale up K OSMOS -1 in terms of model size [13, 14, 60], and integrate the speech [12] capability into K OSMOS -1.”
BackgroundMethodSelf-citationLanguage Models are General-Purpose InterfacesPreprint
Hao, Y., Song, H., Dong, L., Huang, S.-H., Chi, Z., Wang, W.-H., Ma, S.-M., & Wei, F. (2022) · arXiv.org · doi:10.48550/arxiv.2206.06336 · Cited by 117 · see citing works
Self-citationEnabling Multimodal Generation on CLIP via Vision-Language Knowledge Distillation
Dai, W., Hou, L., Shang, L.-F., et al. (2022) · Findings · doi:10.48550/arxiv.2203.06386 · Cited by 115
Self-citationDeepNet: Scaling Transformers to 1,000 Layers
Wang, H., Ma, S., Dong, L., Huang, S., Zhang, D., & Wei, F. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3386927 · Cited by 113 · see citing works
Self-citationAre Elephants Bigger than Butterflies? Reasoning about Sizes of Objects
Bagherinezhad, H., Hajishirzi, H., Choi, Y., et al. (2016) · AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v30i1.10476 · Cited by 54 · cited 2×
“…SST-2 [34] OCR-free sentiment classification Accuracy ✓ HatefulMemes [35] OCR-free meme classification ROC AUC ✓ Cross-modal transfer RelativeSize [36] Commonsense reasoning (object size) Accuracy ✓ MemoryColor [37] Commonsense reasoning (object color) Accuracy ✓ ColorTerms [38] Commonsense…”
BackgroundMethodWebSRC: A Dataset for Web-Based Structural Reading Comprehension
Chen, X., Zhao, Z., Chen, L., Ji, J., Zhang, D., Luo, A., Xiong, Y., & Yu, K. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.343 · Cited by 52 · cited 6× · see citing works
“We compare the performance on the Web-based Structural Reading Comprehension (WebSRC) dataset [43].”
BackgroundMethodInfluentialDeepNet: Scaling Transformers to 1,000 LayersPreprint
Wang, H., Ma, S., Li, D., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2203.00555 · Cited by 52 · see citing works
Self-citationPaLI-X: On Scaling up a Multilingual Vision and Language ModelPreprint
Chen, X., Djolonga, J., Padlewski, P., Mustafa, B., Changpinyo, S., Wu, J., Riquelme, C., Goodman, S., Wang, X., Tay, Y., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.18565 · Cited by 42 · cited 1× · see citing works
“In recent years, vision-language learning and representation models has garnered significant attention [2, 3, 4, 34, 50, 51, 52, 53, 54].”
BackgroundCM3: A Causal Masked Multimodal Model of the Internet
Aghajanyan, A., Huang, B., Ross, C., Karpukhin, V., Xu, H., Goyal, N., Okhonko, D., Joshi, M., Ghosh, G., Lewis, M., et al. (2022) · arXiv · doi:10.48550/arxiv.2201.07520 · Cited by 42 · cited 1× · see citing works
“More importantly, unlocking multimodal input [2, 3, 4, 5, 6, 7] greatly widens the applications of language models to more high-value areas, such as multimodal machine learning, document intelligence, and robotics.”
BackgroundGrounding Language Models to Images for Multimodal Inputs and Outputs
Koh, J. Y., Salakhutdinov, R., & Fried, D. (2023) · arXiv · doi:10.48550/arxiv.2301.13823 · Cited by 38 · see citing works
PIQA: Reasoning about Physical Commonsense in Natural Language
Bisk, Y., Zellers, R., Le bras, R., Gao, J., & Choi, Y. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6239 · Cited by 37 · cited 1× · see citing works
“…Accuracy ✓ ✓ HellaSwag [27] Commonsense NLI Accuracy ✓ ✓ Winograd [28] Word ambiguity Accuracy ✓ ✓ Winogrande [29] Word ambiguity Accuracy ✓ ✓ PIQA [30] Physical commonsense Accuracy ✓ ✓ BoolQ [31] Question answering Accuracy ✓ ✓ CB [32] Textual entailment Accuracy ✓ ✓ COPA [33] Causal reasoning…”
Retrieval-Augmented Multimodal Language ModelingPreprint
Yasunaga, M., Aghajanyan, A., Shi, W., James, R., Leskovec, J., Liang, P., Lewis, M., Zettlemoyer, L., & Yih, W.-T. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2211.12561 · Cited by 37 · see citing works
Foundation TransformersPreprint
Wang, H., Ma, S., Huang, S., Li, D., Wang, W., Peng, Z., Wu, Y., Bajaj, P., Singhal, S., Benhaim, A., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2210.06423 · Cited by 23 · cited 2× · see citing works
“In the future, we would like to scale up K OSMOS -1 in terms of model size [13, 14, 60], and integrate the speech [12] capability into K OSMOS -1.”
MethodSelf-citationAre Elephants Bigger than Butterflies? Reasoning about Sizes of ObjectsPreprint
Bagherinezhad, H., Hajishirzi, H., Choi, Y., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1602.00753 · Cited by 17 · see citing works
Visually-Augmented Language ModelingPreprint
Wang, W., Li, D., Cheng, H., Song, H., Liu, X., Yan, X., Gao, J., & Wei, F. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2205.10178 · Cited by 13 · cited 1× · see citing works
“Model Size Reasoning Color Reasoning RELATIVESIZE MEMORYCOLOR COLORTERMS Using retrieved images VALM [WDC23] 85.”
MethodSelf-citationTransferring Knowledge from Vision to Language: How to Achieve it and how to Measure it?
Norlund, T., Hagström, L., & Johansson, R. (2021) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/2021.blackboxnlp-1.10 · Cited by 10 · cited 3× · see citing works
“Metric Zero-shot Few-shot Language tasks StoryCloze [26] Commonsense reasoning Accuracy ✓ ✓ HellaSwag [27] Commonsense NLI Accuracy ✓ ✓ Winograd [28] Word ambiguity Accuracy ✓ ✓ Winogrande [29] Word ambiguity Accuracy ✓ ✓ PIQA [30] Physical commonsense Accuracy ✓ ✓ BoolQ [31] Question answering Accuracy ✓ ✓ CB [32] Te…”
MethodTorchScale: Transformers at ScalePreprint
Ma, S., Wang, H., Huang, S., Wang, W., Chi, Z., Li, D., Benhaim, A., Patra, B., Chaudhary, V., Song, X., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2211.13184 · Cited by 5 · cited 2× · see citing works
“In the future, we would like to scale up K OSMOS -1 in terms of model size [13, 14, 60], and integrate the speech [12] capability into K OSMOS -1.”
MethodSelf-citationBMJ Publishing Group Ltd, & Royal College of Psychiatrists and British Psychological Society (2008) · Evidence-Based Mental Health · doi:10.1136/ebmh.11.4.102 · cited 3×
“Text Corpora We train our model with The Pile [16] and Common Crawl (CC).”
Method
Flagged first, then by citations
Research works printed but not linked: 5
- Elia Bruni, Gemma Boleda, Marco Baroni, and Nam Khanh Tran. Distributional semantics in technicolor. In ACL, 2012.
- Patricia A Carpenter, Marcel A Just, and Peter Shell. What one intelligence test measures: a theoretical account of the processing in the raven progressive matrices test. Psychological review, 97(3):404, 1990.
- Hector Levesque, Ernest Davis, and Leora Morgenstern. The winograd schema chal- lenge. In Thirteenth International Conference on the Principles of Knowledge Repre- sentation and Reasoning, 2012.
- Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S. Gordon. Choice of plau- sible alternatives: An evaluation of commonsense causal reasoning. In AAAI Spring Symposium, 2011.
- Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. WinoGrande: An adversarial winograd schema challenge at scale. In AAAI, pages 8732-8740, 2020.
Cited by
Sorted by citations
A Survey on Evaluation of Large Language Models
Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., et al. (2024) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3641289 · Cited by 2,842 · see citing works
Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B., et al. (2025) · ACM Transactions on Information Systems · doi:10.1145/3703155 · Cited by 2,142 · see citing works
The rise and potential of large language model based agents: a survey
Xi, Z., Chen, W., Guo, X., He, W., Ding, Y., Hong, B., Zhang, M., Wang, J., Jin, S., Zhou, E., et al. (2025) · Science China Information Sciences · doi:10.1007/s11432-024-4222-0 · Cited by 970 · see citing works
“The second approach to endowing an LLM-based agent with visual understanding capabilities is modality fusion, where a visual encoder is integrated with an LLM at the embedding level [209,210].”
Self-citationVideo-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding
Zhang, H., Li, X. S., & Bing, L. (2023) · Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP) · doi:10.18653/v1/2023.emnlp-demo.49 · Cited by 601 · see citing works
A Survey on Large Language Models: Applications, Challenges, Limitations, and Practical UsagePreprint
Hadi, M. U., Al-Tashi, Q., Qureshi, R., Shah, A., Muneer, A., Irfan, M., Zafar, A., Shaikh, M. B., Akhtar, N., Hassan, S. Z., et al. (2023) · doi:10.36227/techrxiv.23589741.v1 · Cited by 429 · see citing works
A review of deep learning techniques for speech processing
Mehrish, A., Majumder, N., Bharadwaj, R., Mihalcea, R., & Poria, S. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101869 · Cited by 389 · see citing works
Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook.
AlSaad, R., Abd-alrazaq, A., Boughorbel, S., Ahmed, A., Renault, M.-A., Damseh, R., & Sheikh, J. (2024) · Journal of medical Internet research · doi:10.2196/59505 · Cited by 349 · see citing works
“Furthermore, developing more efficient transformer architectures tailored for multimodal data, such as Kosmos-1 [162], Muse [163], and PixArt-α [164], presents a viable solution.”
RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control
Brohan, A., Brown, N., Carbajal, J., Chebotar, Y., Chen, X., Choromanski, K., Ding, T., Driess, D., Dubey, A., Finn, C., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.15818 · Cited by 276 · see citing works
“Finally, we describe how to make these models practical for robot tasks, addressing challenges with model size and inference speed to enable real-time control.”
BackgroundMethodThe Rise and Potential of Large Language Model Based Agents: A Survey
Xi, Z., Chen, W., Guo, X., He, W., Ding, Y., Hong, B., Zhang, M., Wang, J., Jin, S., Zhou, E., et al. (2023) · arXiv · doi:10.48550/arxiv.2309.07864 · Cited by 972 · see citing works
Self-citationMVBench: A Comprehensive Multi-modal Video Understanding Benchmark
Li, K., Wang, Y., He, Y., Li, Y., Wang, Y., Liu, Y., Wang, Z., Xu, J., Chen, G., Lou, P., et al. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.02095 · Cited by 300 · see citing works
“In the past few years, Multi-modal Large Language Models (MLLMs) [1, 16, 26, 39, 42, 47, 59, 104] have gradually driven the advance in vision-language learning, by plugging visual encoders within various pretrained LLMs [10, 15, 58, Figure 1.”
BackgroundMultimodal Large Language Models: A Survey
Wu, J., Gan, W., Chen, Z., Wan, S., & Yu, P. S. (2023) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata59044.2023.10386743 · Cited by 309 · see citing works
“In 2023, KOSMOS-1 was released by Microsoft [25].”
MethodLarge AI Models in Health Informatics: Applications, Challenges, and the Future
Qiu, J., Li, L., Sun, J., Peng, J., Shi, P., Zhang, R., Dong, Y., Lam, K., Lo, F. P.-W., Xiao, B. J., et al. (2023) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2023.3316750 · Cited by 246 · see citing works
“Forrepresentationlearning,thefeaturesfrom differentmodalitiesarethenalignedthroughcontrastivelearn-ing[54],[55],[56]orfusedintoaunifiedrepresentationthrough another encoder on the top of all extracted features [58], [59], [60], [61], [62].”
BackgroundMM-LLMs: Recent Advances in MultiModal Large Language Models
Zhang, D., Yu, Y., Dong, J., Li, C., Su, D., Chu, C., & Yu, D. (2024) · Findings of the Association for Computational Linguistics ACL 2024 · doi:10.18653/v1/2024.findings-acl.738 · Cited by 260 · see citing works
ViperGPT: Visual Inference via Python Execution for Reasoning
Surís, D., Menon, S., & Vondrick, C. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01092 · Cited by 273 · see citing works
“Due to the practical difficulty of using these methods, the field has primarily moved towards end-to-end all-in-one models [1, 22, 23, 30].”
MethodSpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities
Zhang, D., Li, S., Zhang, X., Zhan, J., Wang, P., Zhou, Y., & Qiu, X. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.1055 · Cited by 268 · see citing works
mPLUG-OwI2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration
Ye, Q.-H., Xu, H., Ye, J., Yan, M.-S., Hu, A., Liu, H.-W., Qian, Q., Zhang, J., Huang, F., & Zhou, J.-R. (2023) · Computer Vision and Pattern Recognition · doi:10.1109/cvpr52733.2024.01239 · Cited by 763 · see citing works
“Instruction Tuning with MLLMs.”
Zhang, W., Cai, M., Zhang, T., Zhuang, Y., & Mao, X. (2024) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2024.3409624 · Cited by 213 · see citing works
“Models like VisualGPT [4], BLIP [5], Flamingo [7], and Kosmos-1 [26] showcase strong multimodal reasoning potential after aligning LLMs with image modalities.”
MethodGLaMM: Pixel Grounding Large Multimodal Model
Rasheed, H., Maaz, M., Shaji, S., Shaker, A., Khan, S., Cholakkal, H., Anwer, R. M., Xing, E. P., Yang, M., & Khan, F. S. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01236 · Cited by 205 · see citing works
Vision-Language Models in Remote Sensing: Current progress and future trends
Xiang, L., Wen, C., Hu, Y., Yuan, Z., & Zhu, X. X. (2024) · IEEE Geoscience and Remote Sensing Magazine · doi:10.1109/mgrs.2024.3383473 · Cited by 199 · see citing works
“In particular, to deal with the limited training data, an auxiliary decoder, trained for multilabel scene classification, has been used to assist the encoder in the training process by leveraging its conceptual similarity to image captioning and its ability to highlight semantic classes.”
MethodLet’s Have a Chat! A Conversation with ChatGPT: Technology, Applications, and Limitations
Shahriar, S., & Hayawi, K. (2023) · Artificial Intelligence and Applications · doi:10.47852/bonviewaia3202939 · Cited by 167 · see citing works
Evaluating Large Language Models in Class-Level Code Generation
Du, X., Liu, M., Wang, K., Wang, H., Jun-wei, L., Chen, Y., Feng, J., Sha, C., Peng, X., & Lou, Y. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3597503.3639219 · Cited by 180 · see citing works
Reasoning with Language Model Prompting: A Survey
Qiao, S., Ou, Y., Zhang, N., Chen, X., Yao, Y., Deng, S., Tan, C., Huang, F., & Chen, H. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.294 · Cited by 218 · see citing works
Kuang, W., Qian, B., Li, Z., Chen, D., Gao, D., Pan, X., Xie, Y., Li, Y., Ding, B., & Zhou, J. (2024) · Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining · doi:10.1145/3637528.3671573 · Cited by 142 · see citing works
On the Opportunities and Challenges of Foundation Models for GeoAI (Vision Paper)
Mai, G., Huang, W., Sun, J., Song, S., Mishra, D. R., Liu, N., Gao, S., Liu, T., Cong, G., Hu, Y., et al. (2024) · ACM Transactions on Spatial Algorithms and Systems · doi:10.1145/3653070 · Cited by 275 · see citing works
“Furthermore, we think the urban perception task, as a classic task in urban geography, is more challenging than current visual question-answering tasks commonly used in VLFM research [64, 150] partly due to their partially subjective nature and the rarity of annotated datasets.”
BackgroundMethodInfluentialWang, D., Feng, L., Ye, J., Zou, J., & Zheng, Y. (2023) · MedComm – Future Medicine · doi:10.1002/mef2.43 · Cited by 123 · see citing works
Liu, F., Emerson, G., & Collier, N. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00566 · Cited by 127 · see citing works
Foundation models in smart agriculture: Basics, opportunities, and challenges
Li, J., Xu, M., Xiang, L., Chen, D., Zhuang, W., Yin, X., & Li, Z. (2024) · Computers and Electronics in Agriculture · doi:10.1016/j.compag.2024.109032 · Cited by 141 · see citing works
Yu, T., Yao, Y., Zhang, H., He, T., Han, Y., Cui, G., Hu, J., Liu, Z., Zheng, H.-T., & Sun, M. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01310 · Cited by 156 · see citing works
“Models are pretrained on large-scale image-text pairs [4, 11, 19, 42, 47] or interleaved data [2, 3, 16] to learn the semantic mapping between visual and text signals.”
BackgroundHadi, M. U., Al-Tashi, Q., Qureshi, R., Shah, A., Muneer, A., Irfan, M., Zafar, A., Shaikh, M. B., Akhtar, N., Hassan, S. Z., et al. (2023) · doi:10.36227/techrxiv.23589741.v4 · Cited by 173 · see citing works
Generative Multimodal Models are In-Context Learners
Sun, Q., Cui, Y., Zhang, X., Zhang, F., Yu, Q., Wang, Y., Rao, Y., Liu, J., Huang, T., & Wang, X. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.01365 · Cited by 147 · see citing works
“Recent years have witnessed the rapid growth of large multimodal models [5, 19, 31, 71].”
BackgroundIshmam, M. F., Shovon, M. S. H., Mridha, M. F., & Dey, N. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102270 · Cited by 100 · see citing works
Preventing Zero-Shot Transfer Degradation in Continual Learning of Vision-Language Models
Zheng, Z., Ma, M., Wang, K., Qin, Z., Yue, X., & You, Y. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01752 · Cited by 104 · see citing works
“The deep learning community tends to build large models with a huge dataset [4, 8], including vision-language models [54, 24].”
BackgroundAlpha-CLIP: A CLIP Model Focusing on Wherever you Want
Sun, Z., Fang, Y., Wu, T., Zhang, P., Zang, Y., Kong, S., Xiong, Y., Lin, D., & Wang, J. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01237 · Cited by 119 · see citing works
“By using this pseudo-labeling base-line, Kosmos-2 releases the GRIT dataset and equips multi-modal model [15] with local perception capabilities.”
A Survey of Generative AI Applications
Gozalo-Brizuela, R., & Merchan, E. E. G. (2024) · Journal of Computer Science · doi:10.3844/jcssp.2024.801.818 · Cited by 101 · see citing works
GIT-Mol: A multi-modal large language model for molecular science with graph, image, and text
Liu, P., Ren, Y., Tao, J., & Ren, Z. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.108073 · Cited by 98 · see citing works
“• Modality-Adaptive Learning models like BEIT-3 [7] and KOSMOS-1 [22] use the Mixture of Experts (MoE) [23] and the MAGNETO [24] Transformer.”
MethodWhen brain-inspired AI meets AGI
Zhao, L., Zhang, L., Wu, Z., Chen, Y., Dai, H., Yu, X., Liu, Z., Zhang, T., Hu, X., Jiang, X., et al. (2023) · Meta-Radiology · doi:10.1016/j.metrad.2023.100005 · Cited by 96 · see citing works
The Revolution of Multimodal Large Language Models: A Survey
Caffagni, D., Cocchi, F., Barsellotti, L., Moratelli, N., Sarto, S., Baraldi, L., Baraldi, L., Cornia, M., & Cucchiara, R. (2024) · Findings of the Association for Computational Linguistics ACL 2024 · doi:10.18653/v1/2024.findings-acl.807 · Cited by 98 · see citing works
Hadi, M. U., Al-Tashi, Q., Qureshi, R., Shah, A., Muneer, A., Irfan, M., Zafar, A., Shaikh, M. B., Akhtar, N., Wu, J., et al. (2023) · doi:10.36227/techrxiv.23589741 · Cited by 88 · see citing works
Hallucination Augmented Contrastive Learning for Multimodal Large Language Model
Jiang, C., Xu, H., Dong, M., Chen, J., Ye, W., Yan, M., Ye, Q., Zhang, J., Huang, F., & Zhang, S. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02553 · Cited by 100 · see citing works
Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Action
Lu, J., Clark, C., Lee, S.-H., Zhang, Z., Khosla, S., Marten, R., Hoiem, D., & Kembhavi, A. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02497 · Cited by 98 · see citing works
MMA: Multi-Modal Adapter for Vision-Language Models
Yang, L., Zhang, R., Wang, Y., & Xie, X. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02249 · Cited by 92 · see citing works
“Vision-Language Models (VLMs) [1, 28, 30, 50, 68, 70, 71, 74] have recently been introduced as a class of foundation models.”
BackgroundMethodGenerating meaning: active inference and the scope and limits of passive AI
Pezzulo, G., Parr, T., Cisek, P., Clark, A., & Friston, K. (2024) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2023.10.002 · Cited by 80 · see citing works
“The lack of anchoring on embodied reality motivates novel foundation models for embodied intelligence that include multiple modalities [115] iv or that mimic the visual cortex, rather than starting from language [116].”
ChatGPT: Systematic Review, Applications, and Agenda for Multidisciplinary Research
Singh, H., & Singh, A. (2023) · Journal of Chinese Economic and Business Studies · doi:10.1080/14765284.2023.2210482 · Cited by 77 · see citing works
“The former is mainly used for recognizing the context in classification tasks and language (Huang et al. 2023).”
BackgroundYe, J., Hu, A., Xu, H., Ye, Q., Yan, M., Xu, G., Li, C., Tian, J., Qian, Q., Zhang, J., et al. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.187 · Cited by 82 · see citing works
Retrieving Multimodal Information for Augmented Generation: A Survey
Zhao, R., Chen, H., Wang, W., Jiao, F., Long, X., Qin, C., Ding, B., Guo, X., Li, M., Li, X., et al. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.314 · Cited by 66 · see citing works
Self-citationInstructDiffusion: A Generalist Modeling Interface for Vision Tasks
Geng, Z., Yang, B., Hang, T., Li, C., Gu, S., Zhang, T., Bao, J., Zhang, Z., Li, H., Hu, H., et al. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01208 · Cited by 78 · see citing works
“On the other hand, the success of large language models such as GPT series [7, 53, 56, 57], PaLM [4, 12], and LLaMA [68], has been attracting a lot of research interest [24, 38, 64, 69, 73, 74, 80] in augmenting the large language models with visual capabilities.”
BackgroundGPT-4V(ision) is a Human-Aligned Evaluator for Text-to-3D Generation
Wu, T., Yang, G., Li, Z., Zhang, K., Liu, Z., Guibas, L. J., Lin, D., & Wetzstein, G. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02098 · Cited by 66 · see citing works
“Initial efforts of LMM involve combining computer vision with LLMs by fine-tuning visual encoders to align with language embed-dings [2, 4, 17, 27, 33, 34, 64] or converting visual information to text [26, 58, 66, 71].”
MethodMM1: Methods, Analysis and Insights from Multimodal LLM Pre-training
McKinzie, B., Gan, Z., Fauconnier, J.-P., Dodge, S., Zhang, B., Dufter, P., Shah, D., Du, X., Peng, F., Belyi, A., et al. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73397-0_18 · Cited by 120 · see citing works
Hadi, M. U., Al-Tashi, Q., Qureshi, R., Shah, A., Muneer, A., Irfan, M., Zafar, A., Shaikh, M. B., Akhtar, N., Hassan, S. Z., et al. (2023) · doi:10.36227/techrxiv.23589741.v3 · Cited by 62 · see citing works
The Limitations of Large Language Models for Understanding Human Language and Cognition
Cuskley, C. F., Woods, R. J. H., & Flaherty, M. (2024) · Open Mind · doi:10.1162/opmi_a_00160 · Cited by 77 · see citing works
“…multimodal models that can deal not only with text, but also with data from other modalities, including images and video (Bubeck et al., 2023; Huang, Dong, et al., 2023), and all manner of other modalities that even humans generally cannot readily interpret (e.g., thermal data; Girdhar et…”
BackgroundMethodInstruct-ReID: A Multi-Purpose Person Re-Identification Task with Instructions
He, W., Deng, Y., Tang, S., Chen, Q., Xie, Q., Wang, Y., Bai, L., Zhu, F., Zhao, R., Ouyang, W., et al. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01659 · Cited by 62 · see citing works
“† denotes that the test mode is VIS-to-IR and ‡ denotes IR-to-VIS mode on LLCM. 14.9 63.9 [1] 65.8 [17] 62.9 [17] 40.8 [3] 45.9 [5] 71.8 [4] 93.0 [19] 71.8 [19] 77.7 [11] DeiT [12”
InfluentialUrban attractiveness according to ChatGPT: Contrasting AI and human insights
Malekzadeh, M., Willberg, E. S., Torkko, J., & Toivonen, T. (2024) · Computers Environment and Urban Systems · doi:10.1016/j.compenvurbsys.2024.102243 · Cited by 56 · see citing works
Ghosh, A., Acharya, A., Saha, S., Jain, V., & Chadha, A. R. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4783140 · Cited by 94 · see citing works
On Scaling Up a Multilingual Vision and Language Model
Chen, X., Djolonga, J., Padlewski, P., Mustafa, B., Changpinyo, S., Wu, J., Riquelme, C., Goodman, S., Wang, X., Tay, Y., et al. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01368 · Cited by 140 · see citing works
“[Right] The Pareto frontier between few-shot and fine-tuned performance, comparing PaLI-X with PaLI [5], Flamingo [10], and Kosmos-1 [11]. form the visual input (See Appendix A).”
RegionGPT: Towards Region Understanding Vision Language Model
Guo, Q., Mello, S. D., Yin, H., Byeon, W., Cheung, K. C., Yu, Y., Luo, P., & Liu, S. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01309 · Cited by 54 · see citing works
“Some other interesting works have made remarkable progress by extending LLM to audio [7, 20], medical VQA [31, 57] and control systems [16, 32].”
BackgroundHadi, M. U., Al-Tashi, Q., Qureshi, R., Shah, A., Muneer, A., Irfan, M., Zafar, A., Shaikh, M. B., Akhtar, N., Hassan, S. Z., et al. (2023) · doi:10.36227/techrxiv.23589741.v2 · Cited by 47 · see citing works
Generative AI and Its Educational Implications
Łodzikowski, K., Foltz, P. W., & Behrens, J. T. (2024) · Postdigital Science and Education · doi:10.1007/978-3-031-64487-0_2 · Cited by 45 · see citing works
Zhang, Z., Yao, Y., Zhang, A., Tang, X., Ma, X., He, Z., Wang, Y., Gerstein, M., Wang, R., Liu, G., et al. (2025) · ACM Computing Surveys · doi:10.1145/3719341 · Cited by 51 · see citing works
“More recent studies have also explored modeling the inputs of different modalities into the same vector space, thus resulting in any-to-any representation learning [46, 88, 160] and interleaved multimodal representation learning [57, 190].”
InfluentialHuang, X., Liu, Z. Y., Zhang, M., 桂, 韬., 韩, 先., 户, 保., 黄, 民., 黄, 萱., 刘, 康., 刘, 挺., et al. (2023) · Scientia Sinica Informationis · doi:10.1360/ssi-2023-0113 · Cited by 40 · see citing works
KAM-CoT: Knowledge Augmented Multimodal Chain-of-Thoughts Reasoning
Mondal, D., Modi, S., Panda, S., Singh, R., & Rao, G. S. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i17.29844 · Cited by 51 · see citing works
neuroGPT-X: toward a clinic-ready large language model
Guo, E., Gupta, M., Sinha, S., Rössler, K., Tatagiba, M. S., Akagami, R., Al‐Mefty, O., Sugiyama, T., Stieg, P. E., Pickett, G. E., et al. (2023) · Journal of neurosurgery · doi:10.3171/2023.7.jns23573 · Cited by 48 · see citing works
Breaking Common Sense: WHOOPS! A Vision-and-Language Benchmark of Synthetic and Compositional Images
Bitton-Guetta, N., Bitton, Y., Hessel, J., Schmidt, L., Elovici, Y., Stanovsky, G., & Schwartz, R. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00247 · Cited by 37 · see citing works
UPRISE: Universal Prompt Retrieval for Improving Zero-Shot Evaluation
Cheng, D., Huang, S., Bi, J., Zhan, Y., Liu, J., Wang, Y., Sun, H., Wei, F., Deng, W., & Zhang, Q. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.758 · Cited by 32 · see citing works
Self-citationModel-Driven Prompt Engineering
Clarisó, R., & Cabot, J. (2023) · ACM/IEEE International Conference on Model Driven Engineering Languages and Systems (MODELS) · doi:10.1109/models58315.2023.00020 · Cited by 34 · see citing works
SAM4MLLM: Enhance Multi-Modal Large Language Model for Referring Expression Segmentation
Chen, Y.-C., Li, W., Sun, C., Wang, Y.-C. F., & Chen, C. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73004-7_19 · Cited by 37 · see citing works
Faithful AI in Medicine: A Systematic Review with Large Language Models and BeyondPreprint
Xie, Q., Schenck, E. J., Yang, H. S., Chen, Y., Peng, Y., & Wang, F. (2023) · medRxiv · doi:10.1101/2023.04.18.23288752 · Cited by 57 · see citing works
VCoder: Versatile Vision Encoders for Multimodal Large Language Models
Jain, J., Yang, J., & Shi, H. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02644 · Cited by 35 · see citing works
“Among various open-source MLLMs [5, 23, 33, 34, 73], we chose LLaVA-1.”
BackgroundMethodGenerative language reconstruction from brain recordings
Ye, Z., Ai, Q., Liu, Y., de Rijke, M., Zhang, M., Lioma, C., & Ruotsalo, T. (2025) · Communications Biology · doi:10.1038/s42003-025-07731-7 · Cited by 22 · see citing works
CapsFusion: Rethinking Image-Text Data at Scale
Yu, Q., Sun, Q., Zhang, X., Cui, Y., Zhang, F., Cao, Y. Q., Wang, X., & Liu, J. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01330 · Cited by 27 · see citing works
“Avoid simply concatenating the sentences.”
Generative AI: Overview, Economic Impact, and Applications in Asset ManagementPreprint
Luk, M. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4574814 · Cited by 22 · see citing works
Jaeger, J. (2024) · Neurons Behavior Data analysis and Theory · doi:10.51628/001c.94404 · Cited by 17 · see citing works
“One commonspeculation is that the extremely high-dimensional nature of the weightmatrix (and thus the config-uration space) of an LLM allowsfor the emergence of “higher-level” abilities, such as the detectionof meaning without external referents (see, for example, Piantadosi and Hill (2022)).”
BackgroundKaul, P., Li, Z., Yang, H., Dukler, Y., Swaminathan, A. D., Taylor, C. J., & Soatto, S. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02571 · Cited by 19 · see citing works
“[1, 6, 7, 17, 24, 25, 30, 36, 44, 48, 54] LVLMs take input text and images and generate text responses to enable multi-modal perception and com* prehension.”
Urrutia, F., & Araya, R. (2023) · Journal of educational computing research · doi:10.1177/07356331231191174 · Cited by 20 · see citing works
Pretrained Language Models as Visual Planners for Human Assistance
Patel, D., Eghbal-zadeh, H., Kamra, N., Iuzzolino, M. L., Jain, U., & Desai, R. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01404 · Cited by 18 · see citing works
“For instance, [47, 59, 2, 48, 29, 18] train large transformer based multi-modal sequence prediction models.”
BackgroundA Simple Recipe for Contrastively Pre-Training Video-First Encoders Beyond 16 Frames
Papalampidi, P., Koppula, S., Pathak, S., Chiu, J., Heyward, J., Pătrăucean, V., Shen, J., Miech, A., Zisserman, A., & Nematzdeh, A. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01364 · Cited by 21 · see citing works
“Similar work at smaller scale has additionally included captioning losses [32, 76], more contrastive losses [10, 38, 43, 66], masking/masked autoen-coding [15, 16, 18, 19, 33, 35, 40, 55], and combinations thereof [13, 23, 54, 58, 61, 72, 73, 75, 82].”
BackgroundFashion-GPT: Integrating LLMs with Fashion Retrieval System
Chen, Q., Zhang, T., Nie, M., Wang, Z., Xu, S., Shi, W., & Cao, Z. (2023) · doi:10.1145/3607827.3616844 · Cited by 16 · see citing works
Potential merits and flaws of large language models in epilepsy care: A critical review
Diessen, E. G. A. L. V., van Amerongen, R. A., Zijlmans, M., & Otte, W. M. (2024) · Epilepsia · doi:10.1111/epi.17907 · Cited by 19 · see citing works
“26”
Power-Llava: Large Language and Vision Assistant for Power Transmission Line Inspection
Wang, J., Li, M., Luo, H., Zhu, J., Yang, A., Rong, M., & Wang, X. (2024) · Proceedings - International Conference on Image Processing · doi:10.1109/icip51287.2024.10648271 · Cited by 18 · see citing works
“Targeting at feature alignment in the first stage, we collect 608K brief image-text pairs, consisting of 558K from the public dataset released by LLaVA [12] and 50K generated by VL-models [6, 9, 12]functioning as captioners.”
MethodMiniMedGPT: Efficient Large Vision–Language Model for medical Visual Question Answering
Alsabbagh, A. R., Mansour, T., Al-Kharabsheh, M., Ebdah, A. S., Al-Emaryeen, R., Al-Nahhas, S., Mahafza, W. S., & Al-Kadi, O. S. (2025) · Pattern Recognition Letters · doi:10.1016/j.patrec.2025.01.001 · Cited by 17 · see citing works
A Survey of Reasoning with Foundation ModelsPreprint
Sun, J., Zheng, C., Xie, E., Liu, Z., Chu, R., Liu, J., Xu, J., Ding, M., Li, H., Geng, M., et al. (2023) · doi:10.31219/osf.io/ac4sp · Cited by 51 · see citing works
Self-citationLow-Rank Approximation for Sparse Attention in Multi-Modal LLMs
Lin, S., Chen, Y., Yang, S., Ding, X., Ge, Y., Chen, Y.-C., & Shan, Y. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01306 · Cited by 17 · see citing works
“Additionally, in the multi-modality domain, frameworks like BLIP-2 [26], LLaVA [30, 31], KOSMOS-1 [18], and PaLM-E [10], have advanced multi-modal comprehension by effectively fine-tuning foundational models such as LLaMA [42] and Vi-cuna [7].”
VideoGrounding-DINO: Towards Open-Vocabulary Spatio- Temporal Video Grounding
Wasim, S. T., Naseer, M., Khan, S. A., Yang, M., & Khan, F. S. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01789 · Cited by 14 · see citing works
“We take inspiration from recent foundation models for spatial grounding [4, 8, 10, 12, 14, 29].”
BackgroundMethodInfluentialVisual Instruction Tuning with Polite Flamingo
Chen, D., Liu, J., Dai, W., & Wang, B. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i16.29727 · Cited by 17 · see citing works
Panagopoulou, A., Xue, L., Yu, N., Li, J., Li, D., Joty, S., Xu, R., Savarese, S., Xiong, C., & Niebles, J. C. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72995-9_11 · Cited by 37 · see citing works
Self-citationSignatures of cross-modal alignment in children's early conceptsPreprint
Aho, K., Roads, B. D., & Love, B. C. (2023) · doi:10.31234/osf.io/v6fnq · Cited by 12 · see citing works
ChatNT: A Multimodal Conversational Agent for DNA, RNA and Protein TasksPreprint
Richard, G., de Almeida, B. P., Dalla-Torre, H., Blum, C. F., Hexemer, L., Pandey, P., Laurent, S., Lopez, M., Laterre, A., Lang, M., et al. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.04.30.591835 · Cited by 14 · see citing works
Large Search Model: Redefining Search Stack in the Era of LLMs
Wang, L., Yang, N., Huang, X., Yang, L., Majumder, R., & Wei, F. (2023) · ACM SIGIR Forum · doi:10.1145/3642979.3643006 · Cited by 12 · see citing works
“Techniques like multi-query attention [Shazeer, 2019] and grouped-query attention [Ainslie et al., 2023] are commonly employed to reduce the inference memory footprint at the cost of model quality.”
BackgroundMethodSelf-citationKhanmohammadi, S., Golafshani, E. M., Bai, Y., Li, H., Bazli, M., & Arashpour, M. (2023) · International Journal of Disaster Risk Reduction · doi:10.1016/j.ijdrr.2023.103972 · Cited by 12 · see citing works
Mani, G., & Namomsa, G. B. (2023) · IEEE AFRICON · doi:10.1109/africon55910.2023.10293675 · Cited by 12 · see citing works
“Recent progress (post-ChatGPT) is focused on similar areas: e.g., see offerings from Google [26] and Microsoft [27, 28].”
BackgroundPath to Medical AGI: Unify Domain-specific Medical LLMs with the Lowest CostPreprint
Zhou, J., Chen, X., & Gao, X. (2023) · medRxiv · doi:10.1101/2023.06.23.23291802 · Cited by 15 · see citing works
“To solve the above two challenges, a number of opensource multimodal LLMs were proposed [16], [17], [18], [19], [20], [20], [21], [22], [23], [24], [25], [26], [27], [28], [29], [30], [31], [32], [33], [34], [35], [36], [37].”
The evolution of language models: From N-Grams to LLMs, and beyond
Raeini, M. (2025) · Natural Language Processing Journal · doi:10.1016/j.nlp.2025.100168 · Cited by 22 · see citing works
Bouyamourn, A. Z. A. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.192 · Cited by 11 · see citing works
Towards More Unified In-Context Visual Understanding
Sheng, D., Chen, D., Tan, Z., Liu, Q., Chu, Q., Bao, J., Gong, T., Liu, B., Xu, S., & Yu, N. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01269 · Cited by 11 · see citing works
“Inspired by the success of ChatGPT as well as GPT-4 [32, 33], recent works primarily concentrate on aligning visual features with the pre-trained LLMS for multimodal comprehension tasks [18,24,26,27,44,45,53,57].”
MethodSelf-citationWang, S., Xie, B., Ding, L., Gao, X., Chen, J., & Xiang, Y. (2024) · ACM Conference on Recommender Systems (RecSys) · doi:10.1145/3640457.3688124 · Cited by 10 · see citing works
“SeCor leverages the advanced representing and transfer capabilities of LLMs[17, 18] to combine collaborative and linguistic information, forming efficient hybrid representations.”
Decoding the Language of Chickens - An Innovative NLP Approach to Enhance Poultry WelfarePreprint
Neethirajan, S. R. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.04.29.591707 · Cited by 10 · see citing works
The limitations of large language models for understanding human language and cognitionPreprint
Cuskley, C. F., Woods, R. J. H., & Flaherty, M. (2024) · doi:10.31234/osf.io/hbwrg · Cited by 10 · see citing works
Generating Meaning: Active Inference and the Scope and Limits of Passive AI
Pezzulo, G., Parr, T., Cisek, P., Clark, A., & Friston, K. (2023) · doi:10.31234/osf.io/8xgzv · Cited by 8 · see citing works
Violet: A Vision-Language Model for Arabic Image Captioning with Gemini Decoder
Mohamed, A., Alwajih, F., Nagoudi, E. M. B., Inciarte, A. A., & Abdul-Mageed, M. (2023) · Arabic Natural Language Processing Conference (ArabicNLP) · doi:10.18653/v1/2023.arabicnlp-1.1 · Cited by 12 · see citing works
Learning Visual Abstract Reasoning through Dual-Stream Networks
Zhao, K., Xu, C., & Si, B. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i15.29641 · Cited by 9 · see citing works
“Models combined with language models can exhibit zero-shot or few-shot learning capabilities in such non-verbal reasoning tasks (Huang et al. 2023), and our framework can flexibly integrate multimodal information in the future.”
BackgroundPark, S.-H., Syazwany, N. S., Nam, J.-H., & Lee, S. (2024) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.109205 · Cited by 8 · see citing works
- Authors
- Huang, S., Dong, L., Wang, W., Hao, Y., Singhal, S., Ma, S., Lv, T., Cui, L., Mohammed, O. K., Patra, B., Liu, Q., Aggarwal, K., Chi, Z., Bjorck, J., Chaudhary, V., Som, S., Song, X., & Wei, F.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- Pages
- 72096-72109
- ISSN
- 1049-5258 (ISSN-L)
- arXiv
- 2302.14045
- OpenAlex
- W7133185515
- Cited by
- 235
- References
- 65
- Citations by year
- 2026: 37 · 2025: 47 · 2024: 95 · 2023: 55 · 2021: 1
- FWCI
- 86.06
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- neural information processing systems (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Categorization, perception, and language
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- language models, perception, multimodal alignment, vision-language models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
