The DOI matches its DataCite record; 69 of the 69 references it cites were found in a registry or an index.
Language Is Not All You Need: Aligning Perception with Language Models
Open accessAbstract
A big convergence of language, multimodal perception, action, and world modeling is a key step toward artificial general intelligence. In this work, we introduce Kosmos-1, a Multimodal Large Language Model (MLLM) that can perceive general modalities, learn in context (i.e., few-shot), and follow instructions (i.e., zero-shot). Specifically, we train Kosmos-1 from scratch on web-scale multimodal corpora, including arbitrarily interleaved text and images, image-caption pairs, and text data. We evaluate various settings, including zero-shot, few-shot, and multimodal chain-of-thought prompting, on a wide range of tasks without any gradient updates or finetuning. Experimental results show that Kosmos-1 achieves impressive performance on (i) language understanding, generation, and even OCR-free NLP (directly fed with document images), (ii) perception-language tasks, including multimodal dialogue, image captioning, visual question answering, and (iii) vision tasks, such as image recognition with descriptions (specifying classification via text instructions). We also show that MLLMs can benefit from cross-modal transfer, i.e., transfer knowledge from language to multimodal, and from multimodal to language. In addition, we introduce a dataset of Raven IQ test, which diagnoses the nonverbal reasoning capability of MLLMs.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Furthermore, developing more efficient transformer architectures tailored for multimodal data, such as Kosmos-1 [162], Muse [163], and PixArt-α [164], presents a viable solution.”
From Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook. (2024) · cites this paper as [162]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Huang et al. [77] introduced KOSMOS-1, which is capable of understanding general patterns, following instructions, and learning based on context.”
From A Survey on Evaluation of Large Language Models (2024) · cites this paper as [77]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · cited 3×
“6 Zero-Shot Image Classification We report the zero-shot image classification performance on ImageNet [DDS09].”
BackgroundMethodMicrosoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460 · cited 2×
“Image Captioning Table 2a shows the captioning performance on COCO [39] Karpathy test split and Flickr30k [40] test set.”
BackgroundLanguage Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515 · cited 1×
“We preprocess the data in the “full-sentence” format [LOG19], which packs each input sequence with full sentences that are sampled continuously from one or more documents.”
MethodLearning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293 · cited 3×
“In recent years, vision-language learning and representation models has garnered significant attention [2, 3, 4, 34, 50, 51, 52, 53, 54].”
BackgroundMethodRecursive Deep Models for Semantic Compositionality Over a Sentiment Treebank
Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A., & Potts, C. (2013) · Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d13-1170 · Cited by 6,878
CIDEr: Consensus-based image description evaluation
Vedantam, R., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7299087 · Cited by 4,845 · cited 3×
“Specifically, our model achieves a CIDEr score of 67.1 on the Flickr30k dataset, compared to 60.6 and 61.5 for the Flamingo-3B and Flamingo-9B models, respectively.”
MethodInfluentialChain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505 · cited 2×
“LLMs also serve as basic reasoners [10], which is complementary to perception modules on complex tasks.”
BackgroundKudo, T., & Richardson, J. (2018) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-2012 · Cited by 4,454 · cited 2×
“We adopt a learning rate warm-up and decay schedule, and use SentencePiece for tokenization.”
MethodChain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263
IEEE Transactions on Pattern Analysis and Machine Intelligence
Fu, K.-S. (2004) · doi:10.1002/0471667196.ess1206 · Cited by 3,552 · cited 1×
“We use the test set of COCO Karpathy split [KFF17], which re-partitions the train2014 and val2014 images [LMB14] into 113,287, 5,000, and 5,000 for the training set, validation set, and test set, respectively.”
MethodGaussian Error Linear Units (GELUs)
Hendrycks, D., & Gimpel, K. (2016) · arXiv · doi:10.48550/arxiv.1606.08415 · Cited by 3,537
PaLM: Scaling Language Modeling with Pathways
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.02311 · Cited by 2,590 · cited 1×
“Apart from the capabilities found in previous LLMs [1, 11], MLLMs enable new usages and possibilities.”
BackgroundYoung, P., Lai, A., Hodosh, M., & Hockenmaier, J. (2014) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00166 · Cited by 2,439 · cited 2×
“Image Captioning Table 2a shows the captioning performance on COCO [39] Karpathy test split and Flickr30k [40] test set.”
BackgroundSPICE: Semantic Propositional Image Caption Evaluation
Anderson, P., Fernando, B., Johnson, M., & Gould, S. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46454-1_24 · Cited by 1,986 · cited 1×
“We use COCOEvalCap4 to compute CIDEr [VLZP15] and SPICE [AFJG16] scores as the evaluation metrics.”
MethodConceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning
Sharma, P., Ding, N., Goodman, S., & Soricut, R. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1238 · Cited by 1,879 · cited 1×
“The image-caption pairs are constructed from several datasets, including English LAION-2B [19], LAION-400M [20], COYO-700M [21], and Conceptual Captions [22, 23].”
MethodInstructBLIP: Towards General-purpose Vision-Language Models with Instruction TuningPreprint
Dai, W., Li, J., Li, D., Tiong, A. M. H., Zhao, J., Wang, W., Li, B., Fung, P., & Hoi, S. C. H. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.06500 · Cited by 1,372 · cited 1×
“Meanwhile, some work [57, 58, 59] introduce visual instruction tuning to enhance instruction following capabilities.”
BackgroundSelf-citationSuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems
Wang, A., Pruksachatkun, Y., Nangia, N., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. R. (2019) · arXiv · doi:10.48550/arxiv.1905.00537 · Cited by 1,051 · cited 1×
“e, PIQA), and three datasets BoolQ, CB, and COPA from the SuperGLUE benchmark [WPN19].”
MethodDeep Visual-Semantic Alignments for Generating Image Descriptions
Karpathy, A., & Fei-Fei, L. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2598339 · Cited by 1,048 · cited 2×
“Metric Zero-shot Few-shot Language tasks StoryCloze [26] Commonsense reasoning Accuracy ✓ ✓ HellaSwag [27] Commonsense NLI Accuracy ✓ ✓ Winograd [28] Word ambiguity Accuracy ✓ ✓ Winogrande [29] Word ambiguity Accuracy ✓ ✓ PIQA [30] Physical commonsense Accuracy ✓ ✓ BoolQ [31] Question answering Accuracy ✓ ✓ CB [32] Te…”
BackgroundLAION-5B: An open large-scale dataset for training next generation image-text models
Schuhmann, C., Beaumont, R., Vencu, R., Gordon, C., Wightman, R., Cherti, M., Coombes, T., Katta, A., Mullis, C., Wortsman, M., et al. (2022) · arXiv · doi:10.48550/arxiv.2210.08402 · Cited by 1,022 · cited 1×
“The image-caption pairs are constructed from several datasets, including English LAION-2B [19], LAION-400M [20], COYO-700M [21], and Conceptual Captions [22, 23].”
MethodFlamingo: a visual language model for few-shot learning
Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1723 · Cited by 966
Li, J., Li, D., Savarese, S., & Hoi, S. (2023) · arXiv · doi:10.48550/arxiv.2301.12597 · Cited by 908 · cited 2×
“More importantly, unlocking multimodal input [2, 3, 4, 5, 6, 7] greatly widens the applications of language models to more high-value areas, such as multimodal machine learning, document intelligence, and robotics.”
BackgroundSelf-citationMegatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
Shoeybi, M., Patwary, M., Puri, R., LeGresley, P., Casper, J., & Catanzaro, B. (2019) · arXiv · doi:10.48550/arxiv.1909.08053 · Cited by 893 · cited 1×
“We also include the Common Crawl snapshots (2020-50 and 2021-04) datasets, CC-Stories, and RealNews datasets [17, 18].”
MethodImageBind One Embedding Space to Bind Them All
Girdhar, R., El-Nouby, A., Liu, Z., Singh, M., Alwala, K. V., Joulin, A., & Misra, I. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01457 · Cited by 863 · cited 1×
“In recent years, vision-language learning and representation models has garnered significant attention [2, 3, 4, 34, 50, 51, 52, 53, 54].”
BackgroundHellaSwag: Can a Machine Really Finish Your Sentence?
Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., & Choi, Y. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1472 · Cited by 830 · cited 1×
“Metric Zero-shot Few-shot Language tasks StoryCloze [26] Commonsense reasoning Accuracy ✓ ✓ HellaSwag [27] Commonsense NLI Accuracy ✓ ✓ Winograd [28] Word ambiguity Accuracy ✓ ✓ Winogrande [29] Word ambiguity Accuracy ✓ ✓ PIQA [30] Physical commonsense Accuracy ✓ ✓ BoolQ [31] Question answering…”
BackgroundConceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts
Changpinyo, S., Sharma, P. K., Ding, N., & Soricut, R. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00356 · Cited by 701
Liu, H., Li, C., Wu, Q., & Lee, Y. J. (2023) · arXiv · doi:10.48550/arxiv.2304.08485 · Cited by 682 · cited 1×
“Meanwhile, some work [57, 58, 59] introduce visual instruction tuning to enhance instruction following capabilities.”
BackgroundBoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions
Clark, C., Lee, K., Chang, M.-W., Kwiatkowski, T., Collins, M., & Toutanova, K. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1300 · Cited by 655 · cited 1×
“…✓ ✓ Winograd [28] Word ambiguity Accuracy ✓ ✓ Winogrande [29] Word ambiguity Accuracy ✓ ✓ PIQA [30] Physical commonsense Accuracy ✓ ✓ BoolQ [31] Question answering Accuracy ✓ ✓ CB [32] Textual entailment Accuracy ✓ ✓ COPA [33] Causal reasoning Accuracy ✓ ✓ Rendered SST-2 [34] OCR-free…”
GIT: A Generative Image-to-text Transformer for Vision and LanguagePreprint
Wang, J., Yang, Z., Hu, X., Li, L., Lin, K., Gan, Z., Liu, Z., Liu, C., & Wang, L. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2205.14100 · Cited by 636 · cited 2×
“KOSMOS-1 significantly outperforms GIT [WYH22] by 4.”
Method, J., & Raven, J. (2003) · Springer eBooks · doi:10.1007/978-1-4615-0153-4_11 · Cited by 631
VizWiz Grand Challenge: Answering Visual Questions from Blind People
Gurari, D., Li, Q., Stangl, A. J., Guo, A., Lin, C., Grauman, K., Luo, J., & Bigham, J. P. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00380 · Cited by 621 · cited 5×
“Visual Question Answering Table 2b reports the visual question answering results on VQAv2 [41] and VizWiz [42].”
BackgroundMethodInfluentialPaLM-E: An Embodied Multimodal Language Model
Driess, D., Xia, F., Sajjadi, M. S. M., Lynch, C., Chowdhery, A., Ichter, B., Wahid, A., Tompson, J., Vuong, Q., Yu, T., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.03378 · Cited by 591 · cited 1×
“Another category of research focus on learning multimodality abilities based on LLMs [7, 55, 56].”
BackgroundThe Flan Collection: Designing Data and Methods for Effective Instruction TuningPreprint
Longpre, S., Hou, L., Vu, T., Webson, A., Chung, H. W., Tay, Y., Zhou, D., Le, Q. V., Zoph, B., Wei, J., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2301.13688 · Cited by 553
Unnatural Instructions: Tuning Language Models with (Almost) No Human LaborPreprint
Honovich, O., Scialom, T., Levy, O., & Schick, T. (2022) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2212.09689 · Cited by 516 · cited 1×
“To improve instruction-following capabilities, we perform language-only instruction tuning using Unnatural Instructions [24] and FLANv2 [25] datasets.”
MethodThe Pile: An 800GB Dataset of Diverse Text for Language Modeling
Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., et al. (2021) · arXiv · doi:10.48550/arxiv.2101.00027 · Cited by 492 · cited 1×
“Text Corpora We train our model with The Pile [16] and Common Crawl (CC).”
MethodMaking the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering
Goyal, Y., Khot, T., Agrawal, A., Summers-Stay, D., Batra, D., & Parikh, D. (2019) · International Journal of Computer Vision · doi:10.1007/s11263-018-1116-0 · Cited by 487 · cited 2×
“Visual Question Answering Table 2b reports the visual question answering results on VQAv2 [41] and VizWiz [42].”
MethodThe CommitmentBank: Investigating projection in naturally occurring discourse
de Marneffe, M.-C., Simons, M., & Tonhauser, J. (2019) · Digital Access to Libraries · doi:10.18148/sub/2019.v23i2.601 · Cited by 471 · cited 1×
“…Accuracy ✓ ✓ Winogrande [29] Word ambiguity Accuracy ✓ ✓ PIQA [30] Physical commonsense Accuracy ✓ ✓ BoolQ [31] Question answering Accuracy ✓ ✓ CB [32] Textual entailment Accuracy ✓ ✓ COPA [33] Causal reasoning Accuracy ✓ ✓ Rendered SST-2 [34] OCR-free sentiment classification Accuracy ✓…”
LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs
Schuhmann, C., Vencu, R., Beaumont, R., Kaczmarczyk, R., Mullis, C., Katta, A., Coombes, T., Jitsev, J., & Komatsuzaki, A. (2021) · arXiv · doi:10.48550/arxiv.2111.02114 · Cited by 358 · cited 3×
“Image-Caption Pairs The image-caption pairs are constructed from several datasets, including English LAION-2B [SBV22], LAION-400M [SVB21], COYO-700M [BPK22], and Conceptual Captions [SDGS18, CSDS21].”
MethodMultimodal Few-Shot Learning with Frozen Language Models
Tsimpoukelli, M., Menick, J., Cabi, S., Eslami, S. M. A., Vinyals, O., & Hill, F. (2021) · arXiv · doi:10.48550/arxiv.2106.13884 · Cited by 355 · cited 2×
“In recent years, vision-language learning and representation models has garnered significant attention [2, 3, 4, 34, 50, 51, 52, 53, 54].”
BackgroundThe hateful memes challenge: Detecting hate speech in multimodal memes
Kiela, D., Firooz, H., Mohan, A., Goswami, V., Singh, A., Ringshia, P., & Testuggine, D. (2020) · arXiv · doi:10.48550/arxiv.2005.04790 · Cited by 352 · cited 1×
“…entailment Accuracy ✓ ✓ COPA [33] Causal reasoning Accuracy ✓ ✓ Rendered SST-2 [34] OCR-free sentiment classification Accuracy ✓ HatefulMemes [35] OCR-free meme classification ROC AUC ✓ Cross-modal transfer RelativeSize [36] Commonsense reasoning (object size) Accuracy ✓ MemoryColor [37]…”
BackgroundSmith, S., Patwary, M., Norick, B., LeGresley, P., Rajbhandari, S., Casper, J., Liu, Z., Prabhumoye, S., Zerveas, G., Korthikanti, V., et al. (2022) · arXiv · doi:10.48550/arxiv.2201.11990 · Cited by 286
Self-citationRosenbloom, L. (2015) · Choice Reviews Online · doi:10.5860/choice.189890 · Cited by 273
Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers
Wang, C., Chen, S., Wu, Y., Zhang, Z., Zhou, L., Liu, S., Chen, Z., Liu, Y., Wang, H., Li, J., et al. (2023) · arXiv · doi:10.48550/arxiv.2301.02111 · Cited by 250
Self-citationPaLI: A Jointly-Scaled Multilingual Language-Image ModelPreprint
Chen, X., Wang, X., Changpinyo, S., Piergiovanni, A., Padlewski, P., Salz, D., Goodman, S., Grycner, A., Mustafa, B., Beyer, L., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2209.06794 · Cited by 235 · cited 1×
“In recent years, vision-language learning and representation models has garnered significant attention [2, 3, 4, 34, 50, 51, 52, 53, 54].”
BackgroundOn the Representation Collapse of Sparse Mixture of Experts
Chi, Z., Dong, L., Huang, S.-H., et al. (2022) · Neural Information Processing Systems · doi:10.48550/arxiv.2204.09179 · Cited by 223 · cited 1×
“In the future, we would like to scale up K OSMOS -1 in terms of model size [13, 14, 60], and integrate the speech [12] capability into K OSMOS -1.”
MethodSelf-citationCompressive Transformers for Long-Range Sequence Modelling
Rae, J. W., Potapenko, A., Jayakumar, S. M., & Lillicrap, T. P. (2019) · arXiv · doi:10.48550/arxiv.1911.05507 · Cited by 205
ZeroCap: Zero-Shot Image-to-Text Generation for Visual-Semantic Arithmetic
Tewel, Y., Shalev, Y., Schwartz, I., & Wolf, L. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01739 · Cited by 194
A Length-Extrapolatable TransformerPreprint
Sun, Y., Dong, L., Patra, B., Ma, S.-M., Huang, S.-H., Benhaim, A., Chaudhary, V., Song, X., & Wei, F. (2022) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2212.10554 · Cited by 192 · cited 1×
“Compared with the standard Transformer architecture, we include the following modifications: We use M AGNETO [14], a Transformer variant, as the backbone architecture and X P OS [15] relative position encoding for better long-context modeling.”
MethodSelf-citationJin, W., Cheng, Y., Shen, Y., Chen, W., & Ren, X. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.197 · Cited by 174
Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language TasksPreprint
Wang, W., Bao, H., Li, D., Björck, J., Peng, Z., Liu, Q., Aggarwal, K., Mohammed, O. K., Singhal, S., Som, S., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2208.10442 · Cited by 163 · cited 3×
“In recent years, vision-language learning and representation models has garnered significant attention [2, 3, 4, 34, 50, 51, 52, 53, 54].”
BackgroundSelf-citationNeural Codec Language Models are Zero-Shot Text to Speech Synthesizers
Chen, S., Wang, C., Wu, Y., Zhang, Z., Zhou, L., Liu, S., Chen, Z., Liu, T., Wang, H., Li, J., et al. (2025) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2025.3530270 · Cited by 162 · cited 2×
“In the future, we would like to scale up K OSMOS -1 in terms of model size [13, 14, 60], and integrate the speech [12] capability into K OSMOS -1.”
BackgroundMethodSelf-citationRetrieval-Augmented Multimodal Language ModelingPreprint
Yasunaga, M., Aghajanyan, A., Shi, W., James, R., Leskovec, J., Liang, P., Lewis, M., Zettlemoyer, L., & Yih, W.-T. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2211.12561 · Cited by 144
LSDSem 2017 Shared Task: The Story Cloze Test
Mostafazadeh, N., Roth, M., Louis, A. P., Chambers, N., & Allen, J. F. (2017) · LSDSem@EACL · doi:10.18653/v1/w17-0906 · Cited by 140 · cited 1×
“Metric Zero-shot Few-shot Language tasks StoryCloze [26] Commonsense reasoning Accuracy ✓ ✓ HellaSwag [27] Commonsense NLI Accuracy ✓ ✓ Winograd [28] Word ambiguity Accuracy ✓ ✓ Winogrande [29] Word ambiguity Accuracy ✓ ✓ PIQA [30] Physical commonsense Accuracy ✓ ✓ BoolQ [31] Question answering…”
BackgroundLanguage Models are General-Purpose InterfacesPreprint
Hao, Y., Song, H., Dong, L., Huang, S.-H., Chi, Z., Wang, W.-H., Ma, S.-M., & Wei, F. (2022) · arXiv.org · doi:10.48550/arxiv.2206.06336 · Cited by 117
Self-citationEnabling Multimodal Generation on CLIP via Vision-Language Knowledge Distillation
Dai, W., Hou, L., Shang, L.-F., et al. (2022) · Findings · doi:10.48550/arxiv.2203.06386 · Cited by 115
Self-citationDeepNet: Scaling Transformers to 1,000 Layers
Wang, H., Ma, S., Dong, L., Huang, S., Zhang, D., & Wei, F. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3386927 · Cited by 113
Self-citationWebSRC: A Dataset for Web-Based Structural Reading Comprehension
Chen, X., Zhao, Z., Chen, L., Ji, J., Zhang, D., Luo, A., Xiong, Y., & Yu, K. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.343 · Cited by 97 · cited 6×
“We compare the performance on the Web-based Structural Reading Comprehension (WebSRC) dataset [43].”
BackgroundMethodInfluentialAre Elephants Bigger than Butterflies? Reasoning about Sizes of Objects
Bagherinezhad, H., Hajishirzi, H., Choi, Y., et al. (2016) · AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v30i1.10476 · Cited by 54 · cited 2×
“…SST-2 [34] OCR-free sentiment classification Accuracy ✓ HatefulMemes [35] OCR-free meme classification ROC AUC ✓ Cross-modal transfer RelativeSize [36] Commonsense reasoning (object size) Accuracy ✓ MemoryColor [37] Commonsense reasoning (object color) Accuracy ✓ ColorTerms [38] Commonsense…”
BackgroundMethodDeepNet: Scaling Transformers to 1,000 LayersPreprint
Wang, H., Ma, S., Li, D., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2203.00555 · Cited by 52
Self-citationPaLI-X: On Scaling up a Multilingual Vision and Language ModelPreprint
Chen, X., Djolonga, J., Padlewski, P., Mustafa, B., Changpinyo, S., Wu, J., Riquelme, C., Goodman, S., Wang, X., Tay, Y., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.18565 · Cited by 42 · cited 1×
“In recent years, vision-language learning and representation models has garnered significant attention [2, 3, 4, 34, 50, 51, 52, 53, 54].”
BackgroundCM3: A Causal Masked Multimodal Model of the Internet
Aghajanyan, A., Huang, B., Ross, C., Karpukhin, V., Xu, H., Goyal, N., Okhonko, D., Joshi, M., Ghosh, G., Lewis, M., et al. (2022) · arXiv · doi:10.48550/arxiv.2201.07520 · Cited by 42 · cited 1×
“More importantly, unlocking multimodal input [2, 3, 4, 5, 6, 7] greatly widens the applications of language models to more high-value areas, such as multimodal machine learning, document intelligence, and robotics.”
BackgroundGrounding Language Models to Images for Multimodal Inputs and Outputs
Koh, J. Y., Salakhutdinov, R., & Fried, D. (2023) · arXiv · doi:10.48550/arxiv.2301.13823 · Cited by 38
PIQA: Reasoning about Physical Commonsense in Natural Language
Bisk, Y., Zellers, R., Le bras, R., Gao, J., & Choi, Y. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6239 · Cited by 37 · cited 1×
“…Accuracy ✓ ✓ HellaSwag [27] Commonsense NLI Accuracy ✓ ✓ Winograd [28] Word ambiguity Accuracy ✓ ✓ Winogrande [29] Word ambiguity Accuracy ✓ ✓ PIQA [30] Physical commonsense Accuracy ✓ ✓ BoolQ [31] Question answering Accuracy ✓ ✓ CB [32] Textual entailment Accuracy ✓ ✓ COPA [33] Causal reasoning…”
Visually-Augmented Language ModelingPreprint
Wang, W., Li, D., Cheng, H., Song, H., Liu, X., Yan, X., Gao, J., & Wei, F. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2205.10178 · Cited by 25 · cited 1×
“Model Size Reasoning Color Reasoning RELATIVESIZE MEMORYCOLOR COLORTERMS Using retrieved images VALM [WDC23] 85.”
MethodSelf-citationFoundation TransformersPreprint
Wang, H., Ma, S., Huang, S., Li, D., Wang, W., Peng, Z., Wu, Y., Bajaj, P., Singhal, S., Benhaim, A., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2210.06423 · Cited by 23 · cited 2×
“In the future, we would like to scale up K OSMOS -1 in terms of model size [13, 14, 60], and integrate the speech [12] capability into K OSMOS -1.”
MethodSelf-citationTransferring Knowledge from Vision to Language: How to Achieve it and how to Measure it?
Norlund, T., Hagström, L., & Johansson, R. (2021) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/2021.blackboxnlp-1.10 · Cited by 17 · cited 3×
“Metric Zero-shot Few-shot Language tasks StoryCloze [26] Commonsense reasoning Accuracy ✓ ✓ HellaSwag [27] Commonsense NLI Accuracy ✓ ✓ Winograd [28] Word ambiguity Accuracy ✓ ✓ Winogrande [29] Word ambiguity Accuracy ✓ ✓ PIQA [30] Physical commonsense Accuracy ✓ ✓ BoolQ [31] Question answering Accuracy ✓ ✓ CB [32] Te…”
MethodAre Elephants Bigger than Butterflies? Reasoning about Sizes of ObjectsPreprint
Bagherinezhad, H., Hajishirzi, H., Choi, Y., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1602.00753 · Cited by 17
TorchScale: Transformers at ScalePreprint
Ma, S., Wang, H., Huang, S., Wang, W., Chi, Z., Li, D., Benhaim, A., Patra, B., Chaudhary, V., Song, X., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2211.13184 · Cited by 5 · cited 2×
“In the future, we would like to scale up K OSMOS -1 in terms of model size [13, 14, 60], and integrate the speech [12] capability into K OSMOS -1.”
MethodSelf-citationBMJ Publishing Group Ltd, & Royal College of Psychiatrists and British Psychological Society (2008) · Evidence-Based Mental Health · doi:10.1136/ebmh.11.4.102 · Cited by 3 · cited 3×
“Text Corpora We train our model with The Pile [16] and Common Crawl (CC).”
Method
Research works printed but not linked: 5
- Elia Bruni, Gemma Boleda, Marco Baroni, and Nam Khanh Tran. Distributional semantics in technicolor. In ACL, 2012.
- Patricia A Carpenter, Marcel A Just, and Peter Shell. What one intelligence test measures: a theoretical account of the processing in the raven progressive matrices test. Psychological review, 97(3):404, 1990.
- Hector Levesque, Ernest Davis, and Leora Morgenstern. The winograd schema chal- lenge. In Thirteenth International Conference on the Principles of Knowledge Repre- sentation and Reasoning, 2012.
- Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S. Gordon. Choice of plau- sible alternatives: An evaluation of commonsense causal reasoning. In AAAI Spring Symposium, 2011.
- Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. WinoGrande: An adversarial winograd schema challenge at scale. In AAAI, pages 8732-8740, 2020.
Cited by
Sorted by citations
A Survey on Evaluation of Large Language Models
Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., et al. (2024) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3641289 · Cited by 4,254
Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B., et al. (2025) · ACM Transactions on Information Systems · doi:10.1145/3703155 · Cited by 4,688
The rise and potential of large language model based agents: a survey
Xi, Z., Chen, W., Guo, X., He, W., Ding, Y., Hong, B., Zhang, M., Wang, J., Jin, S., Zhou, E., et al. (2025) · Science China Information Sciences · doi:10.1007/s11432-024-4222-0 · Cited by 970
“The second approach to endowing an LLM-based agent with visual understanding capabilities is modality fusion, where a visual encoder is integrated with an LLM at the embedding level [209,210].”
Self-citationVideo-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding
Zhang, H., Li, X. S., & Bing, L. (2023) · Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP) · doi:10.18653/v1/2023.emnlp-demo.49 · Cited by 744
A Survey on Large Language Models: Applications, Challenges, Limitations, and Practical UsagePreprint
Hadi, M. U., Al-Tashi, Q., Qureshi, R., Shah, A., Muneer, A., Irfan, M., Zafar, A., Shaikh, M. B., Akhtar, N., Hassan, S. Z., et al. (2023) · doi:10.36227/techrxiv.23589741.v1 · Cited by 475
Metadata
- Authors
- Huang, S., Dong, L., Wang, W., Hao, Y., Singhal, S., Ma, S., Lv, T., Cui, L., Mohammed, O. K., Patra, B., Liu, Q., Aggarwal, K., Chi, Z., Bjorck, J., Chaudhary, V., Som, S., Song, X., & Wei, F.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- Pages
- 72096-72109
- ISSN
- 1049-5258 (ISSN-L)
- arXiv
- 2302.14045
- OpenAlex
- W7133185515
- Cited by
- 235
- References
- 65
- Citations by year
- 2026: 37 · 2025: 47 · 2024: 95 · 2023: 55
- FWCI
- 86.06
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- neural information processing systems (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Categorization, perception, and language
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- language models, perception, multimodal alignment, vision-language models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
