The DOI matches its DataCite record; 69 of the 69 references it cites were found in a registry or an index.
Language Is Not All You Need: Aligning Perception with Language Models
Open accessAbstract
A big convergence of language, multimodal perception, action, and world modeling is a key step toward artificial general intelligence. In this work, we introduce Kosmos-1, a Multimodal Large Language Model (MLLM) that can perceive general modalities, learn in context (i.e., few-shot), and follow instructions (i.e., zero-shot). Specifically, we train Kosmos-1 from scratch on web-scale multimodal corpora, including arbitrarily interleaved text and images, image-caption pairs, and text data. We evaluate various settings, including zero-shot, few-shot, and multimodal chain-of-thought prompting, on a wide range of tasks without any gradient updates or finetuning. Experimental results show that Kosmos-1 achieves impressive performance on (i) language understanding, generation, and even OCR-free NLP (directly fed with document images), (ii) perception-language tasks, including multimodal dialogue, image captioning, visual question answering, and (iii) vision tasks, such as image recognition with descriptions (specifying classification via text instructions). We also show that MLLMs can benefit from cross-modal transfer, i.e., transfer knowledge from language to multimodal, and from multimodal to language. In addition, we introduce a dataset of Raven IQ test, which diagnoses the nonverbal reasoning capability of MLLMs.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Furthermore, developing more efficient transformer architectures tailored for multimodal data, such as Kosmos-1 [162], Muse [163], and PixArt-α [164], presents a viable solution.”
From Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook. (2024) · cites this paper as [162]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Huang et al. [77] introduced KOSMOS-1, which is capable of understanding general patterns, following instructions, and learning based on context.”
From A Survey on Evaluation of Large Language Models (2024) · cites this paper as [77]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · cited 3×
“6 Zero-Shot Image Classification We report the zero-shot image classification performance on ImageNet [DDS09].”
BackgroundMethodMicrosoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460 · cited 2×
“Image Captioning Table 2a shows the captioning performance on COCO [39] Karpathy test split and Flickr30k [40] test set.”
BackgroundLanguage Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515 · cited 1×
“We preprocess the data in the “full-sentence” format [LOG19], which packs each input sequence with full sentences that are sampled continuously from one or more documents.”
MethodLearning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293 · cited 3×
“In recent years, vision-language learning and representation models has garnered significant attention [2, 3, 4, 34, 50, 51, 52, 53, 54].”
BackgroundMethodRecursive Deep Models for Semantic Compositionality Over a Sentiment Treebank
Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A., & Potts, C. (2013) · Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d13-1170 · Cited by 6,878
CIDEr: Consensus-based image description evaluation
Vedantam, R., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7299087 · Cited by 4,845 · cited 3×
“Specifically, our model achieves a CIDEr score of 67.1 on the Flickr30k dataset, compared to 60.6 and 61.5 for the Flamingo-3B and Flamingo-9B models, respectively.”
MethodInfluentialChain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505 · cited 2×
“LLMs also serve as basic reasoners [10], which is complementary to perception modules on complex tasks.”
BackgroundKudo, T., & Richardson, J. (2018) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-2012 · Cited by 4,454 · cited 2×
“We adopt a learning rate warm-up and decay schedule, and use SentencePiece for tokenization.”
MethodChain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263
Flagged first, then by citations
Research works printed but not linked: 5
- Elia Bruni, Gemma Boleda, Marco Baroni, and Nam Khanh Tran. Distributional semantics in technicolor. In ACL, 2012.
- Patricia A Carpenter, Marcel A Just, and Peter Shell. What one intelligence test measures: a theoretical account of the processing in the raven progressive matrices test. Psychological review, 97(3):404, 1990.
- Hector Levesque, Ernest Davis, and Leora Morgenstern. The winograd schema chal- lenge. In Thirteenth International Conference on the Principles of Knowledge Repre- sentation and Reasoning, 2012.
- Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S. Gordon. Choice of plau- sible alternatives: An evaluation of commonsense causal reasoning. In AAAI Spring Symposium, 2011.
- Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. WinoGrande: An adversarial winograd schema challenge at scale. In AAAI, pages 8732-8740, 2020.
Cited by
Sorted by citations
Items 101–167 of 167
Park, S.-H., Syazwany, N. S., Nam, J.-H., & Lee, S. (2024) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.109205 · Cited by 8
Xu, J., Zhou, X., Shen, Y., Gu, X., Arnab, A., Sun, C., Wang, X., & Schmid, C. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01238 · Cited by 23
“Building large vision and language models that are generally capable of visual captioning or question answering is a trending research topic [7, 10, 19, 39, 44, 46, 54, 57].”
BackgroundTeikari, P., & Pietrusz, A. (2021) · SportRxiv (OSF Preprints) · doi:10.31236/osf.io/w734a · Cited by 14
Tudor, I., Dlab, M. H., Đurović, G., & Horvat, M. (2025) · Electronics · doi:10.3390/electronics14112281 · Cited by 7
Automatic Prompt Engineering for Automatic Scoring
Xue, M., Liu, Y., Xiao, X., & Wilson, M. R. (2025) · Journal of Educational Measurement · doi:10.1111/jedm.70002 · Cited by 11
Do Multimodal Large Language Models and Humans Ground Language Similarly?
Jones, C. R., Bergen, B. K., & Trott, S. (2024) · Computational Linguistics · doi:10.1162/coli_a_00531 · Cited by 11
Prospects and technology of embodied intelligent humanoid robots driven by AI large models
Kai, Z. E. N. G., Wang, Y., Tan, H., Fang, Q., Wang, Y., & Yuan, L. (2025) · Scientia Sinica Informationis · doi:10.1360/ssi-2024-0350 · Cited by 6
Effective Generative AI: The Human-Algorithm CentaurPreprint
Saghafian, S. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4587250 · Cited by 7
Zhi, Z., Liu, Z., Elbadawi, M., Daneshmend, A., Orlu, M., Basit, A. W., Demosthenous, A., & Rodrigues, M. R. D. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.130502 · Cited by 8
Unraveling the thread: understanding and addressing sequential failures in human-robot interaction
Tisserand, L., Stephenson, B., Baldauf-Quilliatre, H., Lefort, M., & Armetta, F. (2024) · Frontiers in Robotics and AI · doi:10.3389/frobt.2024.1359782 · Cited by 9
IMAD: Image-Augmented Multi-Modal Dialogue
Moskvoretskii, V., Frolov, A., & Kuznetsov, D. (2024) · Journal of Mathematical Sciences · doi:10.1007/s10958-024-07434-0 · Cited by 8
“Modern models [26, 28, 27] focus on connecting multiple modalities in one model.”
Zhang, S., Wu, Y., Zhang, X., Feng, Z., Wan, L., & Zhuang, Z. (2024) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2024.3385436 · Cited by 5
“KOSMOS-1 [42] uses a transformer to align and integrate image and text information, allowing for cross-modal interaction after encoding.”
MethodA Comprehensive Evaluation of Multimodal Large Language Models in Hydrological ApplicationsPreprint
Kadiyala, L., Mermer, O., Samuel, R. D. J., Sermet, Y., & Demir, İ. (2024) · doi:10.31223/x5tq37 · Cited by 5
Zeng, X., Wang, X., Zhang, T., Yan, Y., & Chen, Y. (2024) · Virtual Reality & Intelligent Hardware · doi:10.1016/j.vrih.2024.10.001 · Cited by 4
From Pixels to Insights: Unsupervised Knowledge Graph Generation with Large Language Model
Chen, L., Chen, Z., Wei, Y., Liu, S., & Li, Y. (2025) · Information · doi:10.3390/info16050335 · Cited by 3
“The integration of LLMs as decoders in vision–language tasks has surged in popularity, leveraging cross-modal learning to facilitate knowledge exchange between the language and visual domains [23–25].”
Towards an Exhaustive Evaluation of Vision-Language Foundation Models
Salin, E., Ayache, S., & Favre, B. (2023) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw60793.2023.00041 · Cited by 4
“In this paper, we specifically consider vision-language foundation models, which use visual and textual inputs [92, 17, 49, 39, 97, 2, 53].”
MethodHierarchical Perceptual and Predictive Analogy-Inference Network for Abstract Visual Reasoning
He, W., Ren, J., Bai, R., & Jiang, X. (2024) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3664647.3681246 · Cited by 4
Wang, H., Guo, B., Zeng, Y., Chen, M., Ding, Y., Zhang, Y., Yao, L., & Yu, Z. (2025) · ACM Transactions on Information Systems · doi:10.1145/3715098 · Cited by 7
Understand and Detect: Multi-step zero-shot detection with image-level specific prompt
Guo, M., Wu, K., Jiang, Z., Wang, H., & Men, A. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.113083 · Cited by 4
A Multimodal Large Language Model for Forestry in pest and disease recognitionPreprint
Zhang, M., Li, X., Du, Y., Rao, Z., Chen, S., Wang, W., Chen, X., Huang, D., & Wang, S. (2023) · Research Square · doi:10.21203/rs.3.rs-3444472/v1 · Cited by 5
Self-citationArtificial General Intelligence (AGI) for Medical Education and TrainingPreprint
Lema, K. G. (2023) · AfricArXiv (OSF Preprints) · doi:10.31730/osf.io/xytzm · Cited by 5
The Evolution of Language Models: From N-Grams to LLMs, and BeyondPreprint
Raeini, M. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4625356 · Cited by 4
Mai, G. (2024) · Wiley eBooks · doi:10.1002/9781118786352.wbieg2206 · Cited by 6
Mao, S., Zhang, C., Su, H., Song, H., Shalyminov, I., & Cai, W. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72973-7_27 · Cited by 5
Evaluating Human-Large Language Model Alignment in Group Process
He, Y., Liu, Y., Ouyang, C., Liu, H., Han, W., Gao, Y., Zhu, C., Tang, Y., Zhong, J., Zhou, S., et al. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-9434-8_32 · Cited by 3
Zero-Shot Transfer for Wildlife Bioacoustics DetectionPreprint
Miao, Z., Elizalde, B., Deshmukh, S., Kitzes, J. A., Wang, H., Dodhia, R. M., & Ferres, J. L. (2023) · Research Square · doi:10.21203/rs.3.rs-3180218/v1 · Cited by 3
SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Robotics
Shukor, M., Aubakirova, D., Capuano, F., Kooijmans, P., Palma, S., Zouitine, A., Aractingi, M., Pascal, C., Russi, M., Marafioti, A., et al. (2025) · arXiv · doi:10.48550/arxiv.2506.01844 · Cited by 695
“…has thus been extended to other modalities, sparking interest towards multi-modal vision-language (VLMs) (Alayrac et al., 2022; Chen et al., 2023; Huang et al., 2023; Liu et al., 2023b; Chen et al., 2024; Shukor et al., 2023b) and audio-language models (ALMs) (Défossez et al., 2024; Das et al.,…”
VLIS: Unimodal Language Models Guide Multimodal Language Generation
Chung, J., & Yu, Y. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.46 · Cited by 2
Embedding an Ethical Mind: Aligning Text-to-Image Synthesis via Lightweight Value Optimization
Wang, X., Yi, X., Xie, X., & Jia, J. (2024) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3664647.3681652 · Cited by 3
de Vicente, A., Camperos, M. C., & de la Mata, D. C. (2024) · European Public & Social Innovation Review · doi:10.31637/epsir-2025-782 · Cited by 12
Yin, K., Ji, X., Ding, Z., Hou, S., & Wang, Z. (2025) · Information Sciences · doi:10.1016/j.ins.2025.122601 · Cited by 3
Gatha: Relational Loss for enhancing text-based style transfer
Jandial, S., Deshmukh, S., Java, A., Shahid, S., & Krishnamurthy, B. (2023) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw59228.2023.00362 · Cited by 4
Multimodal Learning for Accurate Visual Question Answering: An Attention-based Approach
Bhardwaj, J., Balakrishnan, A., Pathak, S., Unnarkar, I., Gawande, A., & Ahmadnia, B. (2023) · International conference Recent advances in natural language processing · doi:10.26615/978-954-452-092-2_020 · Cited by 2
“Huang et al. (2023) introduced a framework called “Kosmos-1” for VQA task that aligns perception with language models.”
MethodCan Cross-Lingual Transferability of Multilingual Transformers Be Activated Without End-Task Data?
Chi, Z., Huang, H., & Mao, X.-L. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.796 · Cited by 1
Self-citationVisual Storytelling with Question-Answer Plans
Liu, D., Lapata, M., & Keller, F. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.386 · Cited by 6
Hayat, A., Akil, S., Martinez, H., Khan, B., & Hasan, M. R. (2024) · ASEE Annual Conference & Exposition · doi:10.18260/1-2--46841 · Cited by 1
Grounding Deliberate Reasoning in Multimodal Large Language Models
Chen, J., Liu, Y., Li, D., An, X., Deng, W., Feng, Z., Zhao, Y., & Yin, X. (2024) · Lecture notes in computer science · doi:10.1007/978-981-96-2061-6_2 · Cited by 29
Self-citationFoundation Model Based Camouflaged Object Detection
Chen, Z., Li, Z., Xue, Y., et al. (2025) · IET Computer Vision · doi:10.1049/cvi2.70009 · Cited by 1
Exploring Alternative Approaches to Language Modeling for Learning from Data and Knowledge
Zi, Y., Roy, K., Narayanan, V., & Sheth, A. (2024) · Proceedings of the AAAI Symposium Series · doi:10.1609/aaaiss.v3i1.31211 · Cited by 1
“The implicit representation of concept features by language model parame-Copyright ters, like fruits providing natural sugars, presents challenges in aligning the model’s parameter space to achieve desired outcomes (Huang et al. 2023).”
BackgroundMulti-modal Auto-regressive Modeling via Visual Tokens
Peng, T., Li, Z., Zhang, L., Zhao, H., Wang, P. Q., & Du, B. (2024) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3664647.3681685 · Cited by 6
Каково это — быть рассуждающей языковой моделью?
Д.В., З. (2024) · Логико-философские штудии · doi:10.52119/lphs.2024.51.82.003 · Cited by 1
Continually learn to map visual concepts to language models in resource-constrained environments
Rebillard, C., Hurtado, J., Krutsylo, A., Passaro, L., & Lomonaco, V. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.131013 · Cited by 2
“This fundamental insight has been further refined and expanded [31, 32, 33] achieving state-of-the-art performance in multiple tasks [34].”
Towards Unified Objectives for Self-Reflective AIPreprint
Samwald, M., Praas, R., & Hebenstreit, K. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4446991 · Cited by 2
A Multimodal Commodity Hybrid Recommender System Incorporating Μ2net+(Vit-L/16)Preprint
Gao, L. T., Li, H., Chen, Q., et al. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4825404 · Cited by 1
ECHo: A Visio-Linguistic Dataset for Event Causality Inference via Human-Centric Reasoning
Xie, Y., Li, G., & Kan, M. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.268 · Cited by 4
Lightening the Load: Lightweighting Multimodal Understanding for Visual Grounding Tasks
Pragati, P., & Singh, N. (2024) · International Conference on Data Engineering and Communication Systems (ICDECS) · doi:10.1109/icdecs59733.2023.10503418
Pill: Plug into Llm with Adapter Expert and Attention GatePreprint
Yin, Y., Zhang, F., Wu, Z., et al. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4813421
Research on power system applications based on ChatGPT
Li, B., Jiang, Y. H., Song, X., Sheng, Z., & Wei, Y. (2024) · International Conference on Energy System, Electricity, and Power (ESEP) · doi:10.1117/12.3024345
“In the current year, Microsoft introduced Kosmos-1[33], Google unveiled the PaLM-E[34] visual language model, and on April 18, APUS released its in-house multi-modal AI large model, "AiLMe."”
MethodSelf-citationA Chit-Chat between Llama2 and Chatgpt for the Automated Creation of ExploitsPreprint
Romano, S. P., Caturano, F., & Ciotola, J. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4886571
Rzepka, R., Shinto, R., & Araki, K. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-65572-2_16
Automatic Emotion Analysis in Movies: Matteo Garrone’s Dogman as a Case Study
Hromei, C. D., Forciniti, A., Margiotta, D., & Locati, S. (2024) · Communications in computer and information science · doi:10.1007/978-3-031-66705-3_6
Unified Human-Centric Model, Framework and Benchmark: A Survey
Zhao, X., Sulaiman, S., & Leng, W. Y. (2024) · IEEE Access · doi:10.1109/access.2024.3450123
“…cloud and so on, to interpret human behavior [5], pose estimation [6], [7], [8], gestures [9], interactions [10], [11], ReID [12], human parsing [13], [14], pedestrian detection [15], language involved task [16], [17], [18], etc. Research on UniHCM has made significant progress in recent years.”
BackgroundBoosting Representation Learning via Similarity-based Active Data Sampling from the Web
Ueno, S., & Kato, K. (2024) · International Workshop on Intelligent Systems (IWIS) · doi:10.1109/iwis62722.2024.10706063
Advancing Instance-Based Learning: Dynamic Environment Representation with Vision Language ModelsPreprint
bagherzadehkhorasani, A., & Tehranchi, F. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.5055985
Similarity based task timely termination method for image based intelligent agents
Jie, S., Huang, X., Jing, C., Jiang, X., & Dong, L. (2024) · Scientific Reports · doi:10.1038/s41598-024-83463-8
Self-citationEfficient Training of Lightweight Multimodal Models and Their Application to Downstream Tasks
梁瀬, 和., JIKUYA, K., OMOTE, H., TSUCHIDA, H., & Kato, K. (2025) · Journal of the Japan Society for Precision Engineering · doi:10.2493/jjspe.91.81
Stimulating conversation-style emergencies of multi-modal LMs
湯浅, 順., Liu, B., Sun, C., Xu, Z., & Wang, B. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103047
Efficient Patch Pruning for Vision Transformers via Patch SimilarityPreprint
Han, H., & Xiuying, S. (2025) · Preprints.org · doi:10.20944/preprints202504.0596.v1
Small pre-trained model for background understanding in multi-round question answering
Huang, X., Song, H., & Lu, M. (2025) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2024.1308206
VisCon-100K: Leveraging Contextual Web Data for Fine-Tuning Vision Language Models
Kumar, G. K., Chaabane, I., & Wu, K. (2025) · Lecture notes in computer science · doi:10.1007/978-981-96-8298-0_31
A Novel Visual-Enhanced Dual Stream Long-Term Decision Framework for Large Language Model Agents
Wang, X., Zhao, J., Shi, J., Li, L., & Zeng, D. D. (2025) · Lecture notes in computer science · doi:10.1007/978-981-96-6599-0_23
Bunzel, N. (2025) · IEEE/IFIP International Conference on Dependable Systems and Networks Workshops (DSN-W) · doi:10.1109/dsn-w65791.2025.00072
“Both the image and text embeddings must be aligned and combined to produce the final output [21], [3], [22].”
Kodela, K. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5360155
Zhou, M., Qi, Z., Zhu, T., Vijg, J., & Huang, X. (2025) · Mathematics · doi:10.3390/math13172739
FiT: Feature Integration Transformer with Universal Language Interface for Multi-Task VisionPreprint
Cheema, S., Gilanie, G., Alsahfi, T., Alesawi, S., Alsini, R. A. I., & Daud, A. Y. (2025) · Research Square · doi:10.21203/rs.3.rs-7602995/v1
Kim, H. R., & Ko, B. C. (2025) · Sensors · doi:10.3390/s25206353 · Cited by 1
“Kosmos-1 [14] jointly trains images and text within a single transformer architecture to support end-to-end grounding, captioning, and OCR.”
Back into Plato's Cave: Examining Cross-modal Representational Convergence at Scale
Koepke, A. S., Zverev, D., Ginosar, S., & Efros, A. A. (2026) · arXiv · doi:10.48550/arxiv.2604.18572
“These caveats echo grounding arguments that text-only learning may be insufficient to recover perceptual structure [7, 53], and motivate multimodal foundation models that integrate perception and language at scale [41, 4, 68, 37].”
Metadata
- Authors
- Huang, S., Dong, L., Wang, W., Hao, Y., Singhal, S., Ma, S., Lv, T., Cui, L., Mohammed, O. K., Patra, B., Liu, Q., Aggarwal, K., Chi, Z., Bjorck, J., Chaudhary, V., Som, S., Song, X., & Wei, F.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- Pages
- 72096-72109
- ISSN
- 1049-5258 (ISSN-L)
- arXiv
- 2302.14045
- OpenAlex
- W7133185515
- Cited by
- 235
- References
- 65
- Citations by year
- 2026: 37 · 2025: 47 · 2024: 95 · 2023: 55
- FWCI
- 86.06
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- neural information processing systems (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Categorization, perception, and language
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- language models, perception, multimodal alignment, vision-language models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
