The DOI matches its Crossref record; 1 of the 1 references it cites were found in a registry or an index.
Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Models
Open accessAbstract
Today’s most advanced vision-language models (VLMs) remain proprietary. The strongest open-weight models rely heavily on synthetic data from proprietary VLMs to achieve good performance, effectively distilling these closed VLMs into open ones. As a result, the community has been missing foundational knowledge about how to build performant VLMs from scratch. We present Molmo, a new family of VLMs that are state-of-the-art in their class of openness. Our key contribution is a collection of new datasets called PixMo, including a dataset of highly detailed image captions for pre-training, a free-form image Q&A dataset for fine-tuning, and an innovative 2D pointing dataset, all collected without the use of external VLMs. The success of our approach relies on careful modeling choices, a well- tuned training pipeline, and, most critically, the quality of our newly collected datasets. Our best-in-class 72B model not only outperforms others in the class of open weight and data models, but also outperforms larger proprietary models including Claude 3.5 Sonnet, and Gemini 1.5 Pro and Flash, second only to GPT-4o based on both academic benchmarks and on a large human evaluation. Our model weights, new datasets, and source code are available at https://molmo.allenai.org/blog.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Llama from Meta [53] 4 and Molmo from the Allen Institute for AI [35], which both exhibit comparable performance to closed-source industry models on benchmarks.”
From "It's trained by non-disabled people": Evaluating How Image Quality Affects Product Captioning with Vision-Language Models (2026) · cites this paper as [35]
The cited source says what the claim attributes to it.
In the citing paper
“For instance, Molmo already applies an overlapping cropping procedure in its training [35], which we would expect to make it more resistant to misframed images, but our fi ndings demonstrate that further development is needed to address its sensitivity to image framing.”
From "It's trained by non-disabled people": Evaluating How Image Quality Affects Product Captioning with Vision-Language Models (2026) · cites this paper as [35]
The cited source says what the claim attributes to it.
In the citing paper
“VLM performance is heavily shaped by post-training activities, including fine-tuning on specific tasks (e.g., PixMoCap for captioning [35]) and diverse datasets [80], or training to provide answers in specific formats (e.g., instruction tuning [84]).”
From "It's trained by non-disabled people": Evaluating How Image Quality Affects Product Captioning with Vision-Language Models (2026) · cites this paper as [35]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 125
- Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadal- lah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219, 2024. 6, 14, 20 · arXiv
- Manoj Acharya, Kushal Kafle, and Christopher Kanan. TallyQA: Answering complex counting questions. In AAAI, 2019. 5
- Pravesh Agrawal, Szymon Antoniak, Emma Bou Hanna, Devendra Chaplot, Jessica Chudnovsky, Saurabh Garg, Theophile Gervet, So- ham Ghosh, Amélie Héliou, Paul Jacob, et al. Pixtral 12b. arXiv preprint arXiv:2410.07073, 2024. 6, 14, 20 · arXiv
- AI, :, Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, Kaidong Yu, Peng Liu, and more. Yi: Open foundation models by 01.ai. arXiv preprint arXiv:2403.04652, 2024. 14 · arXiv
- Meta AI. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. 6, 12, 14, 15, 17, 19, 20 · arXiv
- Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Mil- lican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. In NeurIPS, 2022. 19
- Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. In NeurIPS Deep Learning Symposium, 2016. 10
- Lucas Beyer, Andreas Steiner, André Susano Pinto, Alexander Kolesnikov, Xiao Wang, Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, Thomas Unterthiner, Daniel Keysers, Skanda Koppula, Fangyu Liu, Adam Grycner, Alexey Gritsenko, Neil Houlsby, Manoj Kumar, Keran Rong, Julian Eisenschlos, Rishabh Kabra, Matthias Bauer, Matko Bošnjak, Xi Chen, Matthias Minderer, Paul Voigtlaender, Ioana Bica, Ivana Balazevic, Joan Puigcerver, Pinelopi Papalam- pidi, Olivier Henaff, Xi Xiong, Radu Soricut, Jeremiah Harmsen, and Xiaohua Zhai. PaliGemma: A versatile 3B VLM for transfer. arXiv preprint arXiv:2407.07726, 2024. 1, 4, 5, 6, 12, 14, 17, 19, 20 · arXiv
- Ali Furkan Biten, Ruben Tito, Andres Mafla, Lluis Gomez, Marc ¸al Rusinol, Ernest Valveny, CV Jawahar, and Dimosthenis Karatzas. Scene text visual question answering. In ICCV, 2019. 5
- Junbum Cha, Wooyoung Kang, Jonghwan Mun, and Byungseok Roh. Honeybee: Locality-enhanced projector for multimodal llm. In CVPR, 2024. 19
- Guiming Hardy Chen, Shunian Chen, Ruifei Zhang, Junying Chen, Xiangbo Wu, Zhiyi Zhang, Zhihong Chen, Jianquan Li, Xiang Wan, and Benyou Wang. Allava: Harnessing gpt4v-synthesized data for a lite vision-language model. arXiv preprint arXiv:2402.11684, 2024. 20 · arXiv
- Kaibing Chen, Dong Shen, Hanwen Zhong, Huasong Zhong, Kui Xia, Di Xu, Wei Yuan, Yifei Hu, Bin Wen, Tianke Zhang, Changyi Liu, Dewen Fan, Huihui Xiao, Jiahong Wu, Fan Yang, Size Li, and Di Zhang. Evlm: An efficient vision-language model for visual understanding. arXiv preprint arXiv:2407.14177, 2024. 19 · arXiv
- Lin Chen, Jisong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Ji- aqi Wang, Feng Zhao, and Dahua Lin. ShareGPT4V: Improv- ing large multi-modal models with better captions. arXiv preprint arXiv:2311.12793, 2023. 1, 17, 20 · arXiv
- Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Hen- rique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cum- mings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Pe- ter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language mod- els trained on code. arXiv preprint arXiv:2107.03374, 2021. 16 · arXiv
- Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. Microsoft COCO captions: Data collection and evaluation server. arXiv preprint arXiv:1504.00325, 2015. 2, 4, 11 · arXiv
- Xi Chen, Xiao Wang, Lucas Beyer, Alexander Kolesnikov, Jialin Wu, Paul Voigtlaender, Basil Mustafa, Sebastian Good- man, Ibrahim Alabdulmohsin, Piotr Padlewski, et al. Pali-3 vi- sion language models: Smaller, faster, stronger. arXiv preprint arXiv:2310.09199, 2023. 19 · arXiv
- Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, Ji Ma, Jiaqi Wang, Xiaoyi Dong, Hang Yan, Hewei Guo, Conghui He, Botian Shi, Zhenjiang Jin, Chao Xu, Bin Wang, Xingjian Wei, Wei Li, Wenjian Zhang, Bo Zhang, Pinlong Cai, Licheng Wen, Xi- angchao Yan, Min Dou, Lewei Lu, Xizhou Zhu, Tong Lu, Dahua Lin, Yu Qiao, Jifeng Dai, and Wenhai Wang. How far are we to GPT-4V? closing the gap to commercial multimodal models with open-source suites. arXiv preprint arXiv:2404.16821, 2024. 3 · arXiv
- Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Worts- man, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. In CVPR, 2023. 18
- Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E Gonzalez, and Ion Stoica. Chatbot arena: An open platform for evaluating LLMs by human preference. In ICML, 2024. 6, 13
- Xiangxiang Chu, Limeng Qiao, Xinyang Lin, Shuang Xu, Yang Yang, Yiming Hu, Fei Wei, Xinyu Zhang, Bo Zhang, Xiaolin Wei, and Chunhua Shen. Mobilevlm : A fast, strong and open vision language assistant for mobile devices. arXiv preprint arXiv:2312.16886, 2023. 19 · arXiv
- Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sab- harwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018. 16 · arXiv
- Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Ja- cob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schul- man. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. 16 · arXiv
- David F Crouse. On implementing 2d rectangular assignment algo- rithms. IEEE Transactions on Aerospace and Electronic Systems, 2016. 14
- Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Albert Li, Pascale Fung, and Steven C. H. Hoi. Instructblip: Towards general-purpose vision- language models with instruction tuning. In NeurIPS, 2023. 18
- Wenliang Dai, Nayeon Lee, Boxin Wang, Zhuoling Yang, Zihan Liu, Jon Barker, Tuomas Rintamaki, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. NVLM: Open frontier-class multimodal LLMs. arXiv preprint arXiv:2409.11402, 2024. 5 · arXiv
- Tri Dao. FlashAttention-2: Faster attention with better parallelism and work partitioning. In ICLR, 2024. 10
- Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. FlashAttention: Fast and memory-efficient exact attention with IO-awareness. In NeurIPS, 2022. 10
- Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Songyang Zhang, Haodong Duan, Wenwei Zhang, Yining Li, Hang Yan, Yang Gao, Zhe Chen, Xinyue Zhang, Wei Li, Jingwen Li, Wenhai Wang, Kai Chen, Conghui He, Xingcheng Zhang, Jifeng Dai, Yu Qiao, Dahua Lin, and Jiaqi Wang. InternLM- XComposer2-4KHD: A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HD. arXiv preprint arXiv:2404.06512, 2024. 3 · arXiv
- Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa De- hghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021. 3
- Yunhao Fang, Ligeng Zhu, Yao Lu, Yan Wang, Pavlo Molchanov, Jan Kautz, Jang Hyun Cho, Marco Pavone, Song Han, and Hongxu Yin. Vila 2 : Vila augmented vila. arXiv preprint arXiv:2407.17453, 2024. · arXiv
- Andrea Frome, Greg S Corrado, Jon Shlens, Samy Bengio, Jeff Dean, Marc'Aurelio Ranzato, and Tomas Mikolov. Devise: A deep visual-semantic embedding model. NeurIPS, 2013. 18
- Chaoyou Fu, Haojia Lin, Zuwei Long, Yunhang Shen, Meng Zhao, Yifan Zhang, Xiong Wang, Di Yin, Long Ma, Xiawu Zheng, et al. Vita: Towards open-source interactive omni multimodal llm. arXiv preprint arXiv:2408.05211, 2024. 19 · arXiv
- Tao Ge, Xin Chan, Xiaoyang Wang, Dian Yu, Haitao Mi, and Dong Yu. Scaling synthetic data creation with 1,000,000,000 personas. arXiv preprint arXiv:2406.20094, 2024. 18 · arXiv
- Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the V in VQA matter: Elevating the role of image understanding in visual question answering. In CVPR, 2017. 5, 6
- Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rod- ney Kinney, Oyvind Tafjord, A. Jha, Hamish Ivison, Ian Magnus- son, Yizhong Wang, Shane Arora, David Atkinson, Russell Au- thur, Khyathi Raghavi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack Hessel, Tushar Khot, William Mer- rill, Jacob Daniel Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew E. Peters, Valentina Pyatkin, Abhilasha Ravichander, Dustin Schwenk, Saurabh Shah, Will Smith, Emma Strubell, Nishant Subramani, Mitchell Wortsman, Pradeep Dasigi, Nathan Lambert, Kyle Richardson, Luke Zettlemoyer, Jesse Dodge, Kyle Lo, Luca Soldaini, Noah A. Smith, and Hanna Hajishirzi. OLMo: Accelerating the science of language models. In ACL, 2024. 3, 10, 20
- Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In ICLR, 2021. 16
- Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Mea- suring mathematical problem solving with the math dataset. In NeurIPS Track on Datasets and Benchmarks, 2021. 16
- Joeri R Hermans, Gerasimos Spanakis, and Rico Möckel. Accu- mulated gradient normalization. In Asian Conference on Machine Learning, pages 439-454. PMLR, 2017. 10
- Wenyi Hong, Weihan Wang, Ming Ding, Wenmeng Yu, Qingsong Lv, Yan Wang, Yean Cheng, Shiyu Huang, Junhui Ji, Zhao Xue, et al. Cogvlm2: Visual language models for image and video un- derstanding. arXiv preprint arXiv:2408.16500, 2024. 18 · arXiv
- Anwen Hu, Haiyang Xu, Jiabo Ye, Mingshi Yan, Liang Zhang, Bo Zhang, Chen Li, Ji Zhang, Qin Jin, Fei Huang, and Jingren Zhou. mplug-docowl 1.5: Unified structure learning for ocr-free document understanding. In Findings of EMNLP, 2024.
- Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In ICML, 2021. 18
- Dongfu Jiang, Xuan He, Huaye Zeng, Cong Wei, Max Ku, Qian Liu, and Wenhu Chen. Mantis: Interleaved multi-image instruction tuning. arXiv preprint arXiv:2405.01483, 2024. 20 · arXiv
- Kushal Kafle, Brian Price, Scott Cohen, and Christopher Kanan. DVQA: Understanding data visualizations via question answering. In CVPR, 2018. 5, 11, 20
- Samira Ebrahimi Kahou, Vincent Michalski, Adam Atkinson, Ákos Kádár, Adam Trischler, and Yoshua Bengio. FigureQA: An annotated figure dataset for visual reasoning. arXiv preprint arXiv:1710.07300, 2017. 5, 20 · arXiv
- Siddharth Karamcheti, Suraj Nair, Ashwin Balakrishna, Percy Liang, Thomas Kollar, and Dorsa Sadigh. Prismatic vlms: Inves- tigating the design space of visually-conditioned language models. arXiv preprint arXiv:2402.07865, 2024. 5, 19 · arXiv
- Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. A diagram is worth a dozen images. In ECCV, 2016. 5, 6
- Diederik P Kingma. Adam: A method for stochastic optimization. In ICLR, 2015. 5, 9
- Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexan- der C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick. Segment anything. In ICCV, 2023. 7, 14
- Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Li Fei-Fei. Vi- sual genome: Connecting language and vision using crowdsourced dense image annotations. International Journal of Computer Vision, 123:32 -73, 2016. 11, 20
- Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, et al. The open images dataset v4: Unified image classification, object detection, and visual relation- ship detection at scale. IJCV, 2020. 13
- Hugo Laurenc ¸on, Andrés Marafioti, Victor Sanh, and Léo Tron- chon. Building and better understanding vision-language models: insights and future directions. arXiv preprint arXiv:2408.12637, 2024. 20 · arXiv
- Bo Li, Peiyuan Zhang, Jingkang Yang, Yuanhan Zhang, Fanyi Pu, and Ziwei Liu. Otterhd: A high-resolution multi-modality model. arXiv preprint arXiv:2311.04219, 2023. 19 · arXiv
- Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Fanyi Pu, Jingkang Yang, C. Li, and Ziwei Liu. Mimic-it: Multi-modal in- context instruction tuning. arXiv preprint arXiv:2306.05425, 2023. 20 · arXiv
- Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Jingkang Yang, and Ziwei Liu. Otter: A multi-modal model with in-context instruction tuning. arXiv preprint arXiv:2305.03726, 2023. 19 · arXiv
- Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. LLaVA-OneVision: Easy visual task transfer. arXiv preprint arXiv:2408.03326, 2024. 5, 6, 14, 18, 20 · arXiv
- Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Boot- strapping language-image pre-training for unified vision-language understanding and generation. In ICML, 2022.
- Junyan Li, Delin Chen, Yining Hong, Zhenfang Chen, Peihao Chen, Yikang Shen, and Chuang Gan. Covlm: Composing visual entities and relationships in large language models via communicative de- coding. In ICLR, 2024. 20
- Wei Li, William Bishop, Alice Li, Chris Rawles, Folawiyo Campbell-Ajala, Divya Tyamagundlu, and Oriana Riva. On the effects of data scale on computer control agents. arXiv preprint arXiv:2406.03679, 2024. 5, 7 · arXiv
- Zhang Li, Biao Yang, Qiang Liu, Zhiyin Ma, Shuo Zhang, Jingxu Yang, Yabo Sun, Yuliang Liu, and Xiang Bai. Monkey: Image resolution and text label are important things for large multi-modal models. CVPR, 2024.
- Bin Lin, Zhenyu Tang, Yang Ye, Jiaxi Cui, Bin Zhu, Peng Jin, Junwu Zhang, Munan Ning, and Li Yuan. Moe-llava: Mix- ture of experts for large vision-language models. arXiv preprint arXiv:2401.15947, 2024. 19 · arXiv
- Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Mi- crosoft coco: Common objects in context. In ECCV, 2014. 13
- Chang Liu, Henghui Ding, and Xudong Jiang. GRES: Generalized referring expression segmentation. In CVPR, 2023. 18, 20
- Dongyang Liu, Renrui Zhang, Longtian Qiu, Siyuan Huang, Weifeng Lin, Shitian Zhao, Shijie Geng, Ziyi Lin, Peng Jin, Kaipeng Zhang, Wenqi Shao, Chao Xu, Conghui He, Junjun He, Hao Shao, Pan Lu, Yu Qiao, Hongsheng Li, and Peng Gao. SPHINX-x: Scaling data and parameters for a family of multi- modal large language models. In ICML, 2024. 19
- Fuxiao Liu, Xiaoyang Wang, Wenlin Yao, Jianshu Chen, Kaiqiang Song, Sangwoo Cho, Yaser Yacoob, and Dong Yu. Mmc: Ad- vancing multimodal chart understanding with large-scale instruc- tion tuning. arXiv preprint arXiv:2311.10774, 2023. 20 · arXiv
- Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurIPS, 2023. 1, 3, 5, 6, 14, 18, 19, 20
- Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In CVPR, 2024. 3
- Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.03983, 2016. 9 · arXiv
- Ilya Loshchilov and Frank Hutter. Decoupled weight decay regu- larization. In ICLR, 2019. 5, 9
- Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Hao Yang, et al. Deepseek-vl: towards real-world vision-language understanding. arXiv preprint arXiv:2403.05525, 2024. 19 · arXiv
- Jiasen Lu, Christopher Clark, Sangho Lee, Zichen Zhang, Savya Khosla, Ryan Marten, Derek Hoiem, and Aniruddha Kembhavi. Unified-io 2: Scaling autoregressive multimodal models with vision language audio and action. In CVPR, 2024. 20
- Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for sci- ence question answering. In NeurIPS, 2022. 5
- Pan Lu, Liang Qiu, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, Tanmay Rajpurohit, Peter Clark, and Ashwin Kalyan. Dy- namic prompt learning via policy gradient for semi-structured math- ematical reasoning. In ICLR, 2023. 5
- Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Han- naneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. MathVista: Evaluating mathematical reasoning of foundation models in visual contexts. In ICLR, 2024. 6
- Gen Luo, Yiyi Zhou, Tianhe Ren, Shengxin Chen, Xiaoshuai Sun, and Rongrong Ji. Cheap and quick: Efficient vision-language in- struction tuning for large language models. In NeurIPS, 2023. 19
- Chaitanya Malaviya, Subin Lee, Sihao Chen, Elizabeth Sieber, Mark Yatskar, and Dan Roth. ExpertQA: Expert-curated questions and attributed answers. arXiv preprint arXiv:2309.07852, 2023. 20 · arXiv
- Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. OK-VQA: A visual question answering benchmark re- quiring external knowledge. In CVPR, 2019. 5
- Ahmed Masry, Do Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. ChartQA: A benchmark for question answering about charts with visual and logical reasoning. In ACL, 2022. 5, 6
- Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. DocVQA: A dataset for VQA on document images. In WACV, 2021. 5, 6
- Minesh Mathew, Viraj Bagal, Rubèn Tito, Dimosthenis Karatzas, Ernest Valveny, and CV Jawahar. InfographicVQA. In WACV, 2022. 5, 6
- Brandon McKinzie, Zhe Gan, Jean-Philippe Fauconnier, Sam Dodge, Bowen Zhang, Philipp Dufter, Dhruti Shah, Xianzhi Du, Futang Peng, Floris Weers, Anton Belyi, Haotian Zhang, Karan- jeet Singh, Doug Kang, Ankur Jain, Hongyu He, Max Schwarzer, Tom Gunter, Xiang Kong, Aonan Zhang, Jianyu Wang, Chong Wang, Nan Du, Tao Lei, Sam Wiseman, Guoli Yin, Mark Lee, Zirui Wang, Ruoming Pang, Peter Grasch, Alexander Toshev, and Yinfei Yang. Mm1: Methods, analysis & insights from multimodal llm pre-training. arXiv preprint arXiv:2403.09611, 2024. 3, 18 · arXiv
- Nitesh Methani, Pritha Ganguly, Mitesh M Khapra, and Pratyush Kumar. PlotQA: Reasoning over scientific plots. In WACV, 2020. 5, 20
- Niklas Muennighoff, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Jacob Morrison, Sewon Min, Weijia Shi, Pete Walsh, Oyvind Tafjord, Nathan Lambert, Yuling Gu, Shane Arora, Akshita Bha- gia, Dustin Schwenk, David Wadden, Alexander Wettig, Binyuan Hui, Tim Dettmers, Douwe Kiela, Ali Farhadi, Noah A. Smith, Pang Wei Koh, Amanpreet Singh, and Hannaneh Hajishirzi. OL- MoE: Open mixture-of-experts language models. arXiv preprint arXiv:2409.02060, 2024. 3 · arXiv
- OpenAI. GPT-4 technical report. arXiv preprint arXiv:2303.08774, 2023. 1, 6, 14 · arXiv
- OpenAI. GPT-4o system card. arXiv preprint arXiv:2410.21276, 2024. 3, 6, 14, 19 · arXiv
- Maxime Oquab, Timothée Darcet, Theo Moutakanni, Huy V. Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haz- iza, Francisco Massa, Alaaeldin El-Nouby, Russell Howes, Po-Yao Huang, Hu Xu, Vasu Sharma, Shang-Wen Li, Wojciech Galuba, Mike Rabbat, Mido Assran, Nicolas Ballas, Gabriel Synnaeve, Is- han Misra, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. DINOv2: Learning robust visual fea- tures without supervision. arXiv preprint arXiv:2304.07193, 2023. 19 · arXiv
- Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world. arXiv preprint arXiv:2306.14824, 2023. 18, 20 · arXiv
- Jordi Pont-Tuset, Jasper Uijlings, Soravit Changpinyo, Radu Sori- cut, and Vittorio Ferrari. Connecting vision and language with lo- calized narratives. In ECCV, 2020. 4, 11
- Shraman Pramanick, Guangxing Han, Rui Hou, Sayan Nag, Ser- Nam Lim, Nicolas Ballas, Qifan Wang, Rama Chellappa, and Am- jad Almahairi. Jack of all tasks, master of many: Designing general- purpose coarse-to-fine vision-language model. In CVPR, 2023. 20
- Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervi- sion. In ICML, 2021. 3, 18
- Hanoona Rasheed, Muhammad Maaz, Sahal Shaji, Abdelrahman Shaker, Salman Khan, Hisham Cholakkal, Rao M Anwer, Eric Xing, Ming-Hsuan Yang, and Fahad S Khan. GLaMM: Pixel grounding large multimodal model. In CVPR, 2024. 20
- Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. LAION-5B: An open large-scale dataset for training next generation image-text models. arXiv preprint arXiv:2210.08402, 2022. 17 · arXiv
- Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A-OKVQA: A benchmark for vi- sual question answering using world knowledge. In ECCV, 2022. 5
- Amanpreet Singh, Vivek Natarjan, Meet Shah, Yu Jiang, Xinlei Chen, Devi Parikh, and Marcus Rohrbach. Towards VQA models that can read. In CVPR, 2019. 5, 6
- Quan Sun, Yufeng Cui, Xiaosong Zhang, Fan Zhang, Qiying Yu, Zhengxiong Luo, Yueze Wang, Yongming Rao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative multimodal models are in- context learners. In CVPR, 2024. [103] Gemini Team. Gemini 1.5: Unlocking multimodal under- standing across millions of tokens of context. arXiv preprint arXiv:2403.05530, 2024. 3, 6, 14, 19 · arXiv
- Rahul Thapa, Kezhen Chen, Ian Covert, Rahul Chalamala, Ben Athiwaratkun, Shuaiwen Leon Song, and James Zou. Dragonfly: Multi-resolution zoom-in encoding enhances vision-language mod- els. arXiv preprint arXiv:2406.00977, 2024. 18 · arXiv
- Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, et al. Cambrian-1: A fully open, vision-centric exploration of multimodal LLMs. In NeurIPS, 2024. 5, 6, 12, 14, 19, 20
- Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcom- ings of multimodal LLMs. In CVPR, 2024. 18
- Maria Tsimpoukelli, Jacob Menick, Serkan Cabi, S. M. Ali Eslami, Oriol Vinyals, Felix Hill, and Zacharias Janssen. Multimodal few- shot learning with frozen language models. In NeurIPS, 2021.
- Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, 2017. 3, 18
- Junke Wang, Lingchen Meng, Zejia Weng, Bo He, Zuxuan Wu, and Yu-Gang Jiang. To see is to believe: Prompting gpt-4v for better visual instruction tuning. arXiv preprint arXiv:2311.07574, 2023. 20 · arXiv
- Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024. 6, 7, 14, 20 · arXiv
- Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, Jiazheng Xu, Bin Xu, Juanzi Li, Yuxiao Dong, Ming Ding, and Jie Tang. Cogvlm: Visual expert for pretrained language models. In NeurIPS, 2024. 18
- Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Chandu, David Wadden, Kelsey MacMillan, Noah A Smith, Iz Beltagy, et al. How far can camels go? exploring the state of instruction tuning on open resources. In NeurIPS Track on Datasets and Benchmarks, 2023. 15
- Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. MMLU-Pro: A more robust and challenging multi-task language understanding benchmark. In NeurIPS Track on Datasets and Benchmarks, 2024. 16
- Jason Weston, Samy Bengio, and Nicolas Usunier. Wsabie: Scaling up to large vocabulary image annotation. In IJCAI, 2011. 18
- Bin Xiao, Haiping Wu, Weijian Xu, Xiyang Dai, Houdong Hu, Yu- mao Lu, Michael Zeng, Ce Liu, and Lu Yuan. Florence-2: Advanc- ing a unified representation for a variety of vision tasks. In CVPR, 2024. 20
- Hu Xu, Saining Xie, Xiaoqing Tan, Po-Yao Huang, Russell Howes, Vasu Sharma, Shang-Wen Li, Gargi Ghosh, Luke Zettlemoyer, and Christoph Feichtenhofer. Demystifying CLIP data. In ICLR, 2024. 3, 18
- Le Xue, Manli Shu, Anas Awadalla, Jun Wang, An Yan, Senthil Purushwalkam, Honglu Zhou, Viraj Prabhu, Yutong Dai, Michael S Ryoo, et al. xGen-MM (BLIP-3): A family of open large multi- modal models. arXiv preprint arXiv:2408.08872, 2024. 6, 14, 20 · arXiv
- An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tian- hao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zhihao Fan. Qwen2 technical report. arXiv preprint arXiv:2407.10671, 2024. 3 · arXiv
- Charig Yang, Weidi Xie, and Andrew Zisserman. It's about time: Analog clock reading in the wild. In CVPR, 2022. 7, 13, 14, 20
- Huanjin Yao, Wenhao Wu, Taojiannan Yang, Yuxin Song, Mengxi Zhang, Haocheng Feng, Yifan Sun, Zhiheng Li, Wanli Ouyang, and Jingdong Wang. Dense connector for mllms. In NeurIPS, 2024.
- Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. Minicpm-v: A gpt-4v level mllm on your phone. arXiv preprint arXiv:2408.01800, 2024. 19 · arXiv
- Jiabo Ye, Anwen Hu, Haiyang Xu, Qinghao Ye, Ming Yan, Guohai Xu, Chenliang Li, Junfeng Tian, Qi Qian, Ji Zhang, et al. UReader: Universal OCR-free visually-situated language understanding with multimodal large language model. In Findings of EMNLP, 2023. 3
- Haoxuan You, Haotian Zhang, Zhe Gan, Xianzhi Du, Bowen Zhang, Zirui Wang, Liangliang Cao, Shih-Fu Chang, and Yinfei Yang. Ferret: Refer and ground anything anywhere at any granular- ity. arXiv preprint arXiv:2310.07704, 2023. 20 · arXiv
- Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L Berg. Modeling context in referring expressions. In ECCV, 2016. 18, 20
- Yuqian Yuan, Wentong Li, Jian Liu, Dongqi Tang, Xinjie Luo, Chi Qin, Lei Zhang, and Jianke Zhu. Osprey: Pixel understanding with visual instruction tuning. In CVPR, 2024. 20
- Zhengqing Yuan, Zhaoxu Li, Weiran Huang, Yanfang Ye, and Lichao Sun. Tinygpt-v: Efficient multimodal large language model via small backbones. In ICML Workshop on Advancing Neural Net- work Training, 2024. 19
- Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yux- uan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. MMMU: A massive multi- discipline multimodal understanding and reasoning benchmark for expert AGI. In CVPR, 2024. 6
- Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In ICCV, 2023. 3
- Haotian Zhang, Mingfei Gao, Zhe Gan, Philipp Dufter, Nina Wen- zel, Forrest Huang, Dhruti Shah, Xianzhi Du, Bowen Zhang, Yang- hao Li, Sam Dodge, Keen You, Zhen Yang, Aleksei Timofeev, Mingze Xu, Hong-You Chen, Jean-Philippe Fauconnier, Zhengfeng Lai, Haoxuan You, Zirui Wang, Afshin Dehghan, Peter Grasch, and Yinfei Yang. Mm1.5: Methods, analysis & insights from multi- modal llm fine-tuning. arXiv preprint arXiv:2409.20566, 2024. 18 · arXiv
- Pan Zhang, Xiaoyi Dong, Yuhang Zang, Yuhang Cao, Rui Qian, Lin Chen, Qipeng Guo, Haodong Duan, Bin Wang, Linke Ouyang, et al. Internlm-xcomposer-2.5: A versatile large vision language model supporting long-contextual input and output. arXiv preprint arXiv:2407.03320, 2024. 20 · arXiv
- Renrui Zhang, Jiaming Han, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, Peng Gao, and Yu Jiao Qiao. Llama- adapter: Efficient fine-tuning of large language models with zero- initialized attention. In ICLR, 2024. 19
- Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, et al. Pytorch fsdp: Experiences on scaling fully sharded data parallel. arXiv preprint arXiv:2304.11277, 2023. 10 · arXiv
- Xingyi Zhou, Rohit Girdhar, Armand Joulin, Philipp Krähenbühl, and Ishan Misra. Detecting twenty-thousand classes using image- level supervision. In ECCV, 2022. 4
- Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. In ICLR, 2024. 19, 20
Cited by
Sorted by citations
Items 201–300 of 558
OpenVision: A Fully-Open, Cost-Effective Family of Advanced Vision Encoders for Multimodal Learning
Li, X., Liu, Y., Tu, H., & Xie, C. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.00379 · Cited by 9
Corvid: Improving Multimodal Large Language Models Towards Chain-of-Thought Reasoning
Jiang, J., Ma, C., Song, X., Zhang, H., & Luo, J. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.00291 · Cited by 10
GLOV: Guided Large Language Models as Implicit Optimizers for Vision Language ModelsPreprint
Mirza, M. J., Zhao, M., Mao, Z., Doveh, S., Lin, W., Gavrikov, P., Dorkenwald, M., Yang, S., Jha, S., Wakaki, H., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2410.06154 · Cited by 9
KITAB-Bench: A Comprehensive Multi-Domain Benchmark for Arabic OCR and Document UnderstandingPreprint
Heakl, A., Sohail, A., Ranjan, M., Hossam, R., Ahmad, G. S., El-Geish, M., Maher, O., Shen, Z., Khan, F. S., & Khan, S. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2502.14949 · Cited by 9
The Spatial Blindspot of Vision-Language ModelsPreprint
Alam, N., Murali, L. K., Bharadwaj, S., Liu, P., Chung, T. S., Sharma, D., A, A. M., Kiran, K., Tam, W., & Vegesna, B. K. S. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2601.09954 · Cited by 9
Green-VLA: Staged Vision-Language-Action Model for Generalist RobotsPreprint
Apanasevich, I., Artemyev, M., Babakyan, R., Fedotova, P., Grankin, D., Kupryashin, E., Misailidi, A., Nerus, D., Nutalapati, A., Sidorov, G., et al. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2602.00919 · Cited by 9
Gemini Robotics: Bringing AI into the Physical World
Gemini Robotics Team, Abeyruwan, S., Ainslie, J., Alayrac, J.-B., Arenas, M. G., Armstrong, T., Balakrishna, A., Baruch, R., Bauza, M., Blokzijl, M., et al. (2025) · arXiv · doi:10.48550/arxiv.2503.20020 · Cited by 8
Agrawal, P., Antoniak, S., Hanna, E. B., Bout, B., Chaplot, D., Chudnovsky, J., Costa, D., De Monicault, B., Garg, S., Gervet, T., et al. (2024) · arXiv · doi:10.48550/arxiv.2410.07073 · Cited by 8
FieldWorkArena: Agentic AI Benchmark for Real Field Work Tasks
Takahashi, J., Moteki, A., Uchida, A., Masui, S., Yang, F., Uchino, K., Song, Y., Bisk, Y., Neubig, G., Kusajima, I., et al. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-31663-9_7 · Cited by 8
SAGE: A Visual Language Model for Anomaly Detection via Fact Enhancement and Entropy-aware Alignment
Zang, G., Li, X., Di, D., Nie, L., Zhan, D., Song, Y., & Fan, L. (2025) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3746027.3755725 · Cited by 8
How To Teach Large Multimodal Models New Skills?
Zhu, Z., Gong, Y., Xiao, Y., Liu, Y., & Hoiem, D. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37556-8_23 · Cited by 8
Kim, J., Park, J., Park, J., Lee, S. K., Chung, J., Kim, J., Joung, J. H., & Yu, Y. (2026) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2026.acl-long.251 · Cited by 8
Self-citationVIEW2SPACE: Studying Multi-view Visual Reasoning from Sparse Observations
Ke, F., Cai, Z., Li, B., Chen, L., Lin, B., Wang, W., Haghighi, P. D., Haffari, G., & Rezatofighi, H. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37531-5_29 · Cited by 8
Kan, Z., Liu, Y., Yin, K., Jiang, X., Li, X. Y., Cao, H., Liu, Y., Jiang, D., Sun, X. W., Liao, Q., et al. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2505.20777 · Cited by 8
SpinBench: Perspective and Rotation as a Lens on Spatial Reasoning in VLMsPreprint
Zhang, Y., Corcodel, R., Hori, C., Cherian, A., & Zhao, D. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2509.25390 · Cited by 8
Semantic World ModelsPreprint
Berg, J., Zhu, C., Bao, Y., Durugkar, I., & Gupta, A. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2510.19818 · Cited by 8
OmniBridge: Unified Multimodal Understanding, Generation, and Retrieval via Latent Space AlignmentPreprint
Xiao, T., Li, Z., & Zhang, L. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2509.19018 · Cited by 8
LExI: Layer-Adaptive Active Experts for Efficient MoE Model InferencePreprint
Chitty-Venkata, K. T., Madireddy, S., Emani, M. K., & Vishwanath, V. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2509.02753 · Cited by 8
Xing, Y., Wu, J., Ozdemir, S., Zhang, Y., Yang, Y., Shao, W., & Gong, K. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2601.09879 · Cited by 8
Self-citationST4VLA: Spatially Guided Training for Vision-Language-Action ModelsPreprint
Ye, J., Wang, F., Gao, N., Yu, J., Zhu, Y., Wang, B., Zhang, J., Jin, W., Fu, Y., Zheng, F., et al. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2602.10109 · Cited by 8
Self-citationAutomatically Detecting Online Deceptive Patterns
Nayak, A., Wani, Y. R., Zhang, S., Khandelwal, R., & Fawaz, K. (2025) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/3719027.3765191 · Cited by 7
OWLViz: An Open-World Benchmark for Visual Question Answering
Nguyen, T., Nguyen, D., Nguyen, H. H., Luong, T., Dernoncourt, F., Dang, L. H., & Lai, V. D. (2026) · Adaptive Agents and Multi-Agents Systems · doi:10.65109/epvo9609 · Cited by 7
Affogato: Open-Vocabulary Affordance Grounding with Automated Data Generation at Scale
Lee, J.-H., Park, E., Park, C., Kang, D., & Cho, M. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37419-6_28 · Cited by 7
DynamicVerse: A Physically-Aware Multimodal Framework for 4D World Modeling
Wen, K., Huang, Y., Chen, R., Zheng, H., Lin, Y., Pan, P., Li, C., Cong, W., Zhang, J., Lü, J., et al. (2025) · neural information processing systems · doi:10.52202/085713-3621 · Cited by 7
Picek, L., Kahl, S., Goëau, H., Adam, L., Larcher, T., Leblanc, C., Servajean, M., Janoušková, K., Matas, J., Čermák, V., et al. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-04354-2_19 · Cited by 7
Jiang, Q., Wu, L., Zeng, Z., Ren, T., Xiong, Y., Chen, Y., Qin, L., & Zhang, L. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.02012 · Cited by 7
Self-citationRAcQUEt: Unveiling the Dangers of Overlooked Referential Ambiguity in Visual LLMs
Testoni, A., Plank, B., & Fernández, R. (2025) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2025.emnlp-main.1206 · Cited by 7
How Well Do Vision-Language Models Understand Movies?
Zaranis, E., Farinhas, A., Santos, S., Canaverde, B., Ramos, M. M., Mohammed, W., Attanasio, G., Zerva, C., Sivakumaran, N., Yu, S., et al. (2026) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/2026.emnlp-main.1702 · Cited by 7
Understanding and Evaluating Hallucinations in 3D Visual Language ModelsPreprint
Peng, R., Li, K., Zhang, W., Gao, C., Chen, X., & Li, Y. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2502.15888 · Cited by 7
Behind Maya: Building a Multilingual Vision Language ModelPreprint
Alam, N., Kanjula, K. R., Guthikonda, S., Chung, T., Vegesna, B. K. S., Das, A., Susevski, A., Chan, R. S.-Y., Uddin, S. M. I., Islam, S. B., et al. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2505.08910 · Cited by 7
Vision-Speech Models: Teaching Speech Models to Converse about ImagesPreprint
Royer, A., Böhle, M., de Marmiesse, G., Mazaré, L., Zeghidour, N., Défossez, A., & Pérez, P. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2503.15633 · Cited by 7
UAV-VLPA*: A Vision-Language-Path-Action System for Optimal Route Generation on a Large ScalesPreprint
Sautenkov, O., Akhmetkazy, A., Yaqoot, Y., Mustafa, M. A., Tadevosyan, G., Lykov, A., & Tsetserukou, D. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2503.02454 · Cited by 7
Evaluation of Safety Cognition Capability in Vision-Language Models for Autonomous DrivingPreprint
Zhang, E., Gong, P., Dai, X., Huang, M., Yisheng, L. V., & Miao, Q. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2503.06497 · Cited by 7
WebMMU: A Benchmark for Multimodal Multilingual Website Understanding and Code GenerationPreprint
Awal, R., Massoud, M., Feizi, A., Li, Z., Wang, S., Pal, C., Agrawal, A., Vázquez, D., Reddy, S., Rodriguez, J. A., et al. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2508.16763 · Cited by 7
LocateAnything3D: Vision-Language 3D Detection with Chain-of-SightPreprint
Man, Y., Wang, S., Zhang, G., Bjorck, J., Li, Z., Gui, L.-Y., Fan, J., Kautz, J., Wang, Y.-X., & Yu, Z. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2511.20648 · Cited by 7
CoME: Empowering Channel-of-Mobile-Experts with Informative Hybrid-Capabilities ReasoningPreprint
Liu, Y., Xu, W., Huang, K., Chen, C., Zhao, J., Gao, P., Liu, W., Luan, J., Shang, S., Du, B., et al. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2602.24142 · Cited by 7
Tang, Y., Zhang, L., Zhang, S., Zhao, Y., & Hao, X. (2025) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3746027.3758209 · Cited by 6
Benchmarking and Mitigating MCQA Selection Bias of Large Vision-Language Models
Atabuzzaman, M., Asgarov, A., & Thomas, C. (2025) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2025.emnlp-main.1703 · Cited by 6
Unified Reinforcement and Imitation Learning for Vision-Language Models
Lee, B., Hachiuma, R., Ro, Y. M., Wang, Y.-C. F., & Wu, Y.-H. (2025) · neural information processing systems · doi:10.52202/085713-5231 · Cited by 6
From Panels to Prose: Generating Literary Narratives from Comics
Sachdeva, R., & Zisserman, A. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.02030 · Cited by 6
Leveraging Vision-Language Models for Open-Vocabulary Instance Segmentation and Tracking
Pätzold, B., Nogga, J., & Behnke, S. (2025) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2025.3606363 · Cited by 6
Li, B., & Zhang, W. (2026) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/2026.findings-emnlp.87 · Cited by 6
ViPlan: A Benchmark for Visual Planning with Symbolic Predicates and Vision-Language Models
Merler, M., Dainese, N., Alakuijala, M., Bonetta, G., Ferrazzi, P., Tian, Y., Magnini, B., & Marttinen, P. (2026) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/2026.findings-emnlp.1017 · Cited by 6
Vision‑Language‑Vision Auto‑Encoder: Scalable Knowledge Distillation from Diffusion Models
Zhang, T., Li, Y., Chou, Y.-C., Chen, J., Yuille, A., Wei, C., & Xiao, J. (2025) · neural information processing systems · doi:10.52202/085713-2648 · Cited by 6
Self-citationVARCO-VISION: Expanding Frontiers in Korean Vision-Language ModelsPreprint
Ju, J., Kim, D., Park, S., & Kim, Y. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2411.19103 · Cited by 6
$β$-CLIP: Text-Conditioned Contrastive Learning for Multi-Granular Vision-Language AlignmentPreprint
Zohra, F., Zhao, C., Itani, H., & Ghanem, B. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2512.12678 · Cited by 6
PairBench: Are Vision-Language Models Reliable at Comparing What They See?Preprint
Feizi, A., Rajeswar, S., Romero, A., Rabbany, R., Zantedeschi, V., Gella, S., & Monteiro, J. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2502.15210 · Cited by 6
VisCoder2: Building Multi-Language Visualization Coding AgentsPreprint
Ni, Y., Cai, S., Chen, X., Liang, J., Lyu, Z., Deng, J., Zou, K., Nie, P., Yuan, F., Yue, X., et al. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2510.23642 · Cited by 6
$π\mathbf{R}^2$: Reactive Real-time Flow PoliciesPreprint
Park, S., & Tulsiani, S. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2607.26055 · Cited by 6
VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understanding
Zhang, B., Li, K., Cheng, Z., Hu, Z., Yuan, Y., Chen, G., Leng, S., Jiang, Y., Zhang, H., Li, X., et al. (2025) · arXiv · doi:10.48550/arxiv.2501.13106 · Cited by 5
Ayesha, S., Aslam, A., Zaheer, M. H., & Khan, M. B. (2025) · Sensors · doi:10.3390/s25185845 · Cited by 5
Digital collections explorer: An open-source, multimodal viewer for searching digital collections
Huang, Y., & Lee, B. C. G. (2025) · Computational Humanities Research · doi:10.1017/chr.2025.10017 · Cited by 5
Köksal, A., & Alatan, A. A. (2025) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2025.3637115 · Cited by 5
MapleGrasp: Mask-guided Feature Pooling for Language-driven Efficient Robotic Grasping
Bhat, V., Patel, N., Krishnamurthy, P., Karri, R., & Khorrami, F. (2026) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv61042.2026.00729 · Cited by 5
RotBench: Evaluating Multi-modal Large Language Models on Identifying Image Rotation
Niu, T., Cho, J., Stengel-Eskin, E., & Bansal, M. (2026) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2026.eacl-long.259 · Cited by 5
InfiGUIAgent: A Multimodal Generalist GUI Agent with Native Reasoning and Reflection
Liu, Y., Li, P., Wei, Z., Xie, C., Hu, X., Xu, X., Zhang, S., Han, X., Yang, H., & Wu, F. (2026) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2026.eacl-long.45 · Cited by 5
MedicalNarratives: Connecting Medical Vision and Language with Localized Narratives
Ikezogwo, W. O., Zhang, K., & Seyfioğlu, M. S. (2025) · neural information processing systems · doi:10.52202/085713-4440 · Cited by 5
TIPO: Text to Image with Text Presampling for Prompt OptimizationPreprint
Yeh, S.-Y., Li, Y., Park, S., Oh, G., Wang, X., Song, M. K., Yu, Y., & Lai, S.-H. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2411.08127 · Cited by 5
Wu, X., Ding, Y., Li, B., Lu, P., Yin, D., Chang, K., & Peng, N. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2412.02172 · Cited by 5
Fast, Secure, and High-Capacity Image Watermarking with Autoencoded Text VectorsPreprint
Evennou, G., Chappelier, V., & Ewa Kijak, C. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2510.00799 · Cited by 5
Mind the Gap! Static and Interactive Evaluations of Large Audio ModelsPreprint
Li, M., Held, W. B., Ryan, M. J., Pipatanakul, K., Manakul, P., Zhu, H., & Yang, D. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2502.15919 · Cited by 5
MapIQ: Evaluating Multimodal Large Language Models for Map Question AnsweringPreprint
Srivastava, V., Fan, L., Mukhopadhyay, S., Gupta, V., & Maciejewski, R. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2507.11625 · Cited by 5
Mohanty, A., Parthasarathy, V. B., & Shahid, A. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2504.10179 · Cited by 5
Visual serial processing deficits explain divergences in human and VLM reasoningPreprint
Budny, N., Ghods, K., Campbell, D. I., Marjieh, R., Joshi, A., Kumar, S., Cohen, J. D., Webb, T. W., & Griffiths, T. L. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2509.25142 · Cited by 5
Promptception: How Sensitive Are Large Multimodal Models to Prompts?Preprint
Ismithdeen, M. I., Khattak, M. U., & Khan, S. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2509.03986 · Cited by 5
Diagnosing Bottlenecks in Data Visualization Understanding by Vision-Language ModelsPreprint
Tartaglini, A. R., Grant, S., Wurgaft, D., Potts, C., & Fan, J. E. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2510.21740 · Cited by 5
Same or Not? Enhancing Visual Perception in Vision-Language ModelsPreprint
Marsili, D., Mehta, A., Lin, R. Y., & Gkioxari, G. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2512.23592 · Cited by 5
Action-Sketcher: From Reasoning to Action via Visual Sketches for Long-Horizon Robotic ManipulationPreprint
Tan, H., Co, P. A., Xu, Y., Rong, S., Ji, Y., Chi, C., Chen, X., Zhang, Q., Zhao, Z., Wang, P., et al. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2601.01618 · Cited by 5
TrajTok: Learning Trajectory Tokens enables better Video UnderstandingPreprint
Zheng, C., Zhang, J., Zhang, J., Huang, W., Kumar, A., Kong, Q., Tuzel, O., Li, C.-L., & Krishna, R. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2602.22779 · Cited by 5
Self-citationFunctionality understanding and segmentation in 3D scenes
Corsetti, J., Giuliari, F., Fasoli, A., Boscaini, D., & Poiesi, F. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.02286 · Cited by 4
Confidence Calibration for Multimodal LLMs: An Empirical Study Through Medical VQA
Du, Y., Wang, Y., Kong, M. C., Liang, T., Long, Q., Chen, B., & Zhu, Q. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-04978-0_9 · Cited by 4
RESAnything: Attribute Prompting for Arbitrary Referring Segmentation
Wang, R., & Zhang, H. (2025) · neural information processing systems · doi:10.52202/085713-2243 · Cited by 4
Zero-Shot Fine-Grained Image Classification Using Large Vision-Language Models
Atabuzzaman, M., Zhang, A., & Thomas, C. (2025) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2025.findings-emnlp.1280 · Cited by 4
Entropy-Gradient Grounding: Training-Free Evidence Retrieval in Vision-Language Models
Gröpl, M., Jung, J., Kim, S., Pollefeys, M., & Hong, S. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37595-7_31 · Cited by 4
Transferability Between Understanding and Generation in Unified Multimodal Models
Kang, J., Yoon, H., Jung, J., Min, J., Jeon, M., Hwang, B., Jung, S., & Kim, S. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37132-4_35 · Cited by 4
Learning from Primitive: Probing Visual Reasoning of LVLMs via Counting
Che, L., Xue, Z., Quan, Y., Liu, B., Shi, Z., Tang, R., Krishna, R., & Pavlović, V. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37023-5_37 · Cited by 4
Self-citationViLU: Learning Vision-Language Uncertainties for Failure Prediction
Lafon, M., Karmim, Y., Silva-Rodríguez, J., Couairon, P., Rambour, C., Fournier-S’niehotta, R., Ayed, I. B., Dolz, J., & Thome, N. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.01654 · Cited by 4
Hyperphantasia: A Benchmark for Evaluating the Mental Visualization Capabilities of Multimodal LLMs
Sepehri, M. S., Tınaz, B., Fabian, Z., & Soltanolkotabi, M. (2025) · neural information processing systems · doi:10.52202/085713-0132 · Cited by 4
Direct Visual Grounding by Directing Attention of Visual Tokens
Esmaeilkhani, P., & Latecki, L. J. (2026) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv61042.2026.00561 · Cited by 4
VersaViT: Enhancing MLLM Vision Backbones via Task-Guided Optimization
Liu, Y., Liu, Y., Di, S., Wang, H., Zhao, Z., Tian, L., Zhou, X., Zhou, J., Yao, J., Xie, W., et al. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37577-3_5 · Cited by 4
Fu, R., Luo, Z., Lin, H., Ye, Z., & Ma, J. (2025) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2025.naacl-short.57 · Cited by 4
Language-Guided Reinforcement Learning for Hard Attention in Few-Shot LearningPreprint
Nikpour, B., & Armanfard, N. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.07800 · Cited by 4
ELITE: Enhanced Language-Image Toxicity Evaluation for SafetyPreprint
Lee, W., Lee, D., Choi, E., Yu, S., Yousefpour, A., Park, H., Ham, B., & Kim, S. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2502.04757 · Cited by 4
VIP: Visual Information Protection through Adversarial Attacks on Vision-Language ModelsPreprint
Meftah, H. F. Z. B., Hamidouche, W., Fezza, S. A., & Olivier Déforges, A. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2507.08982 · Cited by 4
VL-GenRM: Enhancing Vision-Language Verification via Vision Experts and Iterative TrainingPreprint
Zhang, J., Miao, K., Pi, R., Wang, Z., Liu, R., Pan, R. N., & Zhang, T. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2506.13888 · Cited by 4
MoTE: Mixture of Ternary Experts for Memory-efficient Large Multimodal ModelsPreprint
Wang, H., Xu, J., Wang, R., Feng, Y., Zhai, Y., Pei, P., Cai, X., & Chen, X. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2506.14435 · Cited by 4
CoRI: Communication of Robot Intent for Physical Human-Robot InteractionPreprint
Wang, J., Küçüktabak, E. B., Zarrin, R. S., & Erickson, Z. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2505.20537 · Cited by 4
Grounding Multilingual Multimodal LLMs With Cultural KnowledgePreprint
Nyandwi, J. D. D., Song, Y., Khanuja, S., & Neubig, G. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2508.07414 · Cited by 4
SAILViT: Towards Robust and Generalizable Visual Backbones for MLLMs via Gradual Feature RefinementPreprint
Yin, W., Yang, D., Dong, H., Kang, Z., Wang, J., Liang, X. M., Feng, C., & Ran, J. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2507.01643 · Cited by 4
Unlocking Speech Instruction Data Potential with Query RewritingPreprint
Hei, Y., Yan, Y., Liu, S., Zhou, H., Zhang, L., & Hu, X. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2507.08603 · Cited by 4
Free-form language-based robotic reasoning and graspingPreprint
Jiao, R., Fasoli, A., Giuliari, F., Bortolon, M., Povoli, S., Mei, G., Wang, Y., & Poiesi, F. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2503.13082 · Cited by 4
Color Names in Vision-Language ModelsPreprint
Gomez-Villa, A., Hernández-Cámara, P., Butt, M. A., Laparra, V., Malo, J., & Vázquez-Corral, J. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2509.22524 · Cited by 4
Masking Teacher and Reinforcing Student for Distilling Vision-Language ModelsPreprint
Lee, B.-K., Wang, Y.-C. F., & Hachiuma, R. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2512.22238 · Cited by 4
Encode Once, Decode Never: Reusing Audio LM Internals for Efficient Temporal LocalizationPreprint
An, J., Keung, P., Wang, J., Ahia, O., & Smith, N. A. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2602.10230 · Cited by 4
Self-citationUnderstanding the Fine-Grained Knowledge Capabilities of Vision-Language ModelsPreprint
Ghosh, D., Zhang, Y., & Schmidt, L. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2602.17871 · Cited by 4
Falcon PerceptionPreprint
Bevli, A., Chaybouti, S., Dahou, Y., Hacid, H., Huynh, N. D., Khac, P. H. L., Narayan, S., Para, W. R., & Singh, A. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2603.27365 · Cited by 4
Zone of Proximal Policy Optimization: Teacher in Prompts, Not GradientsPreprint
Lee, B.-K., Lu, X., Diao, S., Kang, M., Muralidharan, S., Sapra, K., Tao, A., Wang, Y.-C. F., & Hachiuma, R. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2606.18216 · Cited by 4
Lingshu: A Generalist Foundation Model for Unified Multimodal Medical Understanding and Reasoning
Xu, W., Chan, H. P., Li, L., Aljunied, M., Yuan, R., Wang, J., Xiao, C., Chen, G., Liu, C., Li, Z., et al. (2025) · arXiv · doi:10.48550/arxiv.2506.07044 · Cited by 4
Garg, K., Tang, X., Heo, J., Morgan, D. R., Gergle, D., Sudderth, E. B., & Piper, A. M. (2026) · Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791309 · Cited by 3
“Finally, we include two recently released open-source models: Llama from Meta [53] 4 and Molmo from the Allen Institute for AI [35], which both exhibit comparable performance to closed-source industry models on benchmarks.”
Influential$π_{0.5}$: a Vision-Language-Action Model with Open-World Generalization
Intelligence, P., Black, K., Brown, N., Darpinian, J., Dhabalia, K., Driess, D., Esmail, A., Equi, M., Finn, C., Fusai, N., et al. (2025) · arXiv · doi:10.48550/arxiv.2504.16054 · Cited by 3
Metadata
- Authors
- Deitke, M., Clark, C., Lee, S., Tripathi, R., Yang, Y., Park, J. S., Salehi, M., Muennighoff, N., Lo, K., Soldaini, L., Lu, J., Anderson, T., Bransom, E., Ehsani, K., Ngo, H., Chen, Y., Patel, A., Yatskar, M., Callison-Burch, C., Head, A., Hendrix, R., Bastani, F., VanderBilt, E., Lambert, N., Chou, Y., Chheda, A., Sparks, J., Skjonsberg, S., Schmitz, M., Sarnat, A., Bischoff, B., Walsh, P., Newell, C., Wolters, P., Gupta, T., Zeng, K.-H., Borchardt, J., Groeneveld, D., Nam, C., Lebrecht, S., Wittlif, C., Schoenick, C., Michel, O., Krishna, R., Weihs, L., Smith, N. A., Hajishirzi, H., Girshick, R., Farhadi, A., & Kembhavi, A.
- Affiliations
- Allen Institute for Artificial Intelligence (US)
- Allen Institute for AI
- University of Washington (US)
- University of Pennsylvania (US)
- ORCID
- Clark, Christopher
- Yang, Yue
- Park, Jae Sung
- Muennighoff, Niklas
- Lo, Kyle
- Soldaini, Luca
- Bransom, Erin
- Patel, Ajay
- Callison-Burch, Chris
- Head, Andrew
- Bastani, Favyen
- Lambert, Nathan
- Chou, Yvonne
- Newell, Chris
- Gupta, Tanmay
- Groeneveld, Dirk
- Michel, Oscar
- Krishna, Ranjay
- Weihs, Luca
- Smith, Noah A.
- Hajishirzi, Hannaneh
- Farhadi, Ali
- Kembhavi, Aniruddha
- Venue
- 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2025-06-10
- Pages
- 91-104
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W4413147844
- Cited by
- 67
- References
- 0
- Citations by year
- 2026: 37 · 2025: 30
- FWCI
- 50.28
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- vision-language models, open weights, image captioning, visual question answering, dataset collection, pre-training, fine-tuning, open-source models, human evaluation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
