The DOI matches its DataCite record; 2 of the 2 references it cites were found in a registry or an index.
Visual Instruction Tuning
Open accessAbstract
Instruction tuning large language models (LLMs) using machine-generated instruction-following data has improved zero-shot capabilities on new tasks, but the idea is less explored in the multimodal field. In this paper, we present the first attempt to use language-only GPT-4 to generate multimodal language-image instruction-following data. By instruction tuning on such generated data, we introduce LLaVA: Large Language and Vision Assistant, an end-to-end trained large multimodal model that connects a vision encoder and LLM for general-purpose visual and language understanding.Our early experiments show that LLaVA demonstrates impressive multimodel chat abilities, sometimes exhibiting the behaviors of multimodal GPT-4 on unseen images/instructions, and yields a 85.1% relative score compared with GPT-4 on a synthetic multimodal instruction-following dataset. When fine-tuned on Science QA, the synergy of LLaVA and GPT-4 achieves a new state-of-the-art accuracy of 92.53%. We make GPT-4 generated visual instruction tuning data, our model and code base publicly available.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“For example, LLaVA (Liu et al., 2023) demonstrated state-ofthe-art results by projecting visual features into a language model with a 2-layer MLP.”
From The Platonic Representation Hypothesis (2024) · cites this paper as Liu et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“For example, BLIP-2 employs the Query Transformer to optimize vision-language information extraction and adapts to instructions for image captioning and VQA.”
From Multimodal fusion and vision–language models: A survey for robot vision (2026) · cites this paper as [218]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“For instance, researchers leverage the common sense of large language models for task planning [257], robot control [82], and image understanding [122].”
From Understanding World or Predicting Future? A Comprehensive Survey of World Models (2026) · cites this paper as [122]
The cited source says what the claim attributes to it.
In the citing paper
“The emergence of Large Language Models (LLMs) [2,95,187,191] and the subsequent transition to combined language/vision models (Vision Large Language Models, VLLMs [2,9,14,123,124,196]) has revolutionized many AI-related research areas and applications.”
From Past, Present, and Future of Sensor-based Human Activity Recognition Using Wearables: A Surveying Tutorial on a Still Challenging Task (2025) · cites this paper as [124]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Existing works [29, 50] evaluate both close-source LLMs(e.g GPT3.5/4) and open-source LLMs(e.g Qwen[3], Baichuan[52], etc.) in financial sentiment analysis.”
From Large Language Model Agent in Financial Trading: A Survey (2026) · cites this paper as [29]
Experimental — accuracy not yet measured
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Our survey[8, 29, 53, 57] shows that LLM agents have achieved annualized return ranging from 15% to 30% over the strongest baseline during backtesting period with real market data, which demonstrates the great potential of using LLM in financial trading.”
From Large Language Model Agent in Financial Trading: A Survey (2026) · cites this paper as [29]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“They also backtest simple longshort strategy based on these sentiment scores, demonstrating the effectiveness of trading using such strategy.”
From Large Language Model Agent in Financial Trading: A Survey (2026) · cites this paper as [29]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“In both[18] and [29], portfolios weighted by market capitalization have shown slightly higher returns than those that are equally weighted.”
From Large Language Model Agent in Financial Trading: A Survey (2026) · cites this paper as [29]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“A similar approach is adopted in [18, 29], where the long-short strategy has shown to outperform both long-only and short-only strategies in backtesting.”
From Large Language Model Agent in Financial Trading: A Survey (2026) · cites this paper as [29]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 51
- Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. arXiv preprint arXiv:2204.14198, 2022. 2, 4 · arXiv
- Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton Van Den Hengel. Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2018. 2
- Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, et al. A general language assistant as a laboratory for alignment. arXiv preprint arXiv:2112.00861, 2021. 1 · arXiv
- Tim Brooks, Aleksander Holynski, and Alexei A Efros. Instruct pix2pix: Learning to follow image editing instructions. arXiv preprint arXiv:2211.09800, 2022. 2 · arXiv
- Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020. 2
- Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In CVPR, 2021. 2
- Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311, 2022. 2 · arXiv
- Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416, 2022. 2 · arXiv
- Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. PaLM-E: An embodied multimodal language model. arXiv preprint arXiv:2303.03378, 2023. 2 · arXiv
- Fartash Faghri, Hadi Pouransari, Sachin Mehta, Mehrdad Farajtabar, Ali Farhadi, Mohammad Rastegari, and Oncel Tuzel. Reinforce data, multiply impact: Improved model accuracy and robustness with dataset reinforcement. arXiv preprint arXiv:2303.08983, 2023. 2 · arXiv
- Oran Gafni, Adam Polyak, Oron Ashual, Shelly Sheynin, Devi Parikh, and Yaniv Taigman. Make-a-scene: Scene-based text-to-image generation with human priors. ArXiv, abs/2203.13131, 2022. · arXiv
- Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli. Chatgpt outperforms crowd-workers for text-annotation tasks. arXiv preprint arXiv:2303.15056, 2023. 3 · arXiv
- Tanmay Gupta and Aniruddha Kembhavi. Visual programming: Compositional visual reasoning without training. arXiv preprint arXiv:2211.11559, 2022. 2 · arXiv
- Weituo Hao, Chunyuan Li, Xiujun Li, Lawrence Carin, and Jianfeng Gao. Towards learning a generic agent for vision-and-language navigation via pre-training. In CVPR, 2020. 2
- Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Qiang Liu, et al. Language is not all you need: Aligning perception with language models. arXiv preprint arXiv:2302.14045, 2023. 2 · arXiv
- Srinivasan Iyer, Xi Victoria Lin, Ramakanth Pasunuru, Todor Mihaylov, Dániel Simig, Ping Yu, Kurt Shuster, Tianlu Wang, Qing Liu, Punit Singh Koura, et al. Opt-iml: Scaling language model instruction meta learning through the lens of generalization. arXiv preprint arXiv:2212.12017, 2022. 2 · arXiv
- Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In ECCV, 2022. 2
- Jing Yu Koh, Ruslan Salakhutdinov, and Daniel Fried. Grounding language models to images for multimodal generation. arXiv preprint arXiv:2301.13823, 2023. 2 · arXiv
- Boyi Li, Kilian Q Weinberger, Serge Belongie, Vladlen Koltun, and René Ranftl. Language- driven semantic segmentation. ICLR, 2022. 1
- Chunyuan Li, Zhe Gan, Zhengyuan Yang, Jianwei Yang, Linjie Li, Lijuan Wang, and Jianfeng Gao. Multimodal foundation models: From specialists to general-purpose assistants. arXiv preprint arXiv:2309.10020, 2023. 1 · arXiv
- Chunyuan Li, Haotian Liu, Liunian Harold Li, Pengchuan Zhang, Jyoti Aneja, Jianwei Yang, Ping Jin, Houdong Hu, Zicheng Liu, Yong Jae Lee, and Jianfeng Gao. ELEVATER: A bench- mark and toolkit for evaluating language-augmented visual models. In NeurIPS Track on Datasets and Benchmarks, 2022. 1
- Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597, 2023. 1, 2, 4, 6, 7 · arXiv
- Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, et al. Grounded language-image pre-training. In CVPR, 2022. 1
- Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jae Lee. Gligen: Open-set grounded text-to-image generation. arXiv preprint arXiv:2301.07093, 2023. 1 · arXiv
- Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft COCO: Common objects in context. In ECCV, 2014. 3
- Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499, 2023. 1 · arXiv
- Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 2022. 2, 5, 8, 9
- Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744, 2022. 2
- Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with GPT-4. arXiv preprint arXiv:2304.03277, 2023. 1, 4 · arXiv
- Hieu Pham, Zihang Dai, Golnaz Ghiasi, Kenji Kawaguchi, Hanxiao Liu, Adams Wei Yu, Jiahui Yu, Yi-Ting Chen, Minh-Thang Luong, Yonghui Wu, et al. Combined scaling for open-vocabulary image classification. arXiv preprint arXiv: 2111.10050, 2021. 1 · arXiv
- Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 2020. 2
- Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. ArXiv, abs/2204.06125, 2022. 1 · arXiv
- Robin Rombach, A. Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. CVPR, pages 10674-10685, 2022. 1
- Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L. Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, Seyedeh Sara Mahdavi, Raphael Gontijo Lopes, Tim Salimans, Jonathan Ho, David J. Fleet, and Mohammad Norouzi. Photorealistic text-to-image diffusion models with deep language understanding. ArXiv, abs/2205.11487, 2022. · arXiv
- Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion- 5b: An open large-scale dataset for training next generation image-text models. arXiv preprint arXiv:2210.08402, 2022. 2 · arXiv
- Dídac Surís, Sachit Menon, and Carl Vondrick. Vipergpt: Visual inference via python execution for reasoning. arXiv preprint arXiv:2303.08128, 2023. 2 · arXiv
- Andrew Szot, Alex Clegg, Eric Undersander, Erik Wijmans, Yili Zhao, John Turner, Noah Maestre, Mustafa Mukadam, Devendra Chaplot, Oleksandr Maksymets, Aaron Gokaslan, Vladimir Vondrus, Sameer Dharur, Franziska Meier, Wojciech Galuba, Angel Chang, Zsolt Kira, Vladlen Koltun, Jitendra Malik, Manolis Savva, and Dhruv Batra. Habitat 2.0: Training home assistants to rearrange their habitat. In Advances in Neural Information Processing Systems (NeurIPS), 2021. 2
- Jianfeng Wang, Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, and Lijuan Wang. Git: A generative image-to-text transformer for vision and language. arXiv preprint arXiv:2205.14100, 2022. 1 · arXiv
- Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language model with self generated instruc- tions. arXiv preprint arXiv:2212.10560, 2022. 2 · arXiv
- Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Anjana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap, et al. Benchmarking generalization via in-context instructions on 1,600+ language tasks. arXiv preprint arXiv:2204.07705, 2022. 2 · arXiv
- Chenfei Wu, Shengming Yin, Weizhen Qi, Xiaodong Wang, Zecheng Tang, and Nan Duan. Visual chatgpt: Talking, drawing and editing with visual foundation models. arXiv preprint arXiv:2303.04671, 2023. 2 · arXiv
- Jianwei Yang, Chunyuan Li, Pengchuan Zhang, Bin Xiao, Lu Yuan, Ce Liu, and Jianfeng Gao. Unified contrastive learning in image-text-label space. CVPR, 2022. 1
- Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Ehsan Azarnasab, Faisal Ahmed, Zicheng Liu, Ce Liu, Michael Zeng, and Lijuan Wang. Mm-react: Prompting chatgpt for multimodal reasoning and action. arXiv preprint arXiv:2303.11381, 2023. 2 · arXiv
- Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, Benton C. Hutchinson, Wei Han, Zarana Parekh, Xin Li, Han Zhang, Jason Baldridge, and Yonghui Wu. Scaling autoregressive models for content-rich text-to-image generation. ArXiv, abs/2206.10789, 2022. 1 · arXiv
- Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, et al. Florence: A new foundation model for computer vision. arXiv preprint arXiv:2111.11432, 2021. 1 · arXiv
- Hao Zhang, Feng Li, Xueyan Zou, Shilong Liu, Chunyuan Li, Jianfeng Gao, Jianwei Yang, and Lei Zhang. A simple framework for open-vocabulary segmentation and detection. arXiv preprint arXiv:2303.08131, 2023. 1 · arXiv
- Renrui Zhang, Jiaming Han, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, Peng Gao, and Yu Qiao. Llama-adapter: Efficient fine-tuning of language models with zero-init attention. arXiv preprint arXiv:2303.16199, 2023. 2, 8, 9 · arXiv
- Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. OPT: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022. 2 · arXiv
- Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis, and Alex Smola. Multi- modal chain-of-thought reasoning in language models. arXiv preprint arXiv:2302.00923, 2023. 8, 9 · arXiv
- Yiwu Zhong, Jianwei Yang, Pengchuan Zhang, Chunyuan Li, Noel Codella, Liunian Harold Li, Luowei Zhou, Xiyang Dai, Lu Yuan, Yin Li, et al. Regionclip: Region-based language-image pretraining. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16793-16803, 2022. 1
- Xueyan Zou, Zi-Yi Dou, Jianwei Yang, Zhe Gan, Linjie Li, Chunyuan Li, Xiyang Dai, Harkirat Behl, Jianfeng Wang, Lu Yuan, et al. Generalized decoding for pixel, image, and language. arXiv preprint arXiv:2212.11270, 2022. 1, 2 · arXiv
Cited by
Sorted by citations
Items 101–200 of 730
RULE: Reliable Multimodal RAG for Factuality in Medical Vision Language Models
Xia, P., Zhu, K., Li, H., Zhu, H., Li, Y., Li, G., Zhang, L., & Yao, H. (2024) · Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2024.emnlp-main.62 · Cited by 67
PathAsst: A Generative Foundation AI Assistant towards Artificial General Intelligence of Pathology
Sun, Y., Zhu, C., Zheng, S., Zhang, K., Sun, L., Shui, Z., Zhang, Y., Li, H., & Yang, L. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i5.28308 · Cited by 100
OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments
Xie, T., Zhang, D., Chen, J., Li, X., Zhao, S., Cao, R., Hua, T. J., Cheng, Z., Shin, D., Lei, F., et al. (2024) · arXiv · doi:10.48550/arxiv.2404.07972 · Cited by 64
MultiModal-GPT: a vision and language model for dialogue with humans
Gong, T., Lyu, C., Zhang, S., Wang, Y., Zheng, M., Zhao, Q., Liu, K., Zhang, W., Luo, P., & Chen, K. (2023) · arXiv · doi:10.48550/arxiv.2305.04790 · Cited by 70
Harnessing large vision and language models in agriculture: a review
Zhu, H., Qin, S., Su, M., Lin, C., Li, A., & Gao, J. (2025) · Frontiers in Plant Science · doi:10.3389/fpls.2025.1579355 · Cited by 97
Extracting Financial Data from Unstructured Sources: Leveraging Large Language Models
Li, H., Gao, H., Wu, C., & Vasarhelyi, M. A. (2024) · Journal of Information Systems · doi:10.2308/isys-2023-047 · Cited by 76
Cognitive Mirage: A Review of Hallucinations in Large Language Models
Ye, H., Liu, T., Zhang, A., Hua, W., & Jia, W. (2023) · arXiv · doi:10.48550/arxiv.2309.06794 · Cited by 61
VideoAgent: A Memory-augmented Multimodal Agent for Video Understanding
Fan, Y., Ma, X., Wu, R., Du, Y., Li, J., Gao, Z., & Li, Q. (2024) · arXiv · doi:10.48550/arxiv.2403.11481 · Cited by 60
Li, W., Qu, C., Chen, X., Bassi, P. R. A. S., Shi, Y., Lai, Y., Qian, Y., Xue, H., Chen, Y., Lin, X., et al. (2024) · Medical Image Analysis · doi:10.1016/j.media.2024.103285 · Cited by 74
Harnessing the Power of MLLMs for Transferable Text-to-Image Person ReID
Tan, W., Ding, C., Jiang, J., Wang, F., Zhan, Y., & Tao, D. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01621 · Cited by 77
Understanding World or Predicting Future? A Comprehensive Survey of World Models
Ding, J., Zhang, Y., Shang, Y., Zhang, Y., Zong, Z., Feng, J., Yuan, Y., Su, H., Li, N., Sukiennik, N., et al. (2026) · ACM Computing Surveys · doi:10.1145/3746449 · Cited by 169
Prompting Large Language Models for Topic Modeling
Wang, H., Prakash, N., Hoang, N. K., Hee, M. S., Naseem, U., & Lee, R. K.-W. (2023) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata59044.2023.10386113 · Cited by 71
Ghosh, A., Acharya, A., Saha, S., Jain, V., & Chadha, A. R. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4783140 · Cited by 169
Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions?
Chen, Y., Hu, H., Luan, Y., Sun, H., Changpinyo, S., Ritter, A., & Chang, M.-W. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.925 · Cited by 184
Seyfioğlu, M. S., Ikezogwo, W. O., Ghezloo, F., Krishna, R., & Shapiro, L. G. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01252 · Cited by 81
Scaling Vision-Language Models with Sparse Mixture of Experts
Shen, S., Yao, Z., Li, C., Darrell, T., Keutzer, K., & He, Y. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.758 · Cited by 113
Self-citationOn Large Visual Language Models for Medical Imaging Analysis: An Empirical Study
Van, M.-H., Verma, P., & Wu, X. (2024) · IEEE/ACM Conference on Connected Health: Applications, Systems and Engineering Technologies · doi:10.1109/chase60773.2024.00029 · Cited by 58
Foundation Models in Agriculture: A Comprehensive Review
Yin, S., Xi, Y., Zhang, X., Sun, C., & Mao, Q. (2025) · Agriculture · doi:10.3390/agriculture15080847 · Cited by 65
Vision Language Models are blind
Rahmanzadehgervi, P., Bolton, L., Taesiri, M. R., & Nguyen, A. T. (2025) · Lecture Notes in Computer Science · doi:10.1007/978-981-96-0917-8_17 · Cited by 54
Urban attractiveness according to ChatGPT: Contrasting AI and human insights
Malekzadeh, M., Willberg, E. S., Torkko, J., & Toivonen, T. (2024) · Computers Environment and Urban Systems · doi:10.1016/j.compenvurbsys.2024.102243 · Cited by 63
Zhang, Y., Pan, Y., Zhong, T., Dong, P., Xie, K., Liu, Y., Jiang, H., Wu, Z., Liu, Z., Zhao, W., et al. (2024) · Meta-Radiology · doi:10.1016/j.metrad.2024.100103 · Cited by 52
CoVR: Learning Composed Video Retrieval from Web Video Captions
Ventura, L., Yang, A., Schmid, C., & Varol, G. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i6.28334 · Cited by 62
Budhiraja, R., Joshi, I., Challa, J. S., Akolekar, H. D., & Kumar, D. (2024) · Proceedings of the 26th Australasian Computing Education Conference · doi:10.1145/3636243.3636257 · Cited by 52
Going Denser with Open-Vocabulary Part Segmentation
Sun, P., Chen, S., Zhu, C., Xiao, F., Luo, P., Xie, S., & Yan, Z. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01417 · Cited by 68
Scene-Driven Multimodal Knowledge Graph Construction for Embodied AI
Song, Y., Sun, P., Liu, H., Li, Z., Song, W., Xiao, Y., & Zhou, X. (2024) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2024.3399746 · Cited by 61
Self-citationChen, Z., Chen, H., Imani, M., Chen, R., & Imani, F. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.125769 · Cited by 61
Zhang, R., Wang, B., Zhang, J., Bian, Z., Feng, C., & Özbay, K. M. A. (2025) · Accident Analysis & Prevention · doi:10.1016/j.aap.2025.108077 · Cited by 68
From screens to scenes: A survey of embodied AI in healthcare
Liu, Y., Cao, X., Chen, T., Jiang, Y., You, J., Wu, M., Wang, X., Feng, M., Jin, Y., & Chen, J. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103033 · Cited by 76
Tao, L., Zhang, H., Jing, H., Liu, Y., Yan, D., Wei, G., & Xue, X. (2025) · Remote Sensing · doi:10.3390/rs17010162 · Cited by 63
HalluciDoctor: Mitigating Hallucinatory Toxicity in Visual Instruction Data
Yu, Q., Li, J., Wei, L., Pang, L., Ye, W., Qin, B., Tang, S., Tian, Q., & Zhuang, Y. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.01230 · Cited by 59
VLAAD: Vision and Language Assistant for Autonomous Driving
Park, S., Lee, M., Kang, J., Choi, H., Park, Y., Cho, J., Lee, A., & Kim, D. (2024) · 2024 IEEE/CVF Winter Conference on Applications of Computer Vision Workshops (WACVW) · doi:10.1109/wacvw60836.2024.00107 · Cited by 49
MAPLM: A Real-World Large-Scale Vision-Language Benchmark for Map and Traffic Scene Understanding
Cao, X., Zhou, T. H., Ma, Y., Ye, W., Cui, C., Tang, K., Cao, Z., Liang, K., Wang, Z., Rehg, J. M., et al. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02061 · Cited by 58
RegionGPT: Towards Region Understanding Vision Language Model
Guo, Q., Mello, S. D., Yin, H., Byeon, W., Cheung, K. C., Yu, Y., Luo, P., & Liu, S. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01309 · Cited by 74
The False Promise of Imitating Proprietary LLMs
Gudibande, A., Wallace, E., Snell, C., Geng, X., Liu, H., Abbeel, P., Levine, S., & Song, D. (2023) · arXiv · doi:10.48550/arxiv.2305.15717 · Cited by 48
“For example, the training process of many recent vision-language model (Li et al., 2022; Liu et al., 2023; Ye et al., 2023; Zhu et al., 2023) includes ChatGPT or GPT-4 outputs at some stages.”
MethodSelf-citationLLMs are Good Action Recognizers
Qu, H., Cai, Y., & Liu, J. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01741 · Cited by 53
IdealGPT: Iteratively Decomposing Vision and Language Reasoning via Large Language Models
You, H., Sun, R., Wang, Z., Chen, L., Wang, G., Ayyubi, H., Chang, K.-W., & Chang, S. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.755 · Cited by 75
A survey on occupancy perception for autonomous driving: The information fusion perspective
Xu, H., Chen, J., Meng, S., Wang, Y., & Chau, L. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102671 · Cited by 64
VideoBooth: Diffusion-based Video Generation with Image Prompts
Jiang, Y., Wu, T., Yang, S., Si, C., Lin, D., Qiao, Y., Loy, C. C., & Liu, Z. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00639 · Cited by 73
DiffEditor: Boosting Accuracy and Flexibility on Diffusion-Based Image Editing
Mou, C., Wang, X., Song, J., Shan, Y., & Zhang, J. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00811 · Cited by 64
OpenVLA: An Open-Source Vision-Language-Action Model
Kim, M. J., Pertsch, K., Karamcheti, S., Xiao, T., Balakrishna, A., Nair, S., Rafailov, R., Foster, E., Lam, G., Sanketi, P., et al. (2024) · arXiv · doi:10.48550/arxiv.2406.09246 · Cited by 62
“While early work explored various architectures for cross-attending between vision and language features [37–41], new open-source VLMs [20, 42–44] have converged on a simpler “patch-as-token” approach, in which patch features from pretrained visual transformers are treated as tokens, and are then…”
BackgroundMethodInfluentialKAM-CoT: Knowledge Augmented Multimodal Chain-of-Thoughts Reasoning
Mondal, D., Modi, S., Panda, S., Singh, R., & Rao, G. S. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i17.29844 · Cited by 67
Generative AI in the context of assistive technologies: Trends, limitations and future directions
Fu, B., Hadid, A., & Damer, N. (2024) · Image and Vision Computing · doi:10.1016/j.imavis.2024.105347 · Cited by 57
Holistic Autonomous Driving Understanding by Bird'View Injected Multi-Modal Large Models
Ding, X., Han, J., Xu, H., Liang, X., Zhang, W., & Li, X. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01297 · Cited by 66
A Current Review of Generative AI in Medicine: Core Concepts, Applications, and Current Limitations
Rouzrokh, P., Khosravi, B., Faghani, S., Moassefi, M., Shariatnia, M. M., Rouzrokh, P., & Erickson, B. J. (2025) · Current Reviews in Musculoskeletal Medicine · doi:10.1007/s12178-025-09961-y · Cited by 58
Multimodal fusion and vision–language models: A survey for robot vision
Han, X., Chen, S., Fu, Z., Feng, Z., Fan, L., An, D., Wang, C., Guo, L., Meng, W., Zhang, X., et al. (2026) · Information Fusion · doi:10.1016/j.inffus.2025.103652 · Cited by 105
“LLaVA [220] also uses similar strategies to generate more natural, user-friendly responses.”
InfluentialA Systematic Review of ChatGPT and Other Conversational Large Language Models in HealthcarePreprint
Wang, L., Wan, Z., Ni, C., Song, Q., Li, Y., Clayton, E. W., Malin, B., & Yin, Z. (2024) · medRxiv · doi:10.1101/2024.04.26.24306390 · Cited by 41
MolCA: Molecular Graph-Language Modeling with Cross-Modal Projector and Uni-Modal Adapter
Liu, Z., Li, S., Luo, Y., Fei, H., Cao, Y., Kawaguchi, K., Wang, X., & Chua, T. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.966 · Cited by 122
Chen, Q. H., & Yin, X. (2025) · Advanced Engineering Informatics · doi:10.1016/j.aei.2025.103478 · Cited by 43
Embodied Multi-Modal Agent trained by an LLM from a Parallel TextWorld
Yang, Y., Zhou, T., Li, K., Tao, D., Li, L., Shen, L., He, X., Jiang, J., & Shi, Y. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02482 · Cited by 51
Visual Program Distillation: Distilling Tools and Programmatic Reasoning into Vision-Language Models
Hu, Y., Stretcu, O., Lu, C.-T., Viswanathan, K., Hata, K., Luo, E., Krishna, R., & Fuxman, A. D. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00916 · Cited by 63
Organizational preparedness for the use of large language models in pathology informatics
Hart, S. N., Hoffman, N. G., Gershkovich, P., Christenson, C., McClintock, D. S., Miller, L. J., Jackups, R., Azimi, V., Spies, N. C., & Brodsky, V. (2023) · Journal of Pathology Informatics · doi:10.1016/j.jpi.2023.100338 · Cited by 38
Breaking Common Sense: WHOOPS! A Vision-and-Language Benchmark of Synthetic and Compositional Images
Bitton-Guetta, N., Bitton, Y., Hessel, J., Schmidt, L., Elovici, Y., Stanovsky, G., & Schwartz, R. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00247 · Cited by 37
Clinical insights: A comprehensive review of language models in medicine
Neveditsin, N., Lingras, P., & Mago, V. (2025) · PLOS Digital Health · doi:10.1371/journal.pdig.0000800 · Cited by 37
A survey of deep-learning-based radiology report generation using multimodal inputs
Wang, X., Figueredo, G. P., Li, R., Zhang, W. E., Chen, W., & Chen, X. (2025) · Medical Image Analysis · doi:10.1016/j.media.2025.103627 · Cited by 65
MELTing Point: Mobile Evaluation of Language Transformers
Laskaridis, S., Katevas, K., Minto, L., & Haddadi, H. (2024) · Proceedings of the 30th Annual International Conference on Mobile Computing and Networking · doi:10.1145/3636534.3690668 · Cited by 37
Wickramasekara, A., Breitinger, F., & Scanlon, M. (2025) · Forensic Science International Digital Investigation · doi:10.1016/j.fsidi.2024.301859 · Cited by 58
Video-Bench: A Comprehensive Benchmark and Toolkit for Evaluating Video-based Large Language Models
Ning, M., Zhu, B., Xie, Y., Lin, B., Cui, J., Yuan, L., Chen, D., & Yuan, L. (2023) · arXiv · doi:10.48550/arxiv.2311.16103 · Cited by 36
EarthVQANet: Multi-task visual question answering for remote sensing image understanding
Wang, J., Ma, A., Chen, Z., Zheng, Z., Wan, Y., Zhang, L., & Zhong, Y. (2024) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2024.05.001 · Cited by 45
OpenBias: Open-Set Bias Detection in Text-to-Image Generative Models
Moreno D’Incà, Peruzzo, E., Mancini, M., Xu, D., Goe, V., Xu, X., Wang, Z., Shi, H., & Sebe, N. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01162 · Cited by 54
Embodied navigation with multi-modal information: A survey from tasks to methodology
Wu, Y., Zhang, P., Gu, M., Zheng, J., & Bai, X. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102532 · Cited by 34
LimSim++: A Closed-Loop Platform for Deploying Multimodal LLMs in Autonomous Driving
Fu, D., Lei, W., Wen, L., Cai, P., Mao, S., Dou, M., Shi, B., & Qiao, Y. (2024) · IEEE Intelligent Vehicles Symposium · doi:10.1109/iv55156.2024.10588848 · Cited by 52
A comprehensive overview of Generative AI (GAI): Technologies, applications, and challenges
Zhang, Z., Zhang, J., Zhang, X., & Mai, W. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.129645 · Cited by 45
TheoremQA: A Theorem-driven Question Answering Dataset
Chen, W., Yin, M., Ku, M., Lu, P., Wan, Y., Ma, X., Xu, J., Wang, X., & Xia, T. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.489 · Cited by 141
Artificial general intelligence for radiation oncology
Liu, C., Liu, Z., Holmes, J., Zhang, L., Zhang, L., Ding, Y., Shu, P., Wu, Z., Dai, H., Li, Y., et al. (2023) · Meta-Radiology · doi:10.1016/j.metrad.2023.100045 · Cited by 39
Guo, R., Wei, J., Sun, L., Yu, B., Chang, G., Liu, D., Zhang, S., Yao, Z., Xu, M., & Bu, L. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.108709 · Cited by 58
EmoVIT: Revolutionizing Emotion Insights with Visual Instruction Tuning
Xie, H., Peng, C.-J., Tseng, Y., Chen, H.-J., Hsu, C.-F., Shuai, H.-H., & Cheng, W.-H. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02511 · Cited by 48
Yi, Z., Xiao, T., & Albert, M. V. (2025) · Information · doi:10.3390/info16020136 · Cited by 43
VIGC: Visual Instruction Generation and Correction
Wang, B., Wu, F., Han, X. H., Peng, J., Zhong, H., Zhang, P., Dong, X., Li, W., Li, W., Wang, J., et al. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i6.28338 · Cited by 65
Understanding natural language: Potential application of large language models to ophthalmology
Yang, Z., Wang, D., Zhou, F., Song, D., Zhang, Y., Jiang, J., Kong, K., Liu, X., Qiao, Y., Chang, R. T., et al. (2024) · Asia-Pacific Journal of Ophthalmology · doi:10.1016/j.apjo.2024.100085 · Cited by 38
Groundhog Grounding Large Language Models to Holistic Segmentation
Zhang, Y., Ma, Z., Gao, X., Shakiah, S. M., Gao, Q., & Chai, J. Y. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01349 · Cited by 49
Wu, H., Mu, W., Zhong, D., Du, Z., Li, H., & Tao, C. (2025) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2025.05.010 · Cited by 36
Wu, D., Zheng, P., Zhao, Q., Zhang, S., Qi, J., Hu, J., Zhu, G., & Wang, L. (2025) · Robotics and Computer-Integrated Manufacturing · doi:10.1016/j.rcim.2025.103064 · Cited by 38
PneumoLLM: Harnessing the power of large language model for pneumoconiosis diagnosis
Song, M., Wang, J., Yu, Z., Wang, J., Yang, L., Lu, Y., Li, B., Wang, X., Wang, X., Huang, Q., et al. (2024) · Medical Image Analysis · doi:10.1016/j.media.2024.103248 · Cited by 31
VCoder: Versatile Vision Encoders for Multimodal Large Language Models
Jain, J., Yang, J., & Shi, H. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02644 · Cited by 53
The Neglected Tails in Vision-Language Models
Parashar, S., Lin, Z., Liu, T., Dong, X., Li, Y., Ramanan, D., Caverlee, J., & Kong, S. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01234 · Cited by 55
VLM-MSGraph: Vision Language Model-enabled Multi-hierarchical Scene Graph for robotic assembly
Li, S., Yan, Z., Wang, Z., & Gao, Y. (2025) · Robotics and Computer-Integrated Manufacturing · doi:10.1016/j.rcim.2025.102978 · Cited by 34
OphGLM: An ophthalmology large language-and-vision assistant
Deng, Z., Gao, W., Chen, C. H., Niu, Z., Gong, Z., Zhang, R., Cao, Z., Li, F., Ma, Z., Wei, W., et al. (2024) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2024.103001 · Cited by 35
Object detection with multimodal large vision-language models: An in-depth review
Sapkota, R., & Karkee, M. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103575 · Cited by 61
Intelligent Edge Computing and Machine Learning: A Survey of Optimization and Applications
Cajas, S. A., Samanta, J., Suárez‐Cetrulo, A. L., & Carbajo, R. S. (2025) · Future Internet · doi:10.3390/fi17090417 · Cited by 47
Instruction-ViT: Multi-modal prompts for instruction learning in vision transformer
Xiao, Z., Chen, Y., Yao, J., Zhang, L., Liu, Z., Wu, Z., Yu, X., Pan, Y., Zhao, L., Ma, C., et al. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.102204 · Cited by 51
Zhang, Z., Wu, H., Zhang, E., Zhai, G., & Lin, W. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3445770 · Cited by 59
Towards Language-Driven Video Inpainting via Multimodal Large Language Models
Wu, J., Li, X., Si, C., Zhou, S., Yang, J., Zhang, J., Li, Y., Chen, K., Tong, Y., Liu, Z., et al. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01188 · Cited by 40
IMPRINT: Generative Object Compositing by Learning Identity-Preserving Representation
Song, Y., Zhang, Z., Lin, Z., Cohen, S. D., Price, B. L., Zhang, J., Kim, S. Y., Zhang, H., Xiong, W., & Aliaga, D. G. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00769 · Cited by 43
Deep Unrestricted Document Image Rectification
Feng, H., Liu, S., Deng, J., Zhou, W., & Li, H. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3347094 · Cited by 29
THE COLOSSEUM: A Benchmark for Evaluating Generalization for Robotic Manipulation
Pumacay, W., Singh, I., Duan, J., Krishna, R., Thomason, J., & Fox, D. (2024) · Robotics: Science and Systems XX · doi:10.15607/rss.2024.xx.133 · Cited by 29
Enhancing Visual Document Understanding with Contrastive Learning in Large Visual-Language Models
Li, X., Wu, Y., Jiang, X., Guo, Z., Gong, M., Cao, H., Liu, Y., Jiang, D., & Sun, X. W. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01472 · Cited by 33
Pink: Unveiling the Power of Referential Comprehension for Multi-modal LLMs
Xuan, S., Guo, Q., Yan, S., & Zhang, S. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01313 · Cited by 45
From Detection to Action: A Multimodal AI Framework for Traffic Incident Response
Ahmed, A., Farhan, M., Eesaar, H., Chong, K. T., & Tayara, H. (2024) · Drones · doi:10.3390/drones8120741 · Cited by 33
LLMs are one-shot URL classifiers and explainers
Rashid, F., Ranaweera, N., Doyle, B., & Seneviratne, S. (2024) · Computer Networks · doi:10.1016/j.comnet.2024.111004 · Cited by 33
The Platonic Representation Hypothesis
Huh, M., Cheung, B., Wang, T., & Isola, P. (2024) · arXiv · doi:10.48550/arxiv.2405.07987 · Cited by 35
“For example, LLaVA (Liu et al., 2023) demonstrated state-of-the-art results by projecting visual features into a language model with a 2-layer MLP.”
BackgroundMethodResultHydra: Multi-head low-rank adaptation for parameter efficient fine-tuning
Kim, S., Yang, H.-M., Kim, Y., Hong, Y., & Park, E. (2024) · Neural Networks · doi:10.1016/j.neunet.2024.106414 · Cited by 36
DrugChat: Towards Enabling ChatGPT-Like Capabilities on Drug Molecule GraphsPreprint
Liang, Y., Zhang, R., Zhang, L., & Xie, P. (2023) · doi:10.36227/techrxiv.22945922 · Cited by 33
On the foundations of Earth foundation models
Zhu, X. X., Xiong, Z., Wang, Y., Stewart, A. J., Heidler, K., Wang, Y., Yuan, Z., Dujardin, T., Xu, Q., & Shi, Y. (2026) · Communications Earth & Environment · doi:10.1038/s43247-025-03127-x · Cited by 26
Chen, Y., Zhang, S., Han, T., Du, Y., Zhang, W., & Li, J. (2024) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2024.04.024 · Cited by 30
OpenECAD: An efficient visual language model for editable 3D-CAD design
Yuan, Z., Shi, J., & Huang, Y. (2024) · Computers & Graphics · doi:10.1016/j.cag.2024.104048 · Cited by 40
Exploring Embodied Multimodal Large Models: Development, datasets, and future directions
Chen, S., Wu, Z., Zhang, K., Li, C., Zhang, B., Ma, F., Yu, F. R., & Li, Q. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103198 · Cited by 39
Self-citationTouch100k: A large-scale touch-language-vision dataset for touch-centric multimodal representation
Cheng, N., Xu, J., Guan, C., Gao, J., Wang, W., Li, Y., Meng, F., Zhou, J., Fang, B., & Han, W. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103305 · Cited by 49
Tu, N. V., Yoo, J., & Hong, J. W. (2024) · IEEE/IFIP Network Operations and Management Symposium (NOMS) · doi:10.1109/noms59830.2024.10575237 · Cited by 25
What Makes Multimodal In-Context Learning Work?
Baldassini, F. B., Shukor, M., Cord, M., Soulier, L., & Piwowarski, B. (2024) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw63382.2024.00161 · Cited by 35
Vision-enhanced multi-modal learning framework for non-destructive pavement damage detection
Zhang, Y., & Liu, C. (2025) · Automation in Construction · doi:10.1016/j.autcon.2025.106389 · Cited by 25
Metadata
- Authors
- Liu, H., Li, C., Wu, Q., & Lee, Y. J.
- Affiliations
- University of Wisconsin–Madison (US)
- Microsoft (United States) (US)
- Columbia University (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2304.08485
- OpenAlex
- W4366330503
- Cited by
- 682
- References
- 0
- Citations by year
- 2026: 43 · 2025: 250 · 2024: 330 · 2023: 58
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- instruction tuning, multimodal large language models, LLaVA, GPT-4, vision encoder, visual language understanding, zero-shot generalization
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
