The DOI matches its DataCite record; 2 of the 2 references it cites were found in a registry or an index.
MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models
Open accessAbstract
The recent GPT-4 has demonstrated extraordinary multi-modal abilities, such as directly generating websites from handwritten text and identifying humorous elements within images. These features are rarely observed in previous vision-language models. However, the technical details behind GPT-4 continue to remain undisclosed. We believe that the enhanced multi-modal generation capabilities of GPT-4 stem from the utilization of sophisticated large language models (LLM). To examine this phenomenon, we present MiniGPT-4, which aligns a frozen visual encoder with a frozen advanced LLM, Vicuna, using one projection layer. Our work, for the first time, uncovers that properly aligning the visual features with an advanced large language model can possess numerous advanced multi-modal abilities demonstrated by GPT-4, such as detailed image description generation and website creation from hand-drawn drafts. Furthermore, we also observe other emerging capabilities in MiniGPT-4, including writing stories and poems inspired by given images, teaching users how to cook based on food photos, and so on. In our experiment, we found that the model trained on short image caption pairs could produce unnatural language outputs (e.g., repetition and fragmentation). To address this problem, we curate a detailed image description dataset in the second stage to finetune the model, which consequently improves the model's generation reliability and overall usability. Our code, pre-trained model, and collected dataset are available at https://minigpt-4.github.io/.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Cited by
Sorted by citations
Items 101–200 of 497
Understanding LLMs: A comprehensive overview from training to inference
Liu, Y., He, H., Han, T., Zhang, X., Liu, M., Tian, J., Zhang, Y., Wang, J., Gao, X., Zhong, T., et al. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.129190 · Cited by 81
MultiModal-GPT: a vision and language model for dialogue with humans
Gong, T., Lyu, C., Zhang, S., Wang, Y., Zheng, M., Zhao, Q., Liu, K., Zhang, W., Luo, P., & Chen, K. (2023) · arXiv · doi:10.48550/arxiv.2305.04790 · Cited by 70
DetGPT: Detect What You Need via Reasoning
Pi, R., Gao, J., Diao, S., Pan, R., Dong, H., Zhang, J., Yao, L., Han, J., Xu, H., Kong, L., et al. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.876 · Cited by 127
Navgpt-2: Unleashing navigational reasoning capability for large vision-language models
Zhou, G., Hong, Y., Wang, Z., Wang, X. E., & Wu, Q. (2025) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-72667-5_15 · Cited by 63
Chat with the Environment: Interactive Multimodal Perception Using Large Language Models
Zhao, X., Li, M., Weber, C., Hafez, M. B., & Wermter, S. (2023) · 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros55552.2023.10342363 · Cited by 62
Cognitive Mirage: A Review of Hallucinations in Large Language Models
Ye, H., Liu, T., Zhang, A., Hua, W., & Jia, W. (2023) · arXiv · doi:10.48550/arxiv.2309.06794 · Cited by 61
Foundation models and intelligent decision-making: Progress, challenges, and perspectives
Huang, J., Xu, Y., Wang, Q., Wang, Q., Liang, X., Wang, F., Zhang, Z., Wei, W., Zhang, B., Huang, L., et al. (2025) · The Innovation · doi:10.1016/j.xinn.2025.100948 · Cited by 87
Self-citationHalluciDoctor: Mitigating Hallucinatory Toxicity in Visual Instruction Data
Yu, Q., Li, J., Wei, L., Pang, L., Ye, W., Qin, B., Tang, S., Tian, Q., & Zhuang, Y. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.01230 · Cited by 59
Harnessing the Power of MLLMs for Transferable Text-to-Image Person ReID
Tan, W., Ding, C., Jiang, J., Wang, F., Zhan, Y., & Tao, D. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01621 · Cited by 77
Ghosh, A., Acharya, A., Saha, S., Jain, V., & Chadha, A. R. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4783140 · Cited by 169
PMC-VQA: Visual Instruction Tuning for Medical Visual Question Answering
Zhang, X., Wu, C., Zhao, Z., Lin, W., Zhang, Y., Wang, Y., & Xie, W. (2023) · arXiv · doi:10.48550/arxiv.2305.10415 · Cited by 66
Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions?
Chen, Y., Hu, H., Luan, Y., Sun, H., Changpinyo, S., Ritter, A., & Chang, M.-W. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.925 · Cited by 184
Seyfioğlu, M. S., Ikezogwo, W. O., Ghezloo, F., Krishna, R., & Shapiro, L. G. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01252 · Cited by 81
The role of large language models in agriculture: harvesting the future with LLM intelligence
Shaikh, T. A., Rasool, T., Veningston, K., & Yaseen, S. M. (2024) · Progress in Artificial Intelligence · doi:10.1007/s13748-024-00359-4 · Cited by 67
Scaling Vision-Language Models with Sparse Mixture of Experts
Shen, S., Yao, Z., Li, C., Darrell, T., Keutzer, K., & He, Y. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.758 · Cited by 113
Hadi, M. U., Al-Tashi, Q., Qureshi, R., Shah, A., Muneer, A., Irfan, M., Zafar, A., Shaikh, M. B., Akhtar, N., Hassan, S. Z., et al. (2023) · doi:10.36227/techrxiv.23589741.v3 · Cited by 71
Zhang, Y., Pan, Y., Zhong, T., Dong, P., Xie, K., Liu, Y., Jiang, H., Wu, Z., Liu, Z., Zhao, W., et al. (2024) · Meta-Radiology · doi:10.1016/j.metrad.2024.100103 · Cited by 52
Scene-Driven Multimodal Knowledge Graph Construction for Embodied AI
Song, Y., Sun, P., Liu, H., Li, Z., Song, W., Xiao, Y., & Zhou, X. (2024) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2024.3399746 · Cited by 61
Li, J., Vo, D. M., Sugimoto, A., & Nakayama, H. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01303 · Cited by 61
From screens to scenes: A survey of embodied AI in healthcare
Liu, Y., Cao, X., Chen, T., Jiang, Y., You, J., Wu, M., Wang, X., Feng, M., Jin, Y., & Chen, J. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103033 · Cited by 76
Self-citationOn the Robustness of Large Multimodal Models Against Image Adversarial Attacks
Cui, X., Aparcedo, A., Jang, Y. K., & Lim, S.-N. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02325 · Cited by 76
VLAAD: Vision and Language Assistant for Autonomous Driving
Park, S., Lee, M., Kang, J., Choi, H., Park, Y., Cho, J., Lee, A., & Kim, D. (2024) · 2024 IEEE/CVF Winter Conference on Applications of Computer Vision Workshops (WACVW) · doi:10.1109/wacvw60836.2024.00107 · Cited by 49
Multimodal vision-based human action recognition using deep learning: a review
Shafizadegan, F., Naghsh‐Nilchi, A. R., & Shabaninia, E. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10730-5 · Cited by 57
A survey on occupancy perception for autonomous driving: The information fusion perspective
Xu, H., Chen, J., Meng, S., Wang, Y., & Chau, L. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102671 · Cited by 64
Self-citationMAPLM: A Real-World Large-Scale Vision-Language Benchmark for Map and Traffic Scene Understanding
Cao, X., Zhou, T. H., Ma, Y., Ye, W., Cui, C., Tang, K., Cao, Z., Liang, K., Wang, Z., Rehg, J. M., et al. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02061 · Cited by 58
RegionGPT: Towards Region Understanding Vision Language Model
Guo, Q., Mello, S. D., Yin, H., Byeon, W., Cheung, K. C., Yu, Y., Luo, P., & Liu, S. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01309 · Cited by 74
The False Promise of Imitating Proprietary LLMs
Gudibande, A., Wallace, E., Snell, C., Geng, X., Liu, H., Abbeel, P., Levine, S., & Song, D. (2023) · arXiv · doi:10.48550/arxiv.2305.15717 · Cited by 48
“For example, the training process of many recent vision-language model (Li et al., 2022; Liu et al., 2023; Ye et al., 2023; Zhu et al., 2023) includes ChatGPT or GPT-4 outputs at some stages.”
MethodConnecting Speech Encoder and Large Language Model for ASR
Yu, W., Tang, C., Sun, G., Chen, X., Tan, T., Li, W., Lu, L., Ma, Z., & Zhang, C. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10445874 · Cited by 79
IdealGPT: Iteratively Decomposing Vision and Language Reasoning via Large Language Models
You, H., Sun, R., Wang, Z., Chen, L., Wang, G., Ayyubi, H., Chang, K.-W., & Chang, S. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.755 · Cited by 75
LION : Empowering Multimodal Large Language Model with Dual-Level Visual Knowledge
Chen, G., Shen, L., Shao, R., Deng, X., & Nie, L. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02506 · Cited by 73
Artificial intelligence generated content (AIGC) in medicine: A narrative review
Shao, L., Chen, B., Zhang, Z., Zhang, Z., & Chen, X. (2024) · Mathematical Biosciences & Engineering · doi:10.3934/mbe.2024073 · Cited by 51
Romero, O. J., Zimmerman, J., Steinfeld, A. M., & Tomasic, A. (2024) · Proceedings of the AAAI Symposium Series · doi:10.1609/aaaiss.v2i1.27706 · Cited by 56
Holistic Autonomous Driving Understanding by Bird'View Injected Multi-Modal Large Models
Ding, X., Han, J., Xu, H., Liang, X., Zhang, W., & Li, X. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01297 · Cited by 66
Self-citationTransferable Multimodal Attack on Vision-Language Pre-training Models
Wang, H., Dong, K., Zhu, Z., Qin, H., Liu, A., Fang, X., Wang, J., & Liu, X. (2024) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp54263.2024.00102 · Cited by 53
Feng, H., Liu, Q., Liu, H., Tang, J., Zhou, W., Li, H., & Huang, C. (2024) · Science China Information Sciences · doi:10.1007/s11432-024-4250-y · Cited by 92
Evaluating large language models for software testing
Li, Y., Liu, P., Wang, H., Chu, J., & Wong, W. E. (2025) · Computer Standards & Interfaces · doi:10.1016/j.csi.2024.103942 · Cited by 42
Multimodal fusion and vision–language models: A survey for robot vision
Han, X., Chen, S., Fu, Z., Feng, Z., Fan, L., An, D., Wang, C., Guo, L., Meng, W., Zhang, X., et al. (2026) · Information Fusion · doi:10.1016/j.inffus.2025.103652 · Cited by 105
“By integrating visual, depth, audio, and linguistic data, these datasets provide abundant resources for comprehensive model training, evaluation, and optimization, while laying a solid foundation for future research in multimodal fusion and cross-domain transfer learning.”
TokenPacker: Efficient Visual Projector for Multimodal LLM
Li, W., Yuan, Y., Liu, J., Tang, D., Wang, S., Qin, J., Zhu, J., & Zhang, L. (2025) · International Journal of Computer Vision · doi:10.1007/s11263-025-02491-7 · Cited by 159
Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMs
You, K., Zhang, H., Schoop, E., Weers, F., Swearngin, A., Nichols, J., Yang, Y., & Gan, Z. (2024) · arXiv · doi:10.48550/arxiv.2404.05719 · Cited by 40
MolCA: Molecular Graph-Language Modeling with Cross-Modal Projector and Uni-Modal Adapter
Liu, Z., Li, S., Luo, Y., Fei, H., Cao, Y., Kawaguchi, K., Wang, X., & Chua, T. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.966 · Cited by 122
GPT4Point: A Unified Framework for Point-Language Understanding and Generation
Qi, Z., Fang, Y., Sun, Z., Wu, X., Wu, T., Wang, J., Lin, D., & Zhao, H. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02495 · Cited by 59
CLIPSyntel: CLIP and LLM Synergy for Multimodal Question Summarization in Healthcare
Ghosh, A., Acharya, A., Jain, R., Saha, S., Chadha, A. R., & Sinha, S. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i20.30206 · Cited by 50
Embodied Multi-Modal Agent trained by an LLM from a Parallel TextWorld
Yang, Y., Zhou, T., Li, K., Tao, D., Li, L., Shen, L., He, X., Jiang, J., & Shi, Y. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02482 · Cited by 51
Li, F., Jiang, Q., Zhang, H., Ren, T., Liu, S., Zou, X., Xu, H., Li, H., Yang, J., Li, C., et al. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01222 · Cited by 54
MuseChat: A Conversational Music Recommendation System for Videos
Dong, Z., Liu, X., Chen, B., Polak, P., & Zhang, P. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01214 · Cited by 46
Visual Program Distillation: Distilling Tools and Programmatic Reasoning into Vision-Language Models
Hu, Y., Stretcu, O., Lu, C.-T., Viswanathan, K., Hata, K., Luo, E., Krishna, R., & Fuxman, A. D. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00916 · Cited by 63
LGR-NET: Language Guided Reasoning Network for Referring Expression Comprehension
Lu, M., Li, R., Feng, F., Ma, Z., & Wang, X. (2024) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2024.3374786 · Cited by 39
Exploring Open-Vocabulary Semantic Segmentation from CLIP Vision Encoder Distillation Only
Chen, J., Zhu, D., Qian, G., Ghanem, B., Yan, Z., Zhu, C., Xiao, F., Culatana, S. C., & Elhoseiny, M. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00071 · Cited by 53
Self-citationLLM4SGG: Large Language Models for Weakly Supervised Scene Graph Generation
Kim, K., Yoon, K., Jeon, J., In, Y., Moon, J., Kim, D., & Park, C. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02674 · Cited by 39
VIAssist: Adapting Multi-Modal Large Language Models for Users with Visual Impairments
Yang, B., He, L., Liu, K., & Yan, Z. (2024) · doi:10.1109/fmsys62467.2024.00010 · Cited by 39
Digital twins of Earth and the computing challenge of human interaction
Bauer, P., Hoefler, T., Stevens, B., & Hazeleger, W. (2024) · Nature Computational Science · doi:10.1038/s43588-024-00599-3 · Cited by 37
Artificial general intelligence for radiation oncology
Liu, C., Liu, Z., Holmes, J., Zhang, L., Zhang, L., Ding, Y., Shu, P., Wu, Z., Dai, H., Li, Y., et al. (2023) · Meta-Radiology · doi:10.1016/j.metrad.2023.100045 · Cited by 39
Self-citationGuo, R., Wei, J., Sun, L., Yu, B., Chang, G., Liu, D., Zhang, S., Yao, Z., Xu, M., & Bu, L. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.108709 · Cited by 58
VIGC: Visual Instruction Generation and Correction
Wang, B., Wu, F., Han, X. H., Peng, J., Zhong, H., Zhang, P., Dong, X., Li, W., Li, W., Wang, J., et al. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i6.28338 · Cited by 65
Groundhog Grounding Large Language Models to Holistic Segmentation
Zhang, Y., Ma, Z., Gao, X., Shakiah, S. M., Gao, Q., & Chai, J. Y. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01349 · Cited by 49
Guo, Y., Siddiqui, F., Zhao, Y., Chellappa, R., & Lo, S.-Y. (2025) · International Journal of Computer Vision · doi:10.1007/s11263-025-02495-3 · Cited by 39
Several Categories of Large Language Models (LLMs): A Short Survey
Pahune, S. A., & Chandrasekharan, M. (2023) · International Journal for Research in Applied Science and Engineering Technology · doi:10.22214/ijraset.2023.54677 · Cited by 54
VCoder: Versatile Vision Encoders for Multimodal Large Language Models
Jain, J., Yang, J., & Shi, H. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02644 · Cited by 53
Object detection with multimodal large vision-language models: An in-depth review
Sapkota, R., & Karkee, M. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103575 · Cited by 61
Wang, M., Su, L., Yan, C., Xu, S. H., Yuan, P., Jiang, X., & Zhang, B. (2024) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2024.3439395 · Cited by 38
Zhang, Z., Wu, H., Zhang, E., Zhai, G., & Lin, W. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3445770 · Cited by 59
Vlogger: Make Your Dream A Vlog
Zhuang, S., Li, K., Chen, X., Wang, Y., Liu, Z., Qiao, Y., & Wang, Y. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00841 · Cited by 62
Zhu, Y., Li, L., Chen, K., Liu, C., Zhou, F., & Shi, Z. (2024) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2024.3497338 · Cited by 46
Generative Multi-Modal Knowledge Retrieval with Large Language Models
Long, X., Zeng, J., Meng, F., Ma, Z., Zhang, K., Zhou, B., & Zhou, J. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i17.29837 · Cited by 34
Liang, H., Zhang, J., Li, Y., Wang, B., & Huang, J. (2024) · IEEE Access · doi:10.1109/access.2024.3408843 · Cited by 31
Towards Language-Driven Video Inpainting via Multimodal Large Language Models
Wu, J., Li, X., Si, C., Zhou, S., Yang, J., Zhang, J., Li, Y., Chen, K., Tong, Y., Liu, Z., et al. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01188 · Cited by 40
Self-citationIMPRINT: Generative Object Compositing by Learning Identity-Preserving Representation
Song, Y., Zhang, Z., Lin, Z., Cohen, S. D., Price, B. L., Zhang, J., Kim, S. Y., Zhang, H., Xiong, W., & Aliaga, D. G. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00769 · Cited by 43
Jailbreak Vision Language Models via Bi-Modal Adversarial Prompt
Ying, Z., Liu, A., Zhang, T., Yu, Z., Liang, S., Liu, X., & Tao, D. (2025) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2025.3583249 · Cited by 107
A Survey on Efficient Vision‐Language Models
Shinde, G., Ravi, A., Dey, E., Sakib, S., Rampure, M., & Roy, N. (2025) · Wiley Interdisciplinary Reviews Data Mining and Knowledge Discovery · doi:10.1002/widm.70036 · Cited by 46
Enhancing Visual Document Understanding with Contrastive Learning in Large Visual-Language Models
Li, X., Wu, Y., Jiang, X., Guo, Z., Gong, M., Cao, H., Liu, Y., Jiang, D., & Sun, X. W. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01472 · Cited by 33
Self-citationLTGC: Long-Tail Recognition via Leveraging LLMs-Driven Generated Content
Zhao, Q., Dai, Y., Li, H., Hu, W., Zhang, F., & Liu, J. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01845 · Cited by 35
Pink: Unveiling the Power of Referential Comprehension for Multi-modal LLMs
Xuan, S., Guo, Q., Yan, S., & Zhang, S. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01313 · Cited by 45
A Comprehensive Study of GPT-4V’s Multimodal Capabilities in Medical ImagingPreprint
Li, Y., Liu, Y., Wang, Z., Liang, X., Liu, L., Wang, L., Cui, L., Tu, Z., Wang, L., & Zhou, L. (2023) · medRxiv · doi:10.1101/2023.11.03.23298067 · Cited by 28
Large Language Model-Empowered Agents for Simulating Macroeconomic ActivitiesPreprint
Li, N., Gao, C., Li, Y., & Liao, Q. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4606937 · Cited by 31
CapsFusion: Rethinking Image-Text Data at Scale
Yu, Q., Sun, Q., Zhang, X., Cui, Y., Zhang, F., Cao, Y. Q., Wang, X., & Liu, J. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01330 · Cited by 58
Luo, S., Qian, Y., Bai, L., Fan, Y., Wang, Y., & Kong, W. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103291 · Cited by 34
DrugChat: Towards Enabling ChatGPT-Like Capabilities on Drug Molecule GraphsPreprint
Liang, Y., Zhang, R., Zhang, L., & Xie, P. (2023) · doi:10.36227/techrxiv.22945922 · Cited by 33
Chen, Y., Zhang, S., Han, T., Du, Y., Zhang, W., & Li, J. (2024) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2024.04.024 · Cited by 30
Exploring Embodied Multimodal Large Models: Development, datasets, and future directions
Chen, S., Wu, Z., Zhang, K., Li, C., Zhang, B., Ma, F., Yu, F. R., & Li, Q. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103198 · Cited by 39
Nguyen, T. T. H., Nguyen, L. P. T., & Cao, H. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102782 · Cited by 34
Prospective Role of Foundation Models in Advancing Autonomous Vehicles
Wu, J., Gao, B., Gao, J., Yu, J., Chu, H., Yu, Q., Gong, X., Chang, Y. M., Tseng, H. E., Chen, H., et al. (2024) · Research · doi:10.34133/research.0399 · Cited by 27
Self-citationZhang, X., Zeng, F., & Gu, C. (2024) · Neural Networks · doi:10.1016/j.neunet.2024.107059 · Cited by 27
Zhao, Y., Zhang, M., Yang, B., Zhang, Z., Kang, J., & Gong, J. (2025) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2025.02.009 · Cited by 26
AutoAD III: The Prequel - Back to the Pixels
Han, T., Bain, M., Nagrani, A., Varol, G., Xie, W., & Zisserman, A. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01720 · Cited by 31
Explainability and vision foundation models: A survey
Kazmierczak, R., Berthier, É., Frehse, G., & Franchi, G. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103184 · Cited by 45
Task-Oriented Semantic Communication in Large Multimodal Models-Based Vehicle Networks
Du, B., Du, H., Niyato, D. T., & Li, R. (2025) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2025.3564543 · Cited by 28
GPT-4V Takes the Wheel: Promises and Challenges for Pedestrian Behavior Prediction
Huang, J., Jiang, P., Gautam, A., & Saripalli, S. (2024) · Proceedings of the AAAI Symposium Series · doi:10.1609/aaaiss.v3i1.31192 · Cited by 22
Prompting Large Language Models for Zero-Shot Domain Adaptation in Speech Recognition
Li, Y., Wu, Y., Li, J., & Liu, S. (2023) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru57964.2023.10389732 · Cited by 37
Inoue, Y., Yada, Y., Tanahashi, K., & Yamaguchi, Y. (2024) · IEEE/CVF Winter Conference on Applications of Computer Vision Workshops (WACVW) · doi:10.1109/wacvw60836.2024.00104 · Cited by 27
Chen, Y., Sikka, K., Cogswell, M., Ji, H., & Divakaran, A. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01350 · Cited by 56
VL-Trojan: Multimodal Instruction Backdoor Attacks against Autoregressive Visual Language Models
Liang, J., Liang, S., Liu, A., & Cao, X. (2025) · International Journal of Computer Vision · doi:10.1007/s11263-025-02368-9 · Cited by 68
Towards Responsible and Reliable Traffic Flow Prediction with Large Language ModelsPreprint
Guo, X., Zhang, Q., Jiang, J., Peng, M., Yang, H., & Zhu, M. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4805901 · Cited by 22
Du, S. Q., Tang, S., Wang, W., Li, X., & Guo, R. (2023) · The international archives of the photogrammetry, remote sensing and spatial information sciences/International archives of the photogrammetry, remote sensing and spatial information sciences · doi:10.5194/isprs-archives-xlviii-1-w2-2023-1729-2023 · Cited by 40
Self-citationYang, X., Wu, W., Feng, S., Wang, M., Wang, D., Li, Y., Sun, Q., Zhang, Y., Fu, X., & Poria, S. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103204 · Cited by 29
BT-Adapter: Video Conversation is Feasible Without Video Instruction Tuning
Liu, R., Li, C., Ge, Y., Li, T. H., Shan, Y., & Li, G. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01296 · Cited by 54
Interactive Continual Learning: Fast and Slow Thinking
Qi, B., Chen, X., Gao, J., Dong, L., Liu, J., Wu, L., & Zhou, B. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01224 · Cited by 21
Feedback-Guided Autonomous Driving
Zhang, J., Huang, Z., Ray, A., & Ohn-Bar, E. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01421 · Cited by 24
SALMONN: Towards Generic Hearing Abilities for Large Language Models
Tang, C., Yu, W., Sun, G., Chen, X., Tan, T., Li, W., Lu, L., Ma, Z., & Zhang, C. (2023) · arXiv · doi:10.48550/arxiv.2310.13289 · Cited by 67
“In addition to audio, multimodal LLMs are more widely studied in visual modalities, such as image (Zhu et al., 2023; Li et al., 2023a), video (Maaz et al., 2023; Chen et al., 2023b) and audio-visual (Su et al., 2023; Lyu et al., 2023; Sun et al., 2023).”
BackgroundMethodFabGPT: An Efficient Large Multimodal Model for Complex Wafer Defect Knowledge Queries
Jiang, Y., Lu, X., Jin, Q., Sun, Q. L., Wu, H., & Zhuo, C. (2024) · doi:10.1145/3676536.3676750 · Cited by 29
Incorporating Structured Representations into Pretrained Vision & Language Models Using Scene Graphs
Herzig, R., Mendelson, A., Karlinsky, L., Arbelle, A., Feris, R., Darrell, T., & Globerson, A. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.870 · Cited by 27
Metadata
- Authors
- Zhu, D., Chen, J., Shen, X., Li, X., & Elhoseiny, M.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2304.10592
- OpenAlex
- W4366850747
- Cited by
- 469
- References
- 0
- Citations by year
- 2026: 18 · 2025: 125 · 2024: 260 · 2023: 65
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by)
- King Abdullah University of Science and Technology Repository (King Abdullah University of Science and Technology) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- vision-language models, GPT-4, vicuña, frozen large language models, image description generation, image captioning, fine-tuning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
