The DOI matches its DataCite record; 2 of the 2 references it cites were found in a registry or an index.
MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models
Open accessAbstract
The recent GPT-4 has demonstrated extraordinary multi-modal abilities, such as directly generating websites from handwritten text and identifying humorous elements within images. These features are rarely observed in previous vision-language models. However, the technical details behind GPT-4 continue to remain undisclosed. We believe that the enhanced multi-modal generation capabilities of GPT-4 stem from the utilization of sophisticated large language models (LLM). To examine this phenomenon, we present MiniGPT-4, which aligns a frozen visual encoder with a frozen advanced LLM, Vicuna, using one projection layer. Our work, for the first time, uncovers that properly aligning the visual features with an advanced large language model can possess numerous advanced multi-modal abilities demonstrated by GPT-4, such as detailed image description generation and website creation from hand-drawn drafts. Furthermore, we also observe other emerging capabilities in MiniGPT-4, including writing stories and poems inspired by given images, teaching users how to cook based on food photos, and so on. In our experiment, we found that the model trained on short image caption pairs could produce unnatural language outputs (e.g., repetition and fragmentation). To address this problem, we curate a detailed image description dataset in the second stage to finetune the model, which consequently improves the model's generation reliability and overall usability. Our code, pre-trained model, and collected dataset are available at https://minigpt-4.github.io/.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Cited by
Sorted by citations
Items 201–300 of 497
Equivariant Similarity for Vision-Language Foundation Models
Wang, T., Lin, K., Li, L., Lin, C.-C., Yang, Z., Zhang, H., Liu, Z., & Wang, L. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01102 · Cited by 32
Developing ChatGPT for biology and medicine: a complete review of biomedical question answering
Li, Q., Li, L., & Li, Y. (2024) · Biophysics Reports · doi:10.52601/bpr.2024.240004 · Cited by 27
An Advanced Driving Agent with the Multimodal Large Language Model for Autonomous Vehicles
Chen, J., & Lu, S. (2024) · IEEE International Conference on Mobility, Operations, Services and Technologies (MOST) · doi:10.1109/most60774.2024.00009 · Cited by 22
Self-citationVisual Fact Checker: Enabling High-Fidelity Detailed Caption Generation
Ge, Y., Zeng, X., Huffman, J. S., Lin, T., Liu, M.-Y., & Cui, Y. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01331 · Cited by 27
Pre-trained Multimodal Large Language Model Enhances Dermatological Diagnosis using SkinGPT-4Preprint
Zhou, J., He, X., Sun, L., Xu, J., Chen, X., Chu, Y., Zhou, L., Liao, X., Zhang, B., & Gao, X. (2023) · medRxiv · doi:10.1101/2023.06.10.23291127 · Cited by 20
Bai, L., Zhang, X., Wang, H., & Du, S. (2024) · Remote Sensing of Environment · doi:10.1016/j.rse.2024.114573 · Cited by 22
Ma, Y., Ye, W., Cui, C., Zhang, H., Xing, S., Ke, F., Wang, J., Miao, C., Chen, J., Rezatofighi, H., et al. (2025) · IEEE/CVF Winter Conference on Applications of Computer Vision Workshops (WACVW) · doi:10.1109/wacvw65960.2025.00114 · Cited by 23
Self-citationGe, J., Zhang, X., Zheng, Y., Guo, K., & Liang, J. (2025) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2025.05.002 · Cited by 22
Foundation models and Transformers for anomaly detection: A survey
Ammar, M. B., Mendoza, A., Belkhir, N., Antoine Manzanera, B., & Franchi, G. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103517 · Cited by 25
Can We Edit Multimodal Large Language Models?
Cheng, S., Tian, B., Liu, Q., Chen, X., Wang, Y., Chen, H., & Zhang, N. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.856 · Cited by 55
MtArtGPT: A Multi-Task Art Generation System With Pre-Trained Transformer
Jin, C., Zhu, R., Zhu, Z., Yang, L., Min, Y., & Luo, J. (2024) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2024.3349567 · Cited by 21
Kaul, P., Li, Z., Yang, H., Dukler, Y., Swaminathan, A. D., Taylor, C. J., & Soatto, S. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02571 · Cited by 37
A Simple Recipe for Contrastively Pre-Training Video-First Encoders Beyond 16 Frames
Papalampidi, P., Koppula, S., Pathak, S., Chiu, J., Heyward, J., Pătrăucean, V., Shen, J., Miech, A., Zisserman, A., & Nematzdeh, A. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01364 · Cited by 29
MP5: A Multi-modal Open-ended Embodied System in Minecraft via Active Perception
Qin, Y., Zhou, E., Liu, Q., Yin, Z., Lu, S., Zhang, R., Qiao, Y., & Shao, J. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01543 · Cited by 48
ChatEarthNet: a global-scale image–text dataset empowering vision–language geo-foundation models
Yuan, Z., Xiong, Z., Mou, L., & Zhu, X. X. (2025) · Earth system science data · doi:10.5194/essd-17-1245-2025 · Cited by 36
Koala: Key Frame-Conditioned Long Video-LLM
Tan, R., Sun, X., Hu, P., Wang, J.-H., Deilamsalehy, H., Plummer, B. A., Russell, B. C., & Saenko, K. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01289 · Cited by 25
MemeCap: A Dataset for Captioning and Interpreting Memes
Hwang, E., & Shwartz, V. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.89 · Cited by 41
Yang, J., Mark, M. S., Vu, B., Sharma, A., Bohg, J., & Finn, C. (2024) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra57147.2024.10610421 · Cited by 35
Jack of All Tasks, Master of Many: Designing General-purpose Coarse-to-Fine Vision-Language Model
Pramanick, S., Han, G., Hou, R., Nag, S., Lim, S.-N., Ballas, N., Wang, Q., Chellappa, R., & Almahairi, A. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01335 · Cited by 35
A Survey of Reasoning with Foundation ModelsPreprint
Sun, J., Zheng, C., Xie, E., Liu, Z., Chu, R., Liu, J., Xu, J., Ding, M., Li, H., Geng, M., et al. (2023) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/ac4sp · Cited by 143
Self-citationExploring the Transferability of Visual Prompting for Multimodal Large Language Models
Zhang, Y., Dong, Y., Zhang, S., Min, T., Su, H., & Zhu, J. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.02508 · Cited by 17
Seeing the Unseen: Visual Common Sense for Semantic Placement
Ramrakhya, R., Kembhavi, A., Batra, D., Kira, Z., Zeng, K.-H., & Weihs, L. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01540 · Cited by 19
VL-FAS: Domain Generalization via Vision-Language Model For Face Anti-Spoofing
Fang, H., Liu, A., Jiang, N., Lü, Q., Zhao, G., & Wan, J. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10448156 · Cited by 17
Beyond Text: Frozen Large Language Models in Visual Signal Comprehension
Zhu, L., Wei, F., & Lu, Y. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02554 · Cited by 24
Power-Llava: Large Language and Vision Assistant for Power Transmission Line Inspection
Wang, J., Li, M., Luo, H., Zhu, J., Yang, A., Rong, M., & Wang, X. (2024) · Proceedings - International Conference on Image Processing · doi:10.1109/icip51287.2024.10648271 · Cited by 22
ADAGENT: Anomaly Detection Agent With Multimodal Large Models in Adverse Environments
Zhang, M., Shen, Y., Yin, J., Lü, S., & Wang, X. (2024) · IEEE Access · doi:10.1109/access.2024.3480250 · Cited by 20
MiniMedGPT: Efficient Large Vision–Language Model for medical Visual Question Answering
Alsabbagh, A. R., Mansour, T., Al-Kharabsheh, M., Ebdah, A. S., Al-Emaryeen, R., Al-Nahhas, S., Mahafza, W. S., & Al-Kadi, O. S. (2025) · Pattern Recognition Letters · doi:10.1016/j.patrec.2025.01.001 · Cited by 17
DDFAV: Remote Sensing Large Vision Language Models Dataset and Evaluation Benchmark
Li, H., Zhang, X., & Qu, H. (2025) · Remote Sensing · doi:10.3390/rs17040719 · Cited by 18
DrugChat: Towards Enabling ChatGPT-Like Capabilities on Drug Molecule GraphsPreprint
Liang, Y., Zhang, R., Zhang, L., & Xie, P. (2023) · doi:10.36227/techrxiv.22945922.v1 · Cited by 19
Konwer, A., Yang, Z., Bas, E., Xiao, C., Prasanna, P., Bhatia, P., & Kass-Hout, T. (2025) · 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52734.2025.01955 · Cited by 17
SkinGPT-4: an interactive dermatology diagnostic system with visual large language model
Zhou, J., He, X., Sun, L., Xu, J., Chen, X., Chu, Y., Zhou, L., Liao, X., Zhang, B., & Gao, X. (2023) · arXiv · doi:10.48550/arxiv.2304.10691 · Cited by 17
Low-Rank Approximation for Sparse Attention in Multi-Modal LLMs
Lin, S., Chen, Y., Yang, S., Ding, X., Ge, Y., Chen, Y.-C., & Shan, Y. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01306 · Cited by 20
Composed Video Retrieval via Enriched Context and Discriminative Embeddings
Thawakar, O., Naseer, M., Anwer, R. M., Khan, S., Felsberg, M., Shah, M., & Khan, F. S. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02540 · Cited by 20
Multi-Modal Instruction Tuned LLMs with Fine-Grained Visual Perception
He, J., Wang, Y., Wang, L., Lu, H., He, J.-Y., Lan, J.-P., Luo, B., & Xie, X. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01326 · Cited by 26
Automatic Medical Report Generation: Methods and Applications
Li, G., Tahir, A. M., Zhang, D., Wang, Z. J., & Ward, R. K. (2024) · APSIPA Transactions on Signal and Information Processing · doi:10.1561/116.20240044 · Cited by 20
Deng, Z., Ma, Y., Liu, Y., Guo, R., Zhang, G., Chen, W., Huang, W., & Benetos, E. (2024) · Findings of the Association for Computational Linguistics: NAACL 2024 · doi:10.18653/v1/2024.findings-naacl.231 · Cited by 15
Guo, M., Wu, M., Shen, Y., Li, H., & Tao, C. (2025) · Pattern Recognition · doi:10.1016/j.patcog.2025.111672 · Cited by 18
Towards Inadequately Pre-trained Models in Transfer Learning
Deng, A., Li, X., Hu, D., Wang, T., Xiong, H., & Xu, C. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01777 · Cited by 17
Self-citationAn Overview of Text-Based Person Search: Recent Advances and Future Directions
Niu, K., Liu, Y., Long, Y., Huang, Y., Wang, L., & Zhang, Y. (2024) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2024.3376373 · Cited by 18
Visual Instruction Tuning with Polite Flamingo
Chen, D., Liu, J., Dai, W., & Wang, B. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i16.29727 · Cited by 27
A Vision Check-up for Language Models
Sharma, P., Shaham, T. R., Baradad, M., Rodriíuez-Muñoz, A., Duggal, S., Isola, P., Torralba, A., & Fu, S. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.01366 · Cited by 15
ImageNet-D: Benchmarking Neural Network Robustness on Diffusion Synthetic Object
Zhang, C., Pan, F., Kim, J., Kweon, I. S., & Mao, C. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02055 · Cited by 21
ModaVerse: Efficiently Transforming Modalities with LLMs
Wang, X., Zhuang, B., & Wu, Q. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02512 · Cited by 26
What If the TV was off? Examining Counterfactual Reasoning Abilities of Multi-modal Language Models
Zhang, L., Zhai, X., Zhao, Z., Zong, Y., Wen, X., & Zhao, B. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02064 · Cited by 19
When Large Vision-Language Models Meet Person Re-Identification
Wang, Q., Li, B., & Xue, X. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11460593 · Cited by 26
IMFND: In-context multimodal fake news detection with large visual-language models
Jiang, Y., & Wang, Y. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.113880 · Cited by 27
Chakraborty, M., Tonmoy, S. M. T. I., Zaman, S. M. M., Gautam, S., Kumar, T., Sharma, K., Barman, N. R., Gupta, C., Jain, V., Chadha, A. R., et al. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.136 · Cited by 15
GPTSee: Enhancing Moment Retrieval and Highlight Detection via Description-Based Similarity Features
Sun, Y., Xu, Y., Xie, Z., Shu, Y., & Du, S. (2023) · IEEE Signal Processing Letters · doi:10.1109/lsp.2023.3340103 · Cited by 15
Zhai, Y., Chen, L., & Deng, M. (2024) · Briefings in Bioinformatics · doi:10.1093/bib/bbae188 · Cited by 14
A survey for table recognition based on deep learning
Yu, C., Li, W., Li, W., Zhu, Z., Liu, R., Hou, B., & Jiao, L. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.128154 · Cited by 16
Prompt Highlighter: Interactive Control for Multi-Modal LLMs
Zhang, Y., Qian, S., Peng, B., Liu, S., & Jia, J. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01255 · Cited by 25
Zhang, L., Awal, R., & Agrawal, A. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01307 · Cited by 25
Zhang, Y., & Chen, X. (2023) · Research Square · doi:10.21203/rs.3.rs-3778538/v1 · Cited by 15
ChatNT: A Multimodal Conversational Agent for DNA, RNA and Protein TasksPreprint
Richard, G., de Almeida, B. P., Dalla-Torre, H., Blum, C. F., Hexemer, L., Pandey, P., Laurent, S., Lopez, M., Laterre, A., Lang, M., et al. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.04.30.591835 · Cited by 14
Compound Text-Guided Prompt Tuning via Image-Adaptive Cues
Tan, H., Li, J., Zhou, Y., Wan, J., Lei, Z., & Zhang, X. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i5.28311 · Cited by 15
Toward Automated Programming for Robotic Assembly Using ChatGPT
Macaluso, A., Cote, N., & Chitta, S. (2024) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra57147.2024.10610554 · Cited by 19
VIT-LENS: Towards Omni-modal Representations
Lei, W., Ge, Y., Yi, K., Zhang, J., Gao, D., Sun, D., Ge, Y., Shan, Y., & Shou, M. Z. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02516 · Cited by 21
Semantic Alignment for Multimodal Large Language Models
Wu, T., Li, M., Chen, J., Ji, W., Wang, L., Gao, J., Kuang, K., Zhao, Z., & Wu, F. Q. (2024) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3664647.3681014 · Cited by 13
Self-citationConsistent prompt learning for vision-language models
Zhang, Y., & Tian, X. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.112974 · Cited by 15
ProteinChat: Towards Achieving ChatGPT-Like Functionalities on Protein 3D StructuresPreprint
Guo, H., Huo, M., Zhang, R., & Xie, P. (2023) · doi:10.36227/techrxiv.23120606 · Cited by 15
Is It Safe to Cross? Interpretable Risk Assessment with GPT-4V for Safety-Aware Street Crossing
Hwang, H., Kwon, S., Kim, Y., & Kim, D. (2024) · International Conference on Ubiquitous Robots (UR) · doi:10.1109/ur61395.2024.10597464 · Cited by 16
Codebook Transfer with Part-of-Speech for Vector-Quantized Image Modeling
Zhang, B., Wang, H., Luo, C., Li, X., Liang, G., Ye, Y., Qi, X., & He, Y. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00741 · Cited by 13
Self-citationLink-Context Learning for Multimodal LLMs
Tai, Y., Fan, W., Zhang, Z., & Liu, Z. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02566 · Cited by 20
Wu, Y., Zhou, S., Wu, Z., Chen, Z., Hu, X., & Li, J. (2024) · Computers and Electronics in Agriculture · doi:10.1016/j.compag.2024.109452 · Cited by 14
Label correlated contrastive learning for medical report generation
Liu, X., Xin, J., Dai, B. T., Shen, Q., Huang, Z., & Wang, Z. (2024) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2024.108482 · Cited by 15
Vehicle Damage Detection Using Artificial Intelligence: A Systematic Literature Review
Hasan, M. J., Nguyen, C. K., Boo, Y. L., Jahani, H., & Ong, K. (2025) · Wiley Interdisciplinary Reviews Data Mining and Knowledge Discovery · doi:10.1002/widm.70027 · Cited by 19
Language-Conditioned Robotic Manipulation with Fast and Slow Thinking
Zhu, M., Zhu, Y., Li, J., Wen, J., Xu, Z., Che, Z., Shen, C., Peng, Y., Liu, D., Feng, F., et al. (2024) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra57147.2024.10611525 · Cited by 21
Adaptive Query Contextualization Algorithm for Enhanced Information Retrieval in Alpaca LLMPreprint
Kuo, C.-W., Huang, Y.-F., & Tsai, H.-C. (2023) · Research Square · doi:10.21203/rs.3.rs-3806145/v1 · Cited by 12
Traffic-IT: Enhancing traffic scene understanding for multimodal large language models
Kuang, S., Liu, Y., Qu, X., & Wei, Y. (2025) · Transportation Research Part C Emerging Technologies · doi:10.1016/j.trc.2025.105325 · Cited by 16
AVI-Talking: Learning Audio-Visual Instructions for Expressive 3D Talking Face Generation
Sun, Y., Chu, W., Zhou, H., Wang, K., & Koike, H. (2024) · IEEE Access · doi:10.1109/access.2024.3390182 · Cited by 17
Towards More Unified In-Context Visual Understanding
Sheng, D., Chen, D., Tan, Z., Liu, Q., Chu, Q., Bao, J., Gong, T., Liu, B., Xu, S., & Yu, N. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01269 · Cited by 14
HRVDA: High-Resolution Visual Document Assistant
Liu, C., Yin, K., Cao, H., Jiang, X., Li, X., Liu, Y., Jiang, D., Sun, X. W., & Xu, L. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01471 · Cited by 23
Self-citationEnhanced Motion-Text Alignment for Image-to-Video Transfer Learning
Zhang, W., Wan, C., Liu, T., Tian, X., Shen, X., & Ye, J. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01751 · Cited by 13
Self-citationCo-LLaVA: Efficient Remote Sensing Visual Question Answering via Model Collaboration
Liu, F., Dai, W., Zhang, C., Zhu, J., Yao, L. X., & Li, X. (2025) · Remote Sensing · doi:10.3390/rs17030466 · Cited by 13
Self-citationZhou, Y., Wang, Z., Jiang, Y., Ma, B., Wang, R., Liu, Y., Zhao, Y., & Tian, Z. (2025) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3735645 · Cited by 17
Path to Medical AGI: Unify Domain-specific Medical LLMs with the Lowest CostPreprint
Zhou, J., Chen, X., & Gao, X. (2023) · medRxiv · doi:10.1101/2023.06.23.23291802 · Cited by 21
Thejaswin, S., Prakash, A., Nambiar, A., & Bernadino, A. (2024) · IEEE Transactions on Biometrics Behavior and Identity Science · doi:10.1109/tbiom.2024.3405081 · Cited by 10
Prompting Multi-Modal Tokens to Enhance End-to-End Autonomous Driving Imitation Learning with LLMs
Duan, Y., Zhang, Q., & Xu, R. (2024) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra57147.2024.10611614 · Cited by 18
Wang, Y., Liu, C., Qu, Y., Cao, H., Jiang, D., & Xu, L. (2024) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3664647.3680779 · Cited by 10
Unifying Text, Tables, and Images for Multimodal Question Answering
Luo, H., Shen, Y. J., & Deng, Y. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.626 · Cited by 11
Autorepo: A General Framework for Multimodal Llm-Based Automated Construction ReportingPreprint
Pu, H., Yang, X., Li, J., & Guo, R. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4678046 · Cited by 10
Multi-Modal Large Language Model Enables Protein Function PredictionPreprint
Huo, M., Guo, H. W., Cheng, X., Singh, D., Rahmani, H., Li, S., Gerlof, P., Ideker, T., Grotjahn, D. A., Villa, E., et al. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.08.19.608729 · Cited by 12
VRSBench: A Versatile Vision-Language Benchmark Dataset for Remote Sensing Image Understanding
Li, X., Ding, J., & Elhoseiny, M. (2024) · arXiv · doi:10.48550/arxiv.2406.12384 · Cited by 47
“In the past few years, the success of large vision-language models (LVLMs)[1, 2, 3, 4, 5] in natural scenes has inspired a trend of applying LVLMs to remote sensing [6].”
Self-citationLinguistic Hallucination for Text-Based Video Retrieval
Fang, S., Dang, T., Wang, S., & Huang, Q. (2024) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2024.3393843 · Cited by 11
Semantics-Aware Motion Retargeting with Vision-Language Models
Zhang, H., Chen, Z., Xu, H., Hao, L., Wu, X., Xu, S., Zhang, Z., Wang, Y., & Xiong, R. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00210 · Cited by 9
Liu, C. X., Guo, S., Dang, F., Qiu, X., & Shao, S. (2024) · Big Data Mining and Analytics · doi:10.26599/bdma.2024.9020029 · Cited by 9
Fatehi, K., Torres, M. T., & Küçükyılmaz, A. (2024) · Speech Communication · doi:10.1016/j.specom.2024.103151 · Cited by 12
Li, X., Hou, X., Ravi, N., Huang, Z., & Gan, Y. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.127833 · Cited by 10
Self-citationLiu, B., Zhang, H., Liu, J., & Wang, Q. (2023) · IEEE International Conference on Sensing, Communication, and Networking (SECON) · doi:10.1109/secon58729.2023.10287530 · Cited by 9
ROME: Evaluating Pre-trained Vision-Language Models on Reasoning beyond Visual Common Sense
Zhou, K., Lai, E., Yeong, W. B. A., Mouratidis, K., & Jiang, J. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.683 · Cited by 19
ProteinChat: Towards Achieving ChatGPT-Like Functionalities on Protein 3D StructuresPreprint
Guo, H., Huo, M., Zhang, R., & Xie, P. (2023) · doi:10.36227/techrxiv.23120606.v1 · Cited by 8
Arabic Mini-ClimateGPT : A Climate Change and Sustainability Tailored Arabic LLM
Mullappilly, S. S., Shaker, A., Thawakar, O., Cholakkal, H., Anwer, R. M., Khan, S. H., & Khan, F. S. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.941 · Cited by 8
SOK-Bench: A Situated Video Reasoning Benchmark with Aligned Open-World Knowledge
Wang, A., Wu, B., Chen, S., Chen, Z., Guan, H., Lee, W.-N., Li, L. E., & Gan, C. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01271 · Cited by 13
Enhancing few-shot KB-VQA with panoramic image captions guided by Large Language Models
Qiang, P., Tan, H., Li, X., Wang, D., Li, R., Sun, X., Zhang, H., & Liang, J. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.129373 · Cited by 10
Self-citationAfzaal, U., Su, Z., Sajjad, U., Stack, T., Lü, H., Niu, S., Akbar, A., Gürcan, M. N., & Niazi, M. K. K. (2025) · Patterns · doi:10.1016/j.patter.2025.101284 · Cited by 9
Explainable medical visual question answering via chain of evidence
Qiu, C., Huang, K., Xie, Z., Liu, M., Gu, J., & Zong, X. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.113672 · Cited by 8
Specialized curricula for training vision language models in retinal image analysis
Holland, R., Taylor, T. R. P., Holmes, C., Riedl, S., Mai, J., Patsiamanidi, M., Mitsopoulou, D., Hager, P., Müller, P., Paetzold, J. C., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01893-8 · Cited by 12
Li, S., Han, C., Yu, P., Edwards, C., Li, M., Wang, X., Fung, Y. R., Yu, C., Tetreault, J., Hovy, E. H., et al. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.799 · Cited by 11
Object-Centric Instruction Augmentation for Robotic Manipulation
Wen, J., Zhu, Y., Zhu, M., Li, J., Xu, Z., Che, Z., Shen, C., Peng, Y., Liu, D., Feng, F., et al. (2024) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra57147.2024.10609992 · Cited by 13
Xu, J., Zhou, X., Shen, Y., Gu, X., Arnab, A., Sun, C., Wang, X., & Schmid, C. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01238 · Cited by 23
Metadata
- Authors
- Zhu, D., Chen, J., Shen, X., Li, X., & Elhoseiny, M.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2304.10592
- OpenAlex
- W4366850747
- Cited by
- 469
- References
- 0
- Citations by year
- 2026: 18 · 2025: 125 · 2024: 260 · 2023: 65
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by)
- King Abdullah University of Science and Technology Repository (King Abdullah University of Science and Technology) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- vision-language models, GPT-4, vicuña, frozen large language models, image description generation, image captioning, fine-tuning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
