Visual Instruction Tuning
The DOI matches its Crossref record.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Furthermore, recent studies have found a striking representational alignment between a wide variety of ANNs and the human brain, despite their having different architectures, network initializations, learning objectives, and even different modalities such as language and vision [12,15–22]”
From The Umwelt Representation Hypothesis: rethinking Universality (2026) · cites this paper as [21]
In the citing paper
“This model extends the general-purpose LLaVA framework82 to the medical domain by adapting it for radiograph interpretation and multimodal instruction following.”
From MoMA: a mixture-of-multimodal-agents architecture for enhancing clinical prediction modelling (2025) · cites this paper as [82]
In the citing paper
“Using other VLMs to synthetically generate annotations is a popular approach [84, 119], but it is likely to perpetuate inaccuracies or biases that the model already has (see distribution shift [110]), rather than capturing important nuances.”
From "It's trained by non-disabled people": Evaluating How Image Quality Affects Product Captioning with Vision-Language Models (2026) · cites this paper as [84]
In the citing paper
“VLM performance is heavily shaped by post-training activities, including fine-tuning on specific tasks (e.g., PixMoCap for captioning [35]) and diverse datasets [80], or training to provide answers in specific formats (e.g., instruction tuning [84]).”
From "It's trained by non-disabled people": Evaluating How Image Quality Affects Product Captioning with Vision-Language Models (2026) · cites this paper as [84]
Citations
Cited by
Improved Baselines with Visual Instruction Tuning
Liu, H., Li, C., Li, Y., & Lee, Y. J. (2023) · arXiv · doi:10.48550/arxiv.2310.03744 · Cited by 1,733
Self-citationImproved Baselines with Visual Instruction Tuning
Liu, H., Li, C., Li, Y., & Lee, Y. J. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.02484 · Cited by 1,748
Self-citationA survey on multimodal large language models
Yin, S., Fu, C., Zhao, S., Li, K., Sun, X., Xu, T., & Chen, E. (2024) · National Science Review · doi:10.1093/nsr/nwae403 · Cited by 893
Intern VL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks
Chen, Z., Wu, J., Wang, W., Su, W., Chen, G., Xing, S., Zhong, M., Zhang, Q., Zhu, X., Lu, L., et al. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.02283 · Cited by 734
ChatGPT for Robotics: Design Principles and Model Abilities
Vemprala, S. H., Bonatti, R., Bucker, A., & Kapoor, A. (2024) · IEEE Access · doi:10.1109/access.2024.3387941 · Cited by 570
LISA: Reasoning Segmentation via Large Language Model
Lai, X., Tian, Z., Chen, Y., Li, Y., Yuan, Y., Liu, S., & Jia, J. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.00915 · Cited by 466
DriveGPT4: Interpretable End-to-End Autonomous Driving Via Large Language Model
Xu, Z., Zhang, Y., Xie, E., Zhao, Z., Guo, Y., Wong, K.-Y. K., Li, Z., & Zhao, H. (2024) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2024.3440097 · Cited by 443
A multimodal generative AI copilot for human pathology
Lu, M. Y., Chen, B., Williamson, D. F. K., Chen, R. J., Zhao, M., Chow, A. K., Ikemura, K., Kim, A., Pouli, D., Patel, A., et al. (2024) · Nature · doi:10.1038/s41586-024-07618-3 · Cited by 461
MMMU: A Massive Multi-Discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI
Yue, X., Ni, Y., Zheng, T., Zhang, K., Liu, R., Zhang, G., Stevens, S., Jiang, D., Ren, W., Sun, Y., et al. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.00913 · Cited by 429
Chen, K., Liu, C., Chen, H., Zhang, H., Li, W., Zou, Z., & Shi, Z. (2024) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2024.3356074 · Cited by 383
A Survey on Multimodal Large Language Models for Autonomous Driving
Cui, C., Ma, Y., Cao, X., Ye, W., Zhou, Y., Liang, K., Chen, J., Lu, J., Yang, Z., Liao, K.-D., et al. (2024) · 2024 IEEE/CVF Winter Conference on Applications of Computer Vision Workshops (WACVW) · doi:10.1109/wacvw60836.2024.00106 · Cited by 369
Atakishiyev, S., Salameh, M., Yao, H., & Goebel, R. (2024) · IEEE Access · doi:10.1109/access.2024.3431437 · Cited by 323
Atakishiyev, S., Salameh, M., Yao, H., & Goebel, R. (2021) · arXiv · doi:10.48550/arxiv.2112.11561 · Cited by 336
How far are we to GPT-4V? Closing the gap to commercial multimodal models with open-source suites
Chen, Z., Wang, W., Tian, H., Ye, S., Gao, Z., Cui, E., Tong, W., Hu, K., Luo, J., Ma, Z., et al. (2024) · Science China Information Sciences · doi:10.1007/s11432-024-4231-5 · Cited by 325
Foundation Models Defining a New Era in Vision: A Survey and Outlook
Awais, M., Naseer, M., Khan, S., Anwer, R. M., Cholakkal, H., Shah, M., Yang, M.-H., & Khan, F. S. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3506283 · Cited by 304
Empowering biomedical discovery with AI agents
Gao, S., Fang, A., Huang, Y., Giunchiglia, V., Noori, A., Schwarz, J. R., Ektefaie, Y., Kondic, J., & Zitnik, M. (2024) · Cell · doi:10.1016/j.cell.2024.09.022 · Cited by 315
MVBench: A Comprehensive Multi-modal Video Understanding Benchmark
Li, K., Wang, Y., He, Y., Li, Y., Wang, Y., Liu, Y., Wang, Z., Xu, J., Chen, G., Lou, P., et al. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.02095 · Cited by 300
SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities
Chen, B., Xu, Z., Kirmani, S., Ichter, B., Sadigh, D., Guibas, L., & Xia, F. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.01370 · Cited by 281
Multimodal Large Language Models: A Survey
Wu, J., Gan, W., Chen, Z., Wan, S., & Yu, P. S. (2023) · 2023 IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata59044.2023.10386743 · Cited by 252
LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Models
Xu, P., Shao, W., Zhang, K., Gao, P., Liu, S., Lei, M., Meng, F., Huang, S., Qiao, Y., & Luo, P. (2023) · arXiv · doi:10.48550/arxiv.2306.09265 · Cited by 273
SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities
Chen, B., Xu, Z., Kirmani, S., Ichter, B., Driess, D., Florence, P., Sadigh, D., Guibas, L., & Xia, F. (2024) · arXiv · doi:10.48550/arxiv.2401.12168 · Cited by 269
RSGPT: A remote sensing vision language model and benchmark
Hu, Y., Yuan, J., Wen, C., Lu, X., Liu, Y., & Li, X. (2025) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2025.03.028 · Cited by 187
RSGPT: A Remote Sensing Vision Language Model and Benchmark
Hu, Y., Yuan, J., Wen, C., Lu, X., & Li, X. (2023) · arXiv · doi:10.48550/arxiv.2307.15266 · Cited by 219
Zhang, W., Cai, M., Zhang, T., Zhuang, Y., & Mao, X. (2024) · arXiv · doi:10.48550/arxiv.2401.16822 · Cited by 203
CogAgent: A Visual Language Model for GUI Agents
Hong, W., Wang, W., Lv, Q., Xu, J., Yu, W., Ji, J., Wang, Y., Wang, Z., Dong, Y., Ding, M., et al. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.01354 · Cited by 202
Zhang, W., Cai, M., Zhang, T., Zhuang, Y., & Mao, X. (2024) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2024.3409624 · Cited by 185
GLaMM: Pixel Grounding Large Multimodal Model
Rasheed, H., Maaz, M., Shaji, S., Shaker, A., Khan, S., Cholakkal, H., Anwer, R. M., Xing, E., Yang, M.-H., & Khan, F. S. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.01236 · Cited by 189
Friha, O., Amine Ferrag, M., Kantarci, B., Cakmak, B., Ozgun, A., & Ghoualmi-Zine, N. (2024) · IEEE Open Journal of the Communications Society · doi:10.1109/ojcoms.2024.3456549 · Cited by 189
LSKNet: A Foundation Lightweight Backbone for Remote Sensing
Li, Y., Li, X., Dai, Y., Hou, Q., Liu, L., Liu, Y., Cheng, M.-M., & Yang, J. (2025) · International Journal of Computer Vision · doi:10.1007/s11263-024-02247-9 · Cited by 184
Vision-Language Models in Remote Sensing: Current progress and future trends
Li, X., Wen, C., Hu, Y., Yuan, Z., & Zhu, X. X. (2024) · IEEE Geoscience and Remote Sensing Magazine · doi:10.1109/mgrs.2024.3383473 · Cited by 178
OCRBench: on the hidden mystery of OCR in large multimodal models
Liu, Y., Li, Z., Huang, M., Yang, B., Yu, W., Li, C., Yin, X.-C., Liu, C.-L., Jin, L., & Bai, X. (2024) · Science China Information Sciences · doi:10.1007/s11432-024-4235-6 · Cited by 167
Self-citationmPLUG-OwI2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration
Ye, Q., Xu, H., Ye, J., Yan, M., Hu, A., Liu, H., Qian, Q., Zhang, J., & Huang, F. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.01239 · Cited by 184
Self-citationLMDrive: Closed-Loop End-to-End Driving with Large Language Models
Shao, H., Hu, Y., Wang, L., Song, G., Waslander, S. L., Liu, Y., & Li, H. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.01432 · Cited by 174
Towards Open Vocabulary Learning: A Survey
Wu, J., Li, X., Xu, S., Yuan, H., Ding, H., Yang, Y., Li, X., Zhang, J., Tong, Y., Jiang, X., et al. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3361862 · Cited by 164
Otter: A Multi-Modal Model With In-Context Instruction Tuning
Li, B., Zhang, Y., Chen, L., Wang, J., Pu, F., Cahyono, J. A., Yang, J., Li, C., & Liu, Z. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3571946 · Cited by 158
Self-citationUnderstanding World or Predicting Future? A Comprehensive Survey of World Models
Ding, J., Zhang, Y., Shang, Y., Zhang, Y., Zong, Z., Feng, J., Yuan, Y., Su, H., Li, N., Sukiennik, N., et al. (2026) · ACM Computing Surveys · doi:10.1145/3746449 · Cited by 169
Guan, T., Liu, F., Wu, X., Xian, R., Li, Z., Liu, X., Wang, X., Chen, L., Huang, F., Yacoob, Y., et al. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.01363 · Cited by 166
(2023) · doi:10.48550/arxiv.2310.14566 · Cited by 166
Zhan, Y., Xiong, Z., & Yuan, Y. (2025) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2025.01.020 · Cited by 330
Large Language Models on Graphs: A Comprehensive Survey
Jin, B., Liu, G., Han, C., Jiang, M., Ji, H., & Han, J. (2024) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2024.3469578 · Cited by 147
Visual Adversarial Examples Jailbreak Aligned Large Language Models
Qi, X., Huang, K., Panda, A., Henderson, P., Wang, M., & Mittal, P. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i19.30150 · Cited by 150
Generative Multimodal Models are In-Context Learners
Sun, Q., Cui, Y., Zhang, X., Zhang, F., Yu, Q., Wang, Y., Rao, Y., Liu, J., Huang, T., & Wang, X. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.01365 · Cited by 147
LLM4TS: Aligning Pre-Trained LLMs as Data-Efficient Time-Series Forecasters
Chang, C., Wang, W., Peng, W.-C., & Chen, T.-F. (2025) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3719207 · Cited by 452
LLM4TS: Aligning Pre-Trained LLMs as Data-Efficient Time-Series Forecasters
Chang, C., Wang, W.-Y., Peng, W.-C., & Chen, T.-F. (2023) · arXiv · doi:10.48550/arxiv.2308.08469 · Cited by 146
Waqas, A., Bui, M. M., Glassy, E. F., El Naqa, I., Borkowski, P., Borkowski, A. A., & Rasool, G. (2023) · Laboratory Investigation · doi:10.1016/j.labinv.2023.100255 · Cited by 140
Zhang, Z., Zhao, T., Guo, Y., & Yin, J. (2024) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2024.3449154 · Cited by 418
TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understanding
Ren, S., Yao, L., Li, S., Sun, X., & Hou, L. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01357 · Cited by 648
A Survey on Video Diffusion Models
Xing, Z., Feng, Q., Chen, H., Dai, Q., Hu, H., Xu, H., Wu, Z., & Jiang, Y. (2024) · ACM Computing Surveys · doi:10.1145/3696415 · Cited by 390
Boosting Continual Learning of Vision-Language Models via Mixture-of-Experts Adapters
Yu, J., Zhuge, Y., Zhang, L., Hu, P., Wang, D., Lu, H., & He, Y. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02191 · Cited by 384
VTimeLLM: Empower LLM to Grasp Video Moments
Huang, B., Wang, X. Y., Chen, H., Song, Z., & Zhu, W. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01353 · Cited by 441
Diffusion Model-Based Image Editing: A Survey
Huang, Y., Huang, J., Liu, Y., Yan, M., Lv, J., Liu, J., Xiong, W., Zhang, H., Cao, L., & Chen, S. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3541625 · Cited by 410
Jin, P., Takanobu, R., Zhang, W., Cao, X., & Li, Y. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01300 · Cited by 575
Large language models for disease diagnosis: a scoping review
Zhou, S., Xu, Z., Zhang, M., Xu, C., Guo, Y., Zhan, Z., Fang, Y., Ding, S., Wang, J., Xu, K., et al. (2025) · npj Artificial Intelligence · doi:10.1038/s44387-025-00011-z · Cited by 122
Woodpecker: hallucination correction for multimodal large language models
Yin, S., Fu, C., Zhao, S., Xu, T., Wang, H., Sui, D., Shen, Y., Li, K., Sun, X., & Chen, E. (2024) · Science China Information Sciences · doi:10.1007/s11432-024-4251-x · Cited by 118
CLIP in medical imaging: A survey
Zhao, Z., Liu, Y., Wu, H., Wang, M., Li, Y., Wang, S., Teng, L., Liu, D., Cui, Z., Wang, Q., et al. (2025) · Medical Image Analysis · doi:10.1016/j.media.2025.103551 · Cited by 282
Lu, J., Yu, L., Li, X., Yang, L., & Zuo, C. (2023) · 2023 IEEE 34th International Symposium on Software Reliability Engineering (ISSRE) · doi:10.1109/issre59848.2023.00026 · Cited by 112
Foundation Model for Advancing Healthcare: Challenges, Opportunities and Future Directions
He, Y., Huang, F., Jiang, X., Nie, Y., Wang, M., Wang, J., & Chen, H. (2024) · IEEE Reviews in Biomedical Engineering · doi:10.1109/rbme.2024.3496744 · Cited by 247
Yu, T., Yao, Y., Zhang, H., He, T., Han, Y., Cui, G., Hu, J., Liu, Z., Zheng, H.-T., & Sun, M. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.01310 · Cited by 113
Yu, T., Yao, Y., Zhang, H., He, T., Han, Y., Cui, G., Hu, J., Liu, Z., Zheng, H.-T., Sun, M., et al. (2023) · arXiv · doi:10.48550/arxiv.2312.00849 · Cited by 117
A comprehensive survey of large language models and multimodal large language models in medicine
Xiao, H., Zhou, F., Liu, X., Liu, T., Li, Z., Liu, X., & Huang, X. (2025) · Information Fusion · doi:10.1016/j.inffus.2024.102888 · Cited by 113
Sequential Modeling Enables Scalable Learning for Large Vision Models
Bai, Y., Geng, X., Mangalam, K., Bar, A., Yuille, A., Darrell, T., Malik, J., & Efros, A. A. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02157 · Cited by 314
LLaVA-MR: Large Language-and-Vision Assistant for Video Moment Retrieval
Li, J. (2024) · Qeios · doi:10.32388/vlxb6m · Cited by 112
BLIVA: A Simple Multimodal LLM for Better Handling of Text-Rich Visual Questions
Hu, W., Xu, Y., Li, Y., Li, W., Chen, Z., & Tu, Z. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i3.27999 · Cited by 105
LVLM-EHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Models
Xu, P., Shao, W., Zhang, K., Gao, P., Liu, S., Lei, M., Meng, F., Huang, S., Qiao, Y., & Luo, P. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3507000 · Cited by 99
Compositional Chain-of-Thought Prompting for Large Multimodal Models
Mitra, C., Huang, B., Darrell, T., & Herzig, R. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01367 · Cited by 390
Evolution and Prospects of Foundation Models: From Large Language Models to Large Multimodal Models
Chen, Z., Xu, L., Zheng, H., Chen, L., Tolba, A. K. R., Zhao, L., Yu, K., & Feng, H. (2024) · Computers, materials & continua/Computers, materials & continua (Print) · doi:10.32604/cmc.2024.052618 · Cited by 179
Materials science in the era of large language models: a perspective
Lei, G., Docherty, R., & Cooper, S. J. (2024) · Digital Discovery · doi:10.1039/d4dd00074a · Cited by 99
BB-GeoGPT: A framework for learning a large language model for geographic information science
Zhang, Y., Wang, Z., He, Z., Li, J., Mai, G., Lin, J., Cheng, W., & Yu, W. (2024) · Information Processing & Management · doi:10.1016/j.ipm.2024.103808 · Cited by 135
LLM Multimodal Traffic Accident Forecasting
de Zarzà, I., de Curtò, J., Roig, G., & Calafate, C. T. (2023) · Sensors · doi:10.3390/s23229225 · Cited by 130
Survey of Different Large Language Model Architectures: Trends, Benchmarks, and Challenges
Shao, M., Basit, A., Karri, R., & Shafique, M. (2024) · IEEE Access · doi:10.1109/access.2024.3482107 · Cited by 179
TinyVLA: Toward Fast, Data-Efficient Vision-Language-Action Models for Robotic Manipulation
Wen, J., Zhu, Y., Li, J., Zhu, M., Tang, Z., Wu, K., Xu, Z., Liu, N., Cheng, R., Shen, C., et al. (2025) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2025.3544909 · Cited by 548
Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces
Yang, J., Yang, S., Gupta, A. W., Han, R., Fei-Fei, L., & Xie, S. (2025) · 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52734.2025.00994 · Cited by 120
Sniffer: Multimodal Large Language Model for Explainable Out-of-Context Misinformation Detection
Qi, P., Yan, Z., Hsu, W., & Lee, M. L. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01240 · Cited by 212
Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teachers
Chen, T.-S., Siarohin, A., Menapace, W., Deyneka, E., Chao, H.-W., Jeon, B. E., Fang, Y., Lee, H.-Y., Ren, J. F., Yang, M.-H., et al. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01265 · Cited by 517
LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasoning, and Planning
Chen, S., Chen, X., Zhang, C., Li, M., Yu, G., Fei, H., Zhu, H., Fan, J., & Chen, T. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.02496 · Cited by 92
ManipLLM: Embodied Multimodal Large Language Model for Object-Centric Robotic Manipulation
Li, X., Zhang, M., Geng, Y., Geng, H., Long, Y., Shen, Y. F., Zhang, R., Liu, J., & Dong, H. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01710 · Cited by 288
OneLLM: One Framework to Align All Modalities with Language
Han, J., Gong, K., Zhang, Y., Wang, J., Zhang, K., Lin, D., Qiao, Y., Gao, P., & Yue, X. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02510 · Cited by 312
Hallucination Augmented Contrastive Learning for Multimodal Large Language Model
Jiang, C., Xu, H., Dong, M., Chen, J., Ye, W., Yan, M., Ye, Q., Zhang, J., Huang, F., & Zhang, S. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02553 · Cited by 315
The Large Language Models on Biomedical Data Analysis: A Survey
Lan, W., Tang, Z., Liu, M., Chen, Q., Peng, W., Chen, Y. P., & Pan, Y. F. (2025) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2025.3530794 · Cited by 113
UAVs meet LLMs: Overviews and perspectives towards agentic low-altitude mobility
Tian, Y., Lin, F., Li, Y., Zhang, T., Zhang, Q., Fu, X., Huang, J., Dai, X., Wang, Y., Tian, C., et al. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103158 · Cited by 175
An overview of large AI models and their applications
Tu, X., He, Z., Huang, Y., Zhang, Z., Yang, M., & Zhao, J. C. (2024) · Visual Intelligence · doi:10.1007/s44267-024-00065-8 · Cited by 134
Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Action
Lu, J., Clark, C., Lee, S.-H., Zhang, Z., Khosla, S., Marten, R., Hoiem, D., & Kembhavi, A. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02497 · Cited by 391
GSVA: Generalized Segmentation via Multimodal Large Language Models
Xia, Z., Han, D., Han, Y., Pan, X., Song, S., & Huang, G. M. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00370 · Cited by 304
SmartEdit: Exploring Complex Instruction-Based Image Editing with Multimodal Large Language Models
Huang, Y., Xie, L., Wang, X., Yuan, Z., Cun, X., Ge, Y., Zhou, J., Dong, C., Huang, R., Zhang, R., et al. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00799 · Cited by 320
CFN-ESA: A Cross-Modal Fusion Network With Emotion-Shift Awareness for Dialogue Emotion Recognition
Li, J., Wang, X., Liu, Y., & Zeng, Z. (2024) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2024.3389453 · Cited by 123
On Decoder-Only Architecture For Speech-to-Text and Large Language Model Integration
Wu, J., Gaur, Y., Chen, Z., Zhou, L., Zhu, Y., Wang, T.-R., Li, J., Liu, S.-J., Ren, B., Liu, L., et al. (2023) · Automatic Speech Recognition & Understanding · doi:10.1109/asru57964.2023.10389705 · Cited by 244
MoE-LLaVA : Mixture of Experts for Large Vision-Language Models
Lin, B., Tang, Z., Ye, Y., Huang, J., Zhang, J., Pang, Y., Jin, P., Ning, M., Luo, J., & Yuan, L. (2026) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2026.3654458 · Cited by 69
DiagLLM: multimodal reasoning with large language model for explainable bearing fault diagnosis
Wang, J., Li, T., Yang, Y., Chen, S., & Zhai, W. (2025) · Science China Information Sciences · doi:10.1007/s11432-024-4333-7 · Cited by 138
On the Adversarial Robustness of Multi-Modal Foundation Models
Schlarmann, C., & Hein, M. (2023) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw60793.2023.00395 · Cited by 228
VideoLLM-online: Online Video Large Language Model for Streaming Video
Chen, J., Lv, Z., Wu, S., Lin, K. Q., Song, C., Gao, D., Liu, J.-W., Gao, Z., Mao, D., & Shou, M. Z. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01742 · Cited by 368
Zhang, J., Li, Y., Fukuda, T., & Wang, B. (2025) · Cities · doi:10.1016/j.cities.2025.106122 · Cited by 115
CXR-LLaVA: a multimodal large language model for interpreting chest X-ray images
Lee, S., Youn, J., Kim, H., Kim, M., & Yoon, S. H. (2025) · European Radiology · doi:10.1007/s00330-024-11339-6 · Cited by 76
A Survey of Robot Intelligence with Large Language Models
Jeong, H., Lee, H., Kim, C., & Shin, S. (2024) · Applied Sciences · doi:10.3390/app14198868 · Cited by 121
InstructDiffusion: A Generalist Modeling Interface for Vision Tasks
Geng, Z., Yang, B., Hang, T., Li, C., Gu, S., Zhang, T., Bao, J., Zhang, Z., Li, H., Hu, H., et al. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01208 · Cited by 247
Self-citationV*: Guided Visual Search as a Core Mechanism in Multimodal LLMs
Wu, P., & Xie, S. (2023) · arXiv · doi:10.48550/arxiv.2312.14135 · Cited by 74
V*: Guided Visual Search as a Core Mechanism in Multimodal LLMs
Wu, P., & Xie, S. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.01243 · Cited by 74
Multi-Modal Hallucination Control by Visual Information Grounding
Favero, A., Zancato, L., Trager, M., Choudhary, S., Perera, P., Achille, A., Swaminathan, A. D., & Soatto, S. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01356 · Cited by 378
Cognitive Architectures for Language Agents
Sumers, T. R., Yao, S., Narasimhan, K., & Griffiths, T. L. (2023) · arXiv · doi:10.48550/arxiv.2309.02427 · Cited by 798
Honeybee: Locality-Enhanced Projector for Multimodal LLM
Cha, J., Kang, W., Mun, J., & Roh, B. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01311 · Cited by 325
Generative AI and process systems engineering: The next frontier
Decardi‐Nelson, B., Alshehri, A. S., Ajagekar, A., & You, F. (2024) · Computers & Chemical Engineering · doi:10.1016/j.compchemeng.2024.108723 · Cited by 97
Metadata
- Authors
- Liu, H., Li, C., Wu, Q., & Lee, Y. J.
- Venue
- Advances in Neural Information Processing Systems 36
- Published
- 2023
- Pages
- 34892-34916
- ISSN
- 1049-5258 (ISSN-L)
- OpenAlex
- W7133193597
- Cited by
- 2,428
- References
- 0
- Citations by year
- 2026: 1374 · 2025: 599 · 2024: 439 · 2023: 16
- FWCI
- 259.66
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- neural information processing systems (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- visual instruction tuning, vision-language models, instruction tuning, visual grounding
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.