The Revolution of Multimodal Large Language Models: A Survey
Open accessAbstract
Davide Caffagni, Federico Cocchi, Luca Barsellotti, Nicholas Moratelli, Sara Sarto, Lorenzo Baraldi, Lorenzo Baraldi, Marcella Cornia, Rita Cucchiara. Findings of the Association for Computational Linguistics: ACL 2024. 2024.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Limitations such as introducing hallucination and omitting necessary signals are inherent to LLMs and may still impact classification performance.
“users should be aware that limitations such as introducing hallucination71 and omitting necessary signals73 are inherent to LLMs and may still impact classification performance, though MoMA is fine-tuned using groundtruth clinical labels.”
From MoMA: a mixture-of-multimodal-agents architecture for enhancing clinical prediction modelling (2025) · cites this paper as [73]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 1× · see citing works
“The introduction of the attention operation and the Transformer architecture (Vaswani et al., 2017) has enabled the creation of models capable of handling various modalities on an increasingly large scale.”
U-Net: Convolutional Networks for Biomedical Image Segmentation
Ronneberger, O., Fischer, P., & Brox, T. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24574-4_28 · Cited by 99,404 · cited 5× · see citing works
“While GILL refrains from fine-tuning both the LLM and the diffusion U-Net, alternative methodologies fine-tune the language model to expand its multimodal generation capabilities.”
InfluentialBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · cited 1× · see citing works
“Initially, this breakthrough was lever-aged for language-specific models (Devlin et al., 2018; Brown et al., 2020) but quickly extended to support diverse modalities (Li et al., 2019; Lu et al., 2019) and facilitate their integration within unified embedding spaces (Radford et al., 2021).”
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460 · cited 8× · see citing works
“COCO (Lin et al., 2014) contains more than 120k images, each associated with five human-generated captions.”
InfluentialLanguage Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 2× · see citing works
“Initially, this breakthrough was lever-aged for language-specific models (Devlin et al., 2018; Brown et al., 2020) but quickly extended to support diverse modalities (Li et al., 2019; Lu et al., 2019) and facilitate their integration within unified embedding spaces (Radford et al., 2021).”
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.11929 · Cited by 21,793 · see citing works
Learning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293 · cited 1× · see citing works
“Initially, this breakthrough was lever-aged for language-specific models (Devlin et al., 2018; Brown et al., 2020) but quickly extended to support diverse modalities (Li et al., 2019; Lu et al., 2019) and facilitate their integration within unified embedding spaces (Radford et al., 2021).”
Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A. C., Lo, W.-Y., et al. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00371 · Cited by 11,953 · cited 3× · see citing works
“For output regions, LISA (Lai et al., 2023), GLaMM, GSVA (Xia et al., 2023b), NeXt-Chat (Zhang et al., 2023a), and LISA++ (Yang et al., 2023b) send the embedding corresponding to special tokens to the mask decoder of SAM, LLaVA-G to OpenSeeD (Zhang et al., 2023c), Lenna (Wei et al., 2023) to Grounding-DINO (Liu et al.…”
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · cited 2× · see citing works
“…used are WebLI (Chen et al., 2023j), composed of 10B images and image-text pairs, and MMC4 (Zhu et al., 2023d), an extension of the text-only C4 (Raffel et al., 2020) dataset composed of 365M documents and 156B tokens relatives to different concepts, and OBELICS (Laurençon et al., 2024), an…”
Emerging Properties in Self-Supervised Vision Transformers
Caron, M., Touvron, H., Misra, I., Jegou, H., Mairal, J., Bojanowski, P., & Joulin, A. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00951 · Cited by 5,638 · cited 5× · see citing works
“DetGPT (Pi et al., 2023) generates a list of categories for Grounding-DINO.”
InfluentialVisual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations
Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L.-J., Shamma, D. A., et al. (2017) · International Journal of Computer Vision · doi:10.1007/s11263-016-0981-7 · Cited by 5,107 · cited 1× · see citing works
“Visual Genome (Krishna et al., 2017) connects structured image concepts to language and comprises 108,077 images along with detailed descriptions of all objects present in them, providing 5.4M region descriptions and 1.7M visual question-answer pairs.”
CIDEr: Consensus-based image description evaluation
Vedantam, R., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7299087 · Cited by 4,845 · cited 2× · see citing works
“As regards the evaluation metric, the CIDEr score (Vedantam et al., 2015), which is the reference metric for the task, is used to compare generated image descriptions with ground-truth captions.”
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602 · cited 1× · see citing works
“Popular choices for the LLM backbone often fall into the LLaMA family (Touvron et al., 2023a,b), given that their weights are freely accessible, they have been trained on public data solely, and they boast different sizes to accommodate various use cases.”
BackgroundScanNet: Richly-Annotated 3D Reconstructions of Indoor Scenes
Dai, A., Chang, A. X., Savva, M., Halber, M., Funkhouser, T., & Niessner, M. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.261 · Cited by 4,368 · cited 9× · see citing works
“3B ♢ CLIP ViT-L Q-Former ✻ ✗ Visual Dialogue, VQA, Captioning LLaMA-Adapter V2 (Gao et al., 2023) LLaMA-7B ▲ CLIP ViT-L Linear ✗ VQA, Captioning OpenFlamingo (Awadalla et al., 2023) MPT-7B ⋆ CLIP ViT-L XAttn LLM ✗ VQA, Captioning Flamingo (Alayrac et al., 2022) Chinchilla-70B ⋆ NFNet-F6 XAttn LLM ✗ Visual Dialogue, VQ…”
InfluentialInstructBLIP: Towards General-purpose Vision-Language Models with Instruction TuningPreprint
Dai, W., Li, J., Li, D., Tiong, A., Zhao, J.-Q., Wang, W., Li, B. A., Fung, P., & Hoi, S. (2023) · Neural Information Processing Systems · doi:10.48550/arxiv.2305.06500 · Cited by 4,191 · cited 4× · see citing works
“Another approach, as demonstrated in Instruct-BLIP (Dai et al., 2023), involves the freezing of the visual encoder and LLM.”
InfluentialGQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answering
Hudson, D. A., & Manning, C. D. (2019) · Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2019.00686 · Cited by 3,703 · cited 1× · see citing works
“IconQA (Lu et al., 2021) tests the visual reasoning abilities of vision-and-language models on three types of questions: multiple-image-choice, multiple-text-choice, and fill-in-the-blank.”
Masked-attention Mask Transformer for Universal Image Segmentation
Cheng, B., Misra, I., Schwing, A. G., Kirillov, A., & Girdhar, R. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00135 · Cited by 3,565 · cited 1× · see citing works
“GELLA (Qi et al., 2024) presents a fusion module in Mask2Former (Cheng et al., 2022) to propose masks based on multi-modal image features and an association module to assign latent embeddings to them.”
Training Language Models to Follow Instructions with Human Feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2011 · Cited by 3,117 · cited 2× · see citing works
“This technique, called instruction-tuning (Chung et al., 2022; Wang et al., 2022b,a; Jiang et al., 2024), turns out to be critical for aligning the behavior of an LLM with that of humans and currently empowers the most advanced LLMs, eventually boosted via reinforcement learning from human feedback (RLHF) (Ouyang et a…”
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-T., Rocktäschel, T., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.11401 · Cited by 3,101 · cited 1× · see citing works
“While retrieval-augmented generation (RAG) is a consolidated technique in LLMs (Lewis et al., 2020; Asai et al., 2023), its application in MLLMs is still under-explored.”
The Power of Scale for Parameter-Efficient Prompt Tuning
Lester, B., Al-Rfou, R., & Constant, N. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.243 · Cited by 3,096 · cited 1× · see citing works
“Among these, prompt-tuning (Hambardzumyan et al., 2021; Lester et al., 2021; Li and Liang, 2021; Liu et al., 2023j) learns a small set of vectors to be fed to the model as soft prompts before the input text.”
Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question AnsweringPreprint
Lu, P., Mishra, S., Xia, T., Qiu, L., Chang, K.-W., Zhu, S.-C., Tafjord, O., Clark, P., & Kalyan, A. (2022) · Neural Information Processing Systems · doi:10.48550/arxiv.2209.09513 · Cited by 2,884 · cited 1× · see citing works
“Several studies (Liu et al., 2023b; Zhu et al., 2023a) show that MLLMs tend to exhibit high hallucination rates, especially when generating longer captions.”
Neural Discrete Representation Learning
Oord, A. V. D., Vinyals, O., & Kavukcuoglu, K. (2017) · arXiv · doi:10.48550/arxiv.1711.00937 · Cited by 2,729 · cited 1× · see citing works
“PaLI-3 (Chen et al., 2023i) converts embeddings into segmentation masks through a VQ-VAE (Van Den Oord et al., 2017) decoder.”
Llama 2: Open Foundation and Fine-Tuned Chat Models
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.09288 · Cited by 2,604 · see citing works
PaLM: Scaling Language Modeling with Pathways
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.02311 · Cited by 2,590 · cited 1× · see citing works
“Brown et al. (2020) discovered that in-context learning, i.e. , prepending the prompt with a few examples demonstrating the desired output of an LLM (Chowdhery et al., 2023; Hoffmann et al., 2022; Tay et al., 2022), improves its performance, especially over unseen tasks.”
LoRA: Low-Rank Adaptation of Large Language Models
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., & Chen, W. (2021) · arXiv · doi:10.48550/arxiv.2106.09685 · Cited by 2,523 · cited 2× · see citing works
“This technique is orthogonal to quantization methods such as QLoRA (Dettmers et al., 2024), which further decreases the memory footprint of the LLM compared to the usual half-precision weights.”
Taming Transformers for High-Resolution Image Synthesis
Esser, P., Rombach, R., & Ommer, B. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01268 · Cited by 2,514 · cited 1× · see citing works
“BuboGPT (Zhao et al., 2023c) leverages a combination of RAM, Grounding-DINO, and SAM and matches tags with nouns in the output sequence.”
Prefix-Tuning: Optimizing Continuous Prompts for Generation
Li, X. L., & Liang, P. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.353 · Cited by 2,473 · cited 1× · see citing works
“Among these, prompt-tuning (Hambardzumyan et al., 2021; Lester et al., 2021; Li and Liang, 2021; Liu et al., 2023j) learns a small set of vectors to be fed to the model as soft prompts before the input text.”
Young, P., Lai, A., Hodosh, M., & Hockenmaier, J. (2014) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00166 · Cited by 2,439 · cited 1× · see citing works
“Flickr30k (Young et al., 2014) comprises 31,783 images, depicting diverse everyday activities, events, and scenes.”
DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation
Ruiz, N., Li, Y., Jampani, V., Pritch, Y., Rubinstein, M., & Aberman, K. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.02155 · Cited by 2,295 · cited 1× · see citing works
“DreamBench (Ruiz et al., 2023) is a benchmark that evaluates the generative capabilities of the models on subject-driven generation.”
Evaluating Object Hallucination in Large Vision-Language ModelsPreprint
Li, Y., Du, Y., Zhou, K., et al. (2023) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2305.10355 · Cited by 2,148 · see citing works
ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks
Lu, J., Batra, D., Parikh, D., & Lee, S. (2019) · arXiv · doi:10.48550/arxiv.1908.02265 · Cited by 2,109 · cited 1× · see citing works
“Initially, this breakthrough was lever-aged for language-specific models (Devlin et al., 2018; Brown et al., 2020) but quickly extended to support diverse modalities (Li et al., 2019; Lu et al., 2019) and facilitate their integration within unified embedding spaces (Radford et al., 2021).”
Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video UnderstandingPreprint
Zhang, H., Li, X., & Bing, L. (2023) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2306.02858 · Cited by 2,019 · see citing works
MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models
Fu, C., Chen, P., Shen, Y.-H., Qin, Y.-L., Zhang, M.-D., Lin, X., Yang, J., Zheng, X.-W., Li, K., Sun, X., et al. (2023) · Neural Information Processing Systems · doi:10.48550/arxiv.2306.13394 · Cited by 1,917 · cited 12×
“TextVQA (VQA T ) (Singh et al., 2019) is a dataset based on pictures from Open Images (Kuznetsova et al., 2020) Comprehensively describing the visual input is another important skill desired in MLLMs.”
InfluentialConceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning
Sharma, P., Ding, N., Goodman, S., & Soricut, R. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1238 · Cited by 1,879 · cited 1× · see citing works
“For instance, the Conceptual Captions 3M (CC3M) dataset (Sharma et al., 2018) is composed of 3M images paired with textual captions specifically designed for image captioning systems.”
Improved Baselines with Visual Instruction Tuning
Liu, H., Li, C., Li, Y., & Lee, Y. J. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.02484 · Cited by 1,748 · cited 3× · see citing works
“Some works (Liu et al., 2023d; You et al., 2023; Chen et al., 2023j) have demonstrated that adopting visual backbones trained on higher resolutions leads to fewer hallucinations and improved multimodal understanding abilities, translating into better performance over tasks that require fine-grained…”
Improved Baselines with Visual Instruction Tuning
Liu, H., Li, C., Li, Y., & Lee, Y. J. (2023) · arXiv · doi:10.48550/arxiv.2310.03744 · Cited by 1,733 · see citing works
mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer
Xue, L., Constant, N., Roberts, A., Kale, M., Al-Rfou, R., Siddhant, A., Barua, A., & Raffel, C. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.41 · Cited by 1,629 · cited 1× · see citing works
“…are OPT (Zhang et al., 2022b), Mag-neto (Wang et al., 2023b), MPT (MosaicML, 2023), and the instruction-tuned (Chung et al., 2022) or multilingual (Xue et al., 2020) flavors of T5 (Raffel et al., 2020), an encoder-decoder language model pre-trained for multiple tasks. et al., 2024; Zhao et al.,…”
Mamba: Linear-Time Sequence Modeling with Selective State Spaces
Gu, A., & Dao, T. (2023) · arXiv · doi:10.48550/arxiv.2312.00752 · Cited by 1,579 · see citing works
Instructpix2pix: Learning to follow image editing instructions
Brooks, T., Holynski, A., & Efros, A. A. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01764 · Cited by 1,466 · see citing works
Scaling Instruction-Finetuned Language Models
Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, Y., Wang, X., Dehghani, M., Brahma, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2210.11416 · Cited by 1,441 · cited 2× · see citing works
“This technique, called instruction-tuning (Chung et al., 2022; Wang et al., 2022b,a; Jiang et al., 2024), turns out to be critical for aligning the behavior of an LLM with that of humans and currently empowers the most advanced LLMs, eventually boosted via reinforcement learning from human feedback…”
Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language ModelsPreprint
Maaz, M., Rasheed, H., Khan, S. H., et al. (2023) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2306.05424 · Cited by 1,339 · see citing works
mPLUG-Owl: Modularization Empowers Large Language Models with MultimodalityPreprint
Ye, Q.-H., Xu, H., Xu, G., Ye, J., Yan, M., Zhou, Y., Wang, J., Hu, A., Shi, P.-C., Shi, Y., et al. (2023) · arXiv.org · doi:10.48550/arxiv.2304.14178 · Cited by 1,318 · cited 4× · see citing works
“…VQA, Captioning PaLI (Chen et al., 2023j) mT5-XXL-13B MiniGPT-4 (Zhu et al., 2023a) Vicuna-13B ⋆ EVA ViT-g Linear ✓ VQA, Captioning mPLUG-Owl (Ye et al., 2023c) LLaMA-7B ▲ CLIP ViT-L Q-Former ✻ ✓ Visual Dialogue, VQA InstructBLIP (Dai et al., 2023) Vicuna-13B ⋆ EVA ViT-g Q-Former ✓ Visual…”
InfluentialLVIS: A Dataset for Large Vocabulary Instance Segmentation
Gupta, A., Dollar, P., & Girshick, R. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00550 · Cited by 1,274 · cited 1× · see citing works
“MLLMs are evaluated across different benchmarks, taking into account both more classic visual comprehension and recognition skills and advanced multimodal conversation capabilities.”
Self-Instruct: Aligning Language Models with Self-Generated Instructions
Wang, Y., Kordi, Y., Mishra, S., Liu, A., Smith, N. A., Khashabi, D., & Hajishirzi, H. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.754 · Cited by 1,243 · see citing works
Generation and Comprehension of Unambiguous Object Descriptions
Mao, J., Huang, J., Toshev, A., Camburu, O., Yuille, A., & Murphy, K. P. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.9 · Cited by 1,211 · cited 1× · see citing works
“Specifically, Shikra (Chen et al., 2023f), DetGPT (Pi et al., 2023), ChatSpot (Zhao et al., 2023b), and PVIT (Chen et al., 2023a) leverage LLMs (Achiam et al., 2023; OpenAI, 2022) to combine regions and captions from datasets that present both annotations ( e.g. , COCO).”
BackgroundVisualBERT: A Simple and Performant Baseline for Vision and Language
Li, L. H., Yatskar, M., Yin, D., Hsieh, C.-J., & Chang, K.-W. (2019) · arXiv · doi:10.48550/arxiv.1908.03557 · Cited by 1,198 · cited 1× · see citing works
“Initially, this breakthrough was lever-aged for language-specific models (Devlin et al., 2018; Brown et al., 2020) but quickly extended to support diverse modalities (Li et al., 2019; Lu et al., 2019) and facilitate their integration within unified embedding spaces (Radford et al., 2021).”
Grounded Language-Image Pre-training
Li, L. H., Zhang, P., Zhang, H., Yang, J., Li, C., Zhong, Y., Wang, L., Yuan, L., Zhang, L., Hwang, J., et al. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01069 · Cited by 1,197 · cited 1× · see citing works
“Differently, Kosmos-2 (Peng et al., 2023) and Ferret (You et al., 2023) exploit an open-vocabulary detector (Li et al., 2022) to ground noun chunks parsed from cap-tions and then reconstruct referring expressions.”
DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection
Zhang, H., Li, F., Liu, S., Zhang, L., Su, H., Zhu, J., Ni, L. M., & Shum, H.-Y. (2022) · arXiv · Cited by 1,154 · see citing works
Shikra: Unleashing Multimodal LLM's Referential Dialogue MagicPreprint
Chen, K., Zhang, Z., Zeng, W., Zhang, R., Zhu, F., & Zhao, R. (2023) · arXiv.org · doi:10.48550/arxiv.2306.15195 · Cited by 1,091 · see citing works
ReferItGame: Referring to Objects in Photographs of Natural Scenes
Kazemzadeh, S., Ordóñez, V., Matten, M., & Berg, T. L. (2014) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1086 · Cited by 1,082 · cited 1× · see citing works
“Differently, Kosmos-2 (Peng et al., 2023) and Ferret (You et al., 2023) exploit an open-vocabulary detector (Li et al., 2022) to ground noun chunks parsed from cap-tions and then reconstruct referring expressions.”
(2016) · doi:10.1007/978-3-319-46475-6_5 · Cited by 1,081 · cited 1× · see citing works
“Instead, for image editing tasks, additional datasets like the one introduced in InstructPix2Pix (Brooks et al., 2023) and MagicBrush (Zhang et al., 2023e) are typically used.”
Deep Visual-Semantic Alignments for Generating Image Descriptions
Karpathy, A., & Fei-Fei, L. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2598339 · Cited by 1,048 · cited 1× · see citing works
“For captioning tasks, the splits defined by Karpathy and Fei-Fei (2015) are typically employed, with 113k, 5k, and 5k images respectively for train, validation and test.”
LAION-5B: An open large-scale dataset for training next generation image-text models
Schuhmann, C., Beaumont, R., Vencu, R., Gordon, C., Wightman, R., Cherti, M., Coombes, T., Katta, A., Mullis, C., Wortsman, M., et al. (2022) · arXiv · doi:10.48550/arxiv.2210.08402 · Cited by 1,022 · cited 1× · see citing works
“Given the amount of training data currently available, approaches like SPHINX-X (Gao et al., 2024) opt to perform a single all-in-one training stage in which to update all model components, possibly also using text-only data to preserve the conversational capabilities of the LLM.”
Gemini: A Family of Highly Capable Multimodal Models
Gemini Team, Anil, R., Borgeaud, S., Alayrac, J.-B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A. M., Hauth, A., Millican, K., et al. (2023) · arXiv · doi:10.48550/arxiv.2312.11805 · Cited by 990 · see citing works
Flamingo: a visual language model for few-shot learning
Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1723 · Cited by 966 · cited 6× · see citing works
“This approach has been proposed in Flamingo (Alayrac et al., 2022) with the integration of dense cross-attention blocks among the existing pre-trained layers of the LLM.”
InfluentialTraining Compute-Optimal Large Language Models
Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. D. L., Hendricks, L. A., Welbl, J., Clark, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.15556 · Cited by 929 · see citing works
Li, J., Li, D., Savarese, S., & Hoi, S. (2023) · arXiv · doi:10.48550/arxiv.2301.12597 · Cited by 908 · see citing works
ImageBind One Embedding Space to Bind Them All
Girdhar, R., El-Nouby, A., Liu, Z., Singh, M., Alwala, K. V., Joulin, A., & Misra, I. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01457 · Cited by 863 · see citing works
mT5: A massively multilingual pre-trained text-to-text transformerPreprint
Xue, L., Constant, N., Roberts, A., Kale, M., Al‐Rfou, R., Siddhant, A., Barua, A., & Raffel, C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.11934 · Cited by 852 · see citing works
Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modeling
Yu, X., Tang, L., Rao, Y., Huang, T., Zhou, J., & Lu, J. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01871 · Cited by 832 · see citing works
Visual7W: Grounded Question Answering in Images
Zhu, Y., Groth, O., Bernstein, M., & Fei-Fei, L. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.540 · Cited by 794 · cited 2× · see citing works
“Instead, for image editing tasks, additional datasets like the one introduced in InstructPix2Pix (Brooks et al., 2023) and MagicBrush (Zhang et al., 2023e) are typically used.”
BackgroundMethodDataComp: In search of the next generation of multimodal datasetsPreprint
Gadre, S., Ilharco, G., Fang, A., et al. (2023) · Neural Information Processing Systems · doi:10.48550/arxiv.2304.14108 · Cited by 781 · cited 1× · see citing works
“Furthermore, DataComp (Gadre et al., 2023) presents an extensive pool of 12.8B filtered image-text pairs sourced from common crawl.”
Towards VQA Models That Can Read
Singh, A., Natarajan, V., Shah, M., Jiang, Y. L., Chen, X., Batra, D., Parikh, D., & Rohrbach, M. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00851 · Cited by 777 · cited 1× · see citing works
“TextVQA (VQA T ) (Singh et al., 2019) is a dataset based on pictures from Open Images (Kuznetsova et al., 2020) Comprehensively describing the visual input is another important skill desired in MLLMs.”
LLaMA-Adapter V2: Parameter-Efficient Visual Instruction ModelPreprint
Gao, P., Han, J., Zhang, R., Lin, Z.-Y., Geng, S., Zhou, A., Zhang, W., Lu, P., He, C.-H., Yue, X., et al. (2023) · arXiv.org · doi:10.48550/arxiv.2304.15010 · Cited by 777 · see citing works
MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningPreprint
Chen, J., Zhu, D., Shen, X., Li, X., Liu, Z.-C., Zhang, P.-C., Krishnamoorthi, R., Chandra, V., Xiong, Y.-Y., & Elhoseiny, M. (2023) · arXiv.org · doi:10.48550/arxiv.2310.09478 · Cited by 764 · see citing works
MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editing
Zhang, K., Mo, L.-B., Chen, W., et al. (2023) · Neural Information Processing Systems · doi:10.48550/arxiv.2306.10012 · Cited by 760 · cited 1×
“MagicBrush (Zhang et al., 2023e) is a benchmark in the area of image editing and contains a collection of 10,000 manually annotated triplets, each consisting of a source image, an editing instruction, and the corresponding target image.”
OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge
Marino, K., Rastegari, M., Farhadi, A., & Mottaghi, R. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00331 · Cited by 749 · cited 1× · see citing works
“There are 107,439 questions, 20% of which makes up for the test split.”
OCR-VQA: Visual Question Answering by Reading Text in Images
Mishra, A., Shekhar, S., Singh, A., & Chakraborty, A. (2019) · IEEE International Conference on Document Analysis and Recognition · doi:10.1109/icdar.2019.00156 · Cited by 746 · see citing works
VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric TasksPreprint
Wang, W., Chen, Z., Chen, X., et al. (2023) · Neural Information Processing Systems · doi:10.48550/arxiv.2305.11175 · Cited by 728 · cited 1× · see citing works
“VisionLLM (Wang et al., 2023e), VistaLLM (Pramanick et al., 2023), LLaFS (Zhu et al., 2023c), and ChatSpot (Zhao et al., 2023b) allow the MLLM to handle polygons by representing them as a series of points.”
LLaMA-VID: An Image is Worth 2 Tokens in Large Language ModelsPreprint
Li, Y., Wang, C.-Y., & Jia, J. (2023) · European Conference on Computer Vision · doi:10.48550/arxiv.2311.17043 · Cited by 684 · see citing works
OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language ModelsPreprint
Awadalla, A., Gao, I., Gardner, J., Hessel, J., Hanafy, Y., Zhu, W., Marathe, K., Bitton, Y., Gadre, S., Sagawa, S., et al. (2023) · arXiv.org · doi:10.48550/arxiv.2308.01390 · Cited by 674 · see citing works
MM-REACT: Prompting ChatGPT for Multimodal Reasoning and ActionPreprint
Yang, Z.-Y., Li, L., Wang, J., Lin, K., Azarnasab, E., Ahmed, F., Liu, Z.-C., Liu, C., Zeng, M., & Wang, L.-J. (2023) · arXiv.org · doi:10.48550/arxiv.2303.11381 · Cited by 660 · see citing works
VizWiz Grand Challenge: Answering Visual Questions from Blind People
Gurari, D., Li, Q., Stangl, A. J., Guo, A., Lin, C., Grauman, K., Luo, J., & Bigham, J. P. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00380 · Cited by 621 · cited 1× · see citing works
“While retrieval-augmented generation (RAG) is a consolidated technique in LLMs (Lewis et al., 2020; Asai et al., 2023), its application in MLLMs is still under-explored.”
PaLM-E: An Embodied Multimodal Language Model
Driess, D., Xia, F., Sajjadi, M. S. M., Lynch, C., Chowdhery, A., Ichter, B., Wahid, A., Tompson, J., Vuong, Q., Yu, T., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.03378 · Cited by 591 · cited 1× · see citing works
“…designed for document analysis and text-intensive visual inputs (Lv et al., 2023; Ye et al., 2023a), those proposed for embodied AI and robotics (Driess et al., 2023; Mu et al., 2023), and those tailored for specific domains such as medicine (Li et al., 2023d) and autonomous driving (Xu et al.,…”
LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day
Li, C., Wong, C., Zhang, S., Usuyama, N., Liu, H., Yang, J., Naumann, T., Poon, H., & Gao, J. (2023) · arXiv · doi:10.48550/arxiv.2306.00890 · Cited by 590 · cited 1× · see citing works
“Similarly, in (Xia et al., 2023a) the LLM is trained to output detailed language-based generation prompts which are employed for generation or editing tasks.”
VQA: Visual Question Answering: www.visualqa.org
Agrawal, A., Lu, J., Antol, S., Mitchell, M., Zitnick, C. L., Parikh, D., & Batra, D. (2017) · International Journal of Computer Vision · doi:10.1007/s11263-016-0966-6 · Cited by 570 · cited 9× · see citing works
“3B ♢ CLIP ViT-L Q-Former ✻ ✗ Visual Dialogue, VQA, Captioning LLaMA-Adapter V2 (Gao et al., 2023) LLaMA-7B ▲ CLIP ViT-L Linear ✗ VQA, Captioning OpenFlamingo (Awadalla et al., 2023) MPT-7B ⋆ CLIP ViT-L XAttn LLM ✗ VQA, Captioning Flamingo (Alayrac et al., 2022) Chinchilla-70B ⋆ NFNet-F6 XAttn LLM ✗ Visual Dialogue, VQ…”
InfluentialKirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A. C., Lo, W.-Y., et al. (2023) · arXiv · doi:10.48550/arxiv.2304.02643 · Cited by 527 · see citing works
QLoRA: Efficient Finetuning of Quantized LLMs
Dettmers, T., Pagnoni, A., Holtzman, A., & Zettlemoyer, L. (2023) · arXiv · doi:10.48550/arxiv.2305.14314 · Cited by 507 · cited 1× · see citing works
“This technique is orthogonal to quantization methods such as QLoRA (Dettmers et al., 2024), which further decreases the memory footprint of the LLM compared to the usual half-precision weights.”
MethodMaking the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering
Goyal, Y., Khot, T., Agrawal, A., Summers-Stay, D., Batra, D., & Parikh, D. (2019) · International Journal of Computer Vision · doi:10.1007/s11263-018-1116-0 · Cited by 487 · cited 2× · see citing works
“This ability is quantitatively evaluated across several visual question-answering datasets, measuring the accuracy (Antol et al., 2015) of the answers provided by the MLLM. VQAv2 (Goyal et al., 2017) is an extended and balanced version of VQA (Antol et al., 2015) built by collecting similar images for the same questio…”
MMBench: Is Your Multi-modal Model an All-around Player?
Liu, Y., Duan, H., Zhang, Y., Li, B., Zhang, S., Zhao, W., Yuan, Y., Wang, J., He, C., Liu, Z., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.06281 · Cited by 487 · see citing works
PandaGPT: One Model To Instruction-Follow Them AllPreprint
Su, Y., Lan, T., Li, H., Xu, J., Wang, Y., & Cai, D. (2023) · Tsinghua Interdisciplinary Workshop on Logic, Language and Meaning · doi:10.48550/arxiv.2305.16355 · Cited by 474 · cited 2× · see citing works
“ChatterBox (Tian et al., 2024b) combines features from the iTPN-B encoder (Tian et al., 2023) and the MLLM and provides them to the DINO detector (Zhang et al., 2022a).”
MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models
Zhu, D., Chen, J., Shen, X., Li, X., & Elhoseiny, M. (2023) · arXiv · doi:10.48550/arxiv.2304.10592 · Cited by 469 · see citing works
SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis
Podell, D., English, Z., Lacey, K., Blattmann, A., Dockhorn, T., Müller, J., Penna, J., & Rombach, R. (2023) · arXiv · doi:10.48550/arxiv.2307.01952 · Cited by 455 · see citing works
PointLLM: Empowering Large Language Models to Understand Point CloudsPreprint
Xu, R.-S., Wang, X., Wang, T., et al. (2023) · European Conference on Computer Vision · doi:10.48550/arxiv.2308.16911 · Cited by 448 · see citing works
MMMU: A Massive Multi-Discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI
Yue, X., Ni, Y., Zheng, T., Zhang, K., Liu, R., Zhang, G., Stevens, S., Jiang, D., Ren, W., Sun, Y., et al. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.00913 · Cited by 429 · cited 1× · see citing works
“Similarly, MiniGPT-5 (Zheng et al., 2023) includes the reconstruction loss of diffusion models in addition to the alignment loss of GILL.”
Robust fine-tuning of zero-shot models
Wortsman, M., Ilharco, G., Kim, J. W., Li, M., Kornblith, S., Roelofs, R., Lopes, R. G., Hajishirzi, H., Farhadi, A., Namkoong, H., et al. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00780 · Cited by 428 · cited 2× · see citing works
“Popular choices are the ViT-L model from CLIP (Rad-ford et al., 2021), the ViT-H backbone from Open-CLIP (Wortsman et al., 2022), and the ViT-g version from EVA-CLIP (Fang et al., 2023).”
MethodGenerating Images with Multimodal Language Models
Koh, J. Y., Fried, D., & Salakhutdinov, R. (2023) · Neural Information Processing Systems · doi:10.48550/arxiv.2305.17216 · Cited by 416 · cited 4×
“While GILL refrains from fine-tuning both the LLM and the diffusion U-Net, alternative methodologies fine-tune the language model to expand its multimodal generation capabilities.”
InfluentialPerceiver: General Perception with Iterative Attention
Jaegle, A., Gimeno, F., Brock, A., Zisserman, A., Vinyals, O., & Carreira, J. (2021) · arXiv · doi:10.48550/arxiv.2103.03206 · Cited by 414 · cited 2× · see citing works
“To reduce computational complexity, this strategy is usually paired with a Perceiver-based component (Jaegle et al., 2021) that reduces the number of visual tokens before they are fed to the LLM.”
BackgroundMethodGPT4RoI: Instruction Tuning Large Language Model on Region-of-InterestPreprint
Zhang, S., Sun, P., Chen, S., Xiao, M., Shao, W., Zhang, W., Liu, Y., Chen, K., & Luo, P. (2023) · ECCV Workshops · doi:10.48550/arxiv.2307.03601 · Cited by 390 · cited 1× · see citing works
“However, recently there has been an interest in addressing more fine-grained visual tasks, such as visual grounding and image generation.”
Backgroundnocaps: novel object captioning at scale
Agrawal, H., Desai, K., Wang, Y., Chen, X., Jain, R., Johnson, M., Batra, D., Parikh, D., Lee, S., & Anderson, P. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00904 · Cited by 386 · cited 1× · see citing works
“Complementing these images are 158,915 captions, obtained through crowd-sourcing techniques. nocaps (Agrawal et al., 2019) represents a benchmark for novel object captioning, boasting an extensive collection of almost 400 novel object categories compared to the COCO dataset.”
LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs
Schuhmann, C., Vencu, R., Beaumont, R., Kaczmarczyk, R., Mullis, C., Katta, A., Coombes, T., Jitsev, J., & Komatsuzaki, A. (2021) · arXiv · doi:10.48550/arxiv.2111.02114 · Cited by 358 · cited 1× · see citing works
“Given the amount of training data currently available, approaches like SPHINX-X (Gao et al., 2024) opt to perform a single all-in-one training stage in which to update all model components, possibly also using text-only data to preserve the conversational capabilities of the LLM.”
GPT4Tools: Teaching Large Language Model to Use Tools via Self-instructionPreprint
Yang, R., Song, L., Li, Y., et al. (2023) · Neural Information Processing Systems · doi:10.48550/arxiv.2305.18752 · Cited by 340 · cited 3× · see citing works
“To reduce the input sequence length to the LLM, and thus the computational cost, MiniGPT-v2 (Chen et al., 2023e) and VILA (Lin et al., 2023a) propose to project multiple visual tokens together into the same token within the embedding space of the LLM.”
LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image UnderstandingPreprint
Zhang, Y., Zhang, R., Gu, J., Zhou, Y., Lipka, N., Yang, D.-Y., & Sun, T.-F. (2023) · arXiv.org · doi:10.48550/arxiv.2306.17107 · Cited by 335 · cited 1× · see citing works
“Given the amount of training data currently available, approaches like SPHINX-X (Gao et al., 2024) opt to perform a single all-in-one training stage in which to update all model components, possibly also using text-only data to preserve the conversational capabilities of the LLM.”
MethodBai, J., Bai, S., Chu, Y., Cui, Z., Dang, K., Deng, X., Fan, Y., Ge, W., Han, Y., Huang, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2309.16609 · Cited by 332 · see citing works
Lin, Z.-Y., Liu, C., Zhang, R., Gao, P., Qiu, L.-T., Xiao, H., Qiu, H., Lin, C., Shao, W., Chen, K.-Q., et al. (2023) · arXiv.org · doi:10.48550/arxiv.2311.07575 · Cited by 316 · see citing works
Multimodal Large Language Models: A Survey
Wu, J., Gan, W., Chen, Z., Wan, S., & Yu, P. S. (2023) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata59044.2023.10386743 · Cited by 309 · cited 1× · see citing works
“In comparison with existing surveys on the topic (Yin et al., 2023a; Wu et al., 2023b; Huang et al., 2023a), our paper exhibits substantial differences.”
Prefix-Tuning: Optimizing Continuous Prompts for Generation
Li, X. L., & Liang, P. (2021) · arXiv · doi:10.48550/arxiv.2101.00190 · Cited by 289 · see citing works
VILA: On Pre-training for Visual Language Models
Lin, J., Yin, H., Ping, W., Molchanov, P. A., Shoeybi, M., & Han, S. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02520 · Cited by 284 · cited 2× · see citing works
“Unified-IO2 (Lu et al., 2023a) uses VQ-GAN decoders for both image and audio modalities to decode output discrete tokens and can generate surface normals, depth, and segmentation masks for the input images.”
AnyGPT: Unified Multimodal LLM with Discrete Sequence Modeling
Zhan, J., Dai, J., Ye, J., et al. (2024) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2402.12226 · Cited by 281 · cited 1×
“AnyGPT (Zhan et al., 2024) interprets all the continuous non-text modalities as discrete tokens in both input and output, using, respectively, multi-modal tokenizers and de-tokenizers.”
LLaVA-Plus: Learning to Use Tools for Creating Multimodal Agents
Liu, S.-L., Cheng, H., Liu, H., et al. (2023) · European Conference on Computer Vision · doi:10.48550/arxiv.2311.05437 · Cited by 266 · cited 1×
“An additional training step is usually performed on an instruction-tuning dataset, such as LLaVa-Instruct (Liu et al., 2023h), to preserve the conversational capabilities of the MLLM.”
Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions?
Chen, Y., Hu, H., Luan, Y., et al. (2023) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2302.11713 · Cited by 266 · cited 1×
“We believe that the emergence of VQA datasets that require external retrieved knowledge (Chen et al., 2023k; Mensink et al., 2023) may enable the development of MLLMs with RAG capabilities (Hu et al., 2023b; Caffagni et al., 2024).”
Huang, T.-H., Ferraro, F., Mostafazadeh, N., Misra, I., Agrawal, A., Devlin, J., Girshick, R., He, X., Kohli, P., Batra, D., et al. (2016) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n16-1147 · Cited by 254 · cited 2× · see citing works
“VIST (Huang et al., 2016) is specifically curated for the task of interleaved image-text generation.”
Otter: A Multi-Modal Model With In-Context Instruction Tuning
Li, B., Zhang, Y., Chen, L., Wang, J., Pu, F., Cahyono, J. A., Yang, J., Li, C., & Liu, Z. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3571946 · Cited by 253 · cited 2× · see citing works
“However, it may lead to performance degradation in other tasks, indicating a degree of forgetting and damage to the general visual representation.”
BackgroundMethodGrounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object DetectionPreprint
Liu, S., Zeng, Z., Ren, T., Li, F., Zhang, H., Yang, J., Jiang, Q., Li, C., Yang, J., Su, H., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2303.05499 · Cited by 247 · cited 4× · see citing works
“For output regions, LISA (Lai et al., 2023), GLaMM, GSVA (Xia et al., 2023b), NeXt-Chat (Zhang et al., 2023a), and LISA++ (Yang et al., 2023b) send the embedding corresponding to special tokens to the mask decoder of SAM, LLaVA-G to OpenSeeD (Zhang et al., 2023c), Lenna (Wei et al., 2023) to Grounding-DINO (Liu et al.…”
InfluentialSuper-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks
Wang, Y., Mishra, S., Alipoormolabashi, P., Kordi, Y., Mirzaei, A., Naik, A., Ashok, A., Dhanasekaran, A. S., Arunkumar, A., Stap, D., et al. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.340 · Cited by 247 · cited 1× · see citing works
“This technique, called instruction-tuning (Chung et al., 2022; Wang et al., 2022b,a; Jiang et al., 2024), turns out to be critical for aligning the behavior of an LLM with that of humans and currently empowers the most advanced LLMs, eventually boosted via reinforcement learning from human feedback…”
WARP: Word-level Adversarial ReProgramming
Hambardzumyan, K., Khachatrian, H., & May, J. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.381 · Cited by 243 · cited 1× · see citing works
“Among these, prompt-tuning (Hambardzumyan et al., 2021; Lester et al., 2021; Li and Liang, 2021; Liu et al., 2023j) learns a small set of vectors to be fed to the model as soft prompts before the input text.”
TextCaps: A Dataset for Image Captioning with Reading Comprehension
Sidorov, O., Hu, R., Rohrbach, M., & Singh, A. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58536-5_44 · Cited by 242 · cited 1× · see citing works
“TextCaps (Sidorov et al., 2020) includes 145k cap-tions aligned with 28k images.”
PaLI: A Jointly-Scaled Multilingual Language-Image ModelPreprint
Chen, X., Wang, X., Changpinyo, S., Piergiovanni, A., Padlewski, P., Salz, D., Goodman, S., Grycner, A., Mustafa, B., Beyer, L., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2209.06794 · Cited by 235 · cited 2× · see citing works
“Given the amount of training data currently available, approaches like SPHINX-X (Gao et al., 2024) opt to perform a single all-in-one training stage in which to update all model components, possibly also using text-only data to preserve the conversational capabilities of the LLM.”
MethodLanguage Is Not All You Need: Aligning Perception with Language Models
Huang, S., Dong, L., Wang, W., Hao, Y., Singhal, S., Ma, S., Lv, T., Cui, L., Mohammed, O. K., Patra, B., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.14045 · Cited by 235 · see citing works
BLIVA: A Simple Multimodal LLM for Better Handling of Text-Rich Visual Questions
Hu, W., Xu, Y., Li, Y., Li, W., Chen, Z., & Tu, Z. (2023) · AAAI Conference on Artificial Intelligence · doi:10.48550/arxiv.2308.09936 · Cited by 233 · cited 1×
“However, employing a frozen visual encoder has some limitations, primarily due to the constrained number of parameters, resulting in an inadequate alignment between the visual and language modalities.”
BackgroundMaking LLaMA SEE and Draw with SEED Tokenizer
Ge, Y.-Y., Zhao, S., Zeng, Z.-Y., et al. (2023) · International Conference on Learning Representations · doi:10.48550/arxiv.2310.01218 · Cited by 227 · cited 1×
“The most common way to output regions is to directly insert them into generated text as a series of coordinates, represented as numbers or as special tokens dedicated to location bins.”
Safe Latent Diffusion: Mitigating Inappropriate Degeneration in Diffusion Models
Schramowski, P., Brack, M., Deiseroth, B., & Kersting, K. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.02157 · Cited by 219 · see citing works
CogVLM: Visual Expert for Pretrained Language Models
Wang, W., Lv, Q., Yu, W., Hong, W., Qi, J., Wang, Y., Ji, J., Yang, Z., Zhao, L., Song, X., et al. (2023) · arXiv · doi:10.48550/arxiv.2311.03079 · Cited by 217 · see citing works
MobileVLM V2: Faster and Stronger Baseline for Vision Language Model
Chu, X., Qiao, L., Zhang, X., et al. (2024) · arXiv.org · doi:10.48550/arxiv.2402.03766 · Cited by 209 · cited 2×
“Effective strategies (Chu et al., 2024) are needed to reduce computational requirements and enable more accessible development of MLLMs.”
GLaMM: Pixel Grounding Large Multimodal Model
Rasheed, H., Maaz, M., Shaji, S., Shaker, A., Khan, S., Cholakkal, H., Anwer, R. M., Xing, E. P., Yang, M., & Khan, F. S. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01236 · Cited by 205 · cited 4× · see citing works
“For output regions, LISA (Lai et al., 2023), GLaMM, GSVA (Xia et al., 2023b), NeXt-Chat (Zhang et al., 2023a), and LISA++ (Yang et al., 2023b) send the embedding corresponding to special tokens to the mask decoder of SAM, LLaVA-G to OpenSeeD (Zhang et al., 2023c), Lenna (Wei et al., 2023) to Grounding-DINO (Liu et al.…”
InfluentialVisual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models
Wu, C., Yin, S., Qi, W., Wang, X., Tang, Z., & Duan, N. (2023) · arXiv · doi:10.48550/arxiv.2303.04671 · Cited by 197 · cited 5× · see citing works
“To facilitate the evaluation, many works propose to leverage state-of-the-art proprietary models ( e.g. , ChatGPT (Ope-nAI, 2022), GPT-4 (Achiam et al., 2023)) to automatically judge candidate answers.”
InfluentialUL2: Unifying Language Learning ParadigmsPreprint
Tay, Y., Dehghani, M., Tran, V. Q., Garcia, X., Wei, J., Wang, X., Chung, H. W., Shakeri, S., Bahri, D., Schuster, T., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2205.05131 · Cited by 194 · cited 1× · see citing works
“Brown et al. (2020) discovered that in-context learning, i.e. , prepending the prompt with a few examples demonstrating the desired output of an LLM (Chowdhery et al., 2023; Hoffmann et al., 2022; Tay et al., 2022), improves its performance, especially over unseen tasks.”
Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond
Bai, J., Bai, S., Yang, S., Wang, S., Tan, S., Wang, P., Lin, J., Zhou, C., & Zhou, J. (2023) · arXiv · doi:10.48550/arxiv.2308.12966 · Cited by 193 · cited 8× · see citing works
“For the same reason, mPLUG-Owl2 (Ye et al., 2023d) and Qwen-VL (Bai et al., 2023b) compress the visual features into fixed-length sequences, independent of the resolution, using learnable queries.”
InfluentialScaling Autoregressive Multi-Modal Models: Pretraining and Instruction TuningPreprint
Yu, L., Shi, B., Pasunuru, R., et al. (2023) · arXiv.org · doi:10.48550/arxiv.2309.02591 · Cited by 181 · see citing works
Monkey: Image Resolution and Text Label are Important Things for Large Multi-Modal Models
Li, Z., Yang, B., Liu, Q., Ma, Z., Zhang, S., Yang, J., Sun, Y., Liu, Y., & Bai, X. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02527 · Cited by 174 · cited 2× · see citing works
“Monkey (Li et al., 2023l) uses a Perceiver-like resampler to extract fixed-length sequences from each sub-image and trains on an image-text dataset curated by several vision expert models integrated by ChatGPT.”
TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understanding
Ren, S., Yao, L., Li, S., Sun, X., & Hou, L. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01357 · Cited by 165 · cited 1× · see citing works
“These models process video frames independently, using CLIP-based backbones to extract frame-level features which are then combined us-ing pooling mechanisms (Li et al., 2023j; Maaz et al., 2023) or Q-Former based solutions (Li et al., 2023h; Ren et al., 2023a).”
PaLI-3 Vision Language Models: Smaller, Faster, Stronger
Chen, X., Wang, X., Beyer, L., et al. (2023) · arXiv.org · doi:10.48550/arxiv.2310.09199 · Cited by 164 · cited 1×
“However, recently there has been an interest in addressing more fine-grained visual tasks, such as visual grounding and image generation.”
BackgroundChen, F., Han, M., Zhao, H., et al. (2023) · arXiv.org · doi:10.48550/arxiv.2305.04160 · Cited by 163 · cited 10× · see citing works
“3B ♢ CLIP ViT-L Q-Former ✻ ✗ Visual Dialogue, VQA, Captioning LLaMA-Adapter V2 (Gao et al., 2023) LLaMA-7B ▲ CLIP ViT-L Linear ✗ VQA, Captioning OpenFlamingo (Awadalla et al., 2023) MPT-7B ⋆ CLIP ViT-L XAttn LLM ✗ VQA, Captioning Flamingo (Alayrac et al., 2022) Chinchilla-70B ⋆ NFNet-F6 XAttn LLM ✗ Visual Dialogue, VQ…”
InfluentialDetGPT: Detect What You Need via Reasoning
Pi, R.-J., Gao, J.-H., Diao, S., et al. (2023) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2305.14167 · Cited by 152 · cited 2×
“Specifically, Shikra (Chen et al., 2023f), DetGPT (Pi et al., 2023), ChatSpot (Zhao et al., 2023b), and PVIT (Chen et al., 2023a) leverage LLMs (Achiam et al., 2023; OpenAI, 2022) to combine regions and captions from datasets that present both annotations ( e.g. , COCO).”
LLaVA-Grounding: Grounded Visual Chat with Large Multimodal ModelsPreprint
Zhang, H., Li, H., Li, F., et al. (2023) · arXiv.org · doi:10.48550/arxiv.2312.02949 · Cited by 152 · see citing works
Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning
Liu, F., Lin, K., Li, L., Wang, J., Yacoob, Y., & Wang, L. (2023) · arXiv · doi:10.48550/arxiv.2306.14565 · Cited by 149 · cited 2× · see citing works
“Several studies (Liu et al., 2023b; Zhu et al., 2023a) show that MLLMs tend to exhibit high hallucination rates, especially when generating longer captions.”
A Simple Framework for Open-Vocabulary Segmentation and Detection
Zhang, H., Li, F., Zou, X., Liu, S., Li, C., Yang, J., & Zhang, L. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00100 · Cited by 149 · cited 2× · see citing works
“For output regions, LISA (Lai et al., 2023), GLaMM, GSVA (Xia et al., 2023b), NeXt-Chat (Zhang et al., 2023a), and LISA++ (Yang et al., 2023b) send the embedding corresponding to special tokens to the mask decoder of SAM, LLaVA-G to OpenSeeD (Zhang et al., 2023c), Lenna (Wei et al., 2023) to Grounding-DINO (Liu et al.…”
Generative Multimodal Models are In-Context Learners
Sun, Q., Cui, Y., Zhang, X., Zhang, F., Yu, Q., Wang, Y., Rao, Y., Liu, J., Huang, T., & Wang, X. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.01365 · Cited by 147 · cited 1× · see citing works
“Shikra (Chen et al., 2023f), Kosmos-2 (Peng et al., 2023), MiniGPT-v2 (Chen et al., 2023e), Ferret (You et al., 2023), CogVLM (Wang et al., 2023c), SPHINX (Lin et al., 2023b), Qwen-VL (Bai et al., 2023b), and Griffon (Zhan et al., Table 2: Summary of MLLMs with components specifically designed for…”
MLLM-Protector: Ensuring MLLM’s Safety without Hurting Performance
Pi, R.-J., Han, T., Zhang, J., et al. (2024) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2401.02906 · Cited by 147 · cited 1×
“Although recent efforts are being made to reduce this phenomenon in text-to-image generative models (Schramowski et al., 2023; Friedrich et al., 2023; Poppi et al., 2024), further exploration is needed to prevent the same behavior in MLLMs (Pi et al., 2024).”
Feng, H., Liu, Q., Liu, H., Tang, J.-Q., Zhou, W.-G., Li, H.-Q., & Huang, C. (2023) · Science China Information Sciences · doi:10.1007/s11432-024-4250-y · Cited by 147 · see citing works
Jiang, A. Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D. S., Casas, D. D. L., Hanna, E. B., Bressand, F., et al. (2024) · arXiv · doi:10.48550/arxiv.2401.04088 · Cited by 143 · see citing works
Cobra: Extending Mamba to Multi-Modal Large Language Model for Efficient Inference
Zhao, H., Zhang, M., Zhao, W., et al. (2024) · AAAI Conference on Artificial Intelligence · doi:10.48550/arxiv.2403.14520 · Cited by 142 · cited 1×
“…OPT (Zhang et al., 2022b), Mag-neto (Wang et al., 2023b), MPT (MosaicML, 2023), and the instruction-tuned (Chung et al., 2022) or multilingual (Xue et al., 2020) flavors of T5 (Raffel et al., 2020), an encoder-decoder language model pre-trained for multiple tasks. et al., 2024; Zhao et al., 2024).”
BuboGPT: Enabling Visual Grounding in Multi-Modal LLMsPreprint
Zhao, Y., Lin, Z.-J., Zhou, D., et al. (2023) · arXiv.org · doi:10.48550/arxiv.2307.08581 · Cited by 137 · cited 2× · see citing works
“BuboGPT (Zhao et al., 2023c) leverages a combination of RAM, Grounding-DINO, and SAM and matches tags with nouns in the output sequence.”
VL-Mamba: Exploring State Space Models for Multimodal Learning
Qiao, Y., Yu, Z., Guo, L.-T., et al. (2024) · ENLSP · doi:10.48550/arxiv.2403.13600 · Cited by 136
Learning 3D Representations from 2D Pre-Trained Models via Image-to-Point Masked Autoencoders
Zhang, R., Wang, L., Qiao, Y., Gao, P., & Li, H. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.02085 · Cited by 132 · see citing works
NExT-GPT: Any-to-Any Multimodal LLMPreprint
Wu, S., Fei, H., Qu, L., Ji, W., & Chua, T. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2309.05519 · Cited by 132 · see citing works
Kosmos-2: Grounding Multimodal Large Language Models to the WorldPreprint
Peng, Z., Wang, W., Dong, L., Hao, Y., Huang, S., Ma, S., & Wei, F. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2306.14824 · Cited by 129 · cited 5× · see citing works
“3B ♢ CLIP ViT-L Q-Former ✻ ✗ Visual Dialogue, VQA, Captioning LLaMA-Adapter V2 (Gao et al., 2023) LLaMA-7B ▲ CLIP ViT-L Linear ✗ VQA, Captioning OpenFlamingo (Awadalla et al., 2023) MPT-7B ⋆ CLIP ViT-L XAttn LLM ✗ VQA, Captioning Flamingo (Alayrac et al., 2022) Chinchilla-70B ⋆ NFNet-F6 XAttn LLM ✗ Visual Dialogue, VQ…”
InfluentialSelf-Instruct: Aligning Language Models with Self-Generated Instructions
Wang, Y., Kordi, Y., Mishra, S., Liu, A., Smith, N. A., Khashabi, D., & Hajishirzi, H. (2022) · arXiv · doi:10.48550/arxiv.2212.10560 · Cited by 118 · see citing works
Semantic-SAM: Segment and Recognize Anything at Any GranularityPreprint
Li, F., Zhang, H., Sun, P., Zou, X., Liu, S., Yang, J., Li, C., Zhang, L., & Gao, J. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2307.04767 · Cited by 115 · cited 1× · see citing works
“However, recently there has been an interest in addressing more fine-grained visual tasks, such as visual grounding and image generation.”
BackgroundVIGC: Visual Instruction Generation and Correction
Wang, B., Wu, F., Han, X., Peng, J., Zhong, H.-P., Zhang, P., Dong, X.-W., Li, W.-J., Li, W., Wang, J., et al. (2023) · AAAI Conference on Artificial Intelligence · doi:10.48550/arxiv.2308.12714 · Cited by 108 · cited 1×
“While some solutions are emerging to mitigate this problem (Liu et al., 2023b; Wang et al., 2023a; Wu et al., 2023d; Yin et al., 2023b; Jing et al., 2023), understanding and correcting the underlying causes of hallucinations remains an important open challenge that is worth addressing to allow the…”
OneLLM: One Framework to Align All Modalities with Language
Han, J., Gong, K., Zhang, Y., Wang, J., Zhang, K., Lin, D., Qiao, Y., Gao, P., & Yue, X. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02510 · Cited by 103 · see citing works
Asai, A., Wu, Z., Wang, Y., Sil, A., & Hajishirzi, H. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.11511 · Cited by 101 · cited 1× · see citing works
“While retrieval-augmented generation (RAG) is a consolidated technique in LLMs (Lewis et al., 2020; Asai et al., 2023), its application in MLLMs is still under-explored.”
Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Action
Lu, J., Clark, C., Lee, S.-H., Zhang, Z., Khosla, S., Marten, R., Hoiem, D., & Kembhavi, A. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02497 · Cited by 98 · see citing works
Hu, Z., İşcen, A., Sun, C., Wang, Z., Chang, K.-W., Sun, Y., Schmid, C., Ross, D. A., & Fathi, A. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.02238 · Cited by 96 · cited 1× · see citing works
“We believe that the emergence of VQA datasets that require external retrieved knowledge (Chen et al., 2023k; Mensink et al., 2023) may enable the development of MLLMs with RAG capabilities (Hu et al., 2023b; Caffagni et al., 2024).”
LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasoning, and Planning
Chen, S., Chen, X., Zhang, C., Li, M., Yu, G., Fei, H., Zhu, H., Fan, J., & Chen, T. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.02496 · Cited by 92 · cited 1× · see citing works
“For input regions, GLaMM (Rasheed et al., 2023), GPT4RoI (Zhang et al., 2023g), ASM (Wang et al., 2023d) and ChatterBox (Tian et al., 2024b) leverage features of the image encoder to perform ROI align on the bounding box, whereas PVIT (Chen et al., 2023a) exploits Re-gionCLIP (Zhong et al., 2022).”
Vista-llama: Reducing Hallucination in Video Language Models via Equal Distance to Visual Tokens
Ma, F., Jin, X.-J., Wang, H., et al. (2023) · Computer Vision and Pattern Recognition · doi:10.1109/cvpr52733.2024.01249 · Cited by 92 · cited 1×
“However, there are also some attempts to develop video-specific adapters (Liu et al., 2023g; Ma et al., 2023a) that can capture fine-grained temporal information.”
Panagopoulou, A., Xue, L., Ning, Y.-X., et al. (2023) · arXiv.org · doi:10.48550/arxiv.2311.18799 · Cited by 92 · cited 1× · see citing works
“This is usually achieved by aligning multimodal features through Transformer blocks such as Q-Former (Chen et al., 2023c; Panagopoulou et al., 2023) and Perceiver (Zhao et al., 2023d), or by utilizing ImageBind to effectively extract features that are inherently multimodal (Su et al., 2023).”
VideoChat: Chat-Centric Video UnderstandingPreprint
Li, K., He, Y., Wang, Y., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.06355 · Cited by 89 · see citing works
Honeybee: Locality-Enhanced Projector for Multimodal LLM
Cha, J., Kang, W., Mun, J., & Roh, B. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01311 · Cited by 88 · cited 1× · see citing works
“A different approach (Cha et al., 2023) proposes to replace linear layers with convolutional ones, demonstrating moderate improvements.”
OtterHD: A High-Resolution Multi-modality ModelPreprint
Li, B., Zhang, P.-Y., Yang, J., et al. (2023) · arXiv.org · doi:10.48550/arxiv.2311.04219 · Cited by 87 · cited 1× · see citing works
“Another solution, namely OtterHD (Li et al., 2023a), can seamlessly deal with any resolution or aspect ratio, as it directly feeds large image patches of 30 × 30 pixels to the LLM, without the need for a visual encoder.”
Otter: A Multi-Modal Model with In-Context Instruction TuningPreprint
Li, B., Zhang, Y., Chen, L., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.03726 · Cited by 86 · see citing works
MM-Interleaved: Interleaved Image-Text Generative Modeling via Multi-modal Feature Synchronizer
Tian, C., Zhu, X., Xiong, Y., et al. (2024) · arXiv.org · doi:10.48550/arxiv.2401.10208 · Cited by 83 · cited 2×
“BuboGPT (Zhao et al., 2023c) leverages a combination of RAM, Grounding-DINO, and SAM and matches tags with nouns in the output sequence.”
ChatSpot: Bootstrapping Multimodal LLMs via Precise Referring Instruction Tuning
Zhao, L., Yu, E., Ge, Z., et al. (2023) · International Joint Conference on Artificial Intelligence · doi:10.48550/arxiv.2307.09474 · Cited by 79
mPLUG-DocOwl: Modularized Multimodal Large Language Model for Document UnderstandingPreprint
Ye, J., Hu, A., Xu, H., Ye, Q., Yan, M., Dan, Y., Zhao, C., Xu, G., Li, C., Tian, J., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2307.02499 · Cited by 74 · cited 1× · see citing works
“A different approach has been studied by Li et al. (2023f) which proposes to fine-tune the LLM by adding two special tokens ( i.e. , <start> and <end> ), and directly encode the generated text be-tween these two tokens using the text encoder in the SD model.”
MM-Vet: Evaluating Large Multimodal Models for Integrated CapabilitiesPreprint
Yu, W., Yang, Z., Li, L., Wang, J., Lin, K., Liu, Z., Wang, X., & Wang, L. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2308.02490 · Cited by 74 · see citing works
OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documents
Laurençon, H., Saulnier, L., Tronchon, L., Bekman, S., Singh, A., Lozhkov, A., Wang, T., Karamcheti, S., Rush, A. M., Kiela, D., et al. (2023) · arXiv · doi:10.48550/arxiv.2306.16527 · Cited by 72 · cited 15× · see citing works
“3B ♢ CLIP ViT-L Q-Former ✻ ✗ Visual Dialogue, VQA, Captioning LLaMA-Adapter V2 (Gao et al., 2023) LLaMA-7B ▲ CLIP ViT-L Linear ✗ VQA, Captioning OpenFlamingo (Awadalla et al., 2023) MPT-7B ⋆ CLIP ViT-L XAttn LLM ✗ VQA, Captioning Flamingo (Alayrac et al., 2022) Chinchilla-70B ⋆ NFNet-F6 XAttn LLM ✗ Visual Dialogue, VQ…”
InfluentialCAT: Enhancing Multimodal Large Language Model to Answer Questions in Dynamic Audio-Visual Scenarios
Ye, Q.-L., Yu, Z.-T., Shao, R., et al. (2024) · European Conference on Computer Vision · doi:10.48550/arxiv.2403.04640 · Cited by 72
MultiModal-GPT: a vision and language model for dialogue with humans
Gong, T., Lyu, C., Zhang, S., Wang, Y., Zheng, M., Zhao, Q., Liu, K., Zhang, W., Luo, P., & Chen, K. (2023) · arXiv · doi:10.48550/arxiv.2305.04790 · Cited by 70 · cited 2× · see citing works
“3B ♢ CLIP ViT-L Q-Former ✻ ✗ Visual Dialogue, VQA, Captioning LLaMA-Adapter V2 (Gao et al., 2023) LLaMA-7B ▲ CLIP ViT-L Linear ✗ VQA, Captioning OpenFlamingo (Awadalla et al., 2023) MPT-7B ⋆ CLIP ViT-L XAttn LLM ✗ VQA, Captioning Flamingo (Alayrac et al., 2022) Chinchilla-70B ⋆ NFNet-F6 XAttn LLM ✗ Visual Dialogue, VQ…”
PMC-VQA: Visual Instruction Tuning for Medical Visual Question Answering
Zhang, X., Wu, C., Zhao, Z., Lin, W., Zhang, Y., Wang, Y., & Xie, W. (2023) · arXiv · doi:10.48550/arxiv.2305.10415 · Cited by 66 · see citing works
TouchStone: Evaluating Vision-Language Models by Language Models
Bai, S., Yang, S.-S., Bai, J., et al. (2023) · arXiv.org · doi:10.48550/arxiv.2308.16890 · Cited by 63
Unified Model for Image, Video, Audio and Language Tasks
Shukor, M., Dancette, C., Ramé, A., et al. (2023) · Trans. Mach. Learn. Res. · doi:10.48550/arxiv.2307.16184 · Cited by 61
Ferret: Refer and Ground Anything Anywhere at Any GranularityPreprint
You, H., Zhang, H., Gan, Z., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.07704 · Cited by 59 · cited 8× · see citing works
“Differently, Kosmos-2 (Peng et al., 2023) and Ferret (You et al., 2023) exploit an open-vocabulary detector (Li et al., 2022) to ground noun chunks parsed from cap-tions and then reconstruct referring expressions.”
InfluentialPG-Video-LLaVA: Pixel Grounding Large Video-Language Models
Munasinghe, S., Thushara, R., Maaz, M., et al. (2023) · arXiv.org · doi:10.48550/arxiv.2311.13435 · Cited by 59 · cited 1×
“In addition to encoding video frames, some works (Munasinghe et al., 2023; Zhang et al., 2023b) also employ audio features to enrich the representation of input video sequences.”
DreamLLM: Synergistic Multimodal Comprehension and CreationPreprint
Dong, R., Han, C., Peng, Y., Qi, Z., Ge, Z., Yang, J., Zhao, L., Sun, J., Zhou, H., Wei, H., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2309.11499 · Cited by 59 · see citing works
EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of ThoughtPreprint
Mu, Y., Zhang, Q., Hu, M., Wang, W., Ding, M., Jin, J., Wang, B., Dai, J., Qiao, Y., & Luo, P. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.15021 · Cited by 58 · cited 1× · see citing works
“…for document analysis and text-intensive visual inputs (Lv et al., 2023; Ye et al., 2023a), those proposed for embodied AI and robotics (Driess et al., 2023; Mu et al., 2023), and those tailored for specific domains such as medicine (Li et al., 2023d) and autonomous driving (Xu et al., 2023c).”
IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language ReasoningPreprint
Lu, P., Qiu, L., Chen, J., Xia, T., Zhao, Y., Zhang, W., Zhou, Y., Liang, X., & Zhu, S. C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.13214 · Cited by 57 · cited 1× · see citing works
“IconQA (Lu et al., 2021) tests the visual reasoning abilities of vision-and-language models on three types of questions: multiple-image-choice, multiple-text-choice, and fill-in-the-blank.”
Visual Instruction Tuning with Polite FlamingoPreprint
Chen, D., Liu, J., Dai, W., et al. (2023) · AAAI Conference on Artificial Intelligence · doi:10.48550/arxiv.2307.01003 · Cited by 57 · cited 3× · see citing works
“3B ♢ CLIP ViT-L Q-Former ✻ ✓ Visual Dialogue, VQA, Captioning, Referring, REC Shikra (Chen et al., 2023f) Vicuna-13B ♦ CLIP ViT-L Linear ✓ Visual Dialogue, VQA, Captioning, Referring, REC, GroundCap Clever Flamingo (Chen et al., 2023b) LLaMA-7B ▲ CLIP ViT-L XAttn LLM ✓ Visual Dialogue, VQA, Captioning SVIT (Zhao et al…”
KOSMOS-2.5: A Multimodal Literate ModelPreprint
Lv, T., Huang, Y., Chen, J., Zhao, Y., Jia, Y., Cui, L., Ma, S., Chang, Y., Huang, S., Wang, W., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2309.11419 · Cited by 55 · cited 1× · see citing works
“Some examples are MLLMs designed for document analysis and text-intensive visual inputs (Lv et al., 2023; Ye et al., 2023a), those proposed for embodied AI and robotics (Driess et al., 2023; Mu et al., 2023), and those tailored for specific domains such as medicine (Li et al., 2023d) and autonomous…”
See, Say, and Segment: Teaching LMMs to Overcome False Premises
Wu, T.-H., Biamby, G., Chan, D., et al. (2023) · Computer Vision and Pattern Recognition · doi:10.1109/cvpr52733.2024.01278 · Cited by 54
LION : Empowering Multimodal Large Language Model with Dual-Level Visual Knowledge
Chen, G., Shen, L., Shao, R., Deng, X., & Nie, L. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02506 · Cited by 53 · cited 1× · see citing works
“It is a Transformer-based model proposed in BLIP-2 (Li et al., 2023g) and then used in several other approaches (Chen et al., 2023d; Dai et al., 2023; Hu et al., 2024).”
(2024) · doi:10.1109/cvprw63382.2024.00188 · Cited by 51 · cited 1× · see citing works
“We believe that the emergence of VQA datasets that require external retrieved knowledge (Chen et al., 2023k; Mensink et al., 2023) may enable the development of MLLMs with RAG capabilities (Hu et al., 2023b; Caffagni et al., 2024).”
Self-citationGuo, Z., Zhang, R., Zhu, X., Tang, Y., Ma, X., Han, J., Chen, K., Gao, P., Li, X., Li, H., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2309.00615 · Cited by 50 · see citing works
VL-GPT: A Generative Pre-trained Transformer for Vision and Language Understanding and Generation
Zhu, J., Ding, X.-H., Ge, Y., et al. (2023) · arXiv.org · doi:10.48550/arxiv.2312.09251 · Cited by 50 · cited 2×
“This is usually achieved by fine-tuning the textual model (Zhu et al., 2023b; Ge et al., 2023a,b).”
Integrally Pre-Trained Transformer Pyramid Networks
Tian, Y., Xie, L.-X., Wang, Z.-Z., Wei, L.-H., Zhang, X.-P., Jiao, J.-B., Wang, Y., Tian, Q., & Ye, Q.-X. (2023) · Computer Vision and Pattern Recognition · doi:10.1109/cvpr52729.2023.01785 · Cited by 49 · see citing works
Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP TasksPreprint
Wang, Y., Mishra, S. R., Alipoormolabashi, P., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2204.07705 · Cited by 46 · see citing works
InternGPT: Solving Vision-Centric Tasks by Interacting with ChatGPT Beyond LanguagePreprint
Liu, Z., He, Y., Wang, W., Wang, W., Wang, Y., Chen, S., Zhang, Q., Lai, Z., Yang, Y., Li, Q., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.05662 · Cited by 46 · see citing works
Cheap and Quick: Efficient Vision-Language Instruction Tuning for Large Language ModelsPreprint
Luo, G., Zhou, Y., Ren, T., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.15023 · Cited by 45 · see citing works
LISA: Reasoning Segmentation via Large Language ModelPreprint
Lai, X. Q., Tian, Z., Chen, Y., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2308.00692 · Cited by 44 · see citing works
PaLI-X: On Scaling up a Multilingual Vision and Language ModelPreprint
Chen, X., Djolonga, J., Padlewski, P., Mustafa, B., Changpinyo, S., Wu, J., Riquelme, C., Goodman, S., Wang, X., Tay, Y., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.18565 · Cited by 42 · see citing works
Griffon: Spelling out All Object Locations at Any Granularity with Large Language ModelsPreprint
Zhan, Y., Zhu, Y., Chen, Z., et al. (2023) · European Conference on Computer Vision · doi:10.48550/arxiv.2311.14552 · Cited by 41 · see citing works
Visual Instruction Tuning towards General-Purpose Multimodal Model: A Survey
Huang, J., Zhang, J., Jiang, K., et al. (2023) · arXiv.org · doi:10.48550/arxiv.2312.16602 · Cited by 37 · cited 1×
“In comparison with existing surveys on the topic (Yin et al., 2023a; Wu et al., 2023b; Huang et al., 2023a), our paper exhibits substantial differences.”
StableLLaVA: Enhanced Visual Instruction Tuning with Synthesized Image-Dialogue DataPreprint
Li, Y., Zhang, C., Yu, G., Wang, Z.-B., Fu, B., Lin, G., Shen, C.-H., Chen, L., & Wei, Y.-C. (2023) · arXiv.org · doi:10.48550/arxiv.2308.10253 · Cited by 37 · cited 1× · see citing works
“Given the amount of training data currently available, approaches like SPHINX-X (Gao et al., 2024) opt to perform a single all-in-one training stage in which to update all model components, possibly also using text-only data to preserve the conversational capabilities of the LLM.”
MethodJointly Training Large Autoregressive Multimodal Models
Aiello, E., Yu, L., Nie, Y., et al. (2023) · International Conference on Learning Representations · doi:10.48550/arxiv.2309.15564 · Cited by 36 · cited 1×
“2024) propose a methodology to mix an LLM architecture with an autoregressive generator, CM3Leon (Yu et al., 2023a), via bi-directional cross-attention across the architectures of both models.”
Encyclopedic VQA: Visual questions about detailed properties of fine-grained categories
Mensink, T., Uijlings, J., Castrejón, L., Goel, A., Cadar, F., Zhou, H., Sha, F., Araújo, A. D. M., & Ferrari, V. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00289 · Cited by 35 · cited 1× · see citing works
“We believe that the emergence of VQA datasets that require external retrieved knowledge (Chen et al., 2023k; Mensink et al., 2023) may enable the development of MLLMs with RAG capabilities (Hu et al., 2023b; Caffagni et al., 2024).”
AnyMAL: An Efficient and Scalable Any-Modality Augmented Language Model
Moon, S., Madotto, A., Lin, Z., Nagarajan, T., Smith, M., Jain, S. S. B. C., Yeh, C.-F., Murugesan, P., Heidari, P., Liu, Y., et al. (2024) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2024.emnlp-industry.98 · Cited by 33 · cited 1× · see citing works
“Additionally, some works also effectively encode 3D data (Yin et al., 2023d) and IMU sensor signals (Moon et al., 2023).”
SCITUNE: Aligning Large Language Models with Human-Curated Scientific Multimodal Instructions
Horawalavithana, S., Munikoti, S., Stewart, I., et al. (2023) · NLP4SCIENCE · doi:10.18653/v1/2024.nlp4science-1 · Cited by 33
LLMGA: Multimodal Large Language Model based Generation Assistant
Xia, B., Wang, S., Tao, Y., et al. (2023) · European Conference on Computer Vision · doi:10.48550/arxiv.2311.16500 · Cited by 31 · cited 1×
“Similarly, in (Xia et al., 2023a) the LLM is trained to output detailed language-based generation prompts which are employed for generation or editing tasks.”
FaithScore: Fine-grained Evaluations of Hallucinations in Large Vision-Language Models
Jing, L., Li, R., Chen, Y., & Du, X. (2023) · arXiv · doi:10.48550/arxiv.2311.01477 · Cited by 31 · see citing works
Pink: Unveiling the Power of Referential Comprehension for Multi-modal LLMs
Xuan, S., Guo, Q., Yan, S., & Zhang, S. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01313 · Cited by 30 · see citing works
ChatterBox: Multi-round Multimodal Referring and Grounding
Tian, Y., Ma, T., Xie, L., et al. (2024) · arXiv.org · doi:10.48550/arxiv.2401.13307 · Cited by 30 · cited 2×
“ChatterBox (Tian et al., 2024b) combines features from the iTPN-B encoder (Tian et al., 2023) and the MLLM and provides them to the DINO detector (Zhang et al., 2022a).”
Tiny LVLM-eHub: Early Multimodal Experiments with BardPreprint
Shao, W., Hu, Y.-T., Gao, P., et al. (2023) · arXiv.org · doi:10.48550/arxiv.2308.03729 · Cited by 30 · cited 1× · see citing works
“…(i) the adaptation issue of switching from small images seen during training to larger ones at inference time and (ii) computational costs provided by the increased number of tokens in both the visual encoder and the LLM, given by the quadratic complexity of the attention-based architectures.”
The All-Seeing Project: Towards Panoptic Visual Recognition and Understanding of the Open WorldPreprint
Wang, W., Shi, M., Li, Q., Wang, W., Huang, Z., Xing, L., Chen, Z., Li, H., Zhu, X., Cao, Z., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2308.01907 · Cited by 29 · cited 2× · see citing works
“ASM (Wang et al., 2023d), GLaMM (Rasheed et al., 2023), and LLaVA-G (Zhang et al., 2023d) propose automated pipelines comprising multiple steps based on off-the-shelf models for generating large corpora of conversations grounded in their corresponding images.”
Chen, Y., Sikka, K., Cogswell, M., Ji, H., & Divakaran, A. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01350 · Cited by 29 · cited 1× · see citing works
“…et al., 2024), turns out to be critical for aligning the behavior of an LLM with that of humans and currently empowers the most advanced LLMs, eventually boosted via reinforcement learning from human feedback (RLHF) (Ouyang et al., 2022; Achiam et al., 2023; Chen et al., 2023l; Bai et al., 2023a).”
BackgroundTextBind: Multi-turn Interleaved Multimodal Instruction-following in the Wild
Li, H., Li, S., Cai, D., et al. (2023) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2309.08637 · Cited by 29 · cited 1×
“However, recently there has been an interest in addressing more fine-grained visual tasks, such as visual grounding and image generation.”
BackgroundCoDi-2: In-Context, Interleaved, and Interactive Any-to-Any Generation
Tang, Z., Yang, Z., Khademi, M., Liu, Y., Zhu, C., & Bansal, M. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02589 · Cited by 28 · cited 1× · see citing works
“Additional research endeavors have been conducted to introduce MLLMs in the field of image editing (Fu et al., 2024; Huang et al., 2023c; Tang et al., 2023).”
mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality CollaborationPreprint
Ye, Q., Xu, H., Ye, J., Yan, M., Hu, A., Liu, H., Qian, Q., Zhang, J., Huang, F., & Zhou, J. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2311.04257 · Cited by 28 · see citing works
Fair Diffusion: Instructing Text-to-Image Generation Models on FairnessPreprint
Friedrich, F., Brack, M., Struppek, L., Hintersdorf, D., Schramowski, P., Luccioni, A. S., & Kristian Kersting, B. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2302.10893 · Cited by 28 · see citing works
Jack of All Tasks, Master of Many: Designing General-purpose Coarse-to-Fine Vision-Language Model
Pramanick, S., Han, G., Hou, R., Nag, S., Lim, S.-N., Ballas, N., Wang, Q., Chellappa, R., & Almahairi, A. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01335 · Cited by 22 · cited 2× · see citing works
“For instance, CoinIt (Pramanick et al., 2023) is a unified set of 14 benchmarks converted into an instruction-tuning format, spanning from single-image coarse-level to multi-image region-level tasks.”
FoodLMM: A Versatile Food Assistant Using Large Multi-Modal Model
Yin, Y., Qi, H., Zhu, B., et al. (2025) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2025.3590924 · Cited by 22 · see citing works
Multimodal C4: An Open, Billion-scale Corpus of Images Interleaved With Text
Zhu, W., Hessel, J., Awadalla, A., Gadre, S. Y., Dodge, J., Fang, A., Yu, Y., Schmidt, L., Wang, W. Y., & Choi, Y. (2023) · arXiv · doi:10.48550/arxiv.2304.06939 · Cited by 21 · cited 3×
“Among these datasets, the most used are WebLI (Chen et al., 2023j), composed of 10B images and image-text pairs, and MMC4 (Zhu et al., 2023d), an extension of the text-only C4 (Raffel et al., 2020) dataset composed of 365M documents and 156B tokens relatives to different concepts, and OBELICS…”
Flagged first, then by citations
Research works printed but not linked: 21
- Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Mar- garet Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. 2015. VQA: Visual Question An- swering. In ICCV.
- Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with im- proved correlation with human judgments. In ACL Workshops.
- Davide Caffagni, Federico Cocchi, Nicholas Moratelli, Sara Sarto, Marcella Cornia, Lorenzo Baraldi, and Rita Cucchiara. 2024. Wiki-LLaVA: Hierarchical Retrieval-Augmented Generation for Multimodal LLMs. In CVPR Workshops.
- Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. 2023. Eva: Exploring the limits of masked visual representation learning at scale. In CVPR.
- Tsu-Jui Fu, Wenze Hu, Xianzhi Du, William Yang Wang, Yinfei Yang, and Zhe Gan. 2024. Guid- ing Instruction-based Image Editing via Multimodal Large Language Models. In ICLR.
- Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. 2022. Masked Au- toencoders Are Scalable Vision Learners. In CVPR.
- Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. 2017. GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium. In NeurIPS.
- Yining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Gan. 2023. 3D-LLM: Injecting the 3D World into Large Language Models. In NeurIPS.
- Jing Yu Koh, Ruslan Salakhutdinov, and Daniel Fried. 2023b. Grounding Language Models to Images for Multimodal Inputs and Outputs. In ICML.
- Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Ui- jlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, et al. 2020. The Open Images Dataset V4: Unified Image Classification, Object Detection, and Visual Relationship Detection at Scale. IJCV, 128:1956- 1981.
- Fangyu Liu, Guy Emerson, and Nigel Collier. 2023a. Visual Spatial Reasoning. TACL, 11:635-651.
- Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. 2023c. Mitigating Hallu- cination in Large Multi-Modal Models via Robust In- struction Tuning. arXiv preprint arXiv:2306.14565. · arXiv
- Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023e. Visual Instruction Tuning. In NeurIPS.
- Fan Ma, Xiaojie Jin, Heng Wang, Yuchen Xian, Jiashi Feng, and Yi Yang. 2023a. Vista-LLaMA: Reliable Video Narrator via Equal Distance to Visual Tokens. arXiv preprint arXiv:2312.08870. · arXiv
- Hongyu Wang, Shuming Ma, Shaohan Huang, Li Dong, Wenhui Wang, Zhiliang Peng, Yu Wu, Payal Bajaj, Saksham Singhal, Alon Benhaim, et al. 2023b. Mag- neto: A Foundation Transformer. In ICML.
- Jingkang Yang, Yi Zhe Ang, Zujin Guo, Kaiyang Zhou, Wayne Zhang, and Ziwei Liu. 2022. Panoptic scene graph generation. In ECCV.
- Jiabo Ye, Anwen Hu, Haiyang Xu, Qinghao Ye, Ming Yan, Guohai Xu, Chenliang Li, Junfeng Tian, Qi Qian, Ji Zhang, et al. 2023b. UReader: Universal OCR-free Visually-situated Language Understand- ing with Multimodal Large Language Model. In EMNLP.
- Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. 2023a. A Survey on Multimodal Large Language Models. arXiv preprint arXiv:2306.13549. · arXiv
- Shukang Yin, Chaoyou Fu, Sirui Zhao, Tong Xu, Hao Wang, Dianbo Sui, Yunhang Shen, Ke Li, Xing Sun, and Enhong Chen. 2023b. Woodpecker: Hallucina- tion correction for multimodal large language models. arXiv preprint arXiv:2310.16045. · arXiv
- Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L Berg. 2016. Modeling Context in Referring Expressions. In ECCV.
- Yiwu Zhong, Jianwei Yang, Pengchuan Zhang, Chun- yuan Li, Noel Codella, Liunian Harold Li, Luowei Zhou, Xiyang Dai, Lu Yuan, Yin Li, et al. 2022. Re- gionCLIP: Region-based Language-Image Pretrain- ing. In CVPR.
Cited by
Sorted by citations
A Survey on the Memory Mechanism of Large Language Model-based Agents
Zhang, Z., Dai, Q., Bo, X., Ma, C., Li, R., Chen, X., Zhu, J., Dong, Z., & Wen, J.-R. (2025) · ACM Transactions on Information Systems · doi:10.1145/3748302 · Cited by 118 · see citing works
Industrial Foundation Models (IFMs) for intelligent manufacturing: A systematic review
Zhao, S., Liu, S., Jiang, Y., Zhao, B. M., Lv, Y., Zhang, J., Wang, L., & Zhong, R. Y. (2025) · Journal of Manufacturing Systems · doi:10.1016/j.jmsy.2025.06.011 · Cited by 46 · see citing works
Wang, B., Zheng, Y., Han, X., Kong, L., Xiao, G., Xiao, Z., & Chen, S. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01715-x · Cited by 34 · see citing works
“Employing Multi-modal Large Language Models 163 , next-generation smart glasses not only enhance the quality of real-time decision-making 164 but also deliver personalized suggestions 165 derived from aggregated life and dietary data.”
Towards Retrieval-Augmented Architectures for Image Captioning
Sarto, S., Cornia, M., Baraldi, L., Nicolosi, A., & Cucchiara, R. (2024) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3663667 · Cited by 20 · see citing works
Self-citationSafe-CLIP: Removing NSFW Concepts from Vision-and-Language Models
Poppi, S., Poppi, T., Cocchi, F., Cornia, M., Baraldi, L., & Cucchiara, R. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73668-1_20 · Cited by 22 · see citing works
Self-citationWei, S., Wang, Z., Li, M., Liu, X., & Wu, B. (2025) · Information Processing & Management · doi:10.1016/j.ipm.2025.104089 · Cited by 19 · see citing works
Deep Learning in Palmprint Recognition: A Comprehensive Survey
Gao, C., Yang, Z., Jia, W., Leng, L., Zhang, B., & Teoh, A. B. J. (2026) · IEEE Transactions on Systems Man and Cybernetics Systems · doi:10.1109/tsmc.2025.3649416 · Cited by 17 · see citing works
“An emerging direction is the adaptation of large language models (LLMs) through multimodal fusion and synthetic palmprint generation [167].”
A Multimodal Framework Embedding Retrieval-Augmented Generation with MLLMs for Eurobarometer Data
Papageorgiou, G., Sarlis, V., Μaragoudakis, M., & Tjortjis, C. (2025) · AI · doi:10.3390/ai6030050 · Cited by 13 · see citing works
“Those approaches advance the models’ contextual understanding and efficiency, making them well suited for applications involving dynamic content generation and interactive AI [22,23].”
Deep Learning in Palmprint Recognition-A Comprehensive SurveyPreprint
Gao, C., Yang, Z., Jia, W., Leng, L., Zhang, B., & Teoh, A. B. J. (2025) · Qeios · doi:10.32388/yz5tfy · Cited by 12 · see citing works
Multimodal language models in agriculture: A tutorial and survey
Haghighat, M., Saleh, A., & Azghadi, M. R. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.104042 · Cited by 13 · see citing works
Large language models for detecting colorectal polyps in endoscopic images
Carlini, L., Massimi, D., Mori, Y., Antonelli, G., Rizkala, T., Spadaccini, M., Lena, C., Parasa, S., Bisschops, R., Renteln, D. A. V., et al. (2025) · Gut · doi:10.1136/gutjnl-2025-335091 · Cited by 9 · see citing works
A Generative AI-Empowered Digital Tutor for Higher Education Courses
Reicher, H., Frenkel, Y., Lavi, M., Nasser, R., Ran-Milo, Y., Sheinin, R., Shtaif, M., & Milo, T. (2025) · Information · doi:10.3390/info16040264 · Cited by 13 · see citing works
“This approach is known as prompt engineering and it is particularly useful with closed source commercial LLMs such as OpenAI’s GPT series, Anthropic’s Claude, Google’s Gemini, and others [27].”
Artificial intelligence performance in pediatric asthma
Yiğit, E. Ş., Taşkırdı, İ., Hacı, İ. A., & Tunçel, T. (2025) · Journal of Asthma · doi:10.1080/02770903.2025.2531500 · Cited by 9 · see citing works
“It answers questions with key-words, images or plugins (7,8).”
Leveraging multimodal large language model for multimodal sequential recommendation
Wang, Z., Liu, B., Huang, W., Hao, T., Zhou, H., & Guo, Y. (2025) · Scientific Reports · doi:10.1038/s41598-025-14251-1 · Cited by 10 · see citing works
“…general knowledge of MLLM, models such as CLIPViT, GPT4V, and LLaMA can effectively integrate and process multimodal data, including text, images, and audio, through pre-training and fine-tuning strategies, demonstrating significant zero-shot recommendation capabilities across various domains 28 .”
Zhu, F., Chen, J.-Y., Wen, J., Yang, Y., Yi, C.-Y., Yun, T., Zhang, P., Cai, J., Niyato, D., & Guizani, M. (2026) · IEEE Communications Surveys and Tutorials · doi:10.1109/comst.2026.3665395 · Cited by 27 · see citing works
“It necessitates the cross-modal fusion of subtle cues, such as language, gestures, and tone, to decipher a user’s implicit intent, which is not fully contained within any single modality [106], [107].”
Computer Audition: From Task-Specific Machine Learning to Foundation Models
Triantafyllopoulos, A., Tsangko, I., Gebhard, A., Mesaros, A., Virtanen, T. I., & Schuller, B. W. (2025) · Proceedings of the IEEE · doi:10.1109/jproc.2025.3593952 · Cited by 12 · see citing works
“Recent surveys have documented this trend [7], [8], [9], [10], [11], [12].”
Multimodal AI for UAV: Vision–Language Models in Human– Machine Collaboration
Krupáš, M., Urblík, Ľ., & Zolotová, I. (2025) · Electronics · doi:10.3390/electronics14173548 · Cited by 12 · see citing works
Can LLMs Generate Behaviors for Embodied Virtual Agents Based on Personality Traits?
Han, B., Kwon, D., Lin, S., Shrestha, K., & Gratch, J. S. (2025) · ACM International Conference on Intelligent Virtual Agents (IVA) · doi:10.1145/3717511.3747060 · Cited by 6 · see citing works
Hao, C., Wang, Z., Mao, X., He, S., Li, B., Liu, H., Peng, F., & Li, W. (2025) · Computers in Industry · doi:10.1016/j.compind.2025.104302 · Cited by 7 · see citing works
BRIDGE: Bridging Gaps in Image Captioning Evaluation with Stronger Visual Cues
Sarto, S., Cornia, M., Baraldi, L., & Cucchiara, R. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73229-4_5 · Cited by 6 · see citing works
Self-citationHyperbolic Deep Learning for Foundation Models: A Survey
He, N., Madhu, H., Bui, N., Yang, M., & Ying, R. (2025) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3711896.3736564 · Cited by 6 · see citing works
Liu, Y., Lerch, L., Palmieri, L., et al. (2025) · IEEE International Conference on Robot and Human Interactive Communication (RO-MAN) · doi:10.1109/ro-man63969.2025.11217803 · Cited by 4 · see citing works
“MLLMs typically consist of three main components [30]: a language model backbone, a vision encoder, and a vision-to-language adapter (see Fig.”
Boosting Text-To-Image Generation via Multilingual Prompting in Large Multimodal Models
Mu, Y., Li, H., Wang, J., Zhou, X., Wang, C., Luo, Y., He, Q., Xiao, T., Chen, G., & Zhu, J. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10889541 · Cited by 3 · see citing works
Multiple Large AI Models’ Consensus for Object Detection—A Survey
Iwanowski, M., & Gahbler, M. (2025) · Applied Sciences · doi:10.3390/app152412961 · Cited by 6 · see citing works
Song, W., Nabahirwa, E., Zhang, M., Fang, Y., & Ni, Z. (2026) · Discover Artificial Intelligence · doi:10.1007/s44163-026-00895-z · Cited by 7 · see citing works
Sun, H., Wang, T., Yuan, L., Zhao, J., & Lv, J. (2026) · Information Fusion · doi:10.1016/j.inffus.2026.104285 · Cited by 4 · see citing works
“Caffagni et al. [20] review earlier stages of MLLM development and analyze their architectural choices, multimodal alignment strategies, and training techniques.”
A Survey on Fine-Grained Multimodal Large Language Models
Peng, Y., Wang, Z., Li, G., Zheng, X., Yin, S., & He, H. (2026) · Chinese Journal of Electronics · doi:10.23919/cje.2025.00.336 · Cited by 5 · see citing works
Fluent and Accurate Image Captioning with a Self-trained Reward Model
Moratelli, N., Cornia, M., Baraldi, L., & Cucchiara, R. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-78456-9_14 · Cited by 3 · see citing works
Self-citationUnlearning Vision Transformers Without Retaining Data via Low-Rank Decompositions
Poppi, S., Sarto, S., Cornia, M., Baraldi, L., & Cucchiara, R. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-78122-3_10 · Cited by 4 · see citing works
“For its efficiency, LoRA has been used across different research fields, like the fine-tuning of large language models to adapt them for various multimodal tasks [5,6] and foundation models fine-tuning for improving their safety [38].”
Self-citationPixels of Faith: Exploiting Visual Saliency to Detect Religious Image Manipulation
Cartella, G., Cuculo, V., Cornia, M., Papasidero, M., Ruozzi, F., & Cucchiara, R. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-91572-7_14 · Cited by 4 · see citing works
Self-citationSheng, Z., Huang, Z., Qu, Y., Chen, J., Luo, Y., Chen, Y., Leng, Y., & Chen, S. (2026) · CHAIN · doi:10.23919/chain.2026.000005 · Cited by 6 · see citing works
“Recent advances in multimodal large language models (MLLMs) have demonstrated remarkable capabilities in understanding and reasoning over visual and linguistic information, enabling a wide range of applications from vi-* sual question answering (QA) to video analysis [2, 24, 44].”
Personalizing Multimodal Large Language Models for Image Captioning: An Experimental Analysis
Bucciarelli, D., Moratelli, N., Cornia, M., Baraldi, L., & Cucchiara, R. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-92089-9_22 · Cited by 11 · see citing works
Self-citationMoMA: a mixture-of-multimodal-agents architecture for enhancing clinical prediction modelling
Gao, J., Rahman, M., Caskey, J., Oguss, M., O’Rourke, A., Brown, R., Stey, A., Mayampurath, A., Churpek, M. M., Chen, G., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-02219-4 · Cited by 2 · see citing works
“…summary overwrote textual information (e.g., “ The specialist reports right basal atelectasis, small right pleural effusion, and cardiomegaly, whereas the Aggregator — guided by the original clinical and radiology notes — states no effusion or atelectasis and does not mention cardiomegaly ” ).”
Adapt to Scarcity: Few-Shot Deepfake Detection via Low-Rank Adaptation
Cappelletti, S., Baraldi, L., Cocchi, F., Cornia, M., Baraldi, L., & Cucchiara, R. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-78305-0_8 · Cited by 2 · see citing works
“Low-Rank Adaptation (LoRA) [27], which originates for parameter efficient fine-tuning of large language models [16,28], has demonstrated its effectiveness in various tasks [7,8,39,48].”
Self-citationWang, Z., Fang, M., Tang, L., et al. (2025) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2025.3558775 · Cited by 2 · see citing works
“Fine-tuning these models with carefully curated datasets has set new performance benchmarks in multiple tasks [8].”
InfluentialA Semantic Compression Framework for Video Surveillance Applications
Karaa, M., Bahria, R., Ghazzai, H., et al. (2024) · IEEE Canadian Conference on Electrical and Computer Engineering (CCECE) · doi:10.1109/ccece59415.2024.10667302 · Cited by 1 · see citing works
“We leverage the advancements of MLLM in video-related tasks such as understanding and captioning [4] to generate captions for surveillance videos.”
MethodLu, B., Wang, X., Ming, W., & Wang, X. (2025) · Journal Of Big Data · doi:10.1186/s40537-025-01290-8 · Cited by 1 · see citing works
Reasoning LLMs for User-Aware Multimodal Conversational Agents
Rahimi, H., Cattoni, J., Beghili, M., et al. (2025) · IEEE International Conference on Robot and Human Interactive Communication (RO-MAN) · doi:10.1109/ro-man63969.2025.11217707 · Cited by 1 · see citing works
“Recent advancements in Multimodal Large Language Models (MLLMs) [6] have significantly enhanced the capacity of robots to interpret textual and visual cues, enabling more intuitive and adaptive interactions [7], [8].”
Omnidirectional Spatial Modeling from Correlated Panoramas
Zhang, X., Fu, T., & Xu, Z. (2025) · ACM International Conference on Multimedia in Asia (MMAsia) · doi:10.1145/3743093.3770977 · Cited by 1 · see citing works
Igualde-Sáez, A., Rhomrasi, L., Ahsini, Y., Vinuesa, R., Hoyas, S., Garcia-Sabater, J. P., Fullana-i-Alfonso, M. J., & Conejero, J. A. (2026) · The Journal of Supercomputing · doi:10.1007/s11227-026-08291-1 · Cited by 1 · see citing works
“…deeper evaluations indicate that many models still default to superficial pattern matching rather than genuinely reasoning about visual content and math tasks [27], such as diagrams in geometry, charts in algebraic contexts, and multilingual notations require robust cross-modal alignment [28].”
Guainazzo, N., Delzanno, G., Ancona, D., & D’Agostino, D. (2026) · Sensors · doi:10.3390/s26051590 · Cited by 1 · see citing works
Deus in Machina: A Conversational AI for Spiritual Experience
Haslbauer, P., Reichherzer, C., Schmid, M., et al. (2025) · doi:10.1145/3773699.3773922 · Cited by 1 · see citing works
Eck, S. O., & Page, K. (2026) · International Conference on Digital Libraries for Musicology (DLfM) · doi:10.1145/3815723.3815726 · Cited by 1 · see citing works
From text to multimodality: A comprehensive survey of LLMs and MLLMs across diverse domains
Anbukkarasi, S., Hemalatha, S., Arunkumar, B., Shanmugavalli, V., Kokila, S., & A, Y. K. (2026) · ICT Express · doi:10.1016/j.icte.2026.07.023 · Cited by 1 · see citing works
Burghardt, M. (2025) · Elsevier eBooks · doi:10.1016/b978-0-443-26629-4.00016-2 · Cited by 2 · see citing works
SoleCoach: Sole Pressure and IMU-based MLLMs for Skill Coaching
Hirano, T., Yoshihara, H., Peng, Y., Liao, C.-C., Wu, E., & Koike, H. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791181
“To overcome these limitations, large language models (LLMs) have been introduced into sports coaching.”
Ye, Z., Wang, X. Q., Zhang, C., Zhang, R., Sun, M., Li, Q., Yu, C., & Shi, Y. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3790922
“Moreover, the models are trained on large-scale coarse-grained caption data crawled from the Internet [13].”
PERFORMANCE COMPARISON OF VISION-LANGUAGE MODELS IN IMAGE CLASSIFICATION
Özeren, D., Yüksel, M. E., & Yüksel, A. S. (2025) · International Journal of 3D Printing Technologies and Digital Industry · doi:10.46519/ij3dptdi.1693015
Enhancing Chinese Character Writing Learning: The Role of MLLM-based Intelligent Tutoring Systems
Su, B., Chen, Q., Peng, J., Tan, W., & Wang, L. (2025) · doi:10.1109/icaie64856.2025.11158428 · Cited by 1 · see citing works
“A. FunctionalArchitectureofChineseCharacterWritingITS The internal operating mechanism of MLLM is mainly divided into three components, including the input layer, the analysis layer, and the output layer [23].”
Optimizing Multimodal Language Models Through Attention-Based Interpretability
Sergeev, A., & Kotelnikov, E. V. (2025) · Communications in computer and information science · doi:10.1007/978-3-032-07175-0_25
Detection and Measurement of Hailstones with Multimodal Large Language Models
Alker, M., Schedl, D. C., & Stöckl, A. (2025) · doi:10.1109/icecer65523.2025.11400929
“Caffagni et al. [11] provide a comprehensive survey of recent multimodal LLMs, detailing how these models integrate visual and textual modalities with dialogue interfaces.”
Adversarial Iterative Pre-enactment Framework for Air Combat Based on Mental Simulation Theory
Han, S., Zhang, X., Hu, T., & Ma, H. (2025) · Lecture notes in computer science · doi:10.1007/978-981-95-3398-5_18
It's All Connected: A Survey for Multimodal Arabic AIPreprint
Aldabbas, F., Elsafty, H., & Sifa, R. (2025) · Research Square · doi:10.21203/rs.3.rs-8007923/v1
Jailbreaking MLLMs via Attention Redirection and Entropy Regularization
Du, J., Dong, F., & Zhang, F. (2026) · Electronics · doi:10.3390/electronics15010237
Evaluating MLLM Security for the Arabic and French Languages
Abdallah, R., Sanan, M., & Nassar, M. (2025) · doi:10.1109/etecom66111.2025.11319129
“…is composed of visual encoders that are responsible for extracting features from image inputs, an LLM serving as the backbone for handling textual data and integrating multi-modal inputs effectively, and adapters that align non-text modalities to the same embedding space as textual inputs [11].”
Multimodal Image Caption Generation with Semantic and Temporal Context
Rafi, S., Das, R., Rao, P. A., et al. (2025) · International Conference on Soft Computing & Machine Intelligence (ISCMI) · doi:10.1109/iscmi67495.2025.11358624
“The Transformer model [7] has since been investigated in various newer methods for vision-language problems suggested using the Transformer architecture in lieu of traditional RNNs to achieve new state-of-the-art performance by greatly enhancing the quality of the sentences produced.”
Fine-Grained Image-Text Multi-Modal Learning for Substation Defect Detection
Wu, G., Li, X., Cheng, L., Zhang, G., Zipei, D. O. N. G., & Dai, L. (2025) · International Conference on Computer, Vision and Intelligent Technology (ICCVIT) · doi:10.1109/iccvit67848.2025.11391367
GIICS: A dataset for gaze-direction-based intention inference in complex visual scenes
Yu, Q., Ling, C., Feng, S., Hu, J., Lu, Y., & Ding, N. (2025) · International Conference on Artificial Intelligence, Human-Computer Interaction and Robotics (AIHCIR) · doi:10.1109/aihcir67580.2025.11404921
“(3) We benchmarked numerous VLLMs (e.g., Llama 3 [6], LLaVA-1.”
OwnerHunter: Multilingual Website Owner Identification Powered by Large Language Model
Tu, C. S., Dong, E., Zhang, Y., Wang, S., Zhang, Z., Zhang, M., Li, Y., Xue, P., & Yang, J. (2026) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2026.3666852
“MLLMs can seamlessly integrate visual and textual modalities while providing a dialogue-based interface and instruction-following capabilities [17].”
Shao, X., Ma, Q., Wang, Y., et al. (2026) · Discover Applied Sciences · doi:10.1007/s42452-026-08539-8
Taranowski, A. O., & Samoylov, V. B. (2026) · Èlektronnoe modelirovanie · doi:10.15407/emodel.48.01.051
A comprehensive survey on table question answering: Datasets, methods and future directions
Xu, W., Liu, Y., Lu, L., Li, H., & Sun, G. (2026) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2026.114459
Литвиненко, М. І., Осієвський, С., Ленець, В. Г., Гармаш, Н. В., Орєхов, С. В., & Дрозд, К. А. (2026) · Системи обробки інформації · doi:10.30748/soi.2026.184.06
Developing a Sinhala Speech Recognition System Using Whisper
Thameera, W. S., Ruwanthilaka, S. D. S. S., Shanaz, A. L. F., et al. (2026) · International Conference on Advanced Research in Computing (ICARC) · doi:10.1109/icarc68737.2026.11454008
“Models such as OpenAI’s GPT-4 and Google’s Gemini integrate text, audio, and visual modalities, enabling tasks that range from natural language generation to multimodal reasoning [4].”
RADI: A Retrieval-Augmented Dynamic In-Context Learning Framework for AIGC Image Detection
Bi, T., Ma, R., Huang, Y., Wang, Y., Liu, J., & Zhang, X. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11462412
Token-Anchored Semantic Drift: Untargeted Transferable Attacks Against Commercial MLLMs
Shen, T. (2026) · International Conference on Generative Artificial Intelligence and Information Security (GAIIS) · doi:10.1109/gaiis69281.2026.11519348
“Commercial MLLMs [1]–[3]are becoming the default interface for image understanding and multimodal assistance.”
Taşkırdı, İ., Yiğit, E. Ş., Akarcan, S. E., & Tunçel, T. (2026) · Journal of Clinical Medicine · doi:10.3390/jcm15114025
FSMVision: Semantic Extraction of Finite State Machines from Diagrams via Multimodal AI
Bakchy, S. C., Islam, M. A., Rahman, S., & Rahman, M. T. (2026) · International Symposium on Quality Electronic Design (ISQED) · doi:10.1109/isqed69900.2026.11534790
“The MLLMs have demonstrated significant capabilities to analyze different media like images, videos, and audio [8].”
Harrouchi, L. E., Moussaoui, H., Karmoudi, M., et al. (2026) · Springer proceedings in mathematics & statistics · doi:10.1007/978-3-032-18042-1_16
SuperVoxelGPT: Adaptive and Ordered 3D Tokenization for Autoregressive Shape Generation
Li, Y., Zhang, C., Gao, X., & Guo, X. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37577-3_22
FLIC: A Real-World Dataset for Visual Estimation of Food Leftovers in Canteens
Piccoli, F., Callegaro, D., Marelli, D., et al. (2026) · Applied Sciences · doi:10.3390/app16115465
Hu, C., Chen, Y., Liang, T., Wei, Y., Li, A., Shi, H., Bian, L., & Shi, X. (2026) · International Conference on Multimedia Retrieval · doi:10.1145/3805622.3810757
Wang, H., & Wan, D. (2026) · International Conference on Multimedia Retrieval · doi:10.1145/3805622.3810610
Unified Multimodal AI Engine for Real-Time Summarization of Text, Images, and Documents
Rakshitha, G., Rajamani, T., Bhardwaj, M., et al. (2026) · doi:10.1109/icici68867.2026.11564932
“A study on architectural improvements in multimodal LLMs has shown that there are still problems with hallucinatory output and coherence when visually, audibly, and textually processed together [2].”
Match me if you can: aligning historical posters and newspaper articles via multimodal deep learning
Xu, F., Düring, M., & Leiva, L. A. (2026) · Digital Scholarship in the Humanities · doi:10.1093/llc/fqag088
García-López, I. M., Espinosa, J. M. M., & Soledad, M. (2026) · IEEE Revista Iberoamericana de Tecnologias del Aprendizaje · doi:10.1109/rita.2026.3701046
Wang, X., & Yang, Y. (2026) · International Journal of Pattern Recognition and Artificial Intelligence · doi:10.1142/s0218001426400215
Same, Same But Different? A Cross-Domain Analysis of Chatbot User Interfaces
Rockstroh, J., Spillner, L., Wenig, N., Mildner, T., Zargham, N., & Malaka, R. (2026) · ACM Conference on Conversational User Interfaces (CUI) · doi:10.1145/3816046.3816229
“Interaction features such as Extended Input or Input Adjustment , as well as richer message formats like Image Chat Message [6] or Sticker Chat Message [37], rely on subsymbolic information processing enabled by state-of-the-art Vision Language Models [21, 22].”
Survey on Multi-Modal Large Language Models: Bridging Text, Vision, and Beyond
Bhashkar, K., & Kumar, S. (2025) · IEEE International Conference on Innovate for Humanitarian: Tech Solutions for Global Challenge (ICIH) · doi:10.1109/icih67754.2025.11608965
“The interface between the various modalities is Multimodal fusion module i.e., the language model is able to take and answer problems on mixed multimodal input [6].”
RaTA-Tool: Retrieval-Based Tool Selection with Multimodal Large Language Models
Mattioli, G., Turri, E., Sarto, S., Baraldi, L., Cornia, M., Baraldi, L., & Cucchiara, R. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-31397-3_21
Self-citationShi, S., Yin, R., Xu, N., et al. (2026) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3770855.3817756
Yu, Z., Xing, J., Zhao, L., Lei, D., Chen, L., Han, Y., & Ren, W. (2026) · Research Square · doi:10.21203/rs.3.rs-10211545/v1
Huang, X., Wang, X., Wang, B., et al. (2026) · 2026 5th International Conference on Energy and Electrical Power Systems (ICEEPS) · doi:10.1109/iceeps70377.2026.11662894
“Recent surveys and technical reports on multimodal large models further demonstrate this trend [2], [3].”
Generative AI Technologies, Techniques and Tensions: A Primer
Behrens, J. T. (2026) · Springer eBooks · doi:10.1007/978-3-032-24819-0_2 · Cited by 1 · see citing works
Zhao, B. (2026) · ITM Web of Conferences · doi:10.1051/itmconf/20269103014
“Interestingly, large-scale multimodal corpora heavily depend on machine-assisted data curation; however, human feedback continues to play a significant role in supervising annotation precision and monitoring overall data quality [7, 8, 13, 14].”
- Authors
- Caffagni, D., Cocchi, F., Barsellotti, L., Moratelli, N., Sarto, S., Baraldi, L., Baraldi, L., Cornia, M., & Cucchiara, R.
- Affiliations
- University of Modena and Reggio Emilia (IT)
- University of Pisa (IT)
- ORCID
- Venue
- Findings of the Association for Computational Linguistics ACL 2024
- Published
- 2024
- Pages
- 13590-13618
- OpenAlex
- W4402670262
- Cited by
- 98
- References
- 171
- Citations by year
- 2026: 44 · 2025: 46 · 2024: 8
- FWCI
- 20.26
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Findings of the Association for Computational Linguistics: ACL (Findings of ACL) (publishedVersion, cc-by)
- Funders
- European Commission (PNRR-M4C2, PE00000013)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- multimodal large language models, multimodal language models, vision-language models, large language models, multimodal representation learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
