The DOI matches its Crossref record; 246 of the 246 references it cites were found in a registry or an index.
The Revolution of Multimodal Large Language Models: A Survey
Open accessAbstract
Davide Caffagni, Federico Cocchi, Luca Barsellotti, Nicholas Moratelli, Sara Sarto, Lorenzo Baraldi, Lorenzo Baraldi, Marcella Cornia, Rita Cucchiara. Findings of the Association for Computational Linguistics: ACL 2024. 2024.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“users should be aware that limitations such as introducing hallucination71 and omitting necessary signals73 are inherent to LLMs and may still impact classification performance, though MoMA is fine-tuned using groundtruth clinical labels.”
From MoMA: a mixture-of-multimodal-agents architecture for enhancing clinical prediction modelling (2025) · cites this paper as [73]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 1×
“The introduction of the attention operation and the Transformer architecture (Vaswani et al., 2017) has enabled the creation of models capable of handling various modalities on an increasingly large scale.”
U-Net: Convolutional Networks for Biomedical Image Segmentation
Ronneberger, O., Fischer, P., & Brox, T. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24574-4_28 · Cited by 99,404 · cited 5×
“While GILL refrains from fine-tuning both the LLM and the diffusion U-Net, alternative methodologies fine-tune the language model to expand its multimodal generation capabilities.”
InfluentialBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 1×
“Initially, this breakthrough was lever-aged for language-specific models (Devlin et al., 2018; Brown et al., 2020) but quickly extended to support diverse modalities (Li et al., 2019; Lu et al., 2019) and facilitate their integration within unified embedding spaces (Radford et al., 2021).”
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.11929 · Cited by 47,472
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460 · cited 8×
“COCO (Lin et al., 2014) contains more than 120k images, each associated with five human-generated captions.”
InfluentialLanguage Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 2×
“Initially, this breakthrough was lever-aged for language-specific models (Devlin et al., 2018; Brown et al., 2020) but quickly extended to support diverse modalities (Li et al., 2019; Lu et al., 2019) and facilitate their integration within unified embedding spaces (Radford et al., 2021).”
Learning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293 · cited 1×
“Initially, this breakthrough was lever-aged for language-specific models (Devlin et al., 2018; Brown et al., 2020) but quickly extended to support diverse modalities (Li et al., 2019; Lu et al., 2019) and facilitate their integration within unified embedding spaces (Radford et al., 2021).”
LoRA: Low-Rank Adaptation of Large Language Models
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., & Chen, W. (2021) · arXiv · doi:10.48550/arxiv.2106.09685 · Cited by 13,453 · cited 2×
“This technique is orthogonal to quantization methods such as QLoRA (Dettmers et al., 2024), which further decreases the memory footprint of the LLM compared to the usual half-precision weights.”
Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A. C., Lo, W.-Y., et al. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00371 · Cited by 11,953 · cited 3×
“For output regions, LISA (Lai et al., 2023), GLaMM, GSVA (Xia et al., 2023b), NeXt-Chat (Zhang et al., 2023a), and LISA++ (Yang et al., 2023b) send the embedding corresponding to special tokens to the mask decoder of SAM, LLaVA-G to OpenSeeD (Zhang et al., 2023c), Lenna (Wei et al., 2023) to Grounding-DINO (Liu et al.…”
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · cited 2×
“…used are WebLI (Chen et al., 2023j), composed of 10B images and image-text pairs, and MMC4 (Zhu et al., 2023d), an extension of the text-only C4 (Raffel et al., 2020) dataset composed of 365M documents and 156B tokens relatives to different concepts, and OBELICS (Laurençon et al., 2024), an…”
Emerging Properties in Self-Supervised Vision Transformers
Caron, M., Touvron, H., Misra, I., Jegou, H., Mairal, J., Bojanowski, P., & Joulin, A. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00951 · Cited by 5,638 · cited 5×
“DetGPT (Pi et al., 2023) generates a list of categories for Grounding-DINO.”
InfluentialVisual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations
Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L.-J., Shamma, D. A., et al. (2017) · International Journal of Computer Vision · doi:10.1007/s11263-016-0981-7 · Cited by 5,107 · cited 1×
“Visual Genome (Krishna et al., 2017) connects structured image concepts to language and comprises 108,077 images along with detailed descriptions of all objects present in them, providing 5.4M region descriptions and 1.7M visual question-answer pairs.”
CIDEr: Consensus-based image description evaluation
Vedantam, R., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7299087 · Cited by 4,845 · cited 2×
“As regards the evaluation metric, the CIDEr score (Vedantam et al., 2015), which is the reference metric for the task, is used to compare generated image descriptions with ground-truth captions.”
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602 · cited 1×
“Popular choices for the LLM backbone often fall into the LLaMA family (Touvron et al., 2023a,b), given that their weights are freely accessible, they have been trained on public data solely, and they boast different sizes to accommodate various use cases.”
BackgroundScanNet: Richly-Annotated 3D Reconstructions of Indoor Scenes
Dai, A., Chang, A. X., Savva, M., Halber, M., Funkhouser, T., & Niessner, M. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.261 · Cited by 4,368 · cited 9×
“3B ♢ CLIP ViT-L Q-Former ✻ ✗ Visual Dialogue, VQA, Captioning LLaMA-Adapter V2 (Gao et al., 2023) LLaMA-7B ▲ CLIP ViT-L Linear ✗ VQA, Captioning OpenFlamingo (Awadalla et al., 2023) MPT-7B ⋆ CLIP ViT-L XAttn LLM ✗ VQA, Captioning Flamingo (Alayrac et al., 2022) Chinchilla-70B ⋆ NFNet-F6 XAttn LLM ✗ Visual Dialogue, VQ…”
InfluentialQLoRA: Efficient Finetuning of Quantized LLMs
Dettmers, T., Pagnoni, A., Holtzman, A., & Zettlemoyer, L. (2023) · arXiv · doi:10.48550/arxiv.2305.14314 · Cited by 3,832 · cited 1×
“This technique is orthogonal to quantization methods such as QLoRA (Dettmers et al., 2024), which further decreases the memory footprint of the LLM compared to the usual half-precision weights.”
MethodMasked-attention Mask Transformer for Universal Image Segmentation
Cheng, B., Misra, I., Schwing, A. G., Kirillov, A., & Girdhar, R. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00135 · Cited by 3,565 · cited 1×
“GELLA (Qi et al., 2024) presents a fusion module in Mask2Former (Cheng et al., 2022) to propose masks based on multi-modal image features and an association module to assign latent embeddings to them.”
Training Language Models to Follow Instructions with Human Feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2011 · Cited by 3,117 · cited 2×
“This technique, called instruction-tuning (Chung et al., 2022; Wang et al., 2022b,a; Jiang et al., 2024), turns out to be critical for aligning the behavior of an LLM with that of humans and currently empowers the most advanced LLMs, eventually boosted via reinforcement learning from human feedback (RLHF) (Ouyang et a…”
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-T., Rocktäschel, T., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.11401 · Cited by 3,101 · cited 1×
“While retrieval-augmented generation (RAG) is a consolidated technique in LLMs (Lewis et al., 2020; Asai et al., 2023), its application in MLLMs is still under-explored.”
The Power of Scale for Parameter-Efficient Prompt Tuning
Lester, B., Al-Rfou, R., & Constant, N. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.243 · Cited by 3,096 · cited 1×
“Among these, prompt-tuning (Hambardzumyan et al., 2021; Lester et al., 2021; Li and Liang, 2021; Liu et al., 2023j) learns a small set of vectors to be fed to the model as soft prompts before the input text.”
Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question AnsweringPreprint
Lu, P., Mishra, S., Xia, T., Qiu, L., Chang, K.-W., Zhu, S.-C., Tafjord, O., Clark, P., & Kalyan, A. (2022) · Neural Information Processing Systems · doi:10.48550/arxiv.2209.09513 · Cited by 2,884 · cited 1×
“Several studies (Liu et al., 2023b; Zhu et al., 2023a) show that MLLMs tend to exhibit high hallucination rates, especially when generating longer captions.”
Neural Discrete Representation Learning
Oord, A. V. D., Vinyals, O., & Kavukcuoglu, K. (2017) · arXiv · doi:10.48550/arxiv.1711.00937 · Cited by 2,729 · cited 1×
“PaLI-3 (Chen et al., 2023i) converts embeddings into segmentation masks through a VQ-VAE (Van Den Oord et al., 2017) decoder.”
Llama 2: Open Foundation and Fine-Tuned Chat Models
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.09288 · Cited by 2,604
PaLM: Scaling Language Modeling with Pathways
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.02311 · Cited by 2,590 · cited 1×
“Brown et al. (2020) discovered that in-context learning, i.e. , prepending the prompt with a few examples demonstrating the desired output of an LLM (Chowdhery et al., 2023; Hoffmann et al., 2022; Tay et al., 2022), improves its performance, especially over unseen tasks.”
Taming Transformers for High-Resolution Image Synthesis
Esser, P., Rombach, R., & Ommer, B. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01268 · Cited by 2,514 · cited 1×
“BuboGPT (Zhao et al., 2023c) leverages a combination of RAM, Grounding-DINO, and SAM and matches tags with nouns in the output sequence.”
Prefix-Tuning: Optimizing Continuous Prompts for Generation
Li, X. L., & Liang, P. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.353 · Cited by 2,473 · cited 1×
“Among these, prompt-tuning (Hambardzumyan et al., 2021; Lester et al., 2021; Li and Liang, 2021; Liu et al., 2023j) learns a small set of vectors to be fed to the model as soft prompts before the input text.”
Young, P., Lai, A., Hodosh, M., & Hockenmaier, J. (2014) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00166 · Cited by 2,439 · cited 1×
“Flickr30k (Young et al., 2014) comprises 31,783 images, depicting diverse everyday activities, events, and scenes.”
DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation
Ruiz, N., Li, Y., Jampani, V., Pritch, Y., Rubinstein, M., & Aberman, K. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.02155 · Cited by 2,295 · cited 1×
“DreamBench (Ruiz et al., 2023) is a benchmark that evaluates the generative capabilities of the models on subject-driven generation.”
ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks
Lu, J., Batra, D., Parikh, D., & Lee, S. (2019) · arXiv · doi:10.48550/arxiv.1908.02265 · Cited by 2,109 · cited 1×
“Initially, this breakthrough was lever-aged for language-specific models (Devlin et al., 2018; Brown et al., 2020) but quickly extended to support diverse modalities (Li et al., 2019; Lu et al., 2019) and facilitate their integration within unified embedding spaces (Radford et al., 2021).”
Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning
Sharma, P., Ding, N., Goodman, S., & Soricut, R. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1238 · Cited by 1,879 · cited 1×
“For instance, the Conceptual Captions 3M (CC3M) dataset (Sharma et al., 2018) is composed of 3M images paired with textual captions specifically designed for image captioning systems.”
GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answering
Hudson, D. A., & Manning, C. D. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00686 · Cited by 1,797 · cited 1×
“IconQA (Lu et al., 2021) tests the visual reasoning abilities of vision-and-language models on three types of questions: multiple-image-choice, multiple-text-choice, and fill-in-the-blank.”
Improved Baselines with Visual Instruction Tuning
Liu, H., Li, C., Li, Y., & Lee, Y. J. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.02484 · Cited by 1,748 · cited 3×
“Some works (Liu et al., 2023d; You et al., 2023; Chen et al., 2023j) have demonstrated that adopting visual backbones trained on higher resolutions leads to fewer hallucinations and improved multimodal understanding abilities, translating into better performance over tasks that require fine-grained…”
Improved Baselines with Visual Instruction Tuning
Liu, H., Li, C., Li, Y., & Lee, Y. J. (2023) · arXiv · doi:10.48550/arxiv.2310.03744 · Cited by 1,733
Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object DetectionPreprint
Liu, S., Zeng, Z., Ren, T., Li, F., Zhang, H., Yang, J., Jiang, Q., Li, C., Yang, J., Su, H., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2303.05499 · Cited by 1,687 · cited 4×
“For output regions, LISA (Lai et al., 2023), GLaMM, GSVA (Xia et al., 2023b), NeXt-Chat (Zhang et al., 2023a), and LISA++ (Yang et al., 2023b) send the embedding corresponding to special tokens to the mask decoder of SAM, LLaVA-G to OpenSeeD (Zhang et al., 2023c), Lenna (Wei et al., 2023) to Grounding-DINO (Liu et al.…”
InfluentialmT5: A Massively Multilingual Pre-trained Text-to-Text Transformer
Xue, L., Constant, N., Roberts, A., Kale, M., Al-Rfou, R., Siddhant, A., Barua, A., & Raffel, C. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.41 · Cited by 1,629 · cited 1×
“…are OPT (Zhang et al., 2022b), Mag-neto (Wang et al., 2023b), MPT (MosaicML, 2023), and the instruction-tuned (Chung et al., 2022) or multilingual (Xue et al., 2020) flavors of T5 (Raffel et al., 2020), an encoder-decoder language model pre-trained for multiple tasks. et al., 2024; Zhao et al.,…”
Mamba: Linear-Time Sequence Modeling with Selective State Spaces
Gu, A., & Dao, T. (2023) · arXiv · doi:10.48550/arxiv.2312.00752 · Cited by 1,579
Instructpix2pix: Learning to follow image editing instructions
Brooks, T., Holynski, A., & Efros, A. A. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01764 · Cited by 1,466
Scaling Instruction-Finetuned Language Models
Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, Y., Wang, X., Dehghani, M., Brahma, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2210.11416 · Cited by 1,441 · cited 2×
“This technique, called instruction-tuning (Chung et al., 2022; Wang et al., 2022b,a; Jiang et al., 2024), turns out to be critical for aligning the behavior of an LLM with that of humans and currently empowers the most advanced LLMs, eventually boosted via reinforcement learning from human feedback…”
InstructBLIP: Towards General-purpose Vision-Language Models with Instruction TuningPreprint
Dai, W., Li, J., Li, D., Tiong, A. M. H., Zhao, J., Wang, W., Li, B., Fung, P., & Hoi, S. C. H. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.06500 · Cited by 1,372 · cited 4×
“Another approach, as demonstrated in Instruct-BLIP (Dai et al., 2023), involves the freezing of the visual encoder and LLM.”
InfluentialVideo-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language ModelsPreprint
Maaz, M., Rasheed, H., Khan, S. H., et al. (2023) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2306.05424 · Cited by 1,339
Grounded Language-Image Pre-training
Li, L. H., Zhang, P., Zhang, H., Yang, J., Li, C., Zhong, Y., Wang, L., Yuan, L., Zhang, L., Hwang, J., et al. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01069 · Cited by 1,336 · cited 1×
“Differently, Kosmos-2 (Peng et al., 2023) and Ferret (You et al., 2023) exploit an open-vocabulary detector (Li et al., 2022) to ground noun chunks parsed from cap-tions and then reconstruct referring expressions.”
mT5: A massively multilingual pre-trained text-to-text transformerPreprint
Xue, L., Constant, N., Roberts, A., Kale, M., Al‐Rfou, R., Siddhant, A., Barua, A., & Raffel, C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.11934 · Cited by 1,324
mPLUG-Owl: Modularization Empowers Large Language Models with MultimodalityPreprint
Ye, Q.-H., Xu, H., Xu, G., Ye, J., Yan, M., Zhou, Y., Wang, J., Hu, A., Shi, P.-C., Shi, Y., et al. (2023) · arXiv.org · doi:10.48550/arxiv.2304.14178 · Cited by 1,318 · cited 4×
“…VQA, Captioning PaLI (Chen et al., 2023j) mT5-XXL-13B MiniGPT-4 (Zhu et al., 2023a) Vicuna-13B ⋆ EVA ViT-g Linear ✓ VQA, Captioning mPLUG-Owl (Ye et al., 2023c) LLaMA-7B ▲ CLIP ViT-L Q-Former ✻ ✓ Visual Dialogue, VQA InstructBLIP (Dai et al., 2023) Vicuna-13B ⋆ EVA ViT-g Q-Former ✓ Visual…”
InfluentialGeneration and Comprehension of Unambiguous Object Descriptions
Mao, J., Huang, J., Toshev, A., Camburu, O., Yuille, A., & Murphy, K. P. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.9 · Cited by 1,281 · cited 1×
“Specifically, Shikra (Chen et al., 2023f), DetGPT (Pi et al., 2023), ChatSpot (Zhao et al., 2023b), and PVIT (Chen et al., 2023a) leverage LLMs (Achiam et al., 2023; OpenAI, 2022) to combine regions and captions from datasets that present both annotations ( e.g. , COCO).”
BackgroundLVIS: A Dataset for Large Vocabulary Instance Segmentation
Gupta, A., Dollar, P., & Girshick, R. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00550 · Cited by 1,274 · cited 1×
“MLLMs are evaluated across different benchmarks, taking into account both more classic visual comprehension and recognition skills and advanced multimodal conversation capabilities.”
Self-Instruct: Aligning Language Models with Self-Generated Instructions
Wang, Y., Kordi, Y., Mishra, S., Liu, A., Smith, N. A., Khashabi, D., & Hajishirzi, H. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.754 · Cited by 1,243
VisualBERT: A Simple and Performant Baseline for Vision and Language
Li, L. H., Yatskar, M., Yin, D., Hsieh, C.-J., & Chang, K.-W. (2019) · arXiv · doi:10.48550/arxiv.1908.03557 · Cited by 1,198 · cited 1×
“Initially, this breakthrough was lever-aged for language-specific models (Devlin et al., 2018; Brown et al., 2020) but quickly extended to support diverse modalities (Li et al., 2019; Lu et al., 2019) and facilitate their integration within unified embedding spaces (Radford et al., 2021).”
DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection
Zhang, H., Li, F., Liu, S., Zhang, L., Su, H., Zhu, J., Ni, L. M., & Shum, H.-Y. (2022) · arXiv · Cited by 1,154
Shikra: Unleashing Multimodal LLM's Referential Dialogue MagicPreprint
Chen, K., Zhang, Z., Zeng, W., Zhang, R., Zhu, F., & Zhao, R. (2023) · arXiv.org · doi:10.48550/arxiv.2306.15195 · Cited by 1,091
ReferItGame: Referring to Objects in Photographs of Natural Scenes
Kazemzadeh, S., Ordóñez, V., Matten, M., & Berg, T. L. (2014) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1086 · Cited by 1,082 · cited 1×
“Differently, Kosmos-2 (Peng et al., 2023) and Ferret (You et al., 2023) exploit an open-vocabulary detector (Li et al., 2022) to ground noun chunks parsed from cap-tions and then reconstruct referring expressions.”
(2016) · doi:10.1007/978-3-319-46475-6_5 · Cited by 1,081 · cited 1×
“Instead, for image editing tasks, additional datasets like the one introduced in InstructPix2Pix (Brooks et al., 2023) and MagicBrush (Zhang et al., 2023e) are typically used.”
Deep Visual-Semantic Alignments for Generating Image Descriptions
Karpathy, A., & Fei-Fei, L. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2598339 · Cited by 1,048 · cited 1×
“For captioning tasks, the splits defined by Karpathy and Fei-Fei (2015) are typically employed, with 113k, 5k, and 5k images respectively for train, validation and test.”
LAION-5B: An open large-scale dataset for training next generation image-text models
Schuhmann, C., Beaumont, R., Vencu, R., Gordon, C., Wightman, R., Cherti, M., Coombes, T., Katta, A., Mullis, C., Wortsman, M., et al. (2022) · arXiv · doi:10.48550/arxiv.2210.08402 · Cited by 1,022 · cited 1×
“Given the amount of training data currently available, approaches like SPHINX-X (Gao et al., 2024) opt to perform a single all-in-one training stage in which to update all model components, possibly also using text-only data to preserve the conversational capabilities of the LLM.”
Gemini: A Family of Highly Capable Multimodal Models
Gemini Team, Anil, R., Borgeaud, S., Alayrac, J.-B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A. M., Hauth, A., Millican, K., et al. (2023) · arXiv · doi:10.48550/arxiv.2312.11805 · Cited by 990
MM-Vet: Evaluating Large Multimodal Models for Integrated CapabilitiesPreprint
Yu, W., Yang, Z., Li, L., Wang, J., Lin, K., Liu, Z., Wang, X., & Wang, L. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2308.02490 · Cited by 984
Flamingo: a visual language model for few-shot learning
Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1723 · Cited by 966 · cited 6×
“This approach has been proposed in Flamingo (Alayrac et al., 2022) with the integration of dense cross-attention blocks among the existing pre-trained layers of the LLM.”
InfluentialTowards VQA Models That Can Read
Singh, A., Natarajan, V., Shah, M., Jiang, Y. L., Chen, X., Batra, D., Parikh, D., & Rohrbach, M. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00851 · Cited by 951 · cited 1×
“TextVQA (VQA T ) (Singh et al., 2019) is a dataset based on pictures from Open Images (Kuznetsova et al., 2020) Comprehensively describing the visual input is another important skill desired in MLLMs.”
Training Compute-Optimal Large Language Models
Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. D. L., Hendricks, L. A., Welbl, J., Clark, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.15556 · Cited by 929
Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video UnderstandingPreprint
Zhang, H., Li, X. S., & Bing, L. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2306.02858 · Cited by 924
Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modeling
Yu, X., Tang, L., Rao, Y., Huang, T., Zhou, J., & Lu, J. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01871 · Cited by 922
Li, J., Li, D., Savarese, S., & Hoi, S. (2023) · arXiv · doi:10.48550/arxiv.2301.12597 · Cited by 908
SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension
Li, B., Wang, R., Wang, G., Ge, Y., Ge, Y., & Shan, Y. (2023) · arXiv · doi:10.48550/arxiv.2307.16125 · Cited by 882
ImageBind One Embedding Space to Bind Them All
Girdhar, R., El-Nouby, A., Liu, Z., Singh, M., Alwala, K. V., Joulin, A., & Misra, I. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01457 · Cited by 863
OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge
Marino, K., Rastegari, M., Farhadi, A., & Mottaghi, R. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00331 · Cited by 845 · cited 1×
“There are 107,439 questions, 20% of which makes up for the test split.”
Visual7W: Grounded Question Answering in Images
Zhu, Y., Groth, O., Bernstein, M., & Fei-Fei, L. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.540 · Cited by 794 · cited 2×
“Instead, for image editing tasks, additional datasets like the one introduced in InstructPix2Pix (Brooks et al., 2023) and MagicBrush (Zhang et al., 2023e) are typically used.”
BackgroundMethodDataComp: In search of the next generation of multimodal datasetsPreprint
Gadre, S., Ilharco, G., Fang, A., et al. (2023) · Neural Information Processing Systems · doi:10.48550/arxiv.2304.14108 · Cited by 781 · cited 1×
“Furthermore, DataComp (Gadre et al., 2023) presents an extensive pool of 12.8B filtered image-text pairs sourced from common crawl.”
MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningPreprint
Chen, J., Zhu, D., Shen, X., Li, X. Y., Liu, Z., Zhang, P., Krishnamoorthi, R., Chandra, V., Xiong, Y., & Elhoseiny, M. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.09478 · Cited by 754
VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric TasksPreprint
Wang, W., Chen, Z., Chen, X., et al. (2023) · Neural Information Processing Systems · doi:10.48550/arxiv.2305.11175 · Cited by 728 · cited 1×
“VisionLLM (Wang et al., 2023e), VistaLLM (Pramanick et al., 2023), LLaFS (Zhu et al., 2023c), and ChatSpot (Zhao et al., 2023b) allow the MLLM to handle polygons by representing them as a series of points.”
LLaMA-VID: An Image is Worth 2 Tokens in Large Language ModelsPreprint
Li, Y., Wang, C.-Y., & Jia, J. (2023) · European Conference on Computer Vision · doi:10.48550/arxiv.2311.17043 · Cited by 684
OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language ModelsPreprint
Awadalla, A., Gao, I., Gardner, J., Hessel, J., Hanafy, Y., Zhu, W., Marathe, K., Bitton, Y., Gadre, S., Sagawa, S., et al. (2023) · arXiv.org · doi:10.48550/arxiv.2308.01390 · Cited by 674
LLaMA-Adapter V2: Parameter-Efficient Visual Instruction ModelPreprint
Gao, P., Han, J., Zhang, R., Lin, Z., Geng, S., Zhou, A., Zhang, W., Lu, P., He, C., Yue, X., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2304.15010 · Cited by 656
VizWiz Grand Challenge: Answering Visual Questions from Blind People
Gurari, D., Li, Q., Stangl, A. J., Guo, A., Lin, C., Grauman, K., Luo, J., & Bigham, J. P. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00380 · Cited by 621 · cited 1×
“While retrieval-augmented generation (RAG) is a consolidated technique in LLMs (Lewis et al., 2020; Asai et al., 2023), its application in MLLMs is still under-explored.”
MM-REACT: Prompting ChatGPT for Multimodal Reasoning and ActionPreprint
Yang, Z., Li, L., Wang, J., Lin, K., Azarnasab, E., Ahmed, F., Liu, Z., Liu, C., Zeng, M., & Wang, L. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2303.11381 · Cited by 597
PaLM-E: An Embodied Multimodal Language Model
Driess, D., Xia, F., Sajjadi, M. S. M., Lynch, C., Chowdhery, A., Ichter, B., Wahid, A., Tompson, J., Vuong, Q., Yu, T., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.03378 · Cited by 591 · cited 1×
“…designed for document analysis and text-intensive visual inputs (Lv et al., 2023; Ye et al., 2023a), those proposed for embodied AI and robotics (Driess et al., 2023; Mu et al., 2023), and those tailored for specific domains such as medicine (Li et al., 2023d) and autonomous driving (Xu et al.,…”
LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day
Li, C., Wong, C., Zhang, S., Usuyama, N., Liu, H., Yang, J., Naumann, T., Poon, H., & Gao, J. (2023) · arXiv · doi:10.48550/arxiv.2306.00890 · Cited by 590 · cited 1×
“Similarly, in (Xia et al., 2023a) the LLM is trained to output detailed language-based generation prompts which are employed for generation or editing tasks.”
VQA: Visual Question Answering: www.visualqa.org
Agrawal, A., Lu, J., Antol, S., Mitchell, M., Zitnick, C. L., Parikh, D., & Batra, D. (2017) · International Journal of Computer Vision · doi:10.1007/s11263-016-0966-6 · Cited by 570 · cited 9×
“3B ♢ CLIP ViT-L Q-Former ✻ ✗ Visual Dialogue, VQA, Captioning LLaMA-Adapter V2 (Gao et al., 2023) LLaMA-7B ▲ CLIP ViT-L Linear ✗ VQA, Captioning OpenFlamingo (Awadalla et al., 2023) MPT-7B ⋆ CLIP ViT-L XAttn LLM ✗ VQA, Captioning Flamingo (Alayrac et al., 2022) Chinchilla-70B ⋆ NFNet-F6 XAttn LLM ✗ Visual Dialogue, VQ…”
InfluentialKirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A. C., Lo, W.-Y., et al. (2023) · arXiv · doi:10.48550/arxiv.2304.02643 · Cited by 527
Otter: A Multi-Modal Model With In-Context Instruction Tuning
Li, B., Zhang, Y., Chen, L., Wang, J., Pu, F., Cahyono, J. A., Yang, J., Li, C., & Liu, Z. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3571946 · Cited by 522 · cited 2×
“However, it may lead to performance degradation in other tasks, indicating a degree of forgetting and damage to the general visual representation.”
BackgroundMethodMaking the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering
Goyal, Y., Khot, T., Agrawal, A., Summers-Stay, D., Batra, D., & Parikh, D. (2019) · International Journal of Computer Vision · doi:10.1007/s11263-018-1116-0 · Cited by 487 · cited 2×
“This ability is quantitatively evaluated across several visual question-answering datasets, measuring the accuracy (Antol et al., 2015) of the answers provided by the MLLM. VQAv2 (Goyal et al., 2017) is an extended and balanced version of VQA (Antol et al., 2015) built by collecting similar images for the same questio…”
MMBench: Is Your Multi-modal Model an All-around Player?
Liu, Y., Duan, H., Zhang, Y., Li, B., Zhang, S., Zhao, W., Yuan, Y., Wang, J., He, C., Liu, Z., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.06281 · Cited by 487
MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models
Zhu, D., Chen, J., Shen, X., Li, X., & Elhoseiny, M. (2023) · arXiv · doi:10.48550/arxiv.2304.10592 · Cited by 469
LISA: Reasoning Segmentation via Large Language Model
Lai, X., Tian, Z., Chen, Y., Li, Y., Yuan, Y., Liu, S., & Jia, J. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.00915 · Cited by 466
SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis
Podell, D., English, Z., Lacey, K., Blattmann, A., Dockhorn, T., Müller, J., Penna, J., & Rombach, R. (2023) · arXiv · doi:10.48550/arxiv.2307.01952 · Cited by 455
VILA: On Pre-training for Visual Language Models
Lin, J., Yin, H., Ping, W., Molchanov, P. A., Shoeybi, M., & Han, S. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02520 · Cited by 448 · cited 2×
“Unified-IO2 (Lu et al., 2023a) uses VQ-GAN decoders for both image and audio modalities to decode output discrete tokens and can generate surface normals, depth, and segmentation masks for the input images.”
PointLLM: Empowering Large Language Models to Understand Point CloudsPreprint
Xu, R.-S., Wang, X., Wang, T., et al. (2023) · European Conference on Computer Vision · doi:10.48550/arxiv.2308.16911 · Cited by 448
MMMU: A Massive Multi-Discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI
Yue, X., Ni, Y., Zheng, T., Zhang, K., Liu, R., Zhang, G., Stevens, S., Jiang, D., Ren, W., Sun, Y., et al. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.00913 · Cited by 429 · cited 1×
“Similarly, MiniGPT-5 (Zheng et al., 2023) includes the reconstruction loss of diffusion models in addition to the alignment loss of GILL.”
Robust fine-tuning of zero-shot models
Wortsman, M., Ilharco, G., Kim, J. W., Li, M., Kornblith, S., Roelofs, R., Lopes, R. G., Hajishirzi, H., Farhadi, A., Namkoong, H., et al. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00780 · Cited by 428 · cited 2×
“Popular choices are the ViT-L model from CLIP (Rad-ford et al., 2021), the ViT-H backbone from Open-CLIP (Wortsman et al., 2022), and the ViT-g version from EVA-CLIP (Fang et al., 2023).”
Methodnocaps: novel object captioning at scale
Agrawal, H., Desai, K., Wang, Y., Chen, X., Jain, R., Johnson, M., Batra, D., Parikh, D., Lee, S., & Anderson, P. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00904 · Cited by 419 · cited 1×
“Complementing these images are 158,915 captions, obtained through crowd-sourcing techniques. nocaps (Agrawal et al., 2019) represents a benchmark for novel object captioning, boasting an extensive collection of almost 400 novel object categories compared to the COCO dataset.”
Generating Images with Multimodal Language Models
Koh, J. Y., Fried, D., & Salakhutdinov, R. (2023) · Neural Information Processing Systems · doi:10.48550/arxiv.2305.17216 · Cited by 416 · cited 4×
“While GILL refrains from fine-tuning both the LLM and the diffusion U-Net, alternative methodologies fine-tune the language model to expand its multimodal generation capabilities.”
InfluentialPerceiver: General Perception with Iterative Attention
Jaegle, A., Gimeno, F., Brock, A., Zisserman, A., Vinyals, O., & Carreira, J. (2021) · arXiv · doi:10.48550/arxiv.2103.03206 · Cited by 414 · cited 2×
“To reduce computational complexity, this strategy is usually paired with a Perceiver-based component (Jaegle et al., 2021) that reduces the number of visual tokens before they are fed to the LLM.”
BackgroundMethodGPT4RoI: Instruction Tuning Large Language Model on Region-of-InterestPreprint
Zhang, S., Sun, P., Chen, S., Xiao, M., Shao, W., Zhang, W., Liu, Y., Chen, K., & Luo, P. (2023) · ECCV Workshops · doi:10.48550/arxiv.2307.03601 · Cited by 390 · cited 1×
“However, recently there has been an interest in addressing more fine-grained visual tasks, such as visual grounding and image generation.”
BackgroundPandaGPT: One Model To Instruction-Follow Them AllPreprint
Su, Y., Lan, T., Li, H., Xu, J., Wang, Y., & Cai, D. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.16355 · Cited by 379 · cited 2×
“ChatterBox (Tian et al., 2024b) combines features from the iTPN-B encoder (Tian et al., 2023) and the MLLM and provides them to the DINO detector (Zhang et al., 2022a).”
Ferret: Refer and Ground Anything Anywhere at Any GranularityPreprint
You, H., Zhang, H., Gan, Z., Du, X., Zhang, B., Wang, Z., Cao, L., Chang, S., & Yang, Y. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.07704 · Cited by 369 · cited 8×
“Differently, Kosmos-2 (Peng et al., 2023) and Ferret (You et al., 2023) exploit an open-vocabulary detector (Li et al., 2022) to ground noun chunks parsed from cap-tions and then reconstruct referring expressions.”
InfluentialMultimodal Large Language Models: A Survey
Wu, J., Gan, W., Chen, Z., Wan, S., & Yu, P. S. (2023) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata59044.2023.10386743 · Cited by 369 · cited 1×
“In comparison with existing surveys on the topic (Yin et al., 2023a; Wu et al., 2023b; Huang et al., 2023a), our paper exhibits substantial differences.”
UL2: Unifying Language Learning ParadigmsPreprint
Tay, Y., Dehghani, M., Tran, V. Q., Garcia, X., Wei, J., Wang, X., Chung, H. W., Shakeri, S., Bahri, D., Schuster, T., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2205.05131 · Cited by 364 · cited 1×
“Brown et al. (2020) discovered that in-context learning, i.e. , prepending the prompt with a few examples demonstrating the desired output of an LLM (Chowdhery et al., 2023; Hoffmann et al., 2022; Tay et al., 2022), improves its performance, especially over unseen tasks.”
LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs
Schuhmann, C., Vencu, R., Beaumont, R., Kaczmarczyk, R., Mullis, C., Katta, A., Coombes, T., Jitsev, J., & Komatsuzaki, A. (2021) · arXiv · doi:10.48550/arxiv.2111.02114 · Cited by 358 · cited 1×
“Given the amount of training data currently available, approaches like SPHINX-X (Gao et al., 2024) opt to perform a single all-in-one training stage in which to update all model components, possibly also using text-only data to preserve the conversational capabilities of the LLM.”
GPT4Tools: Teaching Large Language Model to Use Tools via Self-instructionPreprint
Yang, R., Song, L., Li, Y., et al. (2023) · Neural Information Processing Systems · doi:10.48550/arxiv.2305.18752 · Cited by 340 · cited 3×
“To reduce the input sequence length to the LLM, and thus the computational cost, MiniGPT-v2 (Chen et al., 2023e) and VILA (Lin et al., 2023a) propose to project multiple visual tokens together into the same token within the embedding space of the LLM.”
Bai, J., Bai, S., Chu, Y., Cui, Z., Dang, K., Deng, X., Fan, Y., Ge, W., Han, Y., Huang, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2309.16609 · Cited by 332
GLaMM: Pixel Grounding Large Multimodal Model
Rasheed, H., Maaz, M., Shaji, S., Shaker, A., Khan, S., Cholakkal, H., Anwer, R. M., Xing, E. P., Yang, M., & Khan, F. S. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01236 · Cited by 302 · cited 4×
“For output regions, LISA (Lai et al., 2023), GLaMM, GSVA (Xia et al., 2023b), NeXt-Chat (Zhang et al., 2023a), and LISA++ (Yang et al., 2023b) send the embedding corresponding to special tokens to the mask decoder of SAM, LLaVA-G to OpenSeeD (Zhang et al., 2023c), Lenna (Wei et al., 2023) to Grounding-DINO (Liu et al.…”
InfluentialPrefix-Tuning: Optimizing Continuous Prompts for Generation
Li, X. L., & Liang, P. (2021) · arXiv · doi:10.48550/arxiv.2101.00190 · Cited by 289
TextCaps: A Dataset for Image Captioning with Reading Comprehension
Sidorov, O., Hu, R., Rohrbach, M., & Singh, A. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58536-5_44 · Cited by 283 · cited 1×
“TextCaps (Sidorov et al., 2020) includes 145k cap-tions aligned with 28k images.”
Monkey: Image Resolution and Text Label are Important Things for Large Multi-Modal Models
Li, Z., Yang, B., Liu, Q., Ma, Z., Zhang, S., Yang, J., Sun, Y., Liu, Y., & Bai, X. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02527 · Cited by 283 · cited 2×
“Monkey (Li et al., 2023l) uses a Perceiver-like resampler to extract fixed-length sequences from each sub-image and trains on an image-text dataset curated by several vision expert models integrated by ChatGPT.”
OCR-VQA: Visual Question Answering by Reading Text in Images
Mishra, A., Shekhar, S., Singh, A. K., & Chakraborty, A. (2019) · IAPR International Conference on Document Analysis and Recognition (ICDAR) · doi:10.1109/icdar.2019.00156 · Cited by 281
AnyGPT: Unified Multimodal LLM with Discrete Sequence Modeling
Zhan, J., Dai, J., Ye, J., et al. (2024) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2402.12226 · Cited by 281 · cited 1×
“AnyGPT (Zhan et al., 2024) interprets all the continuous non-text modalities as discrete tokens in both input and output, using, respectively, multi-modal tokenizers and de-tokenizers.”
LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image UnderstandingPreprint
Zhang, Y., Zhang, R., Gu, J., Zhou, Y., Lipka, N., Yang, D., & Sun, T. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2306.17107 · Cited by 278 · cited 1×
“Given the amount of training data currently available, approaches like SPHINX-X (Gao et al., 2024) opt to perform a single all-in-one training stage in which to update all model components, possibly also using text-only data to preserve the conversational capabilities of the LLM.”
MethodTimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understanding
Ren, S., Yao, L., Li, S., Sun, X., & Hou, L. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01357 · Cited by 277 · cited 1×
“These models process video frames independently, using CLIP-based backbones to extract frame-level features which are then combined us-ing pooling mechanisms (Li et al., 2023j; Maaz et al., 2023) or Q-Former based solutions (Li et al., 2023h; Ren et al., 2023a).”
Lin, Z., Liu, C., Zhang, R., Gao, P., Qiu, L., Han, X., Qiu, H., Lin, C., Shao, W., Chen, K., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2311.07575 · Cited by 275
LLaVA-Plus: Learning to Use Tools for Creating Multimodal Agents
Liu, S.-L., Cheng, H., Liu, H., et al. (2023) · European Conference on Computer Vision · doi:10.48550/arxiv.2311.05437 · Cited by 266 · cited 1×
“An additional training step is usually performed on an instruction-tuning dataset, such as LLaVa-Instruct (Liu et al., 2023h), to preserve the conversational capabilities of the MLLM.”
Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions?
Chen, Y., Hu, H., Luan, Y., et al. (2023) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2302.11713 · Cited by 266 · cited 1×
“We believe that the emergence of VQA datasets that require external retrieved knowledge (Chen et al., 2023k; Mensink et al., 2023) may enable the development of MLLMs with RAG capabilities (Hu et al., 2023b; Caffagni et al., 2024).”
Huang, T.-H., Ferraro, F., Mostafazadeh, N., Misra, I., Agrawal, A., Devlin, J., Girshick, R., He, X., Kohli, P., Batra, D., et al. (2016) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n16-1147 · Cited by 254 · cited 2×
“VIST (Huang et al., 2016) is specifically curated for the task of interleaved image-text generation.”
Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks
Wang, Y., Mishra, S., Alipoormolabashi, P., Kordi, Y., Mirzaei, A., Naik, A., Ashok, A., Dhanasekaran, A. S., Arunkumar, A., Stap, D., et al. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.340 · Cited by 247 · cited 1×
“This technique, called instruction-tuning (Chung et al., 2022; Wang et al., 2022b,a; Jiang et al., 2024), turns out to be critical for aligning the behavior of an LLM with that of humans and currently empowers the most advanced LLMs, eventually boosted via reinforcement learning from human feedback…”
WARP: Word-level Adversarial ReProgramming
Hambardzumyan, K., Khachatrian, H., & May, J. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.381 · Cited by 243 · cited 1×
“Among these, prompt-tuning (Hambardzumyan et al., 2021; Lester et al., 2021; Li and Liang, 2021; Liu et al., 2023j) learns a small set of vectors to be fed to the model as soft prompts before the input text.”
MobileVLM V2: Faster and Stronger Baseline for Vision Language ModelPreprint
Chu, X., Qiao, L., Zhang, X., Xu, S., Wei, F., Yang, Y., Sun, X., Hu, Y.-M., Lin, X., Zhang, B., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2402.03766 · Cited by 239 · cited 2×
“Effective strategies (Chu et al., 2024) are needed to reduce computational requirements and enable more accessible development of MLLMs.”
PaLI: A Jointly-Scaled Multilingual Language-Image ModelPreprint
Chen, X., Wang, X., Changpinyo, S., Piergiovanni, A., Padlewski, P., Salz, D., Goodman, S., Grycner, A., Mustafa, B., Beyer, L., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2209.06794 · Cited by 235 · cited 2×
“Given the amount of training data currently available, approaches like SPHINX-X (Gao et al., 2024) opt to perform a single all-in-one training stage in which to update all model components, possibly also using text-only data to preserve the conversational capabilities of the LLM.”
MethodLanguage Is Not All You Need: Aligning Perception with Language Models
Huang, S., Dong, L., Wang, W., Hao, Y., Singhal, S., Ma, S., Lv, T., Cui, L., Mohammed, O. K., Patra, B., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.14045 · Cited by 235
BLIVA: A Simple Multimodal LLM for Better Handling of Text-Rich Visual Questions
Hu, W., Xu, Y., Li, Y., Li, W., Chen, Z., & Tu, Z. (2023) · AAAI Conference on Artificial Intelligence · doi:10.48550/arxiv.2308.09936 · Cited by 233 · cited 1×
“However, employing a frozen visual encoder has some limitations, primarily due to the constrained number of parameters, resulting in an inadequate alignment between the visual and language modalities.”
BackgroundSafe Latent Diffusion: Mitigating Inappropriate Degeneration in Diffusion Models
Schramowski, P., Brack, M., Deiseroth, B., & Kersting, K. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.02157 · Cited by 219
CogVLM: Visual Expert for Pretrained Language Models
Wang, W., Lv, Q., Yu, W., Hong, W., Qi, J., Wang, Y., Ji, J., Yang, Z., Zhao, L., Song, X., et al. (2023) · arXiv · doi:10.48550/arxiv.2311.03079 · Cited by 217
Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models
Wu, C., Yin, S., Qi, W., Wang, X., Tang, Z., & Duan, N. (2023) · arXiv · doi:10.48550/arxiv.2303.04671 · Cited by 197 · cited 5×
“To facilitate the evaluation, many works propose to leverage state-of-the-art proprietary models ( e.g. , ChatGPT (Ope-nAI, 2022), GPT-4 (Achiam et al., 2023)) to automatically judge candidate answers.”
InfluentialUnified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Action
Lu, J., Clark, C., Lee, S.-H., Zhang, Z., Khosla, S., Marten, R., Hoiem, D., & Kembhavi, A. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02497 · Cited by 195
Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond
Bai, J., Bai, S., Yang, S., Wang, S., Tan, S., Wang, P., Lin, J., Zhou, C., & Zhou, J. (2023) · arXiv · doi:10.48550/arxiv.2308.12966 · Cited by 193 · cited 8×
“For the same reason, mPLUG-Owl2 (Ye et al., 2023d) and Qwen-VL (Bai et al., 2023b) compress the visual features into fixed-length sequences, independent of the resolution, using learnable queries.”
InfluentialA Simple Framework for Open-Vocabulary Segmentation and Detection
Zhang, H., Li, F., Zou, X., Liu, S., Li, C., Yang, J., & Zhang, L. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00100 · Cited by 185 · cited 2×
“For output regions, LISA (Lai et al., 2023), GLaMM, GSVA (Xia et al., 2023b), NeXt-Chat (Zhang et al., 2023a), and LISA++ (Yang et al., 2023b) send the embedding corresponding to special tokens to the mask decoder of SAM, LLaVA-G to OpenSeeD (Zhang et al., 2023c), Lenna (Wei et al., 2023) to Grounding-DINO (Liu et al.…”
Guo, Z., Zhang, R., Zhu, X., Tang, Y., Ma, X., Han, J., Chen, K., Gao, P., Li, X., Li, H., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2309.00615 · Cited by 184
Scaling Autoregressive Multi-Modal Models: Pretraining and Instruction TuningPreprint
Yu, L., Shi, B., Pasunuru, R., Müller, B., Golovneva, O., Wang, T., Babu, A., Tang, B., Karrer, B., Sheynin, S., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2309.02591 · Cited by 178
IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language ReasoningPreprint
Lu, P., Qiu, L., Chen, J., Xia, T., Zhao, Y., Zhang, W., Zhou, Y., Liang, X., & Zhu, S. C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.13214 · Cited by 177 · cited 1×
“IconQA (Lu et al., 2021) tests the visual reasoning abilities of vision-and-language models on three types of questions: multiple-image-choice, multiple-text-choice, and fill-in-the-blank.”
PaLI-3 Vision Language Models: Smaller, Faster, StrongerPreprint
Chen, X., Wang, X., Beyer, L., Kolesnikov, A., Wu, J., Voigtlaender, P., Mustafa, B., Goodman, S., Alabdulmohsin, I., Padlewski, P., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.09199 · Cited by 161 · cited 1×
“However, recently there has been an interest in addressing more fine-grained visual tasks, such as visual grounding and image generation.”
BackgroundSVIT: Scaling up Visual Instruction TuningPreprint
Zhao, B., Wu, B., He, M., & Huang, T. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2307.04087 · Cited by 157 · cited 3×
“3B ♢ CLIP ViT-L Q-Former ✻ ✓ Visual Dialogue, VQA, Captioning, Referring, REC Shikra (Chen et al., 2023f) Vicuna-13B ♦ CLIP ViT-L Linear ✓ Visual Dialogue, VQA, Captioning, Referring, REC, GroundCap Clever Flamingo (Chen et al., 2023b) LLaMA-7B ▲ CLIP ViT-L XAttn LLM ✓ Visual Dialogue, VQA, Captioning SVIT (Zhao et al…”
Learning 3D Representations from 2D Pre-Trained Models via Image-to-Point Masked Autoencoders
Zhang, R., Wang, L., Qiao, Y., Gao, P., & Li, H. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.02085 · Cited by 156
Making LLaMA SEE and Draw with SEED TokenizerPreprint
Ge, Y., Zhao, S., Zeng, Z., Ge, Y., Li, C., Wang, X., & Shan, Y. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.01218 · Cited by 155 · cited 1×
“The most common way to output regions is to directly insert them into generated text as a series of coordinates, represented as numbers or as special tokens dedicated to location bins.”
OneLLM: One Framework to Align All Modalities with Language
Han, J., Gong, K., Zhang, Y., Wang, J., Zhang, K., Lin, D., Qiao, Y., Gao, P., & Yue, X. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02510 · Cited by 152
DetGPT: Detect What You Need via Reasoning
Pi, R.-J., Gao, J.-H., Diao, S., et al. (2023) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2305.14167 · Cited by 152 · cited 2×
“Specifically, Shikra (Chen et al., 2023f), DetGPT (Pi et al., 2023), ChatSpot (Zhao et al., 2023b), and PVIT (Chen et al., 2023a) leverage LLMs (Achiam et al., 2023; OpenAI, 2022) to combine regions and captions from datasets that present both annotations ( e.g. , COCO).”
LLaVA-Grounding: Grounded Visual Chat with Large Multimodal ModelsPreprint
Zhang, H., Li, H., Li, F., et al. (2023) · arXiv.org · doi:10.48550/arxiv.2312.02949 · Cited by 152
Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning
Liu, F., Lin, K., Li, L., Wang, J., Yacoob, Y., & Wang, L. (2023) · arXiv · doi:10.48550/arxiv.2306.14565 · Cited by 149 · cited 2×
“Several studies (Liu et al., 2023b; Zhu et al., 2023a) show that MLLMs tend to exhibit high hallucination rates, especially when generating longer captions.”
Generative Multimodal Models are In-Context Learners
Sun, Q., Cui, Y., Zhang, X., Zhang, F., Yu, Q., Wang, Y., Rao, Y., Liu, J., Huang, T., & Wang, X. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.01365 · Cited by 147 · cited 1×
“Shikra (Chen et al., 2023f), Kosmos-2 (Peng et al., 2023), MiniGPT-v2 (Chen et al., 2023e), Ferret (You et al., 2023), CogVLM (Wang et al., 2023c), SPHINX (Lin et al., 2023b), Qwen-VL (Bai et al., 2023b), and Griffon (Zhan et al., Table 2: Summary of MLLMs with components specifically designed for…”
MLLM-Protector: Ensuring MLLM’s Safety without Hurting Performance
Pi, R.-J., Han, T., Zhang, J., et al. (2024) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2401.02906 · Cited by 147 · cited 1×
“Although recent efforts are being made to reduce this phenomenon in text-to-image generative models (Schramowski et al., 2023; Friedrich et al., 2023; Poppi et al., 2024), further exploration is needed to prevent the same behavior in MLLMs (Pi et al., 2024).”
Jiang, A. Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D. S., Casas, D. D. L., Hanna, E. B., Bressand, F., et al. (2024) · arXiv · doi:10.48550/arxiv.2401.04088 · Cited by 143
Cobra: Extending Mamba to Multi-Modal Large Language Model for Efficient Inference
Zhao, H., Zhang, M., Zhao, W., et al. (2024) · AAAI Conference on Artificial Intelligence · doi:10.48550/arxiv.2403.14520 · Cited by 142 · cited 1×
“…OPT (Zhang et al., 2022b), Mag-neto (Wang et al., 2023b), MPT (MosaicML, 2023), and the instruction-tuned (Chung et al., 2022) or multilingual (Xue et al., 2020) flavors of T5 (Raffel et al., 2020), an encoder-decoder language model pre-trained for multiple tasks. et al., 2024; Zhao et al., 2024).”
Honeybee: Locality-Enhanced Projector for Multimodal LLM
Cha, J., Kang, W., Mun, J., & Roh, B. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01311 · Cited by 141 · cited 1×
“A different approach (Cha et al., 2023) proposes to replace linear layers with convolutional ones, demonstrating moderate improvements.”
Chen, F., Han, M., Zhao, H., Zhang, Q., Shi, J., Xu, S. B., & Xu, B. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.04160 · Cited by 140 · cited 10×
“3B ♢ CLIP ViT-L Q-Former ✻ ✗ Visual Dialogue, VQA, Captioning LLaMA-Adapter V2 (Gao et al., 2023) LLaMA-7B ▲ CLIP ViT-L Linear ✗ VQA, Captioning OpenFlamingo (Awadalla et al., 2023) MPT-7B ⋆ CLIP ViT-L XAttn LLM ✗ VQA, Captioning Flamingo (Alayrac et al., 2022) Chinchilla-70B ⋆ NFNet-F6 XAttn LLM ✗ Visual Dialogue, VQ…”
InfluentialVL-Mamba: Exploring State Space Models for Multimodal Learning
Qiao, Y., Yu, Z., Guo, L.-T., et al. (2024) · ENLSP · doi:10.48550/arxiv.2403.13600 · Cited by 136
Planting a SEED of Vision in Large Language ModelPreprint
Ge, Y., Ge, Y., Zeng, Z., Wang, X., & Shan, Y. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2307.08041 · Cited by 135
NExT-GPT: Any-to-Any Multimodal LLMPreprint
Wu, S., Fei, H., Qu, L., Ji, W., & Chua, T. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2309.05519 · Cited by 132
Kosmos-2: Grounding Multimodal Large Language Models to the WorldPreprint
Peng, Z., Wang, W., Dong, L., Hao, Y., Huang, S., Ma, S., & Wei, F. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2306.14824 · Cited by 129 · cited 5×
“3B ♢ CLIP ViT-L Q-Former ✻ ✗ Visual Dialogue, VQA, Captioning LLaMA-Adapter V2 (Gao et al., 2023) LLaMA-7B ▲ CLIP ViT-L Linear ✗ VQA, Captioning OpenFlamingo (Awadalla et al., 2023) MPT-7B ⋆ CLIP ViT-L XAttn LLM ✗ VQA, Captioning Flamingo (Alayrac et al., 2022) Chinchilla-70B ⋆ NFNet-F6 XAttn LLM ✗ Visual Dialogue, VQ…”
InfluentialSPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language ModelsPreprint
Liu, D., Zhang, R., Qiu, L., Huang, S., Lin, W., Zhao, S., Geng, S., Lin, Z., Jin, P., Zhang, K., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2402.05935 · Cited by 127
Hu, Z., İşcen, A., Sun, C., Wang, Z., Chang, K.-W., Sun, Y., Schmid, C., Ross, D. A., & Fathi, A. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.02238 · Cited by 123 · cited 1×
“We believe that the emergence of VQA datasets that require external retrieved knowledge (Chen et al., 2023k; Mensink et al., 2023) may enable the development of MLLMs with RAG capabilities (Hu et al., 2023b; Caffagni et al., 2024).”
LISA++: An Improved Baseline for Reasoning Segmentation with Large Language ModelPreprint
Yang, S., Qu, T., Lai, X. Q., Tian, Z., Peng, B., Liu, S., & Jia, J. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2312.17240 · Cited by 123
KOSMOS-2.5: A Multimodal Literate ModelPreprint
Lv, T., Huang, Y., Chen, J., Zhao, Y., Jia, Y., Cui, L., Ma, S., Chang, Y., Huang, S., Wang, W., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2309.11419 · Cited by 119 · cited 1×
“Some examples are MLLMs designed for document analysis and text-intensive visual inputs (Lv et al., 2023; Ye et al., 2023a), those proposed for embodied AI and robotics (Driess et al., 2023; Mu et al., 2023), and those tailored for specific domains such as medicine (Li et al., 2023d) and autonomous…”
Self-Instruct: Aligning Language Models with Self-Generated Instructions
Wang, Y., Kordi, Y., Mishra, S., Liu, A., Smith, N. A., Khashabi, D., & Hajishirzi, H. (2022) · arXiv · doi:10.48550/arxiv.2212.10560 · Cited by 118
Semantic-SAM: Segment and Recognize Anything at Any GranularityPreprint
Li, F., Zhang, H., Sun, P., Zou, X., Liu, S., Yang, J., Li, C., Zhang, L., & Gao, J. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2307.04767 · Cited by 115 · cited 1×
“However, recently there has been an interest in addressing more fine-grained visual tasks, such as visual grounding and image generation.”
BackgroundBuboGPT: Enabling Visual Grounding in Multi-Modal LLMsPreprint
Zhao, Y., Lin, Z., Zhou, D., Huang, Z., Feng, J., & Kang, B. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2307.08581 · Cited by 112 · cited 2×
“BuboGPT (Zhao et al., 2023c) leverages a combination of RAM, Grounding-DINO, and SAM and matches tags with nouns in the output sequence.”
InternGPT: Solving Vision-Centric Tasks by Interacting with ChatGPT Beyond LanguagePreprint
Liu, Z., He, Y., Wang, W., Wang, W., Wang, Y., Chen, S., Zhang, Q., Lai, Z., Yang, Y., Li, Q., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.05662 · Cited by 112
LAMM: Language-Assisted Multi-Modal Instruction-Tuning Dataset, Framework, and Benchmark
Yin, Z., Wang, J., Cao, J., Shi, Z., Liu, D., Li, M., Sheng, L., Bai, L., Huang, X., Wang, Z., et al. (2023) · arXiv · doi:10.48550/arxiv.2306.06687 · Cited by 108 · cited 1×
“Moreover, it divides the overall training process into two distinct phases: the initial phase concentrates on text-to-image generation, while the subsequent is focused on interleaved vision-and-language generation.”
BackgroundVIGC: Visual Instruction Generation and Correction
Wang, B., Wu, F., Han, X., Peng, J., Zhong, H.-P., Zhang, P., Dong, X.-W., Li, W.-J., Li, W., Wang, J., et al. (2023) · AAAI Conference on Artificial Intelligence · doi:10.48550/arxiv.2308.12714 · Cited by 108 · cited 1×
“While some solutions are emerging to mitigate this problem (Liu et al., 2023b; Wang et al., 2023a; Wu et al., 2023d; Yin et al., 2023b; Jing et al., 2023), understanding and correcting the underlying causes of hallucinations remains an important open challenge that is worth addressing to allow the…”
Feast Your Eyes: Mixture-of-Resolution Adaptation for Multimodal Large Language ModelsPreprint
Luo, G., Zhou, Y., Zhang, Y., Zheng, X., Sun, X., & Ji, R. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2403.03003 · Cited by 106
Asai, A., Wu, Z., Wang, Y., Sil, A., & Hajishirzi, H. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.11511 · Cited by 101 · cited 1×
“While retrieval-augmented generation (RAG) is a consolidated technique in LLMs (Lewis et al., 2020; Asai et al., 2023), its application in MLLMs is still under-explored.”
LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasoning, and Planning
Chen, S., Chen, X., Zhang, C., Li, M., Yu, G., Fei, H., Zhu, H., Fan, J., & Chen, T. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.02496 · Cited by 92 · cited 1×
“For input regions, GLaMM (Rasheed et al., 2023), GPT4RoI (Zhang et al., 2023g), ASM (Wang et al., 2023d) and ChatterBox (Tian et al., 2024b) leverage features of the image encoder to perform ROI align on the bounding box, whereas PVIT (Chen et al., 2023a) exploits Re-gionCLIP (Zhong et al., 2022).”
Feng, H., Liu, Q., Liu, H., Tang, J., Zhou, W., Li, H., & Huang, C. (2024) · Science China Information Sciences · doi:10.1007/s11432-024-4250-y · Cited by 92
Panagopoulou, A., Xue, L., Ning, Y.-X., et al. (2023) · arXiv.org · doi:10.48550/arxiv.2311.18799 · Cited by 92 · cited 1×
“This is usually achieved by aligning multimodal features through Transformer blocks such as Q-Former (Chen et al., 2023c; Panagopoulou et al., 2023) and Perceiver (Zhao et al., 2023d), or by utilizing ImageBind to effectively extract features that are inherently multimodal (Su et al., 2023).”
VideoChat: Chat-Centric Video UnderstandingPreprint
Li, K., He, Y., Wang, Y., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.06355 · Cited by 89
Unified Language-Vision Pretraining in LLM with Dynamic Discrete Visual TokenizationPreprint
Jin, Y., Xu, K., Xu, K., Chen, L., Liao, C., Tan, J., Huang, Q., Chen, B., Lei, C., Liu, A., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2309.04669 · Cited by 89
Otter: A Multi-Modal Model with In-Context Instruction TuningPreprint
Li, B., Zhang, Y., Chen, L., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.03726 · Cited by 86
ChatBridge: Bridging Modalities with Large Language Model as a Language CatalystPreprint
Zhao, Z., Guo, L., Yue, T., Chen, S., Shao, S., Zhu, X., Yuan, Z., & Liu, J. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.16103 · Cited by 80 · cited 5×
“The number of visual tokens is compressed through a Perceiver-like adapter and employs a spatial schema in such a way that the LLM can understand the grid of sub-images.”
InfluentialChatSpot: Bootstrapping Multimodal LLMs via Precise Referring Instruction Tuning
Zhao, L., Yu, E., Ge, Z., et al. (2023) · International Joint Conference on Artificial Intelligence · doi:10.48550/arxiv.2307.09474 · Cited by 79
The All-Seeing Project: Towards Panoptic Visual Recognition and Understanding of the Open WorldPreprint
Wang, W., Shi, M., Li, Q., Wang, W., Huang, Z., Xing, L., Chen, Z., Li, H., Zhu, X., Cao, Z., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2308.01907 · Cited by 75 · cited 2×
“ASM (Wang et al., 2023d), GLaMM (Rasheed et al., 2023), and LLaVA-G (Zhang et al., 2023d) propose automated pipelines comprising multiple steps based on off-the-shelf models for generating large corpora of conversations grounded in their corresponding images.”
OtterHD: A High-Resolution Multi-modality ModelPreprint
Li, B., Zhang, P., Yang, J., Zhang, Y., Pu, F., & Liu, Z. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2311.04219 · Cited by 75 · cited 1×
“Another solution, namely OtterHD (Li et al., 2023a), can seamlessly deal with any resolution or aspect ratio, as it directly feeds large image patches of 30 × 30 pixels to the LLM, without the need for a visual encoder.”
M$^{2}$UGen: Multi-modal Music Understanding and Generation with the Power of Large Language ModelsPreprint
Liu, S., Hussain, A. S., Wu, Q., Sun, C., & Shan, Y. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2311.11255 · Cited by 75
mPLUG-DocOwl: Modularized Multimodal Large Language Model for Document UnderstandingPreprint
Ye, J., Hu, A., Xu, H., Ye, Q., Yan, M., Dan, Y., Zhao, C., Xu, G., Li, C., Tian, J., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2307.02499 · Cited by 74 · cited 1×
“A different approach has been studied by Li et al. (2023f) which proposes to fine-tune the LLM by adding two special tokens ( i.e. , <start> and <end> ), and directly encode the generated text be-tween these two tokens using the text encoder in the SD model.”
MM-Interleaved: Interleaved Image-Text Generative Modeling via Multi-modal Feature SynchronizerPreprint
Tian, C., Zhu, X., Xiong, Y., Wang, W., Chen, Z., Wang, W., Chen, Y., Lu, L., Lü, T., Zhou, J., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2401.10208 · Cited by 74 · cited 2×
“BuboGPT (Zhao et al., 2023c) leverages a combination of RAM, Grounding-DINO, and SAM and matches tags with nouns in the output sequence.”
LION : Empowering Multimodal Large Language Model with Dual-Level Visual Knowledge
Chen, G., Shen, L., Shao, R., Deng, X., & Nie, L. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02506 · Cited by 73 · cited 1×
“It is a Transformer-based model proposed in BLIP-2 (Li et al., 2023g) and then used in several other approaches (Chen et al., 2023d; Dai et al., 2023; Hu et al., 2024).”
OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documents
Laurençon, H., Saulnier, L., Tronchon, L., Bekman, S., Singh, A., Lozhkov, A., Wang, T., Karamcheti, S., Rush, A. M., Kiela, D., et al. (2023) · arXiv · doi:10.48550/arxiv.2306.16527 · Cited by 72 · cited 15×
“3B ♢ CLIP ViT-L Q-Former ✻ ✗ Visual Dialogue, VQA, Captioning LLaMA-Adapter V2 (Gao et al., 2023) LLaMA-7B ▲ CLIP ViT-L Linear ✗ VQA, Captioning OpenFlamingo (Awadalla et al., 2023) MPT-7B ⋆ CLIP ViT-L XAttn LLM ✗ VQA, Captioning Flamingo (Alayrac et al., 2022) Chinchilla-70B ⋆ NFNet-F6 XAttn LLM ✗ Visual Dialogue, VQ…”
InfluentialCAT: Enhancing Multimodal Large Language Model to Answer Questions in Dynamic Audio-Visual Scenarios
Ye, Q.-L., Yu, Z.-T., Shao, R., et al. (2024) · European Conference on Computer Vision · doi:10.48550/arxiv.2403.04640 · Cited by 72
MultiModal-GPT: a vision and language model for dialogue with humans
Gong, T., Lyu, C., Zhang, S., Wang, Y., Zheng, M., Zhao, Q., Liu, K., Zhang, W., Luo, P., & Chen, K. (2023) · arXiv · doi:10.48550/arxiv.2305.04790 · Cited by 70 · cited 2×
“3B ♢ CLIP ViT-L Q-Former ✻ ✗ Visual Dialogue, VQA, Captioning LLaMA-Adapter V2 (Gao et al., 2023) LLaMA-7B ▲ CLIP ViT-L Linear ✗ VQA, Captioning OpenFlamingo (Awadalla et al., 2023) MPT-7B ⋆ CLIP ViT-L XAttn LLM ✗ VQA, Captioning Flamingo (Alayrac et al., 2022) Chinchilla-70B ⋆ NFNet-F6 XAttn LLM ✗ Visual Dialogue, VQ…”
MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image EditingPreprint
Zhang, K., Mo, L., Chen, W., Sun, H., & Su, Y. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2306.10012 · Cited by 69 · cited 1×
“MagicBrush (Zhang et al., 2023e) is a benchmark in the area of image editing and contains a collection of 10,000 manually annotated triplets, each consisting of a source image, an editing instruction, and the corresponding target image.”
FoodLMM: A Versatile Food Assistant Using Large Multi-Modal Model
Yin, Y., Qi, H., Zhu, B., Chen, J., Jiang, Y., & Ngo, C. (2025) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2025.3590924 · Cited by 69
NExT-Chat: An LMM for Chat, Detection and SegmentationPreprint
Zhang, A., Yao, Y., Ji, W., Liu, Z., & Chua, T. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2311.04498 · Cited by 69
Vista-llama: Reducing Hallucination in Video Language Models via Equal Distance to Visual Tokens
Ma, F., Jin, X., Wang, H., Xian, Y., Feng, J., & Yang, Y. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01249 · Cited by 66 · cited 1×
“However, there are also some attempts to develop video-specific adapters (Liu et al., 2023g; Ma et al., 2023a) that can capture fine-grained temporal information.”
PMC-VQA: Visual Instruction Tuning for Medical Visual Question Answering
Zhang, X., Wu, C., Zhao, Z., Lin, W., Zhang, Y., Wang, Y., & Xie, W. (2023) · arXiv · doi:10.48550/arxiv.2305.10415 · Cited by 66
TouchStone: Evaluating Vision-Language Models by Language ModelsPreprint
Bai, S., Yang, S., Bai, J., Wang, P., Zhang, X., Lin, J., Wang, X., Zhou, C., & Zhou, J. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2308.16890 · Cited by 65
Encyclopedic VQA: Visual questions about detailed properties of fine-grained categories
Mensink, T., Uijlings, J., Castrejón, L., Goel, A., Cadar, F., Zhou, H., Sha, F., Araújo, A. D. M., & Ferrari, V. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00289 · Cited by 61 · cited 1×
“We believe that the emergence of VQA datasets that require external retrieved knowledge (Chen et al., 2023k; Mensink et al., 2023) may enable the development of MLLMs with RAG capabilities (Hu et al., 2023b; Caffagni et al., 2024).”
DreamLLM: Synergistic Multimodal Comprehension and CreationPreprint
Dong, R., Han, C., Peng, Y., Qi, Z., Ge, Z., Yang, J., Zhao, L., Sun, J., Zhou, H., Wei, H., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2309.11499 · Cited by 59
EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of ThoughtPreprint
Mu, Y., Zhang, Q., Hu, M., Wang, W., Ding, M., Jin, J., Wang, B., Dai, J., Qiao, Y., & Luo, P. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.15021 · Cited by 58 · cited 1×
“…for document analysis and text-intensive visual inputs (Lv et al., 2023; Ye et al., 2023a), those proposed for embodied AI and robotics (Driess et al., 2023; Mu et al., 2023), and those tailored for specific domains such as medicine (Li et al., 2023d) and autonomous driving (Xu et al., 2023c).”
CoDi-2: In-Context, Interleaved, and Interactive Any-to-Any Generation
Tang, Z., Yang, Z., Khademi, M., Liu, Y., Zhu, C., & Bansal, M. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02589 · Cited by 57 · cited 1×
“Additional research endeavors have been conducted to introduce MLLMs in the field of image editing (Fu et al., 2024; Huang et al., 2023c; Tang et al., 2023).”
PG-Video-LLaVA: Pixel Grounding Large Video-Language ModelsPreprint
Munasinghe, S., Thushara, R., Maaz, M., Rasheed, H., Khan, S. H., Shah, M. A., & Khan, F. S. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2311.13435 · Cited by 57 · cited 1×
“In addition to encoding video frames, some works (Munasinghe et al., 2023; Zhang et al., 2023b) also employ audio features to enrich the representation of input video sequences.”
Visual Instruction Tuning with Polite FlamingoPreprint
Chen, D., Liu, J., Dai, W., et al. (2023) · AAAI Conference on Artificial Intelligence · doi:10.48550/arxiv.2307.01003 · Cited by 57 · cited 3×
“3B ♢ CLIP ViT-L Q-Former ✻ ✓ Visual Dialogue, VQA, Captioning, Referring, REC Shikra (Chen et al., 2023f) Vicuna-13B ♦ CLIP ViT-L Linear ✓ Visual Dialogue, VQA, Captioning, Referring, REC, GroundCap Clever Flamingo (Chen et al., 2023b) LLaMA-7B ▲ CLIP ViT-L XAttn LLM ✓ Visual Dialogue, VQA, Captioning SVIT (Zhao et al…”
Chen, Y., Sikka, K., Cogswell, M., Ji, H., & Divakaran, A. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01350 · Cited by 56 · cited 1×
“…et al., 2024), turns out to be critical for aligning the behavior of an LLM with that of humans and currently empowers the most advanced LLMs, eventually boosted via reinforcement learning from human feedback (RLHF) (Ouyang et al., 2022; Achiam et al., 2023; Chen et al., 2023l; Bai et al., 2023a).”
Background(2024) · doi:10.1109/cvprw63382.2024.00188 · Cited by 51 · cited 1×
“We believe that the emergence of VQA datasets that require external retrieved knowledge (Chen et al., 2023k; Mensink et al., 2023) may enable the development of MLLMs with RAG capabilities (Hu et al., 2023b; Caffagni et al., 2024).”
Self-citationSuper-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP TasksPreprint
Wang, Y., Mishra, S. R., Alipoormolabashi, P., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2204.07705 · Cited by 46
Pink: Unveiling the Power of Referential Comprehension for Multi-modal LLMs
Xuan, S., Guo, Q., Yan, S., & Zhang, S. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01313 · Cited by 45
VL-GPT: A Generative Pre-trained Transformer for Vision and Language Understanding and GenerationPreprint
Zhu, J., Ding, X., Ge, Y., Ge, Y., Zhao, S., Zhao, H., Wang, X., & Shan, Y. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2312.09251 · Cited by 45 · cited 2×
“This is usually achieved by fine-tuning the textual model (Zhu et al., 2023b; Ge et al., 2023a,b).”
Cheap and Quick: Efficient Vision-Language Instruction Tuning for Large Language ModelsPreprint
Luo, G., Zhou, Y., Ren, T., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.15023 · Cited by 45
LISA: Reasoning Segmentation via Large Language ModelPreprint
Lai, X. Q., Tian, Z., Chen, Y., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2308.00692 · Cited by 44
PaLI-X: On Scaling up a Multilingual Vision and Language ModelPreprint
Chen, X., Djolonga, J., Padlewski, P., Mustafa, B., Changpinyo, S., Wu, J., Riquelme, C., Goodman, S., Wang, X., Tay, Y., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.18565 · Cited by 42
Griffon: Spelling out All Object Locations at Any Granularity with Large Language ModelsPreprint
Zhan, Y., Zhu, Y., Chen, Z., et al. (2023) · European Conference on Computer Vision · doi:10.48550/arxiv.2311.14552 · Cited by 41
Integrally Pre-Trained Transformer Pyramid Networks
Tian, Y., Xie, L., Wang, Z., Wei, L., Zhang, X., Jiao, J., Wang, Y., Tian, Q., & Ye, Q. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01785 · Cited by 39
Visual Instruction Tuning towards General-Purpose Multimodal Model: A Survey
Huang, J., Zhang, J., Jiang, K., et al. (2023) · arXiv.org · doi:10.48550/arxiv.2312.16602 · Cited by 37 · cited 1×
“In comparison with existing surveys on the topic (Yin et al., 2023a; Wu et al., 2023b; Huang et al., 2023a), our paper exhibits substantial differences.”
StableLLaVA: Enhanced Visual Instruction Tuning with Synthesized Image-Dialogue DataPreprint
Li, Y., Zhang, C., Yu, G., Wang, Z.-B., Fu, B., Lin, G., Shen, C.-H., Chen, L., & Wei, Y.-C. (2023) · arXiv.org · doi:10.48550/arxiv.2308.10253 · Cited by 37 · cited 1×
“Given the amount of training data currently available, approaches like SPHINX-X (Gao et al., 2024) opt to perform a single all-in-one training stage in which to update all model components, possibly also using text-only data to preserve the conversational capabilities of the LLM.”
MethodJointly Training Large Autoregressive Multimodal Models
Aiello, E., Yu, L., Nie, Y., et al. (2023) · International Conference on Learning Representations · doi:10.48550/arxiv.2309.15564 · Cited by 36 · cited 1×
“2024) propose a methodology to mix an LLM architecture with an autoregressive generator, CM3Leon (Yu et al., 2023a), via bi-directional cross-attention across the architectures of both models.”
Jack of All Tasks, Master of Many: Designing General-purpose Coarse-to-Fine Vision-Language Model
Pramanick, S., Han, G., Hou, R., Nag, S., Lim, S.-N., Ballas, N., Wang, Q., Chellappa, R., & Almahairi, A. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01335 · Cited by 35 · cited 2×
“For instance, CoinIt (Pramanick et al., 2023) is a unified set of 14 benchmarks converted into an instruction-tuning format, spanning from single-image coarse-level to multi-image region-level tasks.”
AnyMAL: An Efficient and Scalable Any-Modality Augmented Language Model
Moon, S., Madotto, A., Lin, Z., Nagarajan, T., Smith, M., Jain, S. S. B. C., Yeh, C.-F., Murugesan, P., Heidari, P., Liu, Y., et al. (2024) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2024.emnlp-industry.98 · Cited by 33 · cited 1×
“Additionally, some works also effectively encode 3D data (Yin et al., 2023d) and IMU sensor signals (Moon et al., 2023).”
SCITUNE: Aligning Large Language Models with Human-Curated Scientific Multimodal Instructions
Horawalavithana, S., Munikoti, S., Stewart, I., et al. (2023) · NLP4SCIENCE · doi:10.18653/v1/2024.nlp4science-1 · Cited by 33
Research works printed but not linked: 21
- Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Mar- garet Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. 2015. VQA: Visual Question An- swering. In ICCV.
- Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with im- proved correlation with human judgments. In ACL Workshops.
- Davide Caffagni, Federico Cocchi, Nicholas Moratelli, Sara Sarto, Marcella Cornia, Lorenzo Baraldi, and Rita Cucchiara. 2024. Wiki-LLaVA: Hierarchical Retrieval-Augmented Generation for Multimodal LLMs. In CVPR Workshops.
- Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. 2023. Eva: Exploring the limits of masked visual representation learning at scale. In CVPR.
- Tsu-Jui Fu, Wenze Hu, Xianzhi Du, William Yang Wang, Yinfei Yang, and Zhe Gan. 2024. Guid- ing Instruction-based Image Editing via Multimodal Large Language Models. In ICLR.
- Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. 2022. Masked Au- toencoders Are Scalable Vision Learners. In CVPR.
- Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. 2017. GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium. In NeurIPS.
- Yining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Gan. 2023. 3D-LLM: Injecting the 3D World into Large Language Models. In NeurIPS.
- Jing Yu Koh, Ruslan Salakhutdinov, and Daniel Fried. 2023b. Grounding Language Models to Images for Multimodal Inputs and Outputs. In ICML.
- Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Ui- jlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, et al. 2020. The Open Images Dataset V4: Unified Image Classification, Object Detection, and Visual Relationship Detection at Scale. IJCV, 128:1956- 1981.
- Fangyu Liu, Guy Emerson, and Nigel Collier. 2023a. Visual Spatial Reasoning. TACL, 11:635-651.
- Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. 2023c. Mitigating Hallu- cination in Large Multi-Modal Models via Robust In- struction Tuning. arXiv preprint arXiv:2306.14565. · arXiv
- Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023e. Visual Instruction Tuning. In NeurIPS.
- Fan Ma, Xiaojie Jin, Heng Wang, Yuchen Xian, Jiashi Feng, and Yi Yang. 2023a. Vista-LLaMA: Reliable Video Narrator via Equal Distance to Visual Tokens. arXiv preprint arXiv:2312.08870. · arXiv
- Hongyu Wang, Shuming Ma, Shaohan Huang, Li Dong, Wenhui Wang, Zhiliang Peng, Yu Wu, Payal Bajaj, Saksham Singhal, Alon Benhaim, et al. 2023b. Mag- neto: A Foundation Transformer. In ICML.
- Jingkang Yang, Yi Zhe Ang, Zujin Guo, Kaiyang Zhou, Wayne Zhang, and Ziwei Liu. 2022. Panoptic scene graph generation. In ECCV.
- Jiabo Ye, Anwen Hu, Haiyang Xu, Qinghao Ye, Ming Yan, Guohai Xu, Chenliang Li, Junfeng Tian, Qi Qian, Ji Zhang, et al. 2023b. UReader: Universal OCR-free Visually-situated Language Understand- ing with Multimodal Large Language Model. In EMNLP.
- Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. 2023a. A Survey on Multimodal Large Language Models. arXiv preprint arXiv:2306.13549. · arXiv
- Shukang Yin, Chaoyou Fu, Sirui Zhao, Tong Xu, Hao Wang, Dianbo Sui, Yunhang Shen, Ke Li, Xing Sun, and Enhong Chen. 2023b. Woodpecker: Hallucina- tion correction for multimodal large language models. arXiv preprint arXiv:2310.16045. · arXiv
- Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L Berg. 2016. Modeling Context in Referring Expressions. In ECCV.
- Yiwu Zhong, Jianwei Yang, Pengchuan Zhang, Chun- yuan Li, Noel Codella, Liunian Harold Li, Luowei Zhou, Xiyang Dai, Lu Yuan, Yin Li, et al. 2022. Re- gionCLIP: Region-based Language-Image Pretrain- ing. In CVPR.
Cited by
Sorted by citations
A Survey on the Memory Mechanism of Large Language Model-based Agents
Zhang, Z., Dai, Q., Bo, X., Ma, C., Li, R., Chen, X., Zhu, J., Dong, Z., & Wen, J.-R. (2025) · ACM Transactions on Information Systems · doi:10.1145/3748302 · Cited by 592
Industrial Foundation Models (IFMs) for intelligent manufacturing: A systematic review
Zhao, S., Liu, S., Jiang, Y., Zhao, B. M., Lv, Y., Zhang, J., Wang, L., & Zhong, R. Y. (2025) · Journal of Manufacturing Systems · doi:10.1016/j.jmsy.2025.06.011 · Cited by 50
Wang, B., Zheng, Y., Han, X., Kong, L., Xiao, G., Xiao, Z., & Chen, S. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01715-x · Cited by 40
“Employing Multi-modal Large Language Models 163 , next-generation smart glasses not only enhance the quality of real-time decision-making 164 but also deliver personalized suggestions 165 derived from aggregated life and dietary data.”
Towards Retrieval-Augmented Architectures for Image Captioning
Sarto, S., Cornia, M., Baraldi, L., Nicolosi, A., & Cucchiara, R. (2024) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3663667 · Cited by 21
Self-citationSafe-CLIP: Removing NSFW Concepts from Vision-and-Language Models
Poppi, S., Poppi, T., Cocchi, F., Cornia, M., Baraldi, L., & Cucchiara, R. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73668-1_20 · Cited by 35
Self-citation
Metadata
- Authors
- Caffagni, D., Cocchi, F., Barsellotti, L., Moratelli, N., Sarto, S., Baraldi, L., Baraldi, L., Cornia, M., & Cucchiara, R.
- Affiliations
- University of Modena and Reggio Emilia (IT)
- University of Pisa (IT)
- ORCID
- Venue
- Findings of the Association for Computational Linguistics ACL 2024
- Published
- 2024
- Pages
- 13590-13618
- OpenAlex
- W4402670262
- Cited by
- 98
- References
- 171
- Citations by year
- 2026: 44 · 2025: 46 · 2024: 8
- FWCI
- 20.26
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Findings of the Association for Computational Linguistics: ACL (Findings of ACL) (publishedVersion, cc-by)
- Funders
- European Commission (PNRR-M4C2, PE00000013)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- multimodal large language models, multimodal language models, vision-language models, large language models, multimodal representation learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
