Multimodal Large Language Models: A Survey
Open accessThe DOI matches its Crossref record; 30 of the 30 references it cites were found in a registry or an index.
Abstract
The exploration of multimodal language models integrates multiple data types, such as images, text, language, audio, and other heterogeneity. While the latest large language models excel in text-based tasks, they often struggle to understand and process other data types. Multimodal models address this limitation by combining various modalities, enabling a more comprehensive understanding of diverse data. This paper begins by defining the concept of multimodal and examining the historical development of multimodal algorithms. Furthermore, we introduce a range of multimodal products, focusing on the efforts of major technology companies. A practical guide is provided, offering insights into the technical aspects of multimodal models. Moreover, we present a compilation of the latest algorithms and commonly used datasets, providing researchers with valuable resources for experimentation and evaluation. Lastly, we explore the applications of multimodal models and discuss the challenges associated with their development. By addressing these aspects, this paper aims to facilitate a deeper understanding of multimodal models and their potentiality in various domains.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460
Efficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568 · cited 1×
“In the case of text, Word2Vec [27] was commonly used for tokenization, including some methods like CBOW and Skip-gram.”
MethodRoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Learning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293 · cited 1×
“In 2021, the CLIP model was introduced [22].”
MethodExploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929
“Librispeech: an ASR corpus based on public domain audio books,”
Panayotov, V., Chen, G., Povey, D., & Khudanpur, S. (2015) · 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2015.7178964 · Cited by 6,072
Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations
Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L.-J., Shamma, D. A., et al. (2017) · International Journal of Computer Vision · doi:10.1007/s11263-016-0981-7 · Cited by 5,107
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Bourdev, L., Girshick, R., Hays, J., Perona, P., Ramanan, D., Zitnick, C. L., & Dollár, P. (2014) · arXiv · doi:10.48550/arxiv.1405.0312 · Cited by 2,579
Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding
Saharia, C., Chan, W., Saxena, S., Li, L., Whang, J., Denton, E., Ghasemipour, S. K. S., Ayan, B. K., Mahdavi, S. S., Lopes, R. G., et al. (2022) · arXiv · doi:10.48550/arxiv.2205.11487 · Cited by 2,528 · cited 1×
“Models such as OpenAI’s DALL-E 2 [23] and Google’s Imagen [51] have made significant breakthroughs in this area, attracting widespread attention.”
MethodHierarchical Text-Conditional Image Generation with CLIP Latents
Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., & Chen, M. (2022) · arXiv · doi:10.48550/arxiv.2204.06125 · Cited by 2,511 · cited 4×
“In 2022, DALL-E 2, a product in Ope-nAI, utilizes a diffusion model conditioned on CLIP image embeddings [23].”
MethodInfluentialLiu, H., Li, C., Wu, Q., & Lee, Y. J. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-1516 · Cited by 2,428
Continual Learning Through Synaptic Intelligence
Zenke, F., Poole, B., & Ganguli, S. (2017) · arXiv · doi:10.48550/arxiv.1703.04200 · Cited by 1,784
Katsigiannis, S., & Ramzan, N. (2018) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2017.2688239 · Cited by 1,135
Align before Fuse: Vision and Language Representation Learning with Momentum Distillation
Li, J., Selvaraju, R. R., Gotmare, A. D., Joty, S., Xiong, C., & Hoi, S. (2021) · arXiv · doi:10.48550/arxiv.2107.07651 · Cited by 1,078 · cited 1×
“Common examples of encoder-only models include CLIP [23] and ALBEF [32], which are suitable for tasks like image-text retrieval but not ideal for tasks like image captioning.”
MethodCommon Voice: A Massively-Multilingual Speech Corpus
Ardila, R., Branson, M., Davis, K., Henretty, M., Kohler, M., Meyer, J., Morais, R., Saunders, L., Tyers, F. M., & Weber, G. (2019) · arXiv · doi:10.48550/arxiv.1912.06670 · Cited by 905
A Lip Sync Expert Is All You Need for Speech to Lip Generation In the Wild
Prajwal, K. R., Mukhopadhyay, R., Namboodiri, V. P., & Jawahar, C. V. (2020) · Proceedings of the 28th ACM International Conference on Multimedia · doi:10.1145/3394171.3413532 · Cited by 897 · cited 1×
“Specifically, natural language generation technology impacts the quality of content in human-computer interactions, while computer vision technology affects the facial expressions and body movements of digital humans, such as lip synchronization [62].”
BackgroundA survey on multimodal large language models
Yin, S., Fu, C., Zhao, S., Li, K., Sun, X., Xu, T., & Chen, E. (2024) · National Science Review · doi:10.1093/nsr/nwae403 · Cited by 893
A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT
White, J., Fu, Q., Hays, S., Sandborn, M., Olea, C., Gilbert, H., Elnashar, A., Spencer-Smith, J., & Schmidt, D. C. (2023) · arXiv · doi:10.48550/arxiv.2302.11382 · Cited by 838
Maximum mutual information estimation of hidden Markov model parameters for speech recognition
Bahl, L., Brown, P., de Souza, P., & Mercer, R. (2005) · ICASSP '86. IEEE International Conference on Acoustics, Speech, and Signal Processing · doi:10.1109/icassp.1986.1169179 · Cited by 817
Liu, H., Li, C., Wu, Q., & Lee, Y. J. (2023) · arXiv · doi:10.48550/arxiv.2304.08485 · Cited by 682
PaLM-E: An Embodied Multimodal Language Model
Driess, D., Xia, F., Sajjadi, M. S. M., Lynch, C., Chowdhery, A., Ichter, B., Wahid, A., Tompson, J., Vuong, Q., Yu, T., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.03378 · Cited by 591 · cited 2×
“Another notable model, namely PaLM-E [26], combines advanced language and vision models, e.g., PaLM and ViT-22B.”
MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models
Zhu, D., Chen, J., Shen, X., Li, X., & Elhoseiny, M. (2023) · arXiv · doi:10.48550/arxiv.2304.10592 · Cited by 469 · cited 1×
“MiniGPT-4 [45] is a reproduction of certain functionalities of GPT-4 based on the combination of BLIP-2 and Vicuna.”
MethodMultimodal Few-Shot Learning with Frozen Language Models
Tsimpoukelli, M., Menick, J., Cabi, S., Eslami, S. M. A., Vinyals, O., & Hill, F. (2021) · arXiv · doi:10.48550/arxiv.2106.13884 · Cited by 355
GPT-4 is here: what scientists think
Sanderson, K. (2023) · Nature · doi:10.1038/d41586-023-00816-5 · Cited by 262 · cited 1×
“GPT-4 [6] serves as an excellent example of a multimodal LLM.”
BackgroundLanguage Is Not All You Need: Aligning Perception with Language Models
Huang, S., Dong, L., Wang, W., Hao, Y., Singhal, S., Ma, S., Lv, T., Cui, L., Mohammed, O. K., Patra, B., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.14045 · Cited by 235 · cited 1×
“In 2023, KOSMOS-1 was released by Microsoft [25].”
MethodAutomated facial expression recognition based on FACS action units
Lien, J. J., Kanade, T., Cohn, J. F., & Li, C.-C. (2002) · Proceedings Third IEEE International Conference on Automatic Face and Gesture Recognition · doi:10.1109/afgr.1998.670980 · Cited by 226
Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models
Wu, C., Yin, S., Qi, W., Wang, X., Tang, Z., & Duan, N. (2023) · arXiv · doi:10.48550/arxiv.2303.04671 · Cited by 197 · cited 2×
“For example, in visual ChatGPT [38], a prompt manager is used to generate informative prompts that facilitate ChatGPT’s understanding and generation of related images.”
BackgroundMethodMusicLM: Generating Music From Text
Agostinelli, A., Denk, T. I., Borsos, Z., Engel, J., Verzetti, M., Caillon, A., Huang, Q., Jansen, A., Roberts, A., Tagliasacchi, M., et al. (2023) · arXiv · doi:10.48550/arxiv.2301.11325 · Cited by 183 · cited 1×
“Moreover, Google’s MusiclM [59] has gained recognition in the audio multimodal domain, as it can generate music based on text inputs.”
BackgroundFrozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval
Bain, M., Nagrani, A., Varol, G., & Zisserman, A. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00175 · Cited by 76
Research works printed but not linked: 48
- 10.1109/BigData59044.2023.10386291 · DOI
- 10.1109/BigData59044.2023.10386351 · DOI
- 10.1017/S1351324920000601 · DOI · Paper page
- Summaira, 2022, A review on methods and applications in multimodal deep learning, ACM Transactions on Multimedia Computing, Communications, and Applications, 19, 76:1
- 10.1007/s11633-022-1410-8 · DOI · Paper page
- 10.1016/j.patrec.2013.07.003 · DOI · Paper page
- 10.1109/TPAMI.2009.76 · DOI · Paper page
- 10.1109/CVPR.1997.609351 · DOI
- 10.1558/cj.v16i3.295-310 · DOI
- 10.1007/11677482_3 · DOI · Paper page
- 10.1109/TASL.2009.2038810 · DOI
- 10.1145/1459359.1459573 · DOI · Paper page
- Ngiam, Multimodal deep learning, 689
- Hinton, 2012, A better way to pretrain deep boltzmann machines, Advances in Neural Information Processing Systems, 25
- 10.1109/CVPR.2016.503 · DOI · Paper page
- Bao, BEiT: BERT pre-training of image transformers
- 10.18653/v1/2020.findings-emnlp.414 · DOI · Paper page
- Yang, Visual concepts tokenization
- 10.1109/CVPR52688.2022.01763 · DOI · Paper page
- 10.1145/3581783.3613848 · DOI
- Wang, SimVLM: Simple visual language model pretraining with weak supervision
- 10.18653/v1/2022.emnlp-main.608 · DOI
- Bao, 2022, Vlmo: Unified vision-language pre-training with mixture-of-modality-experts, Advances in Neural Information Processing Systems, 35, 32 897
- Vaswani, 2017, Attention is all you need, Advances in Neural Information Processing Systems, 30
- Dosovitskiy, An image is worth 16x16 words: Transformers for image recognition at scale
- Yang, 2023, MM-REACT: Prompting chatgpt for multimodal reasoning and action, arXiv preprint, arXiv:2303.11381 · arXiv
- Li, BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models, 19 730
- Zhang, 2023, LLaMA-Adapter: Efficient fine-tuning of language models with zero-init attention, arXiv preprint, arXiv:2303.16199 · arXiv
- 10.1609/aaai.v36i3.20215 · DOI · Paper page
- 10.18653/v1/2022.findings-emnlp.67 · DOI · Paper page
- Guo, 2022, From images to textual prompts: Zero-shot VQA with frozen large language models, arXiv preprint, arXiv:2212.10846 · arXiv
- 10.1007/978-3-030-58520-4_25 · DOI · Paper page
- 10.1007/978-3-030-58621-8_3 · DOI · Paper page
- Forster, Extensions of the sign language recognition and translation corpus RWTH-PHOENIX-Weather, 1911
- 10.1109/MSP.2021.3106895 · DOI · Paper page
- 10.1371/journal.pone.0173392 · DOI · Paper page
- Yan, 2022, Video-text modeling with zero-shot transfer from contrastive captioners, arXiv preprint, arXiv:2212.04979 · arXiv
- 10.18653/v1/2021.emnlp-main.544 · DOI · Paper page
- Xu, mPLUG-2: A modularized multi-modal foundation model across text, image and video, 38 728
- 10.1109/ICCV48922.2021.00121 · DOI · Paper page
- Wu, 2023, AI-generated content (AIGC): A survey, arXiv preprint, arXiv:2304.06632 · arXiv
- 10.1609/aaai.v32i1.12342 · DOI · Paper page
- Dong, 2021, M5Product: A multi-modal pretraining benchmark for e-commercial product downstream tasks, arXiv preprint, arXiv:2109.04275 · arXiv
- 10.1109/CVPR.2016.571 · DOI · Paper page
- Zeng, Distributed training of large language models, 1
- 10.1007/978-3-031-01581-6 · DOI · Paper page
- Zeng, 2023, Large language models for robotics: A survey, arXiv preprint, arXiv:2311.07226 · arXiv
- Chen, 2022, PaLI: A jointly-scaled multilingual language-image model, arXiv preprint, arXiv:2209.06794 · arXiv
Cited by
A Survey on the Memory Mechanism of Large Language Model based Agents
Zhang, Z., Bo, X., Ma, C., Li, R., Chen, X., Dai, Q., Zhu, J., Dong, Z., & Wen, J.-R. (2024) · arXiv · doi:10.48550/arxiv.2404.13501 · Cited by 149
Generative AI in cybersecurity: A comprehensive review of LLM applications and vulnerabilities
Ferrag, M. A., Alwahedi, F., Battah, A., Cherif, B., Mechri, A., Tihanyi, N., Bisztray, T., & Debbah, M. (2025) · Internet of Things and Cyber-Physical Systems · doi:10.1016/j.iotcps.2025.01.001 · Cited by 148
The Revolution of Multimodal Large Language Models: A Survey
Caffagni, D., Cocchi, F., Barsellotti, L., Moratelli, N., Sarto, S., Baraldi, L., Baraldi, L., Cornia, M., & Cucchiara, R. (2024) · Findings of the Association for Computational Linguistics ACL 2024 · doi:10.18653/v1/2024.findings-acl.807 · Cited by 98
“In comparison with existing surveys on the topic (Yin et al., 2023a; Wu et al., 2023b; Huang et al., 2023a), our paper exhibits substantial differences.”
Hallucination of Multimodal Large Language Models: A Survey
Bai, Z., Wang, P., Xiao, T., He, T., Han, Z., Zhang, Z., & Shou, M. Z. (2024) · arXiv · doi:10.48550/arxiv.2404.18930 · Cited by 38
Ding, J.-E., Thao, P. N. M., Peng, W.-C., Wang, J.-Z., Chug, C.-C., Hsieh, M.-C., Tseng, Y.-C., Chen, L., Luo, D., Wu, C., et al. (2024) · Scientific Reports · doi:10.1038/s41598-024-71020-2 · Cited by 36
“Furthermore, researchers have begun to combine modality data with LLMs 20 .”
Method
Metadata
- Authors
- Wu, J., Gan, W., Chen, Z., Wan, S., & Yu, P. S.
- Venue
- 2023 IEEE International Conference on Big Data (BigData)
- Publisher
- IEEE
- Published
- 2023-12-15
- Pages
- 2247-2256
- OpenAlex
- W4391094120
- Cited by
- 252
- References
- 30
- FWCI
- 28.65
- Type
- conference paper
- Language
- en
- Open access
- green
- Funders
- National Natural Science Foundation of China
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.