Multimodal Large Language Models: A Survey
Open accessThe DOI matches its Crossref record; 29 of the 29 references it cites were found in a registry or an index.
Abstract
The exploration of multimodal language models integrates multiple data types, such as images, text, language, audio, and other heterogeneity. While the latest large language models excel in text-based tasks, they often struggle to understand and process other data types. Multimodal models address this limitation by combining various modalities, enabling a more comprehensive understanding of diverse data. This paper begins by defining the concept of multimodal and examining the historical development of multimodal algorithms. Furthermore, we introduce a range of multimodal products, focusing on the efforts of major technology companies. A practical guide is provided, offering insights into the technical aspects of multimodal models. Moreover, we present a compilation of the latest algorithms and commonly used datasets, providing researchers with valuable resources for experimentation and evaluation. Lastly, we explore the applications of multimodal models and discuss the challenges associated with their development. By addressing these aspects, this paper aims to facilitate a deeper understanding of multimodal models and their potentiality in various domains.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460
Efficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Learning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293 · cited 1×
“In 2021, the CLIP model was introduced [22].”
MethodExploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929
“Librispeech: an ASR corpus based on public domain audio books,”
Panayotov, V., Chen, G., Povey, D., & Khudanpur, S. (2015) · 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2015.7178964 · Cited by 6,072
Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations
Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L.-J., Shamma, D. A., et al. (2017) · International Journal of Computer Vision · doi:10.1007/s11263-016-0981-7 · Cited by 5,107
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Bourdev, L., Girshick, R., Hays, J., Perona, P., Ramanan, D., Zitnick, C. L., & Dollár, P. (2014) · arXiv · doi:10.48550/arxiv.1405.0312 · Cited by 2,579
Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding
Saharia, C., Chan, W., Saxena, S., Li, L., Whang, J., Denton, E., Ghasemipour, S. K. S., Ayan, B. K., Mahdavi, S. S., Lopes, R. G., et al. (2022) · arXiv · doi:10.48550/arxiv.2205.11487 · Cited by 2,528 · cited 1×
“Models such as OpenAI’s DALL-E 2 [23] and Google’s Imagen [51] have made significant breakthroughs in this area, attracting widespread attention.”
Method
Research works printed but not linked: 49
- 10.1109/BigData59044.2023.10386291 · DOI
- 10.1109/BigData59044.2023.10386351 · DOI
- 10.1017/S1351324920000601 · DOI · Paper page
- Summaira, 2022, A review on methods and applications in multimodal deep learning, ACM Transactions on Multimedia Computing, Communications, and Applications, 19, 76:1
- 10.1007/s11633-022-1410-8 · DOI · Paper page
- Yin, 2023, A survey on multimodal large language models, arXiv preprint, arXiv:2306.13549 · arXiv
- 10.1016/j.patrec.2013.07.003 · DOI · Paper page
- 10.1109/TPAMI.2009.76 · DOI · Paper page
- 10.1109/CVPR.1997.609351 · DOI
- 10.1558/cj.v16i3.295-310 · DOI
- 10.1007/11677482_3 · DOI · Paper page
- 10.1109/TASL.2009.2038810 · DOI
- 10.1145/1459359.1459573 · DOI · Paper page
- Ngiam, Multimodal deep learning, 689
- Hinton, 2012, A better way to pretrain deep boltzmann machines, Advances in Neural Information Processing Systems, 25
- 10.1109/CVPR.2016.503 · DOI · Paper page
- Bao, BEiT: BERT pre-training of image transformers
- 10.18653/v1/2020.findings-emnlp.414 · DOI · Paper page
- Yang, Visual concepts tokenization
- 10.1109/CVPR52688.2022.01763 · DOI · Paper page
- 10.1145/3581783.3613848 · DOI
- Wang, SimVLM: Simple visual language model pretraining with weak supervision
- 10.18653/v1/2022.emnlp-main.608 · DOI
- Bao, 2022, Vlmo: Unified vision-language pre-training with mixture-of-modality-experts, Advances in Neural Information Processing Systems, 35, 32 897
- Vaswani, 2017, Attention is all you need, Advances in Neural Information Processing Systems, 30
- Dosovitskiy, An image is worth 16x16 words: Transformers for image recognition at scale
- Yang, 2023, MM-REACT: Prompting chatgpt for multimodal reasoning and action, arXiv preprint, arXiv:2303.11381 · arXiv
- Li, BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models, 19 730
- Zhang, 2023, LLaMA-Adapter: Efficient fine-tuning of language models with zero-init attention, arXiv preprint, arXiv:2303.16199 · arXiv
- 10.1609/aaai.v36i3.20215 · DOI · Paper page
- 10.18653/v1/2022.findings-emnlp.67 · DOI · Paper page
- Guo, 2022, From images to textual prompts: Zero-shot VQA with frozen large language models, arXiv preprint, arXiv:2212.10846 · arXiv
- 10.1007/978-3-030-58520-4_25 · DOI · Paper page
- 10.1007/978-3-030-58621-8_3 · DOI · Paper page
- Forster, Extensions of the sign language recognition and translation corpus RWTH-PHOENIX-Weather, 1911
- 10.1109/MSP.2021.3106895 · DOI · Paper page
- 10.1371/journal.pone.0173392 · DOI · Paper page
- Yan, 2022, Video-text modeling with zero-shot transfer from contrastive captioners, arXiv preprint, arXiv:2212.04979 · arXiv
- 10.18653/v1/2021.emnlp-main.544 · DOI · Paper page
- Xu, mPLUG-2: A modularized multi-modal foundation model across text, image and video, 38 728
- 10.1109/ICCV48922.2021.00121 · DOI · Paper page
- Wu, 2023, AI-generated content (AIGC): A survey, arXiv preprint, arXiv:2304.06632 · arXiv
- 10.1609/aaai.v32i1.12342 · DOI · Paper page
- Dong, 2021, M5Product: A multi-modal pretraining benchmark for e-commercial product downstream tasks, arXiv preprint, arXiv:2109.04275 · arXiv
- 10.1109/CVPR.2016.571 · DOI · Paper page
- Zeng, Distributed training of large language models, 1
- 10.1007/978-3-031-01581-6 · DOI · Paper page
- Zeng, 2023, Large language models for robotics: A survey, arXiv preprint, arXiv:2311.07226 · arXiv
- Chen, 2022, PaLI: A jointly-scaled multilingual language-image model, arXiv preprint, arXiv:2209.06794 · arXiv
Cited by
A Survey on the Memory Mechanism of Large Language Model based Agents
Zhang, Z., Bo, X., Ma, C., Li, R., Chen, X., Dai, Q., Zhu, J., Dong, Z., & Wen, J.-R. (2024) · arXiv · doi:10.48550/arxiv.2404.13501 · Cited by 149
Generative AI in cybersecurity: A comprehensive review of LLM applications and vulnerabilities
Ferrag, M. A., Alwahedi, F., Battah, A., Cherif, B., Mechri, A., Tihanyi, N., Bisztray, T., & Debbah, M. (2025) · Internet of Things and Cyber-Physical Systems · doi:10.1016/j.iotcps.2025.01.001 · Cited by 148
The Revolution of Multimodal Large Language Models: A Survey
Caffagni, D., Cocchi, F., Barsellotti, L., Moratelli, N., Sarto, S., Baraldi, L., Baraldi, L., Cornia, M., & Cucchiara, R. (2024) · Findings of the Association for Computational Linguistics ACL 2024 · doi:10.18653/v1/2024.findings-acl.807 · Cited by 98
“In comparison with existing surveys on the topic (Yin et al., 2023a; Wu et al., 2023b; Huang et al., 2023a), our paper exhibits substantial differences.”
Hallucination of Multimodal Large Language Models: A Survey
Bai, Z., Wang, P., Xiao, T., He, T., Han, Z., Zhang, Z., & Shou, M. Z. (2024) · arXiv · doi:10.48550/arxiv.2404.18930 · Cited by 38
Ding, J.-E., Thao, P. N. M., Peng, W.-C., Wang, J.-Z., Chug, C.-C., Hsieh, M.-C., Tseng, Y.-C., Chen, L., Luo, D., Wu, C., et al. (2024) · Scientific Reports · doi:10.1038/s41598-024-71020-2 · Cited by 36
“Furthermore, researchers have begun to combine modality data with LLMs 20 .”
Method
Metadata
- Authors
- Wu, J., Gan, W., Chen, Z., Wan, S., & Yu, P. S.
- Venue
- 2023 IEEE International Conference on Big Data (BigData)
- Publisher
- IEEE
- Published
- 2023-12-15
- Pages
- 2247-2256
- OpenAlex
- W4391094120
- Cited by
- 252
- References
- 29
- FWCI
- 28.65
- Type
- conference paper
- Language
- en
- Open access
- green
- Funders
- National Natural Science Foundation of China
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Multimodal Large Language Models: A Survey. https://app.cochat.ai/papers/10.1109/bigdata59044.2023.10386743
- The paper
- Wu, J., Gan, W., Chen, Z., Wan, S., & Yu, P. S. (2023). Multimodal Large Language Models: A Survey. 2023 IEEE International Conference on Big Data (BigData). https://doi.org/10.1109/bigdata59044.2023.10386743
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.