CoChat
Sign Up
PreprintCited by 21Multimodal Machine Learning ApplicationsTopic ModelingDomain Adaptation and Few-Shot Learning

The DOI matches its DataCite record; 2 of the 2 references it cites were found in a registry or an index.

A Comprehensive Review of Multimodal Large Language Models: Performance and Challenges Across Different Tasks

Open access
Wang, Jiaqi, Jiang, Hanqi, Liu, Yiheng, Ma, Chong, Zhang, Xu, Pan, Yi, Liu, Mengyuan, Gu, Peiran, Xia, Sichen, Chen, Lifeng, Chen, Junhao, Li, Wenjun, Zhang, Yutong, Wu, Zihao, Liu, Zhengliang, Zhong, Tianyang, Ge, Bao, Zhang, Tuo, Qiang, Ning, Hu, Xintao, Jiang, Xi, Zhang, Xin, Zhang, Wei, Shen, Dinggang, Liu, Tianming, Zhang, Shu·2024·arXiv
DOI 10.48550/arxiv.2408.01319arXiv 2408.01319
Open this paper in CoChat — free accountOpen PDF

Abstract

In an era defined by the explosive growth of data and rapid technological advancements, Multimodal Large Language Models (MLLMs) stand at the forefront of artificial intelligence (AI) systems. Designed to seamlessly integrate diverse data types-including text, images, videos, audio, and physiological sequences-MLLMs address the complexities of real-world applications far beyond the capabilities of single-modality systems. In this paper, we systematically sort out the applications of MLLM in multimodal tasks such as natural language, vision, and audio. We also provide a comparative analysis of the focus of different MLLMs in the tasks, and provide insights into the shortcomings of current MLLMs, and suggest potential directions for future research. Through these discussions, this paper hopes to provide valuable insights for the further development and application of MLLM.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2408.01319

Cites

2 of 2 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Correlating information effectively and accurately between modalities while maintaining context is in question.

    “Although correlating information effectively and accurately between modalities while maintaining context is in question (Wang et al. 2024 ).”

    From A review of LLMs and their applications in the architecture, engineering and construction industry (2025) · cites this paper as Wang et al. 2024

    The “Interpretability of multimodal information fusion” section (page unavailable) says understanding how modalities are combined remains challenging, and the “InstructBLIP Architecture” section says cross-modal tasks require models to learn correlated information. The cited passages do not specifically identify maintaining context while correlating modalities accurately as an open question. (LLM, conf=0.79)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Flagged first, then by citations

Research works printed but not linked: 107

  1. J. Wu, W. Gan, Z. Chen, S. Wan, and S. Y. Philip, "Multimodal large language models: A survey," in 2023 IEEE International Conference on Big Data (BigData). IEEE, 2023, pp. 2247-2256.
  2. S. Yin, C. Fu, S. Zhao, K. Li, X. Sun, T. Xu, and E. Chen, "A survey on multimodal large language models," arXiv preprint arXiv:2306.13549, 2023. · arXiv
  3. D. Zhang, Y. Yu, C. Li, J. Dong, D. Su, C. Chu, and D. Yu, "Mm-llms: Recent advances in multimodal large language models," arXiv preprint arXiv:2401.13601, 2024. · arXiv
  4. J. Y. Koh, D. Fried, and R. R. Salakhutdinov, "Generating images with multimodal language models," Advances in Neural Information Processing Systems, vol. 36, 2024.
  5. C. Ma, H. Jiang, W. Chen, Z. Wu, X. Yu, F. Zeng, L. Guo, D. Zhu, T. Zhang, D. Shen et al., "Eye-gaze guided multi-modal alignment framework for radiology," arXiv preprint arXiv:2403.12416, 2024. · arXiv
  6. K. Zhang, J. Yu, Z. Yan, Y. Liu, E. Adhikarla, S. Fu, X. Chen, C. Chen, Y. Zhou, X. Li et al., "Biomedgpt: a unified and generalist biomedical generative pre-trained transformer for vision, language, and multimodal tasks," arXiv preprint arXiv:2305.17100, 2023. · arXiv
  7. X. Mei, L. Yang, D. Gao, X. Cai, J. Han, and T. Liu, "Phraseaug: An augmented medical report generation model with phrasebook," IEEE Transactions on Medical Imaging, 2024.
  8. Z. Xiao, Y. Chen, J. Yao, L. Zhang, Z. Liu, Z. Wu, X. Yu, Y. Pan, L. Zhao, C. Ma et al., "Instruction-vit: Multi-modal prompts for instruction learning in vision transformer," Information Fusion, vol. 104, p. 102204, 2024.
  9. M. Malik, M. K. Malik, K. Mehmood, and I. Makhdoom, "Automatic speech recognition: a survey," Multimedia Tools and Applications, vol. 80, pp. 9411-9457, 2021.
  10. P. Bahar, T. Bieschke, and H. Ney, "A comparative study on end-to-end speech to text translation," in 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU). IEEE, 2019, pp. 792-799.
  11. C. Lyu, M. Wu, L. Wang, X. Huang, B. Liu, Z. Du, S. Shi, and Z. Tu, "Macaw-llm: Multi-modal language modeling with image, audio, video, and text integration," arXiv preprint arXiv:2306.09093, 2023. · arXiv
  12. J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat et al., "Gpt-4 technical report," arXiv preprint arXiv:2303.08774, 2023. · arXiv
  13. L. Zheng, W.-L. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. Xing et al., "Judging llm-as-a-judge with mt- bench and chatbot arena," Advances in Neural Information Processing Systems, vol. 36, 2024.
  14. T. Le Scao, A. Fan, C. Akiki, E. Pavlick, S. Ilić, D. Hesslow, R. Castagné, A. S. Luccioni, F. Yvon, M. Gallé et al., "Bloom: A 176b- parameter open-access multilingual language model," arXiv preprint arXiv:2211.05100, 2023. · arXiv
  15. J. Wang, Z. Wu, Y. Li, H. Jiang, P. Shu, E. Shi, H. Hu, C. Ma, Y. Liu, X. Wang, Y. Yao, X. Liu, H. Zhao, Z. Liu, H. Dai, L. Zhao, B. Ge, X. Li, T. Liu, and S. Zhang, "Large language models for robotics: Opportunities, challenges, and perspectives," 2024. [Online]. Available: https://arxiv.org/abs/2401.04334 · arXiv
  16. J. Wang, Z. Liu, L. Zhao, Z. Wu, C. Ma, S. Yu, H. Dai, Q. Yang, Y. Liu, S. Zhang et al., "Review of large vision models and visual prompt engineering," Meta-Radiology, p. 100047, 2023.
  17. G. Team, R. Anil, S. Borgeaud, Y. Wu, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth et al., "Gemini: a family of highly capable multimodal models," arXiv preprint arXiv:2312.11805, 2023. · arXiv
  18. S. Wu, H. Fei, L. Qu, W. Ji, and T.-S. Chua, "Next-gpt: Any-to-any multimodal llm," arXiv preprint arXiv:2309.05519, 2023. · arXiv
  19. Y. Liu, K. Zhang, Y. Li, Z. Yan, C. Gao, R. Chen, Z. Yuan, Y. Huang, H. Sun, J. Gao et al., "Sora: A review on background, technology, limitations, and opportunities of large vision models," arXiv preprint arXiv:2402.17177, 2024. · arXiv
  20. K. Li, Y. He, Y. Wang, Y. Li, W. Wang, P. Luo, Y. Wang, L. Wang, and Y. Qiao, "Videochat: Chat-centric video understanding," arXiv preprint arXiv:2305.06355, 2023. · arXiv
  21. B. Lin, B. Zhu, Y. Ye, M. Ning, P. Jin, and L. Yuan, "Video-llava: Learning united visual representation by alignment before projection," arXiv preprint arXiv:2311.10122, 2023. · arXiv
  22. S. K. Gaikwad, B. W. Gawali, and P. Yannawar, "A review on speech recognition technique," International Journal of Computer Applica- tions, vol. 10, no. 3, pp. 16-24, 2010.
  23. M. Mulimani and A. Mesaros, "Class-incremental learning for multi- label audio classification," in ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2024, pp. 916-920.
  24. Y. Shen, K. Song, X. Tan, D. Li, W. Lu, and Y. Zhuang, "Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face," Advances in Neural Information Processing Systems, vol. 36, 2024.
  25. R. Huang, M. Li, D. Yang, J. Shi, X. Chang, Z. Ye, Y. Wu, Z. Hong, J. Huang, J. Liu et al., "Audiogpt: Understanding and generating speech, music, sound, and talking head," in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 38, no. 21, 2024, pp. 23 802- 23 804.
  26. C. Tang, W. Yu, G. Sun, X. Chen, T. Tan, W. Li, L. Lu, Z. Ma, and C. Zhang, "Salmonn: Towards generic hearing abilities for large language models," arXiv preprint arXiv:2310.13289, 2023. · arXiv
  27. C. Li, Z. Gan, Z. Yang, J. Yang, L. Li, L. Wang, J. Gao et al., "Multimodal foundation models: From specialists to general-purpose assistants," Foundations and Trends® in Computer Graphics and Vision, vol. 16, no. 1-2, pp. 1-214, 2024.
  28. Y. Liu, H. He, T. Han, X. Zhang, M. Liu, J. Tian, Y. Zhang, J. Wang, X. Gao, T. Zhong et al., "Understanding llms: A comprehensive overview from training to inference," arXiv preprint arXiv:2401.02038, 2024. · arXiv
  29. A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly et al., "An image is worth 16x16 words: Transformers for image recognition at scale," arXiv preprint arXiv:2010.11929, 2020. · arXiv
  30. K. He, X. Zhang, S. Ren, and J. Sun, "Deep residual learning for image recognition," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770-778.
  31. F. Chen, M. Han, H. Zhao, Q. Zhang, J. Shi, S. Xu, and B. Xu, "X- llm: Bootstrapping advanced large language models by treating multi- modalities as foreign languages," arXiv preprint arXiv:2305.04160, 2023. · arXiv
  32. W.-N. Hsu, B. Bolte, Y.-H. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed, "Hubert: Self-supervised speech representation learning by masked prediction of hidden units," IEEE/ACM transactions on audio, speech, and language processing, vol. 29, pp. 3451-3460, 2021.
  33. S. Chen, Y. Wu, C. Wang, S. Liu, D. Tompkins, Z. Chen, and F. Wei, "Beats: Audio pre-training with acoustic tokenizers," arXiv preprint arXiv:2212.09058, 2022. · arXiv
  34. A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, "Robust speech recognition via large-scale weak supervi- sion," in International conference on machine learning. PMLR, 2023, pp. 28 492-28 518.
  35. K. Gadzicki, R. Khamsehashari, and C. Zetzsche, "Early vs late fusion in multimodal convolutional neural networks," in 2020 IEEE 23rd international conference on information fusion (FUSION). IEEE, 2020, pp. 1-6.
  36. Y. Li, M. E. H. Daho, P.-H. Conze, R. Zeghlache, H. Le Boité, R. Tadayoni, B. Cochener, M. Lamard, and G. Quellec, "A review of deep learning-based information fusion techniques for multimodal medical image classification," Computers in Biology and Medicine, p. 108635, 2024.
  37. J. D. M.-W. C. Kenton and L. K. Toutanova, "Bert: Pre-training of deep bidirectional transformers for language understanding," in Proceedings of NAACL-HLT, 2019, pp. 4171-4186.
  38. S. Schneider, A. Baevski, R. Collobert, and M. Auli, "wav2vec: Unsupervised pre-training for speech recognition," Interspeech 2019, 2019.
  39. D. Zhu, J. Chen, X. Shen, X. Li, and M. Elhoseiny, "Minigpt-4: En- hancing vision-language understanding with advanced large language models," arXiv preprint arXiv:2304.10592, 2023. · arXiv
  40. J. Chen, D. Zhu, X. Shen, X. Li, Z. Liu, P. Zhang, R. Krishnamoorthi, V. Chandra, Y. Xiong, and M. Elhoseiny, "Minigpt-v2: large language model as a unified interface for vision-language multi-task learning," arXiv preprint arXiv:2310.09478, 2023. · arXiv
  41. D. Caffagni, F. Cocchi, N. Moratelli, S. Sarto, M. Cornia, L. Baraldi, and R. Cucchiara, "Wiki-llava: Hierarchical retrieval-augmented gen- eration for multimodal llms," in Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, 2024, pp. 1818- 1826.
  42. Z. Li, C. Zhang, X. Wang, R. Ren, Y. Xu, R. Ma, and X. Liu, "3dmit: 3d multi-modal instruction tuning for scene understanding," arXiv preprint arXiv:2401.03201, 2024. · arXiv
  43. Z. Li, Q. Xu, D. Zhang, H. Song, Y. Cai, Q. Qi, R. Zhou, J. Pan, Z. Li, V. T. Vu et al., "Groundinggpt: Language enhanced multi-modal grounding model." CoRR, 2024.
  44. X. Wang, B. Zhuang, and Q. Wu, "Modaverse: Efficiently transforming modalities with llms," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 26 606-26 616.
  45. H. Wei, L. Kong, J. Chen, L. Zhao, Z. Ge, E. Yu, J. Sun, C. Han, and X. Zhang, "Small language model meets with reinforced vision vocabulary," arXiv preprint arXiv:2401.12503, 2024. · arXiv
  46. S. Chen, Z. Jie, and L. Ma, "Llava-mole: Sparse mixture of lora experts for mitigating data conflicts in instruction finetuning mllms," arXiv preprint arXiv:2401.16160, 2024. · arXiv
  47. J. Qi, M. Ding, W. Wang, Y. Bai, Q. Lv, W. Hong, B. Xu, L. Hou, J. Li, Y. Dong et al., "Cogcom: Train large vision-language models diving into details through chain of manipulations," arXiv preprint arXiv:2402.04236, 2024. · arXiv
  48. H. Gao, J. Pei, and H. Huang, "Progan: Network embedding via proximity generative adversarial network," in Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2019, pp. 1308-1316.
  49. C. Tian, X. Zhu, Y. Xiong, W. Wang, Z. Chen, W. Wang, Y. Chen, L. Lu, T. Lu, J. Zhou et al., "Mm-interleaved: Interleaved image- text generative modeling via multi-modal feature synchronizer," arXiv preprint arXiv:2401.10208, 2024. · arXiv
  50. F.-Y. Wang, Z. Huang, X. Shi, W. Bian, G. Song, Y. Liu, and H. Li, "Animatelcm: Accelerating the animation of personalized diffusion models and adapters with decoupled consistency learning," arXiv preprint arXiv:2402.00769, 2024. · arXiv
  51. D. Li, A. Kamko, E. Akhgari, A. Sabet, L. Xu, and S. Doshi, "Playground v2. 5: Three insights towards enhancing aesthetic quality in text-to-image generation," arXiv preprint arXiv:2402.17245, 2024. · arXiv
  52. H. Laurenc ¸on, L. Tronchon, M. Cord, and V. Sanh, "What matters when building vision-language models?" arXiv preprint arXiv:2405.02246, 2024. · arXiv
  53. Y. Li, Y. Zhang, C. Wang, Z. Zhong, Y. Chen, R. Chu, S. Liu, and J. Jia, "Mini-gemini: Mining the potential of multi-modality vision language models," arXiv preprint arXiv:2403.18814, 2024. · arXiv
  54. J. Lin, H. Yin, W. Ping, P. Molchanov, M. Shoeybi, and S. Han, "Vila: On pre-training for visual language models," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 26 689-26 699.
  55. D. Jiang, X. He, H. Zeng, C. Wei, M. Ku, Q. Liu, and W. Chen, "Mantis: Interleaved multi-image instruction tuning," arXiv preprint arXiv:2405.01483, 2024. · arXiv
  56. H. Zhang, X. Li, and L. Bing, "Video-llama: An instruction-tuned audio-visual language model for video understanding," arXiv preprint arXiv:2306.02858, 2023. · arXiv
  57. S. Munasinghe, R. Thushara, M. Maaz, H. A. Rasheed, S. Khan, M. Shah, and F. Khan, "Pg-video-llava: Pixel grounding large video- language models," arXiv preprint arXiv:2311.13435, 2023. · arXiv
  58. Z. Cheng, S. Leng, H. Zhang, Y. Xin, X. Li, G. Chen, Y. Zhu, W. Zhang, Z. Luo, D. Zhao et al., "Videollama 2: Advancing spatial- temporal modeling and audio understanding in video-llms," arXiv preprint arXiv:2406.07476, 2024. · arXiv
  59. Y. Su, T. Lan, H. Li, J. Xu, Y. Wang, and D. Cai, "Pandagpt: One model to instruction-follow them all," arXiv preprint arXiv:2305.16355, 2023. · arXiv
  60. M. Maaz, H. Rasheed, S. Khan, and F. S. Khan, "Video-chatgpt: Towards detailed video understanding via large vision and language models," arXiv preprint arXiv:2306.05424, 2023. · arXiv
  61. Y. Zhao, Z. Lin, D. Zhou, Z. Huang, J. Feng, and B. Kang, "Bubogpt: Enabling visual grounding in multi-modal llms," arXiv preprint arXiv:2307.08581, 2023. · arXiv
  62. R. Luo, Z. Zhao, M. Yang, J. Dong, D. Li, P. Lu, T. Wang, L. Hu, M. Qiu, and Z. Wei, "Valley: Video assistant with large language model enhanced ability," arXiv preprint arXiv:2306.07207, 2023. · arXiv
  63. K. Li, Y. Wang, Y. He, Y. Li, Y. Wang, Y. Liu, Z. Wang, J. Xu, G. Chen, P. Luo et al., "Mvbench: A comprehensive multi-modal video understanding benchmark," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 22 195-22 206.
  64. B. Li, Y. Zhang, L. Chen, J. Wang, F. Pu, J. Yang, C. Li, and Z. Liu, "Mimic-it: Multi-modal in-context instruction tuning," arXiv preprint arXiv:2306.05425, 2023. · arXiv
  65. T. Gong, C. Lyu, S. Zhang, Y. Wang, M. Zheng, Q. Zhao, K. Liu, W. Zhang, P. Luo, and K. Chen, "Multimodal-gpt: A vi- sion and language model for dialogue with humans," arXiv preprint arXiv:2305.04790, 2023. · arXiv
  66. Y. Li, C. Wang, and J. Jia, "Llama-vid: An image is worth 2 tokens in large language models," arXiv preprint arXiv:2311.17043, 2023. · arXiv
  67. A. Panagopoulou, L. Xue, N. Yu, J. Li, D. Li, S. Joty, R. Xu, S. Savarese, C. Xiong, and J. C. Niebles, "X-instructblip: A framework for aligning x-modal instruction-aware representations to llms and emergent cross-modal reasoning," arXiv preprint arXiv:2311.18799, 2023. · arXiv
  68. J. Li, D. Li, S. Savarese, and S. Hoi, "Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models," in International conference on machine learning. PMLR, 2023, pp. 19 730-19 742.
  69. R. Girdhar, A. El-Nouby, Z. Liu, M. Singh, K. V. Alwala, A. Joulin, and I. Misra, "Imagebind: One embedding space to bind them all," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 15 180-15 190.
  70. M. Bain, A. Nagrani, G. Varol, and A. Zisserman, "Frozen in time: A joint video and image encoder for end-to-end retrieval," in Proceedings of the IEEE/CVF international conference on computer vision, 2021, pp. 1728-1738.
  71. P. Sharma, N. Ding, S. Goodman, and R. Soricut, "Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning," in Proceedings of the 56th Annual Meeting of the Asso- ciation for Computational Linguistics (Volume 1: Long Papers), 2018, pp. 2556-2565.
  72. H. Liu, C. Li, Q. Wu, and Y. J. Lee, "Visual instruction tuning," Advances in neural information processing systems, vol. 36, 2024.
  73. S. Changpinyo, P. Sharma, N. Ding, and R. Soricut, "Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts," in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2021, pp. 3558-3568.
  74. D. Xu, Z. Zhao, J. Xiao, F. Wu, H. Zhang, X. He, and Y. Zhuang, "Video question answering via gradually refined attention over ap- pearance and motion," in Proceedings of the 25th ACM international conference on Multimedia, 2017, pp. 1645-1653.
  75. Z. Tang, Z. Yang, C. Zhu, M. Zeng, and M. Bansal, "Any-to-any generation via composable diffusion," Advances in Neural Information Processing Systems, vol. 36, 2024.
  76. J. Zhan, J. Dai, J. Ye, Y. Zhou, D. Zhang, Z. Liu, X. Zhang, R. Yuan, G. Zhang, L. Li et al., "Anygpt: Unified multimodal llm with discrete sequence modeling," arXiv preprint arXiv:2402.12226, 2024. · arXiv
  77. I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, "Generative adversarial net- works," Communications of the ACM, vol. 63, no. 11, pp. 139-144, 2020.
  78. D. P. Kingma and M. Welling, "Auto-encoding variational bayes," arXiv preprint arXiv:1312.6114, 2013. · arXiv
  79. D. Neimark, O. Bar, M. Zohar, and D. Asselmann, "Video transformer network," in Proceedings of the IEEE/CVF international conference on computer vision, 2021, pp. 3163-3172.
  80. J. Ho, T. Salimans, A. Gritsenko, W. Chan, M. Norouzi, and D. J. Fleet, "Video diffusion models," Advances in Neural Information Processing Systems, vol. 35, pp. 8633-8646, 2022.
  81. C. Wu, S. Yin, W. Qi, X. Wang, Z. Tang, and N. Duan, "Visual chatgpt: Talking, drawing and editing with visual foundation models," arXiv preprint arXiv:2303.04671, 2023. · arXiv
  82. L. Tian, Q. Wang, B. Zhang, and L. Bo, "Emo: Emote portrait alive- generating expressive portrait videos with audio2video diffusion model under weak conditions," arXiv preprint arXiv:2402.17485, 2024. · arXiv
  83. H. Liu, W. Yan, M. Zaharia, and P. Abbeel, "World model on million-length video and language with ringattention," arXiv preprint arXiv:2402.08268, 2024. · arXiv
  84. C. D. Kim, B. Kim, H. Lee, and G. Kim, "Audiocaps: Generating captions for audios in the wild," in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 2019, pp. 119-132.
  85. E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen, "Lora: Low-rank adaptation of large language models," arXiv preprint arXiv:2106.09685, 2021. · arXiv
  86. J. Xu, T. Mei, T. Yao, and Y. Rui, "Msr-vtt: A large video description dataset for bridging video and language," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 5288-5296.
  87. J. Xiao, X. Shang, A. Yao, and T.-S. Chua, "Next-qa: Next phase of question-answering to explaining temporal actions," in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2021, pp. 9777-9786.
  88. B. D. Willmore and A. J. King, "Adaptation in auditory processing," Physiological Reviews, vol. 103, no. 2, pp. 1025-1058, 2023.
  89. D. R. Bohnenstiehl, "Automated cataloging of oyster toadfish (opsanus tau) boatwhistle calls using template matching and machine learning," Ecological Informatics, vol. 77, p. 102268, 2023.
  90. S. Deshmukh, B. Elizalde, R. Singh, and H. Wang, "Pengi: An audio language model for audio tasks," Advances in Neural Information Processing Systems, vol. 36, pp. 18 090-18 108, 2023.
  91. W. Pan, H. Li, X. Zhou, J. Jiao, C. Zhu, and Q. Zhang, "Research on pig sound recognition based on deep neural network and hidden markov models," Sensors, vol. 24, no. 4, p. 1269, 2024.
  92. M. Shafieian, "Hidden markov model and persian speech recognition," International Journal of Nonlinear Analysis and Applications, vol. 14, no. 1, pp. 3111-3119, 2023.
  93. N. Zeghidour, A. Luebs, A. Omran, J. Skoglund, and M. Tagliasacchi, "Soundstream: An end-to-end neural audio codec," IEEE/ACM Trans- actions on Audio, Speech, and Language Processing, vol. 30, pp. 495- 507, 2021.
  94. T. Hayashi and S. Watanabe, "Discretalk: Text-to-speech as a machine translation problem," arXiv preprint arXiv:2005.05525, 2020. · arXiv
  95. E. Kharitonov, J. Copet, K. Lakhotia, T. A. Nguyen, P. Tomasello, A. Lee, A. Elkahky, W.-N. Hsu, A. Mohamed, E. Dupoux et al., "textless-lib: A library for textless spoken language processing," arXiv preprint arXiv:2202.07359, 2022. · arXiv
  96. Y. Chu, J. Xu, X. Zhou, Q. Yang, S. Zhang, Z. Yan, C. Zhou, and J. Zhou, "Qwen-audio: Advancing universal audio understand- ing via unified large-scale audio-language models," arXiv preprint arXiv:2311.07919, 2023. · arXiv
  97. P. K. Rubenstein, C. Asawaroengchai, D. D. Nguyen, A. Bapna, Z. Bor- sos, F. d. C. Quitry, P. Chen, D. E. Badawy, W. Han, E. Kharitonov et al., "Audiopalm: A large language model that can speak and listen," arXiv preprint arXiv:2306.12925, 2023. · arXiv
  98. D. Zhang, S. Li, X. Zhang, J. Zhan, P. Wang, Y. Zhou, and X. Qiu, "Speechgpt: Empowering large language models with intrinsic cross-modal conversational abilities," arXiv preprint arXiv:2305.11000, 2023. · arXiv
  99. A. Zhang, H. Fei, Y. Yao, W. Ji, L. Li, Z. Liu, and T.-S. Chua, "Vpgtrans: Transfer visual prompt generator across llms," Advances in Neural Information Processing Systems, vol. 36, 2024.
  100. A. Awadalla, I. Gao, J. Gardner, J. Hessel, Y. Hanafy, W. Zhu, K. Marathe, Y. Bitton, S. Gadre, S. Sagawa et al., "Openflamingo: An open-source framework for training large autoregressive vision- language models," arXiv preprint arXiv:2308.01390, 2023. · arXiv
  101. G. H. Chen, S. Chen, R. Zhang, J. Chen, X. Wu, Z. Zhang, Z. Chen, J. Li, X. Wan, and B. Wang, "Allava: Harnessing gpt4v- synthesized data for a lite vision-language model," arXiv preprint arXiv:2402.11684, 2024. · arXiv
  102. H. Liu, C. Li, Y. Li, and Y. J. Lee, "Improved baselines with visual instruction tuning," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 26 296-26 306.
  103. P. Gao, J. Han, R. Zhang, Z. Lin, S. Geng, A. Zhou, W. Zhang, P. Lu, C. He, X. Yue et al., "Llama-adapter v2: Parameter-efficient visual instruction model," arXiv preprint arXiv:2304.15010, 2023. · arXiv
  104. A. Young, B. Chen, C. Li, C. Huang, G. Zhang, G. Zhang, H. Li, J. Zhu, J. Chen, J. Chang et al., "Yi: Open foundation models by 01. ai," arXiv preprint arXiv:2403.04652, 2024. · arXiv
  105. J. Yu, X. Li, J. Y. Koh, H. Zhang, R. Pang, J. Qin, A. Ku, Y. Xu, J. Baldridge, and Y. Wu, "Vector-quantized image modeling with improved vqgan," arXiv preprint arXiv:2110.04627, 2021. · arXiv
  106. G. Laccetti, R. Montella, C. Palmieri, and V. Pelliccia, "The high performance internet of things: using gvirtus to share high-end gpus with arm based cluster computing nodes," in International Conference on Parallel Processing and Applied Mathematics. Springer, 2013, pp. 734-744.
  107. J. Giorgi, O. Nitski, B. Wang, and G. Bader, "Declutr: Deep contrastive learning for unsupervised textual representations," arXiv preprint arXiv:2006.03659, 2020. · arXiv

Cited by

Sorted by citations

Metadata

Authors
Wang, J., Jiang, H., Liu, Y., Ma, C., Zhang, X., Pan, Y., Liu, M., Gu, P., Xia, S., Chen, L., Chen, J., Li, W., Zhang, Y., Wu, Z., Liu, Z., Zhong, T., Ge, B., Zhang, T., Qiang, N., Hu, X., Jiang, X., Zhang, X., Zhang, W., Shen, D., Liu, T., & Zhang, S.
Affiliations
Northwestern Polytechnical University (CN)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2024
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2408.01319
arXiv
2408.01319
OpenAlex
W4406302098
Cited by
21
References
2
Citations by year
2026: 5 · 2025: 16
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, other-oa)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords
multimodal large language models, multimodal tasks, natural language understanding, audio understanding, cross-modal integration, model performance comparison

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review