Cited by 369Multimodal Machine Learning ApplicationsAutonomous Vehicle Technology and SafetyAdvanced Neural Network Applications

A Survey on Multimodal Large Language Models for Autonomous Driving

Open access
Cui, Can, Ma, Yunsheng, Cao, Xu, Ye, Wenqian, Zhou, Yang, Liang, Kaizhao, Chen, Jintai, Lu, Juanwu +13 more·2024·2024 IEEE/CVF Winter Conference on Applications of Computer Vision Workshops (WACVW)

Abstract

With the emergence of Large Language Models (LLMs) and Vision Foundation Models (VFMs), multimodal AI systems benefiting from large models have the potential to equally perceive the real world, make decisions, and control tools as humans. In recent months, LLMs have shown widespread attention in autonomous driving and map systems. Despite its immense potential, there is still a lack of a comprehensive understanding of key challenges, opportunities, and future endeavors to apply in LLM driving systems. In this paper, we present a systematic investigation in this field. We first introduce the background of Multimodal Large Language Models (MLLMs), the multimodal models development using LLMs, and the history of autonomous driving. Then, we overview existing MLLM tools for driving, transportation, and map systems together with existing datasets and benchmarks. Moreover, we summarized the works in The 1st WACV Workshop on Large Language and Vision Models for Autonomous Driving (LLVM-AD), which is the first workshop of its kind regarding LLMs in autonomous driving. To further promote the development of this field, we also discuss several important problems regarding using MLLMs in autonomous driving systems that need to be solved by both academia and industry.

Integrity signals

Registry

Crossref · 10.1109/wacvw60836.2024.00106

Cites

179 of 179 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Efficient reasoning LLMs are able to greatly improve autonomic driving by helping them understand large amounts of sensor data in a human-like way.

    “Efficient reasoning LLMs are able to greatly improve autonomic driving [30,199-201] by helping them understand large amounts of sensor data in a human-like way.”

    From Stop Overthinking: A Survey on Efficient Reasoning for Large Language Models (2025) · cites this paper as [30]

  • Transformer models have made notable strides in computer vision, as evidenced by vision transformers like EfficientViT and YOLOv8.

    “These models have also made notable strides in computer vision, as evidenced by the development of vision transformers like EfficientViT [145], [146] and YOLOv8 [147], [148], [149].”

Cites

Flagged first, then by citations

Cited by

Sorted by citations

Authors
Cui, C., Ma, Y., Cao, X., Ye, W., Zhou, Y., Liang, K., Chen, J., Lu, J., Yang, Z., Liao, K.-D., Gao, T., Li, E., Tang, K., Cao, Z., Zhou, T., Liu, A., Yan, X., Mei, S., Cao, J., Wang, Z., & Zheng, C.
Affiliations
  • Purdue University West Lafayette (US)
  • University of Illinois Urbana-Champaign (US)
  • University of Virginia (US)
  • New York University (US)
  • SambaNova Systems, Inc,Palo Alto,CA,USA,94303
  • Objective, Inc,San Francisco,CA,USA,94110
  • Tencent (China) (CN)
  • PediaMed AI,Shenzhen,China
Venue
2024 IEEE/CVF Winter Conference on Applications of Computer Vision Workshops (WACVW)
Publisher
IEEE
Published
2024-01-01
Pages
958-979
OpenAlex
W4394862623
Cited by
369
References
179
Citations by year
2026: 83 · 2025: 165 · 2024: 119 · 2023: 2
FWCI
75.56
Type
conference paper
Language
en
Open access
Locations
  • IEEE/CVF Winter Conference on Applications of Computer Vision Workshops (WACVW) (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by-nc-sa)
  • arXiv (Cornell University)
Funders
  • Purdue University
  • Tencent
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Physical Sciences › Engineering › Automotive Engineering › Autonomous Vehicle Technology and Safety
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
Keywords
multimodal large language models, autonomous driving, vision foundation models, driving system, transportation, multimodal dataset, benchmark

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review