A Survey on Multimodal Large Language Models for Autonomous Driving
Open accessAbstract
With the emergence of Large Language Models (LLMs) and Vision Foundation Models (VFMs), multimodal AI systems benefiting from large models have the potential to equally perceive the real world, make decisions, and control tools as humans. In recent months, LLMs have shown widespread attention in autonomous driving and map systems. Despite its immense potential, there is still a lack of a comprehensive understanding of key challenges, opportunities, and future endeavors to apply in LLM driving systems. In this paper, we present a systematic investigation in this field. We first introduce the background of Multimodal Large Language Models (MLLMs), the multimodal models development using LLMs, and the history of autonomous driving. Then, we overview existing MLLM tools for driving, transportation, and map systems together with existing datasets and benchmarks. Moreover, we summarized the works in The 1st WACV Workshop on Large Language and Vision Models for Autonomous Driving (LLVM-AD), which is the first workshop of its kind regarding LLMs in autonomous driving. To further promote the development of this field, we also discuss several important problems regarding using MLLMs in autonomous driving systems that need to be solved by both academia and industry.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Efficient reasoning LLMs are able to greatly improve autonomic driving by helping them understand large amounts of sensor data in a human-like way.
“Efficient reasoning LLMs are able to greatly improve autonomic driving [30,199-201] by helping them understand large amounts of sensor data in a human-like way.”
From Stop Overthinking: A Survey on Efficient Reasoning for Large Language Models (2025) · cites this paper as [30]
Transformer models have made notable strides in computer vision, as evidenced by vision transformers like EfficientViT and YOLOv8.
“These models have also made notable strides in computer vision, as evidenced by the development of vision transformers like EfficientViT [145], [146] and YOLOv8 [147], [148], [149].”
From From Google Gemini to OpenAI Q* (Q-Star): A Survey on Reshaping the Generative Artificial Intelligence (AI) Research Landscape (2025) · cites this paper as [147]
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · see citing works
Jakubův, J., Chvalovský, K., Goertzel, Z., et al. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,648 · see citing works
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works
Efficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568 · see citing works
Sequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766 · see citing works
High-Resolution Image Synthesis with Latent Diffusion Models
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01042 · Cited by 15,880 · see citing works
Are we ready for autonomous driving? The KITTI vision benchmark suite
Geiger, A., Lenz, P., & Urtasun, R. (2012) · 2012 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2012.6248074 · Cited by 14,897 · see citing works
Kaugeranna, R., Kaugeranna, E., & Aion, (. S. 4. 6. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.giscience.2023.43 · Cited by 13,695 · see citing works
ASPECTS OF THE THEORY OF SYNTAX
Chomsky, N. (1964) · doi:10.21236/ad0616323 · Cited by 13,587 · see citing works
Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A. C., Lo, W.-Y., et al. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00371 · Cited by 11,953 · see citing works
Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
Chung, J., Gulcehre, C., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1412.3555 · Cited by 11,008 · see citing works
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · see citing works
Self-citationBidirectional recurrent neural networks
Schuster, M., & Paliwal, K. K. (1997) · IEEE Transactions on Signal Processing · doi:10.1109/78.650093 · Cited by 10,172 · see citing works
PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation
Charles, R., Su, H., Mo, K., & Guibas, L. J. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.16 · Cited by 9,767 · see citing works
nuScenes: A Multimodal Dataset for Autonomous Driving
Caesar, H., Bankiti, V., Lang, A. H., Vora, S., Liong, V. E., Xu, Q., Krishnan, A., Pan, Y., Baldan, G., & Beijbom, O. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01164 · Cited by 7,152 · see citing works
Show and tell: A neural image caption generator
Vinyals, O., Toshev, A., Bengio, S., & Erhan, D. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298935 · Cited by 6,318 · cited 1× · see citing works
“This early attempt bridged vision (understanding block configurations) with language (interpreting and executing instructions), even though it was not based on deep learning.”
BackgroundDeep visual-semantic alignments for generating image descriptions
Karpathy, A., & Fei-Fei, L. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298932 · Cited by 4,902 · see citing works
PointPillars: Fast Encoders for Object Detection From Point Clouds
Lang, A. H., Vora, S., Caesar, H., Zhou, L., Yang, J., & Beijbom, O. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.01298 · Cited by 4,673 · see citing works
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602 · see citing works
Adding Conditional Control to Text-to-Image Diffusion Models
Zhang, L., Rao, A., & Agrawala, M. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00355 · Cited by 4,547 · see citing works
VQA: Visual Question Answering
Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.279 · Cited by 4,487 · see citing works
Self-citationChain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263 · see citing works
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208 · see citing works
Taxonomy and definitions for terms related to driving automation systems for on-road motor vehicles
On-Road Automated Driving (ORAD) Committee (2016) · doi:10.4271/j3016_201609 · Cited by 3,892 · see citing works
Scalability in Perception for Autonomous Driving: Waymo Open Dataset
Sun, P., Kretzschmar, H., Dotiwalla, X., Chouard, A., Patnaik, V., Tsui, P., Guo, J., Zhou, Y., Chai, Y., Caine, B., et al. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00252 · Cited by 3,437 · see citing works
Self-citation(1992) · doi:10.5555/176313.176316 · Cited by 2,892 · see citing works
Llama 2: Open Foundation and Fine-Tuned Chat Models
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.09288 · Cited by 2,604 · see citing works
PaLM: Scaling Language Modeling with Pathways
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.02311 · Cited by 2,590 · see citing works
Liu, H., Li, C., Wu, Q., & Lee, Y. J. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-1516 · Cited by 2,428 · see citing works
Deep reinforcement learning for autonomous driving: A survey
Kiran, B. R., Sobh, I., Talpaert, V., Mannion, P., Sallab, A. A. A., Yogamani, S., & Perez, P. (2022) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2021.3054625 · Cited by 2,342 · see citing works
Stanley: The robot that won the DARPA Grand Challenge
Thrun, S., Montemerlo, M., Dahlkamp, H., Stavens, D., Aron, A., Diebel, J., Fong, P., Gale, J., Halpenny, M., Hoffmann, G., et al. (2006) · Journal of Field Robotics · doi:10.1002/rob.20147 · Cited by 2,237 · see citing works
ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks
Lu, J., Batra, D., Parikh, D., & Lee, S. (2019) · arXiv · doi:10.48550/arxiv.1908.02265 · Cited by 2,109 · see citing works
Self-citationLarge Language Models are Zero-Shot Reasoners
Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., & Iwasawa, Y. (2022) · arXiv · doi:10.48550/arxiv.2205.11916 · Cited by 2,013 · see citing works
A Survey of Autonomous Driving: <i>Common Practices and Emerging Technologies</i>
Yurtsever, E., Lambert, J., Carballo, A., & Takeda, K. (2020) · IEEE Access · doi:10.1109/access.2020.2983149 · Cited by 1,889 · cited 1× · see citing works
“Inspired by current limitations, part of the research on autonomous driving is now focusing on addressing the safety of autonomous systems and enhancing the safety of autonomous systems [200].”
BackgroundA survey of deep learning techniques for autonomous driving
Grigorescu, S., Trasnea, B., Cocias, T., & Macesanu, G. (2020) · Journal of Field Robotics · doi:10.1002/rob.21918 · Cited by 1,809 · see citing works
Flamingo: a Visual Language Model for Few-Shot Learning
Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.14198 · Cited by 1,555 · see citing works
Volumetric and Multi-view CNNs for Object Classification on 3D Data
Qi, C. R., Su, H., NieBner, M., Dai, A., Yan, M., & Guibas, L. J. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.609 · Cited by 1,504 · see citing works
Argoverse: 3D Tracking and Forecasting With Rich Maps
Chang, M.-F., Lambert, J., Sangkloy, P., Singh, J., Bak, S., Hartnett, A., Wang, D., Carr, P., Lucey, S., Ramanan, D., et al. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00895 · Cited by 1,474 · see citing works
Scaling Instruction-Finetuned Language Models
Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, Y., Wang, X., Dehghani, M., Brahma, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2210.11416 · Cited by 1,441 · see citing works
Towards fully autonomous driving: Systems and algorithms
Levinson, J., Askeland, J., Becker, J., Dolson, J., Held, D., Kammel, S., Kolter, J. Z., Langer, D., Pink, O., Pratt, V., et al. (2011) · 2011 IEEE Intelligent Vehicles Symposium (IV) · doi:10.1109/ivs.2011.5940562 · Cited by 1,374 · see citing works
ALVINN: An autonomous land vehicle in a neural network
Pomerleau, D. A. (2018) · Carnegie Mellon University · doi:10.1184/r1/6603146.v1 · Cited by 1,340 · see citing works
mPLUG-Owl: Modularization Empowers Large Language Models with MultimodalityPreprint
Ye, Q.-H., Xu, H., Xu, G., Ye, J., Yan, M., Zhou, Y., Wang, J., Hu, A., Shi, P.-C., Shi, Y., et al. (2023) · arXiv.org · doi:10.48550/arxiv.2304.14178 · Cited by 1,318 · see citing works
Self-citationSpatial as Deep: Spatial CNN for Traffic Scene Understanding
Pan, X., Shi, J., Luo, P., Wang, X., & Tang, X. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.12301 · Cited by 1,213 · see citing works
VisualBERT: A Simple and Performant Baseline for Vision and Language
Li, L. H., Yatskar, M., Yin, D., Hsieh, C.-J., & Chang, K.-W. (2019) · arXiv · doi:10.48550/arxiv.1908.03557 · Cited by 1,198 · see citing works
Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN)Preprint
Mao, J., Xu, W., Yang, Y., Wang, J., Huang, Z., & Yuille, A. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1412.6632 · Cited by 1,039 · see citing works
Li, J., Li, D., Savarese, S., & Hoi, S. (2023) · arXiv · doi:10.48550/arxiv.2301.12597 · Cited by 908 · see citing works
The hierarchical hidden Markov model: Analysis and applications
Fine, S., Singer, Y., & Tishby, N. (1998) · Machine Learning · doi:10.1023/a:1007469218079 · Cited by 906 · see citing works
A survey on multimodal large language models
Yin, S., Fu, C., Zhao, S., Li, K., Sun, X., Xu, T., & Chen, E. (2024) · National Science Review · doi:10.1093/nsr/nwae403 · Cited by 893 · see citing works
Three Decades of Driver Assistance Systems: Review and Future Perspectives
Bengler, K. J., Dietmayer, K., Färber, B., Maurer, M., Stiller, C., & Winner, H. (2014) · IEEE Intelligent Transportation Systems Magazine · doi:10.1109/mits.2014.2336271 · Cited by 870 · see citing works
Arandjelović, R., & Zisserman, A. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.73 · Cited by 836 · see citing works
Code as Policies: Language Model Programs for Embodied Control
Liang, J., Huang, W., Xia, F., Xu, P., Hausman, K., Ichter, B., Florence, P., & Zeng, A. (2023) · 2023 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra48891.2023.10160591 · Cited by 767 · see citing works
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
Min, S., Lyu, X., Holtzman, A., Artetxe, M., Lewis, M., Hajishirzi, H., & Zettlemoyer, L. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.759 · Cited by 755 · see citing works
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
Min, S., Lyu, X., Holtzman, A., Artetxe, M., Lewis, M., Hajishirzi, H., & Zettlemoyer, L. (2022) · arXiv · doi:10.48550/arxiv.2202.12837 · Cited by 752 · see citing works
High-Resolution Image Synthesis with Latent Diffusion Models
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2021) · arXiv · doi:10.48550/arxiv.2112.10752 · Cited by 737 · see citing works
Kendall, A., Hawke, J., Janz, D., Mazur, P., Reda, D., Allen, J.-M., Lam, V.-D., Bewley, A., & Shah, A. (2019) · 2019 International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2019.8793742 · Cited by 694 · cited 1× · see citing works
“Deep Reinforcement Learning (DRL) has additionally played a pivotal role in enhancing the control strategies of autonomous vehicles, refining motion planning, and decision-making processes to adapt to dynamic and uncertain driving conditions [16,75,78,93],.”
BackgroundLiu, H., Li, C., Wu, Q., & Lee, Y. J. (2023) · arXiv · doi:10.48550/arxiv.2304.08485 · Cited by 682 · see citing works
Autoware on Board: Enabling Autonomous Vehicles with Embedded Systems
Kato, S., Tokunaga, S., Maruyama, Y., Maeda, S., Hirabayashi, M., Kitsukawa, Y., Monrroy, A., Ando, T., Fujii, Y., & Azumi, T. (2018) · ACM/IEEE International Conference on Cyber-Physical Systems (ICCPS) · doi:10.1109/iccps.2018.00035 · Cited by 667 · see citing works
MM-REACT: Prompting ChatGPT for Multimodal Reasoning and ActionPreprint
Yang, Z.-Y., Li, L., Wang, J., Lin, K., Azarnasab, E., Ahmed, F., Liu, Z.-C., Liu, C., Zeng, M., & Wang, L.-J. (2023) · arXiv.org · doi:10.48550/arxiv.2303.11381 · Cited by 660 · see citing works
Self-citationDo As I Can, Not As I Say: Grounding Language in Robotic Affordances
Ahn, M., Brohan, A., Brown, N., Chebotar, Y., Cortes, O., David, B., Finn, C., Fu, C., Gopalakrishnan, K., Hausman, K., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.01691 · Cited by 624 · see citing works
ProgPrompt: Generating Situated Robot Task Plans using Large Language Models
Singh, I., Blukis, V., Mousavian, A., Goyal, A., Xu, D., Tremblay, J., Fox, D., Thomason, J., & Garg, A. (2023) · 2023 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra48891.2023.10161317 · Cited by 613 · see citing works
Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity
Lu, Y., Bartolo, M., Moore, A., Riedel, S., & Stenetorp, P. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.556 · Cited by 610 · cited 1× · see citing works
“Notable examples of visual-language models include CLIP [139], ViLBERT [100], VisualBERT [95], SimVLM [181], BLIP-2 [94] and Flamingo [3], which were pre-trained on large-scale cross-modal datasets comprising images and languages.”
MethodVideo-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding
Zhang, H., Li, X. S., & Bing, L. (2023) · Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP) · doi:10.18653/v1/2023.emnlp-demo.49 · Cited by 608 · see citing works
Image Segmentation Using Text and Image Prompts
Lüddecke, T., & Ecker, A. S. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00695 · Cited by 592 · see citing works
PaLM-E: An Embodied Multimodal Language Model
Driess, D., Xia, F., Sajjadi, M. S. M., Lynch, C., Chowdhery, A., Ichter, B., Wahid, A., Tompson, J., Vuong, Q., Yu, T., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.03378 · Cited by 591 · see citing works
ChatGPT for Robotics: Design Principles and Model Abilities
Vemprala, S. H., Bonatti, R., Bucker, A., & Kapoor, A. (2024) · IEEE Access · doi:10.1109/access.2024.3387941 · Cited by 570 · see citing works
An Open Approach to Autonomous Vehicles
Kato, S., Takeuchi, E., Ishiguro, Y., Ninomiya, Y., Takeda, K., & Hamada, T. (2015) · IEEE Micro · doi:10.1109/mm.2015.133 · Cited by 563 · see citing works
Chameleon: Plug-and-Play Compositional Reasoning with Large Language ModelsPreprint
Lu, P., Peng, B., Cheng, H., Galley, M., Chang, K.-W., Wu, Y., Zhu, S.-C., & Gao, J.-F. (2023) · Neural Information Processing Systems · doi:10.48550/arxiv.2304.09842 · Cited by 534 · see citing works
Procedures as a representation for data in a computer program for understanding natural language
Winograd, T. (1971) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 521 · see citing works
SimVLM: Simple Visual Language Model Pretraining with Weak SupervisionPreprint
Wang, Z., Yu, J., Yu, A. W., Dai, Z., Tsvetkov, Y., & Cao, Y. J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.10904 · Cited by 517 · see citing works
Self-citationMilestones in Autonomous Driving and Intelligent Vehicles: Survey of Surveys
Chen, L. F., Li, Y., Huang, C., Li, B., Xing, Y., Tian, D., Li, L., Hu, Z., Na, X., Li, Z., et al. (2022) · IEEE Transactions on Intelligent Vehicles · doi:10.1109/tiv.2022.3223131 · Cited by 517 · see citing works
DriveGPT4: Interpretable End-to-End Autonomous Driving Via Large Language Model
Xu, Z., Zhang, Y., Xie, E., Zhen, Z., Guo, Y. F., Wong, K. K., Li, Z., & Zhao, H. (2024) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2024.3440097 · Cited by 494 · see citing works
HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face
Shen, Y., Song, K., Tan, X., Li, D., Lu, W., & Zhuang, Y. (2023) · arXiv · doi:10.48550/arxiv.2303.17580 · Cited by 477 · see citing works
MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models
Zhu, D., Chen, J., Shen, X., Li, X., & Elhoseiny, M. (2023) · arXiv · doi:10.48550/arxiv.2304.10592 · Cited by 469 · see citing works
Self-citationActBERT: Learning Global-Local Video-Text Representations
Zhu, L., & Yang, Y. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00877 · Cited by 414 · see citing works
Driver Activity Recognition for Intelligent Vehicles: A Deep Learning Approach
Xing, Y., Lv, C., Wang, H., Cao, D., Velenis, E., & Wang, F. (2019) · IEEE Transactions on Vehicular Technology · doi:10.1109/tvt.2019.2908425 · Cited by 404 · see citing works
Simple Open-Vocabulary Object Detection
Minderer, M., Gritsenko, A. A., Stone, A., Neumann, M., Weissenborn, D., Dosovitskiy, A., Mahendran, A., Arnab, A., Dehghani, M., Shen, Z., et al. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-20080-9_42 · Cited by 365 · see citing works
Textual Explanations for Self-Driving Vehicles
Kim, J., Rohrbach, A., Darrell, T., Canny, J., & Akata, Z. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01216-8_35 · Cited by 355 · see citing works
Recent Advancements in End-to-End Autonomous Driving Using Deep Learning: A Survey
Chib, P. S., & Singh, P. P. (2023) · IEEE Transactions on Intelligent Vehicles · doi:10.1109/tiv.2023.3318070 · Cited by 326 · see citing works
The Present Status of Automatic Translation of Languages
Bar‐Hillel, Y. (1960) · Advances in computers · doi:10.1016/s0065-2458(08)60607-5 · Cited by 318 · see citing works
A Survey on Cooperative Longitudinal Motion Control of Multiple Connected and Automated Vehicles
Wang, Z., Bian, Y., Shladover, S. E., Wu, G., Li, S. E., & Barth, M. J. (2020) · IEEE Intelligent Transportation Systems Magazine · doi:10.1109/mits.2019.2953562 · Cited by 316 · see citing works
Self-citationLanguage Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents
Huang, W., Abbeel, P., Pathak, D., & Mordatch, I. (2022) · arXiv · doi:10.48550/arxiv.2201.07207 · Cited by 304 · see citing works
A review on safety failures, security attacks, and available countermeasures for autonomous vehicles
Cui, J. T., Liew, L. S., Sabaliauskaite, G., & Zhou, F. (2018) · Ad Hoc Networks · doi:10.1016/j.adhoc.2018.12.006 · Cited by 297 · see citing works
Inner Monologue: Embodied Reasoning through Planning with Language Models
Huang, W., Xia, F., Xiao, T., Chan, H., Liang, J., Florence, P., Zeng, A., Tompson, J., Mordatch, I., Chebotar, Y., et al. (2022) · arXiv · doi:10.48550/arxiv.2207.05608 · Cited by 287 · see citing works
Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Grounding
Ku, A., Anderson, P., Patel, R., Ie, E., & Baldridge, J. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.356 · Cited by 274 · see citing works
Mapping Instructions and Visual Observations to Actions with Reinforcement Learning
Misra, D. K., Langford, J., & Artzi, Y. (2017) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1106 · Cited by 258 · see citing works
V2V4Real: A Real-World Large-Scale Dataset for Vehicle-to-Vehicle Cooperative Perception
Xu, R., Xia, X., Li, J., Li, H., Zhang, S., Tu, Z., Meng, Z., Xiang, H., Dong, X., Song, R., et al. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01318 · Cited by 254 · see citing works
A Survey on In-context Learning
Dong, Q., Li, L., Dai, D., Zheng, C., Ma, J., Li, R., Xia, H., Xu, J., Wu, Z., Liu, T., et al. (2023) · arXiv · doi:10.48550/arxiv.2301.00234 · Cited by 248 · see citing works
Self-citationSelf-Supervised MultiModal Versatile NetworksPreprint
Alayrac, J.-B., Recasens, A., Schneider, R. G., Arandjelović, R., Ramapuram, J., Fauw, J. D., Smaira, L., Dieleman, S., & Zisserman, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.16228 · Cited by 247 · see citing works
Computer Vision for Autonomous Vehicles: Problems, Datasets and State of the Art
Janai, J., Güney, F., Behl, A., & Geiger, A. (2020) · doi:10.1561/9781680836899 · Cited by 243 · see citing works
OccuSeg: Occupancy-Aware 3D Instance Segmentation
Han, L., Zheng, T., Xu, L., & Fang, L. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00301 · Cited by 239 · see citing works
Visual Programming: Compositional visual reasoning without training
Gupta, T., & Kembhavi, A. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01436 · Cited by 230 · see citing works
Deep multimodal learning for Audio-Visual Speech Recognition
Mroueh, Y., Marcheret, E., & Goel, V. (2015) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2015.7178347 · Cited by 226 · see citing works
Slaney, J., & Thiébaux, S. (2001) · Artificial Intelligence · doi:10.1016/s0004-3702(00)00079-5 · Cited by 226 · see citing works
CLIPort: What and Where Pathways for Robotic Manipulation
Shridhar, M., Manuelli, L., & Fox, D. (2021) · arXiv · doi:10.48550/arxiv.2109.12098 · Cited by 223 · see citing works
Voyager: An Open-Ended Embodied Agent with Large Language Models
Wang, G., Xie, Y., Jiang, Y., Mandlekar, A., Xiao, C., Zhu, Y., Fan, L., & Anandkumar, A. (2023) · arXiv · doi:10.48550/arxiv.2305.16291 · Cited by 222 · cited 3× · see citing works
“Recurrent Neural Networks (RNNs) [148] and Long Short-Term Memory (LSTM) networks [55] were used for various NLP tasks.”
BackgroundMethodTellex, S., Gopalan, N., Kress‐Gazit, H., & Matuszek, C. (2020) · Annual Review of Control Robotics and Autonomous Systems · doi:10.1146/annurev-control-101119-071628 · Cited by 221 · see citing works
A Survey of Reinforcement Learning Informed by Natural Language
Luketina, J., Nardelli, N., Farquhar, G., Foerster, J., Andreas, J., Grefenstette, E., Whiteson, S., & Rocktäschel, T. (2019) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2019/880 · Cited by 208 · see citing works
Masked Autoencoders that Listen
Huang, P.-Y., Xu, H., Li, J., Baevski, A., Auli, M., Galuba, W., Metze, F., & Feichtenhofer, C. (2022) · arXiv · doi:10.48550/arxiv.2207.06405 · Cited by 206 · see citing works
Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language
Zeng, A., Attarian, M., Ichter, B., Choromanski, K., Wong, A., Welker, S., Tombari, F., Purohit, A., Ryoo, M., Sindhwani, V., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.00598 · Cited by 205 · see citing works
Driving with LLMs: Fusing Object-Level Vector Modality for Explainable Autonomous Driving
Chen, L., Sinavski, O., Hünermann, J., Karnsund, A., Willmott, A. J., Birch, D., Maund, D., & Shotton, J. (2024) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra57147.2024.10611018 · Cited by 203 · see citing works
Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models
Wu, C., Yin, S., Qi, W., Wang, X., Tang, Z., & Duan, N. (2023) · arXiv · doi:10.48550/arxiv.2303.04671 · Cited by 197 · see citing works
Drive Like a Human: Rethinking Autonomous Driving with Large Language Models
Fu, D., Li, X., Wen, L., Dou, M., Cai, P., Shi, B., & Qiao, Y. (2024) · IEEE/CVF Winter Conference on Applications of Computer Vision Workshops (WACVW) · doi:10.1109/wacvw60836.2024.00102 · Cited by 195 · see citing works
Large-scale Text-to-Image Generation Models for Visual Artists’ Creative Works
Ko, H.-K., Park, G., Jeon, H., Jo, J., Kim, J., & Seo, J. (2023) · Proceedings of the 28th International Conference on Intelligent User Interfaces · doi:10.1145/3581641.3584078 · Cited by 189 · cited 1× · see citing works
“These works high-light the potential of multimodal learning in capturing complex relationships between different types of data, leading to richer and more accurate representations.”
BackgroundPrompt, Generate, Then Cache: Cascade of Foundation Models Makes Strong Few-Shot Learners
Zhang, R., Hu, X., Li, B., Huang, S., Deng, H., Qiao, Y., Gao, P., & Li, H. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01460 · Cited by 177 · see citing works
NuScenes-QA: A Multi-Modal Visual Question Answering Benchmark for Autonomous Driving Scenario
Qian, T., Chen, J., Zhuo, L., Jiao, Y., & Jiang, Y. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i5.28253 · Cited by 177 · see citing works
Self-citationBC-Z: Zero-Shot Task Generalization with Robotic Imitation Learning
Jang, E., Irpan, A., Khansari, M., Kappler, D., Ebert, F., Lynch, C., Levine, S., & Finn, C. (2022) · arXiv · doi:10.48550/arxiv.2202.02005 · Cited by 167 · see citing works
Ploennigs, J., & Berger, M. (2023) · AI in Civil Engineering · doi:10.1007/s43503-023-00018-y · Cited by 153 · cited 1× · see citing works
“They showcase the potential for us-ing multimodal models in many application scenarios such as healthcare [97], civil engineering [133], robotics [71] and, art [80].”
BackgroundExplainable AI: current status and future directionsPreprint
Gohel, P., Singh, P., & Mohanty, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.07045 · Cited by 148 · see citing works
EEG-Based Attention Tracking During Distracted Driving
Wang, Y., Jung, T., & Lin, C. (2015) · IEEE Transactions on Neural Systems and Rehabilitation Engineering · doi:10.1109/tnsre.2015.2415520 · Cited by 147 · see citing works
Talk2Car: Taking Control of Your Self-Driving Car
Deruyttere, T., Vandenhende, S., Grujicic, D., Gool, L. V., & Moens, M.-F. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1215 · Cited by 135 · see citing works
Cui, C., Ma, Y., Cao, X., Ye, W., & Wang, Z. (2024) · IEEE/CVF Winter Conference on Applications of Computer Vision Workshops (WACVW) · doi:10.1109/wacvw60836.2024.00101 · Cited by 132 · see citing works
Self-citationGenerating Adversarial Driving Scenarios in High-Fidelity Simulators
Abeysirigoonawardena, Y., Shkurti, F., & Dudek, G. (2019) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2019.8793740 · Cited by 130 · see citing works
Video Prediction Recalling Long-term Motion Context via Memory Alignment Learning
Lee, S., Kim, H. G., Hwi Choi, D., Kim, H.-I., & Ro, Y. M. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00307 · Cited by 129 · see citing works
DRAMA: Joint Risk Localization and Captioning in Driving
Malla, S., Choi, C., Dwivedi, I., Choi, J. H., & Li, J. (2023) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv56688.2023.00110 · Cited by 128 · see citing works
Receive, Reason, and React: Drive as You Say, With Large Language Models in Autonomous Vehicles
Cui, C., Ma, Y., Cao, X., Ye, W., & Wang, Z. (2024) · IEEE Intelligent Transportation Systems Magazine · doi:10.1109/mits.2024.3381793 · Cited by 121 · see citing works
Self-citationAutonomous land vehicle project at CMU
Kanade, T., Thorpe, C. E., & Whittaker, W. (1986) · doi:10.1145/324634.325197 · Cited by 118 · see citing works
Toward understanding natural language directions
Kollar, T., Tellex, S., Roy, D., et al. (2010) · ACM/IEEE International Conference on Human-Robot Interaction (HRI) · doi:10.1109/hri.2010.5453186 · Cited by 112 · see citing works
Language Conditioned Imitation Learning Over Unstructured Data
Lynch*, C., & Sermanet*, P. (2021) · Robotics: Science and Systems XVII · doi:10.15607/rss.2021.xvii.047 · Cited by 110 · see citing works
Language as an Abstraction for Hierarchical Deep Reinforcement Learning
Jiang, Y., Gu, S., Murphy, K., & Finn, C. (2019) · arXiv · doi:10.48550/arxiv.1906.07343 · Cited by 109 · see citing works
MotionLM: Multi-Agent Motion Forecasting as Language Modeling
Seff, A., Cera, B., Chen, D., Ng, M., Zhou, A., Nayakanti, N., Refaat, K. S., Al‐Rfou, R., & Sapp, B. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00788 · Cited by 108 · see citing works
Multimodal Safety-Critical Scenarios Generation for Decision-Making Algorithms Evaluation
Ding, W., Chen, B., Li, B., Eun, K. J., & Zhao, D. X. (2021) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2021.3058873 · Cited by 98 · see citing works
Online Prediction of Driver Distraction Based on Brain Activity Patterns
Wang, S., Zhang, Y., Wu, C., Darvas, F., & Chaovalitwongse, W. A. (2014) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2014.2330979 · Cited by 88 · see citing works
Correcting Robot Plans with Natural Language Feedback
Sharma, P., Sundaralingam, B., Blukis, V., Paxton, C., Hermans, T., Torralba, A. B., Andreas, J., & Fox, D. (2022) · Robotics: Science and Systems · doi:10.15607/rss.2022.xviii.065 · Cited by 83 · see citing works
OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization
Iyer, S., Lin, X. V., Pasunuru, R., Mihaylov, T., Simig, D., Yu, P., Shuster, K., Wang, T., Liu, Q., Koura, P. S., et al. (2022) · arXiv · doi:10.48550/arxiv.2212.12017 · Cited by 81 · see citing works
VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models
Huang, W., Wang, C., Zhang, R., Li, Y., Wu, J., & Fei-Fei, L. (2023) · arXiv · doi:10.48550/arxiv.2307.05973 · Cited by 81 · see citing works
Talk2BEV: Language-enhanced Bird’s-eye View Maps for Autonomous Driving
Choudhary, T., Dewangan, V., Chandhok, S., Priyadarshan, S., Jain, A., Singh, A. K., Srivastava, S., Murthy, J. K., & Krishna, K. M. (2024) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra57147.2024.10611485 · Cited by 79 · see citing works
Cai, P., Wang, H., Sun, Y., & Liu, M. (2022) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2022.3184990 · Cited by 78 · see citing works
SayNav: Grounding Large Language Models for Dynamic Planning to Navigation in New Environments
Rajvanshi, A., Sikka, K., Lin, X., Lee, B., Chiu, H.-P., & Velasquez, A. (2024) · Proceedings of the International Conference on Automated Planning and Scheduling · doi:10.1609/icaps.v34i1.31506 · Cited by 78 · see citing works
DiLu: A Knowledge-Driven Approach to Autonomous Driving with Large Language ModelsPreprint
Wen, L., Fu, D., Li, X., Cai, X., Ma, T., Cai, P., Dou, M., Shi, B., He, L., & Qiao, Y. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2309.16292 · Cited by 76 · see citing works
SayNav: Grounding Large Language Models for Dynamic Planning to Navigation in New Environments
Rajvanshi, A., Sikka, K., Lin, X., Lee, B., Chiu, H.-P., & Velasquez, A. (2023) · arXiv · doi:10.48550/arxiv.2309.04077 · Cited by 67 · see citing works
Automatic Language Translation
Oettinger, A. G. (1960) · Harvard University Press eBooks · doi:10.4159/harvard.9780674421974 · Cited by 66 · see citing works
PixL2R: Guiding Reinforcement Learning Using Natural Language by Mapping Pixels to RewardsPreprint
Goyal, P., Niekum, S., & Mooney, R. (2020) · Conference on Robot Learning · doi:10.48550/arxiv.2007.15543 · Cited by 63 · see citing works
Video Action Recognition for Lane-Change Classification and Prediction of Surrounding Vehicles
Biparva, M., Llorca, D. F., Izquierdo, R., & Tsotsos, J. K. (2022) · IEEE Transactions on Intelligent Vehicles · doi:10.1109/tiv.2022.3164507 · Cited by 61 · see citing works
Chain-of-Thought Reasoning in Tabular Language Models
Zheng, M., Yang, H., Jiang, W., Lin, Z., Lyu, Y., She, Q., & Wang, W. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.734 · Cited by 50 · see citing works
ERNIE-GeoL: A Geography-and-Language Pre-trained Model and its Applications in Baidu Maps
Huang, J., Wang, H., Sun, Y., Shi, Y., Huang, Z., Zhuo, A., & Feng, S. (2022) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3534678.3539021 · Cited by 49 · see citing works
LanguageMPC: Large Language Models as Decision Makers for Autonomous DrivingPreprint
Sha, H., Mu, Y., Jiang, Y., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.03026 · Cited by 49 · see citing works
VLAAD: Vision and Language Assistant for Autonomous Driving
Park, S., Lee, M., Kang, J., Choi, H., Park, Y., Cho, J., Lee, A., & Kim, D. (2024) · 2024 IEEE/CVF Winter Conference on Applications of Computer Vision Workshops (WACVW) · doi:10.1109/wacvw60836.2024.00107 · Cited by 49 · see citing works
Waymax: An Accelerated, Data-Driven Simulator for Large-Scale Autonomous Driving Research
Gulino, C., Fu, J., Luo, W., Tucker, G., Bronstein, E., Lu, Y., Harb, J., Pan, X., Wang, Y., Chen, X., et al. (2023) · arXiv · doi:10.48550/arxiv.2310.08710 · Cited by 48 · see citing works
Self-citationAudiovisual Masked Autoencoders
Georgescu, M.-I., Fonseca, E., Ionescu, R. T., Lučić, M., Schmid, C., & Arnab, A. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01479 · Cited by 48 · see citing works
Enhancing Driver Distraction Recognition Using Generative Adversarial Networks
Ou, C., & Karray, F. O. (2019) · IEEE Transactions on Intelligent Vehicles · doi:10.1109/tiv.2019.2960930 · Cited by 46 · see citing works
IEEE Intelligent Vehicles Symposium 2012
(2012) · IEEE Intelligent Transportation Systems Magazine · doi:10.1109/mits.2012.2189995 · Cited by 46 · see citing works
Zero-Shot Learning for IMU-Based Activity Recognition Using Video Embeddings
Tong, C., Ge, J., & Lane, N. D. (2021) · Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies · doi:10.1145/3494995 · Cited by 41 · see citing works
AggPose: Deep Aggregation Vision Transformer for Infant Pose Estimation
Cao, X., Li, X., Ma, L., Huang, Y., Feng, X. J., Chen, Z., Zeng, H., & Cao, J. (2022) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2022/700 · Cited by 40 · see citing works
Self-citationKamath, A., Anderson, P., Wang, S., Koh, J. Y., Ku, A., Waters, A., Yang, Y., Baldridge, J., & Parekh, Z. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01041 · Cited by 39 · see citing works
GPT-Driver: Learning to Drive with GPTPreprint
Mao, J., Qian, Y., Ye, J., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.01415 · Cited by 39 · see citing works
Liu, W., Muramatsu, S., & Okubo, Y. (2018) · International Conference on Intelligent Transport Systems Telecommunications (ITST) · doi:10.1109/itst.2018.8566704 · Cited by 37 · see citing works
Zhong, J., Li, M., Chen, Y., Wei, Z., Yang, F., & Shen, H. (2024) · IEEE/CVF Winter Conference on Applications of Computer Vision Workshops (WACVW) · doi:10.1109/wacvw60836.2024.00131 · Cited by 37 · see citing works
In the Eye of Transformer: Global–Local Correlation for Egocentric Gaze Estimation and Beyond
Lai, B., Liu, M., Ryan, F., & Rehg, J. M. (2023) · International Journal of Computer Vision · doi:10.1007/s11263-023-01879-7 · Cited by 34 · see citing works
Vitasd: Robust Vision Transformer Baselines for Autism Spectrum Disorder Facial Diagnosis
Cao, X., Ye, W., Sizikova, E., Bai, X., Coffee, M. P., Zeng, H., & Cao, J. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10094684 · Cited by 31 · see citing works
Self-citationPrevent the Language Model from being Overconfident in Neural Machine Translation
Miao, M., Meng, F., Liu, Y., Zhou, X., & Zhou, J. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2021.acl-long.268 · Cited by 30 · see citing works
Human-Centric Autonomous Systems With LLMs for User Command Reasoning
Yang, Y., Zhang, Q., Li, C., Marta, D., Batool, N., & Folkesson, J. B. (2024) · IEEE/CVF Winter Conference on Applications of Computer Vision Workshops (WACVW) · doi:10.1109/wacvw60836.2024.00108 · Cited by 30 · see citing works
Jin, Y., Yang, R., Yi, Z., Shen, X., Peng, H., Liu, X., Qin, J., Li, J., Xie, J., Gao, P., et al. (2023) · arXiv · doi:10.48550/arxiv.2309.13193 · Cited by 29 · see citing works
GAIA-1: A Generative World Model for Autonomous Driving
Hu, A., Russell, L., Yeo, H., Murez, Z., Fedoseev, G., Kendall, A., Shotton, J., & Corrado, G. (2023) · arXiv · doi:10.48550/arxiv.2309.17080 · Cited by 28 · see citing works
Talk to the Vehicle: Language Conditioned Autonomous Navigation of Self Driving Cars
Sriram, N. N., Maniar, T., Jayaganesh, K., Gandhi, V., Bhowmick, B., & Krishna, K. M. (2019) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros40897.2019.8967929 · Cited by 27 · see citing works
Language Prompt for Autonomous DrivingPreprint
Wu, D., Han, W., Liu, Y., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2309.04379 · Cited by 22 · see citing works
HiLM-D: Enhancing MLLMs with Multi-Scale High-Resolution Details for Autonomous DrivingPreprint
Ding, X., Han, J., Xu, H., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2309.05186 · Cited by 22 · see citing works
Inoue, Y., Yada, Y., Tanahashi, K., & Yamaguchi, Y. (2024) · IEEE/CVF Winter Conference on Applications of Computer Vision Workshops (WACVW) · doi:10.1109/wacvw60836.2024.00104 · Cited by 22 · see citing works
THMA: Tencent HD Map AI System for Creating HD Map Annotations
Kun, T., Cao, X., Cao, Z., Zhou, T., Li, E., Liu, A., Zou, S., Liu, C., Mei, S., Sizikova, E., et al. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i13.26848 · Cited by 20 · see citing works
Self-citationM2DAR: Multi-View Multi-Scale Driver Action Recognition with Vision Transformer
Ma, Y., Yuan, L., Abdelraouf, A., Han, K., Gupta, R., Li, Z., & Wang, Z.-R. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw59228.2023.00557 · Cited by 20 · see citing works
Self-citationCaption Anything: Interactive Image Description with Diverse Multimodal ControlsPreprint
Wang, T., Zhang, J., Fei, J., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.02677 · Cited by 18 · see citing works
Man-to-machine communication and automatic code translation
Holt, A. W., & Turanski, W. J. (1960) · doi:10.1145/1460361.1460405 · Cited by 13 · see citing works
Keysan, A., Look, A., Kosman, E., Gürsun, G., Wagner, J., Yao, Y., & Rakitsch, B. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2309.05282 · Cited by 13 · see citing works
Ma, Y., Ye, W., Cao, X., et al. (2023) · IEEE International Conference on Intelligent Transportation Systems (ITSC) · doi:10.1109/itsc57777.2023.10421798 · Cited by 13 · see citing works
Self-citationNi, Y. S., Wang, S., Xin, L., Meng, Y., Yin, J., & Sun, J. (2020) · IEEE International Conference on Intelligent Transportation Systems (ITSC) · doi:10.1109/itsc45102.2020.9294501 · Cited by 12 · see citing works
Learning Interactive Real-World Simulators
Yang, S., Du, Y., Ghasemipour, K., Tompson, J., Kaelbling, L., Schuurmans, D., & Abbeel, P. (2023) · arXiv · doi:10.48550/arxiv.2310.06114 · Cited by 12 · see citing works
Navigation with Large Language Models: Semantic Guesswork as a Heuristic for PlanningPreprint
Shah, D., Equi, M., Osiński, B., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.10103 · Cited by 12 · see citing works
Visual Chain of Thought: Bridging Logical Gaps with Multimodal InfillingsPreprint
Rose, D., Himakunthala, V., Ouyang, A., He, R., Mei, A., Lu, Y., Saxon, M., Sonar, C., Mirza, D., & Wang, W. Y. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.02317 · Cited by 10 · see citing works
The Waymo Open Sim Agents ChallengePreprint
Montali, N., Lambert, J., Mougin, P., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.12032 · Cited by 9 · see citing works
Proto-CLIP: Vision-Language Prototypical Network for Few-Shot LearningPreprint
Jishnu Jaykumar, P., Palanisamy, K., Chao, Y.-W., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2307.03073 · Cited by 9 · see citing works
Chain of Thought Prompt Tuning in Vision Language ModelsPreprint
Ge, J., Luo, H., Qian, S., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2304.07919 · Cited by 8 · see citing works
LIP-Loc: LiDAR Image Pretraining for Cross-Modal LocalizationPreprint
Puligilla, S. S., Omama, M., Zaidi, H., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2312.16648 · Cited by 7 · see citing works
Listen to Look Into the Future: Audio-Visual Egocentric Gaze Anticipation
Lai, B., Ryan, F., Jia, W., Liu, M., & Rehg, J. M. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72673-6_11 · Cited by 7 · see citing works
High‐definition map automatic annotation system based on active learning
Zheng, C., Cao, X., Tang, K., Cao, Z., Sizikova, E., Zhou, T. H., Li, E., Liu, A., Zou, S., Yan, X., et al. (2023) · AI Magazine · doi:10.1002/aaai.12139 · Cited by 5 · see citing works
Self-citationA Game of Bundle Adjustment - Learning Efficient Convergence
Belder, A., Vivanti, R., & Tal, A. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00774 · Cited by 5 · see citing works
Latency Driven Spatially Sparse Optimization for Multi-Branch CNNs for Semantic Segmentation
Zampokas, G., Bouganis, C.-S., & Tzovaras, D. K. (2024) · IEEE/CVF Winter Conference on Applications of Computer Vision Workshops (WACVW) · doi:10.1109/wacvw60836.2024.00105 · Cited by 5 · see citing works
ALT-Pilot: Autonomous navigation with Language augmented Topometric mapsPreprint
Omama, M., Inani, P., Paul, P., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.02324 · Cited by 4 · see citing works
Nabhan, M. (2020) · HAL (Le Centre pour la Communication Scientifique Directe) · Cited by 2 · see citing works
PIE: Simulating Disease Progression via Progressive Image EditingPreprint
Liang, K., Cao, X., Liao, K.-D., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2309.11745 · Cited by 1 · see citing works
Self-citationBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805
Flagged first, then by citations
Cited by
Sorted by citations
Generative AI in cybersecurity: A comprehensive review of LLM applications and vulnerabilities
Ferrag, M. A., Alwahedi, F., Battah, A., Cherif, B., Mechri, A., Tihanyi, N., Bisztray, T., & Debbah, M. (2025) · Internet of Things and Cyber-Physical Systems · doi:10.1016/j.iotcps.2025.01.001 · Cited by 148 · see citing works
A Survey on the Memory Mechanism of Large Language Model-based Agents
Zhang, Z., Dai, Q., Bo, X., Ma, C., Li, R., Chen, X., Zhu, J., Dong, Z., & Wen, J.-R. (2025) · ACM Transactions on Information Systems · doi:10.1145/3748302 · Cited by 118 · see citing works
McIntosh, T. R., Susnjak, T., Liu, T., Watters, P., Xu, D., Liu, D., & Halgamuge, M. N. (2025) · Technologies · doi:10.3390/technologies13020051 · Cited by 102 · see citing works
“For autonomous vehicles, multimodality can enhance safety and navigation by synthesizing data from various sensors, including visual, radar, and Light Detection and Ranging (LIDAR) [126], [125], [127].”
BackgroundGenerative AI for self-adaptive systems: State of the art and research roadmap
Li, J., Zhang, M., Li, N., Weyns, D., Jin, Z., & Tei, K. (2024) · ACM Transactions on Autonomous and Adaptive Systems · doi:10.1145/3686803 · Cited by 77 · see citing works
VLAAD: Vision and Language Assistant for Autonomous Driving
Park, S., Lee, M., Kang, J., Choi, H., Park, Y., Cho, J., Lee, A., & Kim, D. (2024) · 2024 IEEE/CVF Winter Conference on Applications of Computer Vision Workshops (WACVW) · doi:10.1109/wacvw60836.2024.00107 · Cited by 49 · see citing works
Westhäußer, R., Minker, W., & Zepf, S. (2026) · Proceedings of the 31st International Conference on Intelligent User Interfaces · doi:10.1145/3742413.3789222 · Cited by 1 · see citing works
“These advancements led to a growing area of research focused on utilizing LLMs to develop AI agents with human-like capabilities, such as decision-making and planning [10, 52].”
Stop Overthinking: A Survey on Efficient Reasoning for Large Language Models
Sui, Y., Chuang, Y.-N., Wang, G., Zhang, J., Zhang, T., Yuan, J., Liu, H., Wen, A., Zhong, S., Zou, N., et al. (2025) · arXiv · Cited by 2 · see citing works
“Efficient reasoning LLMs are able to greatly improve autonomic driving [22] by helping them understand large amounts of sensor data in a human-like way.”
- Authors
- Cui, C., Ma, Y., Cao, X., Ye, W., Zhou, Y., Liang, K., Chen, J., Lu, J., Yang, Z., Liao, K.-D., Gao, T., Li, E., Tang, K., Cao, Z., Zhou, T., Liu, A., Yan, X., Mei, S., Cao, J., Wang, Z., & Zheng, C.
- Affiliations
- Purdue University West Lafayette (US)
- University of Illinois Urbana-Champaign (US)
- University of Virginia (US)
- New York University (US)
- SambaNova Systems, Inc,Palo Alto,CA,USA,94303
- Objective, Inc,San Francisco,CA,USA,94110
- Tencent (China) (CN)
- PediaMed AI,Shenzhen,China
- ORCID
- Venue
- 2024 IEEE/CVF Winter Conference on Applications of Computer Vision Workshops (WACVW)
- Publisher
- IEEE
- Published
- 2024-01-01
- Pages
- 958-979
- OpenAlex
- W4394862623
- Semantic Scholar
- 451539c0d0f5f5785ff58d09ca5e67a5f129f9de
- Cited by
- 369
- References
- 179
- Citations by year
- 2026: 83 · 2025: 165 · 2024: 119 · 2023: 2
- FWCI
- 75.56
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE/CVF Winter Conference on Applications of Computer Vision Workshops (WACVW) (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by-nc-sa)
- arXiv (Cornell University)
- Funders
- Purdue University
- Tencent
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Engineering › Automotive Engineering › Autonomous Vehicle Technology and Safety
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- multimodal large language models, autonomous driving, vision foundation models, driving system, transportation, multimodal dataset, benchmark
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review