Vision-Language Models for Autonomous Driving: CLIP-Based Dynamic Scene Understanding
Open accessThe DOI matches its Crossref record; 29 of the 29 references it cites were found in a registry or an index.
Abstract
Scene understanding is essential for enhancing driver safety, generating human-centric explanations for Automated Vehicle (AV) decisions, and leveraging Artificial Intelligence (AI) for retrospective driving video analysis. This study developed a dynamic scene retrieval system using Contrastive Language–Image Pretraining (CLIP) models, which can be optimized for real-time deployment on edge devices. The proposed system outperforms state-of-the-art in-context learning methods, including the zero-shot capabilities of GPT-4o, particularly in complex scenarios. By conducting frame-level analyses on the Honda Scenes Dataset, which contains a collection of about 80 h of annotated driving videos capturing diverse real-world road and weather conditions, our study highlights the robustness of CLIP models in learning visual concepts from natural language supervision. The results also showed that fine-tuning the CLIP models, such as ViT-L/14 (Vision Transformer) and ViT-B/32, significantly improved scene classification, achieving a top F1-score of 91.1%. These results demonstrate the ability of the system to deliver rapid and precise scene recognition, which can be used to meet the critical requirements of advanced driver assistance systems (ADASs). This study shows the potential of CLIP models to provide scalable and efficient frameworks for dynamic scene understanding and classification. Furthermore, this work lays the groundwork for advanced autonomous vehicle technologies by fostering a deeper understanding of driver behavior, road conditions, and safety-critical scenarios, marking a significant step toward smarter, safer, and more context-aware autonomous driving systems.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“These capabilities are increasingly vital for agents operating in domains such as robotics [88], autonomous vehicles [89,90], and visual content moderation [91,92].”
From AI Agents vs. Agentic AI: A Conceptual taxonomy, applications and challenges (2026) · cites this paper as [89]
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Learning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293
Saito, T., & Rehmsmeier, M. (2015) · PLOS ONE · doi:10.1371/journal.pone.0118432 · Cited by 5,309
Vision Language Models in Autonomous Driving: A Survey and Outlook
Zhou, X., Liu, M., Yurtsever, E., Žagar, B. L., Zimmer, W. M. X., Cao, H., & Knoll, A. (2024) · IEEE Transactions on Intelligent Vehicles · doi:10.1109/tiv.2024.3402136 · Cited by 401
Toward Driving Scene Understanding: A Dataset for Learning Driver Behavior and Causal Reasoning
Ramanishka, V., Chen, Y.-T., Misu, T., & Saenko, K. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00803 · Cited by 314
Analysis of the Performance Impact of Fine-Tuned Machine Learning Model for Phishing URL Detection
Samad, S. R. A., Balasubaramanian, S., Kaabi, A. S. R. A., Sharma, B., Chowdhury, S., Mehbodniya, A., Webber, J. L., & Bostani, A. (2023) · Electronics · doi:10.3390/electronics12071642 · Cited by 137
An Improved Deep Network-Based Scene Classification Method for Self-Driving Cars
Ni, J., Shen, K., Chen, Y., Cao, W., & Yang, S. X. (2022) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2022.3146923 · Cited by 127
Ping, P., Qin, W., Xu, Y., Miyajima, C., & Takeda, K. (2019) · IEEE Access · doi:10.1109/access.2019.2920489 · Cited by 119
Tami, M. A., Ashqar, H. I., Elhenawy, M. M. Z., Glaser, S., & Rakotonirainy, A. (2024) · Vehicles · doi:10.3390/vehicles6030074 · Cited by 100
Self-citationA Survey on Deep Learning Based Approaches for Scene Understanding in Autonomous Driving
Guo, Z., Huang, Y., Hu, X., Wei, H., & Zhao, B. (2021) · Electronics · doi:10.3390/electronics10040471 · Cited by 85
Masello, L., Sheehan, B., Castignani, G., Shannon, D., & Murphy, F. (2023) · Accident Analysis & Prevention · doi:10.1016/j.aap.2023.106969 · Cited by 64
Ashqar, H. I., Alhadidi, T. I., Elhenawy, M. M. Z., & Khanfar, N. O. (2024) · Automation · doi:10.3390/automation5040029 · Cited by 57
Self-citationSceneCut: Joint Geometric and Object Segmentation for Indoor Scenes
Pham, T. T., Do, T.-T., Sünderhauf, N., & Reid, I. D. (2018) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2018.8461108 · Cited by 49
Exploration of the Relationships between Hazard Perception and Eye Movement for Young Drivers
Yang, J., Liu, C., Chu, P., Wen, X., & Zhang, Y. (2021) · Journal of Advanced Transportation · doi:10.1155/2021/6642999 · Cited by 34
Research on Road Scene Understanding of Autonomous Vehicles Based on Multi-Task Learning
Guo, J., Wang, J., Wang, H., Xiao, B., He, Z., & Li, L. (2023) · Sensors · doi:10.3390/s23136238 · Cited by 32
Towards Traffic Scene Description: The Semantic Scene Graph
Zipfl, M., & Zöllner, J. M. (2022) · IEEE International Conference on Intelligent Transportation Systems (ITSC) · doi:10.1109/itsc55140.2022.9922469 · Cited by 30
Khanfar, N. O., Elhenawy, M. M. Z., Ashqar, H. I., Hussain, Q., & Alhajyaseen, W. (2022) · International Journal of Injury Control and Safety Promotion · doi:10.1080/17457300.2022.2103573 · Cited by 28
Self-citationSustainable CNN for Robotic: An Offloading Game in the 3D Vision Computation
Li, L., Ota, K., & Dong, M. (2018) · IEEE Transactions on Sustainable Computing · doi:10.1109/tsusc.2018.2844348 · Cited by 15
Orlovska, J., Novakazi, F., Wickman, C., & Söderberg, R. C. (2019) · Proceedings of the ... International Conference on Engineering Design · doi:10.1017/dsi.2019.186 · Cited by 13
Supervised Contrastive Learning for Detecting Anomalous Driving Behaviours from Multimodal Videos
Khan, S. S., Shen, Z., Sun, H., Patel, A., & Abedi, A. (2022) · Canadian Conference on Computer and Robot Vision · doi:10.1109/crv55824.2022.00011 · Cited by 13
DeGuzman, C. A., Ayas, S., & Donmez, B. (2023) · Transportation Research Record Journal of the Transportation Research Board · doi:10.1177/03611981221151022 · Cited by 12
Improving Minority Class Recall through a Novel Cluster-Based Oversampling Technique
Prexawanprasut, T., & Banditwattanawong, T. (2024) · Informatics · doi:10.3390/informatics11020035 · Cited by 12
Pradhan, A. K., Hungund, A. P., Pai, G., & Pamarthi, J. (2023) · Traffic Safety Research · doi:10.55329/udqk4583 · Cited by 11
Advanced Driver Assistance Systems (ADAS): Who’s Driving What and What’s Driving Use?
Palac, D., Scully, I. D., Jonas, R. K., Campbell, J. L., Young, D., & Cades, D. M. (2021) · Proceedings of the Human Factors and Ergonomics Society Annual Meeting · doi:10.1177/1071181321651234 · Cited by 10
A 3D LiDAR odometry for UGVs using coarse-to-fine deep scene flow estimation
Li, C., Yan, F., Wang, S., & Zhuang, Y. (2022) · Transactions of the Institute of Measurement and Control · doi:10.1177/01423312221105165 · Cited by 9
Unbiased 3D Semantic Scene Graph Prediction in Point Cloud Using Deep Learning
Han, C., Li, H., Xu, J. G., Dong, B., Wang, Y., Zhou, X., & Zhao, S. (2023) · Applied Sciences · doi:10.3390/app13095657 · Cited by 8
Holst, D., Schoepflin, D., & Schüppstuhl, T. (2022) · Lecture notes in mechanical engineering · doi:10.1007/978-3-031-18326-3_28 · Cited by 5
Night-time lane positioning based on camera and LiDAR fusion
Niu, S., Ma, Y., & Wei, C. (2023) · International Conference on Traffic Engineering and Transportation System (ICTETS) · doi:10.1117/12.2668596 · Cited by 2
Pothole and Wet Surface Detection Using Pretrained Models and ML Techniques
Vernekar, P., Singh, A., & Patil, K. (2023) · International Journal for Research in Applied Science and Engineering Technology · doi:10.22214/ijraset.2023.49489 · Cited by 1
Research works printed but not linked: 1
- Dosovitskiy, A. (2020). An image is worth 16x16 words: Transformers for image recognition at scale. arXiv.
Cited by
AI Agents vs. Agentic AI: A Conceptual taxonomy, applications and challenges
Sapkota, R., Roumeliotis, K. I., & Karkee, M. (2026) · Information Fusion · doi:10.1016/j.inffus.2025.103599 · Cited by 671
AI Agents vs. Agentic AI: A Conceptual Taxonomy, Applications and Challenges
Sapkota, R., Roumeliotis, K. I., & Karkee, M. (2025) · arXiv · doi:10.48550/arxiv.2505.10468 · Cited by 295
AI Agents vs. Agentic AI: A Conceptual Taxonomy, Applications and Challenges
Sapkota, R., Roumeliotis, K. I., & Karkee, M. (2025) · SuperIntelligence - Robotics - Safety & Alignment · doi:10.70777/si.v2i3.15161 · Cited by 186
Large Language Models (LLMs) as Traffic Control Systems at Urban Intersections: A New Paradigm
Masri, S., Ashqar, H. I., & Elhenawy, M. M. Z. (2025) · Vehicles · doi:10.3390/vehicles7010011 · Cited by 68
Self-citationHussain, K., Moreira, C., Pereira, J. M., Jardim, S., & Jorge, J. A. (2025) · Smart Cities · doi:10.3390/smartcities8030079 · Cited by 16
Metadata
- Authors
- Elhenawy, M., Ashqar, H. I., Rakotonirainy, A., Alhadidi, T. I., Jaber, A., & Tami, M. A.
- Affiliations
- Queensland University of Technology (AU)
- Arab American University (PS)
- Al-Ahliyya Amman University (JO)
- Budapest University of Technology and Economics (HU)
- Corresponding author
- Jaber, Ahmed
- ORCID
- Venue
- Electronics
- Publisher
- MDPI AG
- Published
- 2025-03-24
- Volume
- 14
- Issue
- 7
- Pages
- 1282
- ISSN
- 2079-9292 (ISSN-L)
- OpenAlex
- W4408781028
- Cited by
- 37
- References
- 29
- Citations by year
- 2026: 21 · 2025: 16
- FWCI
- 13.08
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Electronics (publishedVersion, cc-by)
- Funders
- Australian Government (DP220102598)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Engineering › Automotive Engineering › Autonomous Vehicle Technology and Safety
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- autonomous driving, scene understanding, CLIP, scene classification, edge deployment, in-context learning, zero-shot learning, driver assistance systems, driver behavior
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.