Multimodal end-to-end autonomous driving
Open accessAbstract
A crucial component of an autonomous vehicle (AV) is the artificial intelligence (AI) is able to drive towards a desired destination. Today, there are different paradigms addressing the development of AI drivers. On the one hand, we find modular pipelines, which divide the driving task into sub-tasks such as perception and maneuver planning and control. On the other hand, we find end-to-end driving approaches that try to learn a direct mapping from input raw sensor data to vehicle control signals. The later are relatively less studied, but are gaining popularity since they are less demanding in terms of sensor data annotation. This paper focuses on end-to-end autonomous driving. So far, most proposals relying on this paradigm assume RGB images as input sensor data. However, AVs will not be equipped only with cameras, but also with active sensors providing accurate depth information (e.g., LiDARs). Accordingly, this paper analyses whether combining RGB and depth modalities,i.e.using RGBD data, produces better end-to-end AI drivers than relying on a single modality. We consider multimodality based on early, mid and late fusion schemes, both in multisensory and single-sensor (monocular depth estimation) settings. Using the CARLA simulator and conditional imitation learning (CIL), we show how, indeed, early fusion multimodality outperforms single-modality.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
A common attention mechanism similar to transformer structure has been proposed to improve applicability of different modal data and capture local feature correlations.
“A common attention mechanism similar to transformer structure has been proposed to improve the applicability of different modal data and capture local feature correlations [43].”
From Multimodal fusion and vision–language models: A survey for robot vision (2026) · cites this paper as [43]
The Fusion schemes section (p. 4) describes CNN-based early, mid and late fusion of RGB and depth data. Searches for a transformer-like attention mechanism and local feature correlations found no passage addressing those parts of the claim in our copy. (LLM, conf=0.76)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2015) · arXiv · doi:10.48550/arxiv.1506.01497 · Cited by 70,812 · see citing works
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 56,035 · see citing works
Fully convolutional networks for semantic segmentation
Long, J., Shelhamer, E., & Darrell, T. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298965 · Cited by 38,834 · see citing works
(2016) · Cited by 31,590 · see citing works
Human-level control through deep reinforcement learning
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015) · Nature · doi:10.1038/nature14236 · Cited by 30,357 · see citing works
YOLO9000: Better, Faster, Stronger
Redmon, J., & Farhadi, A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.690 · Cited by 19,892 · see citing works
Are we ready for autonomous driving? The KITTI vision benchmark suite
Geiger, A., Lenz, P., & Urtasun, R. (2012) · 2012 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2012.6248074 · Cited by 14,897 · see citing works
The Cityscapes Dataset for Semantic Urban Scene Understanding
Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., & Schiele, B. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.350 · Cited by 12,433 · see citing works
Simple online and realtime tracking with a deep association metric
Wojke, N., Bewley, A., & Paulus, D. (2017) · 2017 IEEE International Conference on Image Processing (ICIP) · doi:10.1109/icip.2017.8296962 · Cited by 5,286 · see citing works
VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detection
Zhou, Y., & Tuzel, O. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00472 · Cited by 4,813 · see citing works
PointPillars: Fast Encoders for Object Detection From Point Clouds
Lang, A. H., Vora, S., Caesar, H., Zhou, L., Yang, J., & Beijbom, O. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.01298 · Cited by 4,673 · see citing works
Learning Deconvolution Network for Semantic Segmentation
Noh, H., Hong, S., & Han, B. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.178 · Cited by 4,422 · see citing works
CARLA: An open urban driving simulator
Dosovitskiy, A., Ros, G., Codevilla, F., Lopez, A., & Koltun, V. (2017) · arXiv · doi:10.48550/arxiv.1711.03938 · Cited by 3,718 · see citing works
Self-citationMulti-task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics
Cipolla, R., Gal, Y., & Kendall, A. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00781 · Cited by 3,657 · see citing works
Multi-view 3D Object Detection Network for Autonomous Driving
Chen, X., Ma, H., Wan, J., Li, B., & Xia, T. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.691 · Cited by 3,450 · see citing works
Unsupervised Monocular Depth Estimation with Left-Right Consistency
Godard, C., Aodha, O. M., & Brostow, G. J. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.699 · Cited by 3,270 · see citing works
A survey of robot learning from demonstration
Argall, B. D., Chernova, S., Veloso, M., & Browning, B. (2009) · Robotics and Autonomous Systems · doi:10.1016/j.robot.2008.10.024 · Cited by 3,227 · see citing works
End to End Learning for Self-Driving Cars
Bojarski, M., Del Testa, D., Dworakowski, D., Firner, B., Flepp, B., Goyal, P., Jackel, L. D., Monfort, M., Muller, U., Zhang, J., et al. (2016) · arXiv · doi:10.48550/arxiv.1604.07316 · Cited by 3,198 · see citing works
Reinforcement learning in robotics: A survey
Kober, J., Bagnell, J. A., & Peters, J. (2013) · The International Journal of Robotics Research · doi:10.1177/0278364913495721 · Cited by 3,160 · see citing works
Frustum PointNets for 3D Object Detection from RGB-D Data
Qi, C. R., Liu, W., Wu, C., Su, H., & Guibas, L. J. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00102 · Cited by 2,655 · see citing works
A Survey of Motion Planning and Control Techniques for Self-Driving Urban Vehicles
Paden, B. A., Čáp, M., Yong, S. Z., Yershov, D. S., & Frazzoli, E. (2016) · IEEE Transactions on Intelligent Vehicles · doi:10.1109/tiv.2016.2578706 · Cited by 2,601 · see citing works
Fully Convolutional Networks for Semantic SegmentationPreprint
Long, J., Shelhamer, E., & Darrell, T. J. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1411.4038 · Cited by 2,490 · see citing works
Revisiting Unreasonable Effectiveness of Data in Deep Learning Era
Sun, C., Shrivastava, A., Singh, S., & Gupta, A. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.97 · Cited by 2,484 · see citing works
Deep Ordinal Regression Network for Monocular Depth Estimation
Fu, H., Gong, M., Wang, C., Batmanghelich, K., & Tao, D. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00214 · Cited by 1,991 · see citing works
DeepDriving: Learning Affordance for Direct Perception in Autonomous Driving
Chen, C., Seff, A., Kornhauser, A., & Xiao, J. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.312 · Cited by 1,876 · see citing works
Joint 3D Proposal Generation and Object Detection from View Aggregation
Ku, J., Mozifian, M., Lee, J., Harakeh, A., & Waslander, S. L. (2018) · 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2018.8594049 · Cited by 1,697 · see citing works
Deep Reinforcement Learning That Matters
Henderson, P., Islam, R., Bachman, P., Pineau, J., Precup, D., & Meger, D. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11694 · Cited by 1,685 · see citing works
1 year, 1000 km: The Oxford RobotCar dataset
Maddern, W., Pascoe, G., Linegar, C., & Newman, P. (2017) · The International Journal of Robotics Research · doi:10.1177/0278364916679498 · Cited by 1,632 · see citing works
Feng, D., Haase-Schutz, C., Rosenbaum, L., Hertlein, H., Glaser, C., Timm, F., Wiesbeck, W., & Dietmayer, K. (2021) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2020.2972974 · Cited by 1,425 · see citing works
ALVINN: An autonomous land vehicle in a neural network
Pomerleau, D. A. (2018) · Carnegie Mellon University · doi:10.1184/r1/6603146.v1 · Cited by 1,340 · see citing works
PIXOR: Real-time 3D Object Detection from Point Clouds
Yang, B., Luo, W., & Urtasun, R. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00798 · Cited by 1,249 · see citing works
Kalra, N., & Paddock, S. M. (2016) · Transportation Research Part A: Policy and Practice · doi:10.1016/j.tra.2016.09.010 · Cited by 1,221 · see citing works
3D Bounding Box Estimation Using Deep Learning and Geometry
Mousavian, A., Anguelov, D., Flynn, J., & Kosecka, J. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.597 · Cited by 1,129 · see citing works
Traffic-Sign Detection and Classification in the Wild
Zhu, Z., Liang, D., Zhang, S., Huang, X., Li, B., & Hu, S. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.232 · Cited by 1,062 · see citing works
Planning and Decision-Making for Autonomous Vehicles
Schwarting, W., Alonso-Mora, J., & Rus, D. (2018) · Annual Review of Control, Robotics, and Autonomous Systems · doi:10.1146/annurev-control-060117-105157 · Cited by 972 · see citing works
Simultaneous Localization and Mapping: A Survey of Current Trends in Autonomous Driving
Bresson, G., Alsayed, Z., Yu, L., & Glaser, S. (2017) · IEEE Transactions on Intelligent Vehicles · doi:10.1109/tiv.2017.2749181 · Cited by 885 · see citing works
End-to-End Learning of Driving Models from Large-Scale Video Datasets
Xu, H., Gao, Y., Yu, F., & Darrell, T. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.376 · Cited by 884 · see citing works
CNN-SLAM: Real-Time Dense Monocular SLAM with Learned Depth Prediction
Tateno, K., Tombari, F., Laina, I., & Navab, N. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.695 · Cited by 800 · see citing works
A Survey on 3D Object Detection Methods for Autonomous Driving Applications
Arnold, E. H., Al-Jarrah, O. Y., Dianati, M., Fallah, S., Oxtoby, D., & Mouzakitis, A. (2019) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2019.2892405 · Cited by 720 · see citing works
MFNet: Towards real-time semantic segmentation for autonomous vehicles with multi-spectral scenes
Ha, Q., Watanabe, K., Karasawa, T., Ushiku, Y., & Harada, T. (2017) · 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2017.8206396 · Cited by 693 · see citing works
Learning to Track: Online Multi-object Tracking by Decision Making
Yu, X., Alahi, A., & Savarese, S. (2015) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.534 · Cited by 684 · see citing works
Multi-task Self-Supervised Visual Learning
Doersch, C., & Zisserman, A. (2017) · IEEE International Conference on Computer Vision · doi:10.1109/iccv.2017.226 · Cited by 682 · see citing works
Computer Vision for Autonomous Vehicles
Janai, J., Güney, F., Behl, A., & Geiger, A. (2020) · Foundations and Trends® in Computer Graphics and Vision · doi:10.1561/0600000079 · Cited by 664 · see citing works
Deep Watershed Transform for Instance Segmentation
Bai, M., & Urtasun, R. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.305 · Cited by 563 · see citing works
Illumination-aware faster R-CNN for robust multispectral pedestrian detection
Li, C., Song, D., Tong, R., & Tang, M. A. (2018) · Pattern Recognition · doi:10.1016/j.patcog.2018.08.005 · Cited by 550 · see citing works
Unveiling the Power of Deep Tracking
Bhat, G., Johnander, J., Danelljan, M., Khan, F. S., & Felsberg, M. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01216-8_30 · Cited by 514 · see citing works
(2005) · neural information processing systems · Cited by 446 · see citing works
Chabot, F., Chaouch, M., Rabarisoa, J., Teulière, C., & Château, T. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.198 · Cited by 442 · see citing works
Deep Learning Scaling is Predictable, Empirically
Hestness, J., Narang, S., Ardalani, N., Diamos, G., Jun, H., Kianinejad, H., Patwary, M. M. A., Yang, Y., & Zhou, Y. (2017) · arXiv · doi:10.48550/arxiv.1712.00409 · Cited by 427 · see citing works
Explaining How a Deep Neural Network Trained with End-to-End Learning Steers a CarPreprint
Bojarski, M., Yeres, P., Choromanska, A., Choromański, K., Firner, B., Jackel, L. D., & Müller, U. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1704.07911 · Cited by 414 · see citing works
Near-Online Multi-target Tracking with Aggregated Local Flow Descriptor
Choi, W. (2015) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.347 · Cited by 409 · see citing works
Interpretable Learning for Self-Driving Cars by Visualizing Causal Attention
Kim, J., & Canny, J. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.320 · Cited by 373 · see citing works
Structured Attention Guided Convolutional Neural Fields for Monocular Depth Estimation
Xu, D., Wang, W., Tang, H., Liu, H., Sebe, N., & Ricci, E. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00412 · Cited by 358 · see citing works
Pedestrian Detection at Day/Night Time with Visible and FIR Cameras: A Comparison
González, A., Fang, Z., Socarras, Y., Serrat, J., Vázquez, D., Xu, J., & López, A. M. (2016) · Sensors · doi:10.3390/s16060820 · Cited by 325 · see citing works
Kalra, N., & Paddock, S. M. (2016) · RAND Corporation eBooks · doi:10.7249/rr1478 · Cited by 313 · see citing works
TORCS, The Open Racing Car Simulator
Espié, E., Guionneau, C., Wymann, B., Christos Dimitrakakis, A., Coulom, R., & Sumner, A. J. (2005) · Infoscience (Ecole Polytechnique Fédérale de Lausanne) · Cited by 294 · see citing works
CIRL: Controllable Imitative Reinforcement Learning for Vision-Based Self-driving
Liang, X., Wang, T., Yang, L., & Xing, E. P. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01234-2_36 · Cited by 294 · see citing works
End-to-end learning for lane keeping of self-driving cars
Chen, Z., & Huang, X. (2017) · IEEE Intelligent Vehicles Symposium · doi:10.1109/ivs.2017.7995975 · Cited by 282 · see citing works
SGN: Sequential Grouping Networks for Instance Segmentation
Liu, S., Jia, J., Fidler, S., & Urtasun, R. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.378 · Cited by 281 · see citing works
Learning a Driving SimulatorPreprint
Santana, E., & Hotz, G. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1608.01230 · Cited by 226 · see citing works
Multimodal vehicle detection: fusing 3D-LIDAR and color camera data
Asvadi, A., Garrote, L., Premebida, C., Peixoto, P., & J. Nunes, U. (2018) · Pattern Recognition Letters · doi:10.1016/j.patrec.2017.09.038 · Cited by 225 · see citing works
Luo, Y., Ren, J., Lin, M., Pang, J., Sun, W., Li, H., & Lin, L. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00024 · Cited by 205 · see citing works
The Stixel World - A Compact Medium Level Representation of the 3D-World
Badino, H., Franke, U., & Pfeiffer, D. (2009) · Lecture notes in computer science · doi:10.1007/978-3-642-03798-6_6 · Cited by 192 · see citing works
Pedestrian detection combining RGB and dense LIDAR data
Premebida, C., Carreira, J., Batista, J. P., & Nunes, U. (2014) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2014.6943141 · Cited by 189 · see citing works
Beyond Pixels: Leveraging Geometry and Shape Cues for Online Multi-Object Tracking
Sharma, S., Ansari, J. A., Jatavallabhula, K. M., & Krishna, K. M. (2018) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2018.8461018 · Cited by 169 · see citing works
Driving Policy Transfer via Modularity and AbstractionPreprint
Müller, M., Dosovitskiy, A., Ghanem, B., & Koltun, V. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1804.09364 · Cited by 167 · see citing works
End-to-end Multi-Modal Multi-Task Vehicle Control for Self-Driving Cars with Visual Perceptions
Yang, Z., Zhang, Y., Yu, J., Cai, J., & Luo, J. (2018) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr.2018.8546189 · Cited by 157 · see citing works
Pixel-Level Encoding and Depth Layering for Instance-Level Semantic Labeling
Uhrig, J., Cordts, M., Franke, U., & Brox, T. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-45886-1_2 · Cited by 154 · see citing works
A Multilevel Mixture-of-Experts Framework for Pedestrian Classification
Enzweiler, M., & Gavrila, D. M. (2011) · IEEE Transactions on Image Processing · doi:10.1109/tip.2011.2142006 · Cited by 150 · see citing works
Computer Vision for Autonomous Vehicles: Problems, Datasets and State of the Art
Janai, J., Güney, F., Behl, A., & Geiger, A. (2017) · arXiv · doi:10.48550/arxiv.1704.05519 · Cited by 141 · see citing works
End-to-End Driving Via Conditional Imitation Learning
Codevilla, F., Muller, M., Lopez, A., Koltun, V., & Dosovitskiy, A. (2018) · 2018 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2018.8460487 · Cited by 129 · see citing works
Self-citationMonocular Depth Estimation with Affinity, Vertical Pooling, and Label Enhancement
Gan, Y., Xu, X., Sun, W., & Lin, L. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01219-9_14 · Cited by 122 · see citing works
End-to-End Deep Learning for Steering Autonomous Vehicles Considering Temporal DependenciesPreprint
Eraqi, H. M., Moustafa, M. N., & Honer, J. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1710.03804 · Cited by 121 · see citing works
Complex-YOLO: Real-time 3D Object Detection on Point CloudsPreprint
Simón, M., Milz, S., Amende, K., & Groß, H. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1803.06199 · Cited by 120 · see citing works
Direct Visual SLAM Using Sparse Depth for Camera-LiDAR System
Shin, Y.-S., Park, Y. S., & Kim, A. (2018) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2018.8461102 · Cited by 116 · see citing works
On-Board Object Detection: Multicue, Multimodal, and Multiview Random Forest of Local Experts
González, A., Vázquez, D., López, A. M., & Amores, J. (2016) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2016.2593940 · Cited by 108 · see citing works
Imitation learning for locomotion and manipulation
Ratliff, N., Bagnell, J. A., & Srinivasa, S. S. (2007) · IEEE-RAS International Conference on Humanoid Robots (Humanoids) · doi:10.1109/ichr.2007.4813899 · Cited by 104 · see citing works
On Offline Evaluation of Vision-Based Driving Models
Codevilla, F., López, A. M., Koltun, V., & Dosovitskiy, A. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01267-0_15 · Cited by 103 · see citing works
Self-citationLearning to Drive from Simulation without Real World Labels
Bewley, A., Rigley, J., Liu, Y., Hawke, J., Shen, R., Lam, V.-D., & Kendall, A. (2019) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2019.8793668 · Cited by 103 · see citing works
LocNet: Global Localization in 3D Point Clouds for Mobile Vehicles
Yin, H., Tang, L., Ding, X., Wang, Y., & Xiong, R. (2018) · IEEE Intelligent Vehicles Symposium · doi:10.1109/ivs.2018.8500682 · Cited by 101 · see citing works
Amini, A., Schwarting, W., Rosman, G., et al. (2018) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2018.8594386 · Cited by 92 · see citing works
Semantic Stixels: Depth is not enough
Schneider, L., Cordts, M., Rehfeld, T., Pfeiffer, D., Enzweiler, M., Franke, U., Pollefeys, M., & Roth, S. (2016) · IEEE Intelligent Vehicles Symposium · doi:10.1109/ivs.2016.7535373 · Cited by 90 · see citing works
(2012) · Cited by 74 · see citing works
Behavioral Pedestrian Tracking Using a Camera and LiDAR Sensors on a Moving Vehicle
Dimitrievski, M., Veelaert, P., & Philips, W. R. (2019) · Sensors · doi:10.3390/s19020391 · Cited by 74 · see citing works
Stixels estimation without depth map computation
Benenson, R., Timofte, R., & Gool, L. V. (2011) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw.2011.6130495 · Cited by 63 · see citing works
Deep Imitative Models for Flexible Inference, Planning, and ControlPreprint
Rhinehart, N., McAllister, R., & Levine, S. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.06544 · Cited by 56 · see citing works
Optimal Sensor Data Fusion Architecture for Object Detection in Adverse Weather Conditions
Pfeuffer, A., & Dietmayer, K. (2018) · International Conference on Information Fusion (FUSION) · doi:10.23919/icif.2018.8455757 · Cited by 54 · see citing works
Slanted Stixels: Representing San Francisco's Steepest Streets
Hernandez-Juarez, D., Schneider, L., Espinosa, A., Moure, J., Vazquez, D., López, A., Franke, U., & Pollefeys, M. (2017) · Procedings of the British Machine Vision Conference 2017 · doi:10.5244/c.31.87 · Cited by 53 · see citing works
Zhang, X., Rad, A. B., & Wong, Y.-K. (2012) · Sensors · doi:10.3390/s120100429 · Cited by 52 · see citing works
Taking Mobile Multi-object Tracking to the Next Level: People, Unknown Objects, and Carried Items
Mitzel, D., & Leibe, B. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-33715-4_41 · Cited by 46 · see citing works
Robust SLAM system based on monocular vision and LiDAR for robotic urban search and rescue
Chen, X., Zhang, H., Lu, H., Xiao, J., Qiu, Q., & Li, Y. (2017) · IEEE International Symposium on Safety, Security, and Rescue Robotics (SSRR) · doi:10.1109/ssrr.2017.8088138 · Cited by 45 · see citing works
On the Sample Complexity of End-to-end Training vs. Semantic Abstraction TrainingPreprint
Shalev‐Shwartz, S., & Shashua, A. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1604.06915 · Cited by 43 · see citing works
Multimodal Neural Networks: RGB-D for Semantic Segmentation and Object Detection
Schneider, L., Jasch, M., Fröhlich, B., Weber, T., Franke, U., Pollefeys, M., & Rätsch, M. (2017) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-59126-1_9 · Cited by 42 · see citing works
Adding navigation to the equation: Turning decisions for end-to-end vehicle control
Hubschneider, C., Bauer, A., Weber, M., & Zöllner, J. M. (2017) · IEEE International Conference on Intelligent Transportation Systems (ITSC) · doi:10.1109/itsc.2017.8317923 · Cited by 41 · see citing works
End-to-end Multi-Modal Multi-Task Vehicle Control for Self-Driving Cars with Visual PerceptionPreprint
Yang, Z., Zhang, Y., Yu, J., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1801.06734 · Cited by 40 · see citing works
End-To-End Multi-Modal Sensors Fusion System For Urban Automated Driving
Sobh, I., Amin, L., Abdelkarim, S., et al. (2018) · Cited by 36 · see citing works
Li, Z., Motoyoshi, T., Sasaki, K., Ogata, T., & Sugano, S. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.11100 · Cited by 33 · see citing works
Imitation learning for vision-based lane keeping assistance
Innocenti, C., Linden, H., Panahandeh, G., et al. (2017) · IEEE International Conference on Intelligent Transportation Systems (ITSC) · doi:10.1109/itsc.2017.8317915 · Cited by 21 · see citing works
Monocular Depth Estimation by Learning from Heterogeneous Datasets
Gurram, A., Urfalıoǧlu, O., Halfaoui, I., Bouzaraa, F., & López, A. M. (2018) · IEEE Intelligent Vehicles Symposium · doi:10.1109/ivs.2018.8500683 · Cited by 19 · see citing works
Self-citationComputer Vision in Vehicle Technology
López, A. M., Imiya, A., Pajdla, T., et al. (2017) · Wiley eBooks · doi:10.1002/9781118868065 · Cited by 17 · see citing works
Siamese-ResNet: Implementing Loop Closure Detection based on Siamese Network
Qiu, K., Ai, Y., Tian, B., Wang, B., & Cao, D. (2018) · IEEE Intelligent Vehicles Symposium · doi:10.1109/ivs.2018.8500465 · Cited by 15 · see citing works
Visual Place Recognition in Long-term and Large-scale Environment based on CNN Feature
Zhu, J., Ai, Y., Tian, B., Cao, D., & Scherer, S. (2018) · IEEE Intelligent Vehicles Symposium · doi:10.1109/ivs.2018.8500686 · Cited by 14 · see citing works
A 3D Object Detection Based on Multi-Modality Sensors of USV
Wu, Y., Qin, H., Liu, T., et al. (2019) · Applied Sciences · doi:10.3390/app9030535 · Cited by 12 · see citing works
Imitation Learning for End to End Vehicle Longitudinal Control with Forward CameraPreprint
George, L., Buhet, T., Wirbel, É., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1812.05841 · Cited by 11 · see citing works
End-to-end driving simulation via angle branched networkPreprint
Wang, Q., Chen, L. F., & Tian, W. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.07545 · Cited by 10 · see citing works
Towards Self-Supervised High Level Sensor FusionPreprint
Khan, Q., Schön, T., & Wenzel, P. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.04272 · Cited by 2 · see citing works
Improved Semantic Stixels via Multimodal Sensor FusionPreprint
Piewak, F., Pinggera, P., Enzweiler, M., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.08993 · Cited by 1 · see citing works
Flagged first, then by citations
Research works printed but not linked: 3
- [19][19] F. Yu and V. Koltun, 'Multi-scale context aggregation by dilated convolutions,' in Inter. Conf. on Learning Representation (ICLR) , 2016. -
- [56][56] A. Sauer, N. Savinov, and A. Geiger, 'Conditional affordance learning for driving in urban environments,' in Conf. on Robot Learning (CoRL) , 2018. -
- [74][74] C. Doersch and A. Zisserman, 'Multi-task selfsupervised visual learning,' in Inter. Conf. on Computer Vision (ICCV) , 2017. -
Cited by
Sorted by citations
Bayoudh, K., Knani, R., Hamdaoui, F., & Mtibaa, A. (2022) · The Visual Computer · doi:10.1007/s00371-021-02166-7 · Cited by 456 · see citing works
Autonomous driving system: A comprehensive survey
Zhao, J., Zhao, W., Deng, B., Wang, Z., Zhang, F., Zheng, W., Cao, W., Nan, J., Lian, Y., & Burke, A. F. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.122836 · Cited by 361 · see citing works
Deep multimodal fusion for semantic image segmentation: A survey
Zhang, Y., Sidibé, D., Morel, O., & Mériaudeau, F. (2021) · Image and Vision Computing · doi:10.1016/j.imavis.2020.104042 · Cited by 261 · see citing works
A2D2: Audi Autonomous Driving Dataset
Geyer, J., Kassahun, Y., Mahmudi, M., Ricou, X., Durgesh, R., Chung, A. S., Hauswald, L., Pham, V. H., Mühlegg, M., Dorn, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2004.06320 · Cited by 250 · see citing works
A Review on Explainability in Multimodal Deep Neural Nets
Joshi, G., Walambe, R., & Kotecha, K. (2021) · IEEE Access · doi:10.1109/access.2021.3070212 · Cited by 204 · see citing works
Multimodal data integration for oncology in the era of deep neural networks: a review
Waqas, A., Tripathi, A., Ramachandran, R. P., Stewart, P. A., & Rasool, G. (2024) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2024.1408843 · Cited by 165 · see citing works
Robust Multimodal Learning with Missing Modalities via Parameter-Efficient Adaptation
Reza, M. K., Prater-Bennette, A., & Asif, M. S. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3476487 · Cited by 61 · see citing works
“In recent years, MML has achieved remarkable success in a wide range of applications, including image segmentation [6, 55, 70], captioning [71, 65], classification [15, 46], sentiment analysis [50, 28], and autonomous driving [61, 45].”
BackgroundMultimodal fusion and vision–language models: A survey for robot vision
Han, X., Chen, S., Fu, Z., Feng, Z., Fan, L., An, D., Wang, C., Guo, L., Meng, W., Zhang, X., et al. (2026) · Information Fusion · doi:10.1016/j.inffus.2025.103652 · Cited by 41 · see citing works
“Early fusion directly integrates multimodal inputs at the data level, such as concatenating RGB [44] images and depth maps before feeding them into the model.”
- Authors
- Xiao, Y., Codevilla, F., Gurram, A., Urfalioglu, O., & Lopez, A. M.
- Affiliations
- Universitat Autònoma de Barcelona (ES)
- Huawei Technologies (Germany) (DE)
- Venue
- IEEE Transactions on Intelligent Transportation Systems
- Published
- 2022-01
- Volume
- 23
- Issue
- 1
- Pages
- 537-547
- ISSN
- 1524-9050 (ISSN-L)
- 1558-0016
- OpenAlex
- W2948153973
- Cited by
- 319
- References
- 107
- Citations by year
- 2026: 42 · 2025: 73 · 2024: 68 · 2023: 54 · 2022: 35 · 2021: 32 · 2020: 15
- FWCI
- 12.58
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE Transactions on Intelligent Transportation Systems (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- Dipòsit Digital de Documents de la UAB (Universitat Autònoma de Barcelona) (submittedVersion, public-domain)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Engineering › Automotive Engineering › Autonomous Vehicle Technology and Safety
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Vision and Imaging
- Keywords
- end-to-end autonomous driving, multimodal fusion, LiDAR, feature-level fusion, monocular depth estimation, CARLA simulator, sensor data annotation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
