Feature analysis and selection for training an end-to-end autonomous vehicle controller using deep learning approach
Abstract
Deep learning-based approaches have been widely used for training controllers for autonomous vehicles due to their powerful ability to approximate nonlinear functions or policies. However, the training process usually requires large labeled data sets and takes a lot of time. In this paper, we analyze the influences of features on the performance of controllers trained using the convolutional neural networks (CNNs), which gives a guideline of feature selection to reduce computation cost. We collect a large set of data using The Open Racing Car Simulator (TORCS) and classify the image features into three categories (sky-related, roadside-related, and road-related features). We then design two experimental frameworks to investigate the importance of each single feature for training a CNN controller. The first framework uses the training data with all three features included to train a controller, which is then tested with data that has one feature removed to evaluate the feature's effects. The second framework is trained with the data that has one feature excluded, while all three features are included in the test data. Different driving scenarios are selected to test and analyze the trained controllers using the two experimental frameworks. The experiment results show that (1) the road-related features are indispensable for training the controller, (2) the roadside-related features are useful to improve the generalizability of the controller to scenarios with complicated roadside information, and (3) the sky-related features have limited contribution to train an end-to-end autonomous vehicle controller.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Cites
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126 · see citing works
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 74,248 · see citing works
Caffe: Convolutional Architecture for Fast Feature Embedding
Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., & Darrell, T. (2014) · Proceedings of the 22nd ACM international conference on Multimedia · doi:10.1145/2647868.2654889 · Cited by 11,077 · see citing works
End to End Learning for Self-Driving Cars
Bojarski, M., Del Testa, D., Dworakowski, D., Firner, B., Flepp, B., Goyal, P., Jackel, L. D., Monfort, M., Muller, U., Zhang, J., et al. (2016) · arXiv · doi:10.48550/arxiv.1604.07316 · Cited by 3,198 · see citing works
DeepDriving: Learning Affordance for Direct Perception in Autonomous Driving
Chen, C., Seff, A., Kornhauser, A., & Xiao, J. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.312 · Cited by 1,876 · see citing works
A convolutional neural network cascade for face detection
Li, H., Lin, Z., Shen, X., Brandt, J., & Hua, G. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7299170 · Cited by 1,390 · see citing works
End-to-End Learning of Driving Models from Large-Scale Video Datasets
Xu, H., Gao, Y., Yu, F., & Darrell, T. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.376 · Cited by 884 · see citing works
(2005) · neural information processing systems · Cited by 446 · see citing works
Pedestrian detection aided by deep learning semantic tasks
Tian, Y., Luo, P., Wang, X., & Tang, X. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7299143 · Cited by 416 · see citing works
Fused DNN: A Deep Neural Network Fusion Approach to Fast and Robust Pedestrian Detection
Du, X., El‐Khamy, M., Lee, J., & Davis, L. S. (2017) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv.2017.111 · Cited by 280 · see citing works
Very Deep Convolutional Networks for Natural Language Processing.Preprint
Conneau, A., Schwenk, H., Barrault, L., & LeCun, Y. (2016) · arXiv (Cornell University) · Cited by 214 · see citing works
Implementation of Training Convolutional Neural NetworksPreprint
Liu, T., Fang, S., Zhao, Y., Wang, P., & Zhang, J. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.01195 · Cited by 153 · see citing works
Performance evaluation of deep feature learning for RGB-D image/video classification
Shao, L., Cai, Z., Liu, L., & Lü, K. (2017) · Information Sciences · doi:10.1016/j.ins.2017.01.013 · Cited by 92 · see citing works
Are all objects equal? Deep spatio-temporal importance prediction in driving videos
Ohn-Bar, E., & Trivedi, M. M. (2016) · Pattern Recognition · doi:10.1016/j.patcog.2016.08.029 · Cited by 56 · see citing works
On the Sample Complexity of End-to-end Training vs. Semantic Abstraction TrainingPreprint
Shalev‐Shwartz, S., & Shashua, A. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1604.06915 · Cited by 43 · see citing works
End-to-end Learning of Driving Models from Large-scale Video DatasetsPreprint
Xu, H., Gao, Y., Yu, F., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1612.01079 · Cited by 38 · see citing works
A Face Detection and Recognition System for Intelligent Vehicles
Ishak, K. A., Samad, S. A., & Hussain, A. (2006) · Information Technology Journal · doi:10.3923/itj.2006.507.515 · Cited by 15 · see citing works
Fused DNN: A deep neural network fusion approach to fast and robust pedestrian detectionPreprint
Du, X., El‐Khamy, M., Lee, J., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1610.03466 · Cited by 12 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
A survey of deep learning techniques for autonomous driving
Grigorescu, S., Trasnea, B., Cocias, T., & Macesanu, G. (2020) · Journal of Field Robotics · doi:10.1002/rob.21918 · Cited by 1,809 · see citing works
Di, X. S., & Shi, R. (2021) · Transportation Research Part C Emerging Technologies · doi:10.1016/j.trc.2021.103008 · Cited by 240 · see citing works
Huang, Z., Lv, C., Xing, Y., & Wu, J. (2020) · IEEE Sensors Journal · doi:10.1109/jsen.2020.3003121 · Cited by 194 · see citing works
Intelligent and connected vehicles: Current status and future perspectives
Yang, D., Jiang, K., Zhao, D. X., Yu, C., Cao, Z. L., Xie, S., Xiao, Z., Jiao, X., Wang, S., & Zhang, K. (2018) · Science China Technological Sciences · doi:10.1007/s11431-017-9338-1 · Cited by 176 · see citing works
Deep Reinforcement Learning for Autonomous DrivingPreprint
Wang, S., Jia, D., & Weng, X. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.11329 · Cited by 145 · see citing works
A Survey of Deep Learning Applications to Autonomous Vehicle Control
Kuutti, S., Bowden, R., Jin, Y., Barber, P., & Fallah, S. (2021) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2019.2962338 · Cited by 96 · see citing works
Wang, W., & Zhao, D. X. (2018) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2018.2794604 · Cited by 82 · see citing works
Self-citationBioinspired Computational Intelligence and Transportation Systems: A Long Road Ahead
Ser, J. D., Osaba, E., Medina, J., Fister, I., & Fister, I. (2019) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2019.2897377 · Cited by 64 · see citing works
Merriman, S. E., Plant, K. L., Revell, K. M. A., & Stanton, N. A. (2020) · Transportation Research Part F Traffic Psychology and Behaviour · doi:10.1016/j.trf.2020.10.011 · Cited by 43 · see citing works
Steering Angle Prediction Techniques for Autonomous Ground Vehicles: A Review
Saleem, H., Riaz, F., Mostarda, L., Niazi, M. A., Rafiq, A., & Saeed, S. (2021) · IEEE Access · doi:10.1109/access.2021.3083890 · Cited by 36 · see citing works
End-to-End Reinforcement Learning for Self-driving Car
Chopra, R. R., & Roy, S. S. (2020) · Advances in intelligent systems and computing · doi:10.1007/978-981-15-1081-6_5 · Cited by 32 · see citing works
Morooka, F. E., Manoel, A., Sigahi, T. F. A. C., Pinto, J. D. S., Rampasso, I. S., & Anholon, R. (2023) · Machine Learning and Knowledge Extraction · doi:10.3390/make5030041 · Cited by 32 · see citing works
End-to-End Autonomous Mobile Robot Navigation with Model-Based System Support
Carballo, A., Seiya, S., Lambert, J., Darweesh, H., Narksri, P., Morales, L. Y., Akai, N., Takeuchi, E., & Takeda, K. (2018) · Journal of Robotics and Mechatronics · doi:10.20965/jrm.2018.p0563 · Cited by 30 · see citing works
Involvement of Deep Learning for Vision Sensor-Based Autonomous Driving Control: A Review
Khanum, A., Lee, C.-Y., & Yang, C. (2023) · IEEE Sensors Journal · doi:10.1109/jsen.2023.3280959 · Cited by 28 · see citing works
OBJECT DETECTION AND LANE CHANGING FOR SELF DRIVING CAR USING CNN
R, S. M., Archana, N., Reshma, J., Sangeetha, S., & Varalakshmi, M. (2022) · International Conference on Communication, Computing and Internet of Things (IC3IoT) · doi:10.1109/ic3iot53935.2022.9767882 · Cited by 26 · see citing works
End‐to‐end learning for high‐precision lane keeping via multi‐state model
Yuan, W., Yang, M. Z., Li, H., Wang, C., & Wang, B. (2018) · CAAI Transactions on Intelligence Technology · doi:10.1049/trit.2018.1025 · Cited by 24 · see citing works
Dynamically integrated spatiotemporal‐based trajectory planning and control for autonomous vehicles
Li, B., Du, H., Li, W., & Zhang, B. (2018) · IET Intelligent Transport Systems · doi:10.1049/iet-its.2018.5306 · Cited by 22 · see citing works
A survey of Automotive Driving Assistance Systems technologies
Swief, A., & El-Habrouk, M. (2018) · International Artificial Intelligence and Data Processing Symposium (IDAP) · doi:10.1109/idap.2018.8620826 · Cited by 21 · see citing works
Investigation of Steering Wheel Control of an Electric Buggy Car for Designing Fuzzy Controller
Halin, H., Khairunizam, W., Ikram, K., Haris, H., Shahriman, A. B., Razlan, Z. M., Zunaidi, I., & Desa, H. (2018) · Lecture notes in mechanical engineering · doi:10.1007/978-981-10-8788-2_41 · Cited by 17 · see citing works
End-to-End Self-Driving Approach Independent of Irrelevant Roadside Objects With Auto-Encoder
Wang, T., Luo, Y., Liu, J., Chen, R., & Li, K. (2020) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2020.3018473 · Cited by 16 · see citing works
Moral dilemmas for moral machines
LaCroix, T. (2022) · AI and Ethics · doi:10.1007/s43681-022-00134-y · Cited by 15 · see citing works
Safe and feasible motion generation for autonomous driving via constrained policy net
Zhan, W., Li, J., Hu, Y., & Tomizuka, M. (2017) · Annual Conference of the IEEE Industrial Electronics Society (IECON) · doi:10.1109/iecon.2017.8216790 · Cited by 14 · see citing works
Advanced Camera Image Cropping Approach for CNN-Based End-to-End Controls on Sustainable Computing
Sung, Y., Jin, Y., Kwak, J., Lee, S.-G., & Cho, K. (2018) · Sustainability · doi:10.3390/su10030816 · Cited by 14 · see citing works
Wang, S., Fujii, H., & Yoshimura, S. (2022) · Physica A Statistical Mechanics and its Applications · doi:10.1016/j.physa.2022.128172 · Cited by 13 · see citing works
A New Single Point Preview-Based Human-Like Driver Model on Urban Curved Roads
Zhou, X., Jiang, H., Li, A., & Ma, S. (2020) · IEEE Access · doi:10.1109/access.2020.3001208 · Cited by 13 · see citing works
Paniego, S., Calvo-Palomino, R., & Cañas, J. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.128161 · Cited by 11 · see citing works
VRDriving: A Virtual-to-Real Autonomous Driving Framework Based on Adversarial Learning
Yuan, W., Yang, M. Z., Wang, C., & Wang, B. (2020) · IEEE Transactions on Cognitive and Developmental Systems · doi:10.1109/tcds.2020.3006621 · Cited by 11 · see citing works
Liu, D., Zhao, J., Xi, A., Wang, C., Huang, X., Lai, K., & Liu, C. (2020) · Computer Modeling in Engineering & Sciences · doi:10.32604/cmes.2020.08641 · Cited by 10 · see citing works
Self-citationSteeringLoss: A Cost-Sensitive Loss Function for the End-to-End Steering Estimation
Yuan, W., Yang, M. Z., Li, H., Wang, C., & Wang, B. (2020) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2019.2963003 · Cited by 10 · see citing works
Al-Qizwini, M., Bulan, O., Qi, X., Mengistu, Y., Mahesh, S., Hwang, J., & Clifford, D. (2020) · IEEE Sensors Journal · doi:10.1109/jsen.2020.3036532 · Cited by 10 · see citing works
Unifying ground and air: a comprehensive review of deep learning-enabled CAVs and UAVs
Zia, M. U., Xiang, W., Huang, T., Ahmad, J., Chattha, J. N., Naqvi, I. H., & Butt, F. A. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11425-1 · Cited by 9 · see citing works
Comparative Analysis of Pre-trained Deep Neural Networks for Plant Disease Classification
George, R., Thuseethan, S., & Ragel, R. (2024) · International Joint Conference on Computer Science and Software Engineering · doi:10.1109/jcsse61278.2024.10613633 · Cited by 9 · see citing works
End-to-End Navigation with Branch Turning Support Using Convolutional Neural Network
Seiya, S., Carballo, A., Takeuchi, E., Miyajima, C., & Takeda, K. (2018) · IEEE International Conference on Robotics and Biomimetics (ROBIO) · doi:10.1109/robio.2018.8665079 · Cited by 9 · see citing works
Hoshino, S., & Sumiyoshi, J. (2022) · IEEE/SICE International Symposium on System Integration (SII) · doi:10.1109/sii52469.2022.9708748 · Cited by 9 · see citing works
End-to-End Discrete Motion Planner based on Deep Neural Network for Autonomous Mobile Robots
Hoshino, S., & Sumiyoshi, J. (2020) · IEEE/SICE International Symposium on System Integration (SII) · doi:10.1109/sii46433.2020.9026271 · Cited by 8 · see citing works
Siddiqui, M. A. S., Rabbi, M. S., Islam, M. J., & Ahmed, R. U. (2025) · Journal of Advanced Transportation · doi:10.1155/atr/9995539 · Cited by 7 · see citing works
Deep Learning-Based Decision-Making of Autonomous Vehicles to Predict Accidents
Khanum, A., Lee, C.-Y., & Yang, C. (2023) · IEEE International Conference on Multimedia and Expo workshops · doi:10.1109/icmew59549.2023.00022 · Cited by 6 · see citing works
Deep learning approach for super-knock event prediction of petrol engine with sample imbalance
Zhou, Z., Xiong, S., Chen, Y., Zhang, C., & Cao, Y. (2021) · Fuel · doi:10.1016/j.fuel.2021.122509 · Cited by 6 · see citing works
SteeringLoss: Theory and Application for Steering Prediction
Yuan, W., Yang, M. Z., Wang, C., & Wang, B. (2019) · IEEE Intelligent Vehicles Symposium · doi:10.1109/ivs.2019.8814041 · Cited by 5 · see citing works
Ran, C., Xie, Z., Xie, Y., Yin, Y., & Ye, H. (2024) · IEEE Access · doi:10.1109/access.2024.3463967 · Cited by 5 · see citing works
End-to-End Learning with Memory Models for Complex Autonomous Driving Tasks in Indoor Environments
Lai, Z., & Bräunl, T. (2023) · Journal of Intelligent & Robotic Systems · doi:10.1007/s10846-022-01801-2 · Cited by 5 · see citing works
Prasad, B., Kumar, R., & Singh, M. (2023) · International Journal of Electrical and Electronics Research · doi:10.37391/ijeer.110201 · Cited by 4 · see citing works
AGBM: An Adaptive Gradient Balanced Mechanism for the End-to-End Steering Estimation
Yuan, W., Zhuang, H., Wang, C., & Yang, M. Z. (2022) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2022.3147248 · Cited by 4 · see citing works
Anticipating Autonomous Vehicle Driving based on Multi-Modal Multiple Motion Tasks Network
Khanum, A., Lee, C.-Y., Hus, C.-C., & Yang, C. (2022) · Journal of Intelligent & Robotic Systems · doi:10.1007/s10846-022-01677-2 · Cited by 4 · see citing works
A Decision Based Mobility Model for Semi and Fully Autonomous Vehicles
Pandey, V., Slon, C. M., Deschenes, L., Gorsich, D., & Jayakumar, P. (2020) · SAE International Journal of Advances and Current Practices in Mobility · doi:10.4271/2020-01-0747 · Cited by 3 · see citing works
Yuan, W., Yang, M. Z., Wang, C., & Wang, B. (2019) · Communications in computer and information science · doi:10.1007/978-981-13-7986-4_38 · Cited by 2 · see citing works
Simulation Studies of the Hybrid Human-Fuzzy Controller for Path Tracking of an Autonomous Vehicle
Halin, H., Khairunizam, W., Haris, H., Razlan, Z. M., Shahriman, A. B., Zunaidi, I., & Mustafa, W. A. (2020) · Lecture notes in electrical engineering · doi:10.1007/978-981-15-5281-6_71 · Cited by 2 · see citing works
Value of Information in Multi-Attribute Decisions With Applications in Ground Vehicle Autonomy
Kassoumeh, S., Pandey, V., Gorsich, D., & Jayakumar, P. (2021) · Journal of Autonomous Vehicles and Systems · doi:10.1115/1.4054125 · Cited by 1 · see citing works
Amar, V. (2021) · International Journal for Research in Applied Science and Engineering Technology · doi:10.22214/ijraset.2021.35771 · Cited by 1 · see citing works
Effects of environmental feature selection on end‐to‐end vehicle steering controller
Liu, D., Zhao, J., Cao, Z., Huang, X., & Xi, A. (2020) · The Journal of Engineering · doi:10.1049/joe.2019.1153
Incorporating Orientations into End-to-end Driving Model for Steering ControlPreprint
Wan, P., Song, Z., & Lu, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.05846
Deep Learning for Planning and Control in Autonomous Vehicles
Zhao, J., & Burke, A. F. (2025) · IntechOpen eBooks · doi:10.5772/intechopen.1009340
- Authors
- Yang, S., Wang, W., Liu, C., Deng, W., & Hedrick, J. K.
- Affiliations
- Jilin University (CN)
- Beijing Institute of Technology (CN)
- University of California, Berkeley (US)
- Venue
- 2017 IEEE Intelligent Vehicles Symposium (IV)
- Publisher
- IEEE
- Published
- 2017-06
- Pages
- 1033-1038
- ISSN
- 1931-0587 (ISSN-L)
- OpenAlex
- W2605022719
- Cited by
- 54
- References
- 16
- Citations by year
- 2025: 3 · 2024: 4 · 2023: 5 · 2022: 6 · 2021: 7 · 2020: 14 · 2019: 4 · 2018: 10 · 2017: 1
- FWCI
- 8.81
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- IEEE Intelligent Vehicles Symposium (publishedVersion)
- Funders
- National Natural Science Foundation of China (2016YFB0100904, U1564211)
- National Key Research and Development Program of China (2016YFB0100904)
- Topics
- Physical Sciences › Engineering › Automotive Engineering › Autonomous Vehicle Technology and Safety
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- autonomous vehicle control, end-to-end learning, feature selection, convolutional neural networks, TORCS, driving simulator, feature importance, roadside features, computation cost reduction
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
