Driver Activity Recognition for Intelligent Vehicles: A Deep Learning Approach
Open accessThe DOI matches its Crossref record; 46 of the 46 references it cites were found in a registry or an index.
Abstract
Driver decisions and behaviors are essential factors that can affect the driving safety. To understand the driver behaviors, a driver activities recognition system is designed based on the deep convolutional neural networks (CNN) in this paper. Specifically, seven common driving activities are identified, which are the normal driving, right mirror checking, rear mirror checking, left mirror checking, using in-vehicle radio device, texting, and answering the mobile phone, respectively. Among these activities, the first four are regarded as normal driving tasks, while the rest three are classified into the distraction group. The experimental images are collected using a low-cost camera, and ten drivers are involved in the naturalistic data collection. The raw images are segmented using the Gaussian mixture model to extract the driver body from the background before training the behavior recognition CNN model. To reduce the training cost, transfer learning method is applied to fine tune the pre-trained CNN models. Three different pre-trained CNN models, namely, AlexNet, GoogLeNet, and ResNet50 are adopted and evaluated. The detection results for the seven tasks achieved an average of 81.6% accuracy using the AlexNet, 78.6% and 74.9% accuracy using the GoogLeNet and ResNet50, respectively. Then, the CNN models are trained for the binary classification task and identify whether the driver is being distracted or not. The binary detection rate achieved 91.4% accuracy, which shows the advantages of using the proposed deep learning approach. Finally, the real-world application are analyzed and discussed.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
Histograms of Oriented Gradients for Human Detection
Dalal, N., & Triggs, B. (2005) · 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) · doi:10.1109/cvpr.2005.177 · Cited by 33,584
The Infinite Gaussian Mixture Model
Rasmussen, C. E. (1999) · Cited by 1,057
Srivastava, R. K., Greff, K., & Schmidhuber, J. (2015) · arXiv · doi:10.48550/arxiv.1505.00387 · Cited by 912
Towards Understanding Action Recognition
Jhuang, H., Gall, J., Zuffi, S., Schmid, C., & Black, M. J. (2013) · 2013 IEEE International Conference on Computer Vision · doi:10.1109/iccv.2013.396 · Cited by 879
On Convergence Properties of the EM Algorithm for Gaussian Mixtures
Xu, L., & Jordan, M. I. (1996) · Neural Computation · doi:10.1162/neco.1996.8.1.129 · Cited by 756
Driving Style Recognition for Intelligent Vehicle Control and Advanced Driver Assistance: A Survey
Marina Martinez, C., Heucke, M., Wang, F.-Y., Gao, B., & Cao, D. (2018) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2017.2706978 · Cited by 620 · cited 1×
“Previous studies mainly focus on the driver attention [41] and distraction (either physical distraction or cognitive distraction) [36], driver intention [7], [9], driver styles [42], driver drowsiness and fatigue detection [10]–[12], etc.”
BackgroundSelf-citationHuman Activity Recognition Process Using 3-D Posture Data
Gaglio, S., Re, G. L., & Morana, M. (2015) · IEEE Transactions on Human-Machine Systems · doi:10.1109/thms.2014.2377111 · Cited by 354
Real-time Distracted Driver Posture ClassificationPreprint
Abouelnaga, Y., Eraqi, H. M., & Moustafa, M. N. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1706.09498 · Cited by 329
Towards Detection of Bus Driver Fatigue Based on Robust Visual Analysis of Eye State
Mandal, B., Li, L., Wang, G. S., & Lin, J. (2017) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2016.2582900 · Cited by 328
Capturing Car-Following Behaviors by Deep Learning
Wang, X., Jiang, R., Li, L., Lin, Y., Zheng, X., & Wang, F.-Y. (2018) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2017.2706963 · Cited by 325
Self-citationMurphy-Chutorian, E., & Trivedi, M. M. (2010) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2010.2044241 · Cited by 290
(2001) · Cited by 281
Personalized Driver/Vehicle Lane Change Models for ADAS
Butakov, V., & Ioannou, P. (2015) · IEEE Transactions on Vehicular Technology · doi:10.1109/tvt.2014.2369522 · Cited by 281
Intelligence Testing for Autonomous Vehicles: A New Approach
Li, L., Huang, W.-L., Liu, Y., Zheng, N.-N., & Wang, F.-Y. (2016) · IEEE Transactions on Intelligent Vehicles · doi:10.1109/tiv.2016.2608003 · Cited by 267
Self-citationLane Change Intent Analysis Using Robust Operators and Sparse Bayesian Learning
McCall, J. C., Wipf, D. P., Trivedi, M. M., & Rao, B. D. (2007) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2007.902640 · Cited by 254
Chai, R., Naik, G. R., Nguyen, T. N., Ling, S. H., Tran, Y., Craig, A., & Nguyen, H. T. (2017) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2016.2532354 · Cited by 250
Parallel driving in CPSS: a unified approach for transport automation and vehicle intelligence
Wang, F.-Y., Zheng, N.-N., Cao, D., Martinez, C. M., Li, L., & Liu, T. (2017) · IEEE/CAA Journal of Automatica Sinica · doi:10.1109/jas.2017.7510598 · Cited by 239
Self-citationAutomated Detection of Driver Fatigue Based on Entropy and Complexity Measures
Zhang, C., Wang, H., & Fu, R. (2014) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2013.2275192 · Cited by 215
Self-citationDriver Distraction Detection Using Semi-Supervised Machine Learning
Liu, T., Yang, Y., Huang, G.-B., Yeo, Y. K., & Lin, Z. (2016) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2015.2496157 · Cited by 215
Recent Trends in Driver Safety Monitoring Systems: State of the Art and Challenges
Koesdwiady, A., Soua, R., Karray, F., & Kamel, M. S. (2017) · IEEE Transactions on Vehicular Technology · doi:10.1109/tvt.2016.2631604 · Cited by 168
Real‐time detection of distracted driving based on deep learning
Tran, D., Manh, H., Sheng, W., Bai, H., & Chowdhary, G. V. (2018) · IET Intelligent Transport Systems · doi:10.1049/iet-its.2018.5172 · Cited by 164
Driving posture recognition by convolutional neural networks
Yan, C., Coenen, F., & Zhang, B. (2016) · IET Computer Vision · doi:10.1049/iet-cvi.2015.0175 · Cited by 159
Kim, I.-H., Kim, J.-W., Haufe, S., & Lee, S. (2014) · Journal of Neural Engineering · doi:10.1088/1741-2560/12/1/016001 · Cited by 157
Xing, Y., Lv, C., Zhang, Z., Wang, H., Na, X., Cao, D., Velenis, E., & Wang, F. (2017) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2017.2766884 · Cited by 156
Self-citationAnalysis of autopilot disengagements occurring during autonomous vehicle testing
Lv, C., Cao, D., Zhao, Y., Auger, D. J., Sullman, M., Wang, H., Dutka, L. M., Skrypchuk, L., & Mouzakitis, A. (2018) · IEEE/CAA Journal of Automatica Sinica · doi:10.1109/jas.2017.7510745 · Cited by 153
Self-citationHead, Eye, and Hand Patterns for Driver Activity Recognition
Ohn-Bar, E., Martin, S., Tawari, A., & Trivedi, M. M. (2014) · 2014 22nd International Conference on Pattern Recognition · doi:10.1109/icpr.2014.124 · Cited by 139
Modeling and prediction of driver behavior by foot gesture analysis
Tran, C., Doshi, A., & Trivedi, M. M. (2011) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2011.09.008 · Cited by 129
A Kinect-Based Wearable Face Recognition System to Aid Visually Impaired Users
Neto, L. D. S. B., Grijalva, F., Maike, V. R. M. L., Martini, L. C., Florencio, D. A. F., Baranauskas, M. C. C., Rocha, A. D. R., & Goldenstein, S. K. (2016) · IEEE Transactions on Human-Machine Systems · doi:10.1109/thms.2016.2604367 · Cited by 121
Look at the Driver, Look at the Road: No Distraction! No Accident!
Rezaei, M., & Klette, R. (2014) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2014.24 · Cited by 119
Driver distraction detection and recognition using RGB-D sensorPreprint
Craye, C., & Karray, F. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1502.00250 · Cited by 116
On Performance Evaluation of Driver Hand Detection Algorithms: Challenges, Dataset, and Metrics
Das, N., Ohn-Bar, E., & Trivedi, M. M. (2015) · IEEE International Conference on Intelligent Transportation Systems (ITSC) · doi:10.1109/itsc.2015.473 · Cited by 115
Liao, Y. J., Li, S. E., Wang, W., Wang, Y., Li, G., & Cheng, B. (2016) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2015.2506602 · Cited by 115
Abnormal Driving Detection Based on Normalized Driving Behavior
Hu, J., Li, X., He, X., & Meng, W. (2017) · IEEE Transactions on Vehicular Technology · doi:10.1109/tvt.2017.2660497 · Cited by 112
EEG-Based Detection of Driver Emergency Braking Intention for Brain-Controlled Vehicles
Teng, T., Bi, L., & Liu, Y. (2017) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2017.2740427 · Cited by 99
Tawari, A., Sivaraman, S., Trivedi, M. M., Shannon, T., & Tippelhofer, M. (2014) · 2014 IEEE Intelligent Vehicles Symposium Proceedings · doi:10.1109/ivs.2014.6856600 · Cited by 89 · cited 1×
“Previous studies mainly focus on the driver attention [41] and distraction (either physical distraction or cognitive distraction) [36], driver intention [7], [9], driver styles [42], driver drowsiness and fatigue detection [10]–[12], etc.”
BackgroundWang, J., Wang, J., Wang, R., & Hu, C. (2016) · IEEE Transactions on Vehicular Technology · doi:10.1109/tvt.2016.2609154 · Cited by 88
Li, N., & Busso, C. (2015) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2015.2493451 · Cited by 72
How Much of Driving Is Preattentive?
Pugeault, N., & Bowden, R. (2015) · IEEE Transactions on Vehicular Technology · doi:10.1109/tvt.2015.2487826 · Cited by 51
Queuing Network Modeling of Driver EEG Signals-Based Steering Control
Bi, L., Lu, Y. P., Fan, X., Lian, J., & Liu, Y. (2016) · IEEE Transactions on Neural Systems and Rehabilitation Engineering · doi:10.1109/tnsre.2016.2614003 · Cited by 45
Multiview fusion for activity recognition using deep neural networks
Kavi, R., Kulathumani, V., Rohit, F., & Kecojevic, V. (2016) · Journal of Electronic Imaging · doi:10.1117/1.jei.25.4.043010 · Cited by 37
In-vehicle Hand Gesture Recognition using Hidden Markov models
Deo, N., Rangesh, A., & Trivedi, M. M. (2016) · IEEE International Conference on Intelligent Transportation Systems (ITSC) · doi:10.1109/itsc.2016.7795908 · Cited by 31
Cited by
Sensor-based and vision-based human activity recognition: A comprehensive survey
Minh Dang, L., Min, K., Wang, H., Jalil Piran, M., Hee Lee, C., & Moon, H. (2020) · Pattern Recognition · doi:10.1016/j.patcog.2020.107561 · Cited by 730
Self-citationLi, X., Liu, H., Wang, W., Zheng, Y., Lv, H., & Lv, Z. (2022) · Future Generation Computer Systems · doi:10.1016/j.future.2021.10.006 · Cited by 398
A Survey on Multimodal Large Language Models for Autonomous Driving
Cui, C., Ma, Y., Cao, X., Ye, W., Zhou, Y., Liang, K., Chen, J., Lu, J., Yang, Z., Liao, K.-D., et al. (2024) · 2024 IEEE/CVF Winter Conference on Applications of Computer Vision Workshops (WACVW) · doi:10.1109/wacvw60836.2024.00106 · Cited by 369
Xing, Y., Lv, C., & Cao, D. (2020) · IEEE Transactions on Vehicular Technology · doi:10.1109/tvt.2019.2960110 · Cited by 261
Self-citationAli, E. S., Hasan, M. K., Hassan, R., Saeed, R. A., Hassan, M. B., Islam, S., Nafi, N. S., & Bevinakoppa, S. (2021) · Security and Communication Networks · doi:10.1155/2021/8868355 · Cited by 257
“Clustering algorithms provide a way to model approaches such as centroid-based and hierarchical for intelligent localization [61, 62].”
Method
Metadata
- Authors
- Xing, Y., Lv, C., Wang, H., Cao, D., Velenis, E., & Wang, F.-Y.
- Venue
- IEEE Transactions on Vehicular Technology
- Published
- 2019-06
- Volume
- 68
- Issue
- 6
- Pages
- 5379-5390
- ISSN
- 0018-9545 (ISSN-L)
- 1939-9359
- OpenAlex
- W2934625602
- Cited by
- 399
- References
- 46
- FWCI
- 57.45
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.