Recurrent Neural Networks for driver activity anticipation via sensory-fusion architecture
Open accessThe DOI matches its Crossref record; 48 of the 51 references it cites were found in a registry or an index; 12 of the 48 sources its claims cite were checked for support.
Abstract
Anticipating the future actions of a human is a widely studied problem in robotics that requires spatio-temporal reasoning. In this work we propose a deep learning approach for anticipation in sensory-rich robotics applications. We introduce a sensory-fusion architecture which jointly learns to anticipate and fuse information from multiple sensory streams. Our architecture consists of Recurrent Neural Networks (RNNs) that use Long Short-Term Memory (LSTM) units to capture long temporal dependencies. We train our architecture in a sequence-to-sequence prediction manner, and it explicitly learns to predict the future given only a partial temporal context. We further introduce a novel loss layer for anticipation which prevents over-fitting and encourages early anticipation. We use our architecture to anticipate driving maneuvers several seconds before they happen on a natural driving data set of 1180 miles. The context for maneuver anticipation comes from multiple sensors installed on the vehicle. Our approach shows significant improvement over the state-of-the-art in maneuver anticipation by increasing the precision from 77.4% to 90.5% and recall from 71.2% to 87.4%.
Overview
CoChat Summary
TL;DR
The paper proposes a sensory-fusion RNN-LSTM architecture for anticipating driver maneuvers from multiple vehicle sensors, trained sequence-to-sequence with an exponential loss layer. On 1180 miles of natural driving data, it improves maneuver anticipation over prior state-of-the-art, reaching 90.5% precision and 87.4% recall when using 3D head-pose features.
Key findings
- The Fusion-RNN with exponential loss (F-RNN-EL) improved all-maneuver anticipation over AIO-HMM on the same features, increasing precision from 77.4% to 84.5% and recall from 71.2% to 77.1%, with time-to-maneuver 3.58 s.
- Adding CLNF-tracked 68 facial landmarks and 3D head-pose features further increased F-RNN-EL to 90.5% precision and 87.4% recall, with time-to-maneuver 3.16 s.
- Fusing high-level representations from separate sensory RNNs outperformed simple concatenation: in the all-maneuvers setting, F-RNN-EL had about 6% higher precision and recall than S-RNN.
- The exponential loss improved over uniform loss by about 2% precision and recall in the all-maneuvers setting, while time-to-maneuver remained comparable.
Methods
- Architecture: each sensory stream is processed by an independent RNN with LSTM units; high-level representations are combined through a fully connected fusion layer and softmax output. Training is sequence-to-sequence, mapping partial observation sequences to the future event, with an exponential loss \(e^{-(T-t)}\log(y_t^k)\) that penalizes later mistakes more.
- Data: 1180 miles of natural freeway and city driving from 10 drivers, containing 700 annotated events (274 lane changes, 131 turns, 295 straight-driving instances), augmented to 2250 events. Context comes from a driver-facing camera, road-facing camera, GPS, and vehicle dynamics; improved vision uses CLNF for 68 facial landmarks and 3D head pose.
- Evaluation: 5-fold cross-validation with precision, recall, and time-to-maneuver, choosing a prediction threshold that maximizes F1. Baselines include Chance, Random-Forest, SVM, IOHMM, AIO-HMM, S-RNN, F-RNN-UL, and F-RNN-EL.
Limitations
- The paper does not include an explicit limitations section.
- Evident from the text: evaluation is on a single natural driving dataset of 1180 miles from 10 drivers with 5-fold cross-validation; no external or cross-dataset validation is reported.
- Evident from the text: precision and recall exclude the “driving straight” maneuver because anticipation algorithms default to it when not confident, and data augmentation only adds redundancy rather than new information.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures

![Fig. 2 Variations in the data set. Images from the data set [19] for a left lane change. ( Left ) Views from the road facing camera. ( Right ) Driving style of the drivers vary for the same maneuver.](https://figures.cochat.ai/paper-figures/f183cae4b17b4bccb552955c36acb9d6/recurrent-neural-networks-for-driver-activity-anticipation-via-sensory-fusion-figure-2.png)


![Fig. 5 Maneuver anticipation pipeline. Temporal context in maneuver anticipation comes from cameras facing the driver and the road, the GPS, and the vehicle's dynamics. ( a , b and c ) We improve upon the vision pipeline from Jain et al. [19] by tracking 68 landmark points on the driver's face and including the 3D head-pose features. ( d ) Using the Fusion-RNN we combine the sensory streams of features from inside the vehicle (driver facing camera), with the features from outside the vehicle (road camera, GPS, vehicle dynamics). ( e ) The model outputs the predicted probabilities of future maneuvers.](https://figures.cochat.ai/paper-figures/9617ccca82c44db4a16ed8745a46abd8/recurrent-neural-networks-for-driver-activity-anticipation-via-sensory-fusion-figure-5.png)
![Fig. 6 Improved features for maneuver anticipation. We track facial landmark points using the CLNF tracker [3] which results in more consistent 2D trajectories as compared to the KLT tracker [32] used by Jain et al. [19]. Furthermore, the CLNF also gives an estimate of the driver's 3D head pose.](https://figures.cochat.ai/paper-figures/c2c2ce8af37648d3918180e38c0abe7b/recurrent-neural-networks-for-driver-activity-anticipation-via-sensory-fusion-figure-6.png)


Verification
Claims the paper makes
4 Valid · 9 Not valid · 18 Not checked · 47 Own results and interpretations
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“Anticipating human activities has been shown to improve human-robot collaboration [40, 21, 25, 10].”
What the sources say
- Not checked — no full text of the sourcecites [40] [40] Z. Wang, K. M¨ ulling, M. Deisenroth, H. Amor, D. Vogt, B. Sch¨ olkopf, and J. Peters. Probabilistic movement modeling for intention inference in human-robot interaction. IJRR , 2013. - source
- Not matchedcites [21] [21] H. Koppula and A. Saxena. Anticipating human activities using object affordances for reactive robotic response. In RSS , 2013. -
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not checked — no full text of the sourcecites [25] [25] J. Mainprice and D. Berenson. Human-robot collaborative manipulation planning using early prediction of human motion. In IROS , 2013. - source
- Could not verifycites [10] [10] A. Dragan and S. Srinivasa. Formalizing assistive teleoperation. In RSS , 2012. - sourceExperimental — accuracy not yet measured
The Related Work section mentions human activity detection but does not say that anticipating activities improves human-robot collaboration. Searches for activity anticipation and collaborative outcomes returned passages about object manipulation planning instead, so our copy does not address the attributed claim. (LLM, conf=0.76)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“In another application, anticipating driving maneuvers several seconds in advance [19, 27, 38, 36] enables assistive cars to alert drivers before they make a dangerous maneuver.”
What the sources say
- Not checked — no full text of the sourcecites [19] [19] A. Jain, H. S. Koppula, B. Raghavan, S. Soh, and A. Saxena. Car that knows before you do: Anticipating maneuvers via learning temporal driving models. In ICCV (To appear) , 2015. - source
- Could not verifycites [27] [27] B. Morris, A. Doshi, and M. Trivedi. Lane change intent prediction for driver assistance: On-road design and evaluation. In IEEE IVS , 2011. - sourceExperimental — accuracy not yet measured
The passages from §2.2 (pp. 3–4) discuss recurrent-network dynamics, not driving. Searches for maneuver anticipation, driver alerts, and vehicle prediction found no passage in our copy addressing the claim. (LLM, conf=0.76)
- Not checked — no full text of the sourcecites [38] [38] R. Vasudevan, V. Shia, Y. Gao, R. Cervera-Navarro, R. Bajcsy, and F. Borrelli. Safe semi-autonomous control with enhanced driver modeling. In ACC , 2012. - source
- Not checked — no full text of the sourcecites [36] [36] A. Tawari, S. Sivaraman, M. Trivedi, T. Shannon, and M. Tippelhofer. Looking-in and looking-out vision for urban intelligent assistance: Estimation of driver attentive state and dynamic surround for safe merging and braking. In IEEE IVS , 2014. - source
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Furthermore, the time-to-maneuver remains comparable for F-RNN with and without exponential loss.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As shown in Table I, exponentially growing the loss improves performance.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The F1-score does not undergo large variations with changes to the prediction threshold.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
32 more claims
Create a free account to see all 78 claims and to verify the 1 reference still open.
- It may state two separate claims as one. 23
- Probably not a substantial claim. 25
- The quote only partly states this claim. 1
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ VII. CONCLUSION · p. 8
“It improved the precision from 78% to 84.5% and recall from 71.1% to 77.1%.”
§ I. INTRODUCTION · p. 1
“On this data set, our deep learning approach improves the state-ofthe-art in maneuver anticipation by increasing the precision from 77.4% to 84.5% and recall from 71.2% to 77.1% .”
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 4×
“A common remedy to vanishing gradients is to replace tanh non-linearities by Long ShortTerm Memory cells [16].”
BackgroundMethodInfluentialSequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766 · cited 1×
“previous works on RNNs and LSTM from the language [35], speech [15], and vision [8] communities.”
Are we ready for autonomous driving? The KITTI vision benchmark suite
Geiger, A., Lenz, P., & Urtasun, R. (2012) · 2012 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2012.6248074 · Cited by 14,897 · cited 1×
“cs applications is challenging because the contextual information can come from multiple sensors with different data modalities. Examples include autonomous vehicles that reason from multiple sensors [2] or robots that jointly reason over perception and language instructions [26]. In such applications the way informat…”
BackgroundShi, J., & Tomasi (1994) · Proceedings of IEEE Conference on Computer Vision and Pattern Recognition CVPR-94 · doi:10.1109/cvpr.1994.323794 · Cited by 7,724 · cited 2×
“[19], the driver facing camera detects discriminative points on the driver’s face and tracks the detected points across frames using the KLT tracker [32].”
MethodLong-term recurrent convolutional networks for visual recognition and description
Donahue, J., Hendricks, L. A., Guadarrama, S., Rohrbach, M., Venugopalan, S., Darrell, T., & Saenko, K. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298878 · Cited by 5,228 · cited 2×
“We consider RNNs in multimodal setting, which is related to the recent use of RNNs in image-captioning [8].”
MethodOn the difficulty of training recurrent neural networks
Pascanu, R., Mikolov, T., & Bengio, Y. (2012) · arXiv · doi:10.48550/arxiv.1211.5063 · Cited by 4,310 · cited 3×
“RNNs with this choice of f suffer from a well-studied problem of vanishing gradients [29], and hence are poor at capturing long temporal dependencies which are essential for anticipation.”
BackgroundMethodAction Recognition with Improved Trajectories
Wang, H., & Schmid, C. (2013) · 2013 IEEE International Conference on Computer Vision · doi:10.1109/iccv.2013.441 · Cited by 3,407
(2011) · Cited by 2,752
Hierarchical recurrent neural network for skeleton based action recognition
Du, Y., Wang, W., & Wang, L. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298714 · Cited by 1,895 · cited 14×
“The high level representations from both RNNs {(hx1 , ...,hxT ), (hz1, ...,hzT ) are then concatenated at each time step t and passed through a fully connected (fusion) layer which fuses the two representations (11), as shown in Figure 4.”
MethodInfluential
Research works printed but not linked: 2
- jain, 2015, Neural
- 0, Bosch urban
Cited by
Items 101–200 of 272
Learning to Anticipate Future with Dynamic Context Removal
Xu, X., Li, Y., & Lu, C. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01240 · Cited by 31
Temporal Information Fusion Network for Driving Behavior Prediction
Guo, C., Liu, H., Chen, J., & Ma, H. (2023) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2023.3267150 · Cited by 30
Rivoir, D., Bodenstedt, S., Funke, I., von Bechtolsheim, F., Distler, M., Weitz, J., & Speidel, S. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-59716-0_72 · Cited by 30
ERA: Expert Retrieval and Assembly for Early Action Prediction
Foo, L. G., Li, T., Rahmani, H., Ke, Q., & Liu, J. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19830-4_38 · Cited by 32
Big Data Perspective for Driver/Driving Behavior
Terzi, R., Sağıroğlu, Ş., & Demi̇rezen, M. U. (2018) · IEEE Intelligent Transportation Systems Magazine · doi:10.1109/mits.2018.2879220 · Cited by 19
Li, L. (2018) · International Journal of Machine Learning and Computing · doi:10.18178/ijmlc.2018.8.5.724 · Cited by 19
Traffic Risk Assessment: A Two-Stream Approach Using Dynamic-Attention
Corcoran, G.-P., & Clark, J. J. (2019) · Canadian Conference on Computer and Robot Vision · doi:10.1109/crv.2019.00030 · Cited by 25
Dear-Net: Learning Diversities for Skeleton-Based Early Action Recognition
Wang, R., Liu, J., Ke, Q., Peng, D., & Lei, Y. (2021) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2021.3139768 · Cited by 23
E2ENNet: An end-to-end neural network for emotional brain-computer interface
Han, Z., Chang, H., Zhou, X., Wang, J., Wang, L., & Shao, Y. (2022) · Frontiers in Computational Neuroscience · doi:10.3389/fncom.2022.942979 · Cited by 21
Chen, W., Lin, Y., & Chen, W. (2019) · International Conference on Informatics, Electronics & Vision (ICIEV) · doi:10.1109/iciev.2019.8858531 · Cited by 20
Anticipating Daily Intention Using On-wrist Motion Triggered Sensing
Wu, T.-Y., Chien, T.-A., Chan, C.-S., Hu, C.-W., & Sun, M. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.15 · Cited by 22
GA-Net: A Guidance Aware Network for Skeleton-Based Early Activity Recognition
Wang, W., Chang, F., Liu, C., Li, G., & Wang, B. (2021) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2021.3137745 · Cited by 21
Díaz-Álvarez, A., Clavijo, M., Jiménez, F., & Serradilla, F. (2021) · Sensors · doi:10.3390/s21020475 · Cited by 17
Systematic literature review on driving behavior
Vilaca, A., Cunha, P., & Ferreira, A. L. (2017) · IEEE International Conference on Intelligent Transportation Systems (ITSC) · doi:10.1109/itsc.2017.8317786 · Cited by 24
Trajectory Tracking Control for Autonomous Vehicles in Different Cut-in Scenarios
Chen, Y., & Wang, J. (2019) · Proceedings of the ... American Control Conference/Proceedings of the American Control Conference · doi:10.23919/acc.2019.8814985 · Cited by 17
Multi-View Region of Interest Prediction for Autonomous Driving Using Semi-Supervised Labeling
Hofbauer, M., Kuhn, C. B., Meng, J., Petrović, G., & Steinbach, E. G. (2020) · IEEE International Conference on Intelligent Transportation Systems (ITSC) · doi:10.1109/itsc45102.2020.9294387 · Cited by 17
Peng, L., Wang, Y., Zhang, F., et al. (2021) · IET Intelligent Transport Systems · doi:10.1049/itr2.12053 · Cited by 16
Anticipating Future Relations via Graph Growing for Action Prediction
Wu, X., Zhao, J., & Wang, R. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i4.16402 · Cited by 16
Failure Detection Using Proprioceptive, Auditory and Visual Modalities
İnceoğlu, A., İnce, G., Yaslan, Y., & Sarıel, S. (2018) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2018.8594169 · Cited by 17
Enhancing the Tracking of Seedling Growth Using RGB-Depth Fusion and Deep Learning
Garbouge, H., Rasti, P., & Rousseau, D. (2021) · Sensors · doi:10.3390/s21248425 · Cited by 21
Domínguez-Vidal, J. E., Rodríguez, N., & Sanfeliu, A. (2024) · International Journal of Social Robotics · doi:10.1007/s12369-024-01103-7 · Cited by 22
Predicting Human Intent for Cooperative Physical Human-Robot Interaction Tasks
Maithani, H., Ramón, J. A. C., & Mezouar, Y. (2019) · International Conference on Control and Automation · doi:10.1109/icca.2019.8899490 · Cited by 17
Multi-label movie genre classification based on multimodal fusion
Cai, Z.-H., Ding, H., Wu, J., Xi, Y., Wu, X., & Cui, X. (2023) · Multimedia Tools and Applications · doi:10.1007/s11042-023-16121-2 · Cited by 16
DROID: Driver-Centric Risk Object IDentification
Li, C., Chan, S. H., & Chen, Y. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3294305 · Cited by 24
Real-Time Workload Classification during Driving using HyperNetworks
Wang, R., Amadori, P. V., & Demiris, Y. (2018) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2018.8594305 · Cited by 19
Magi-Net: Meta Negative Network for Early Activity Prediction
Wang, W., Chang, F., Zhang, J., Yan, R., Liu, C., Wang, B., & Shou, M. Z. (2023) · IEEE Transactions on Image Processing · doi:10.1109/tip.2023.3279991 · Cited by 17
Driving Maneuver Detection via Sequence Learning from Vehicle Signals and Video Images
Peng, X., Liu, R., Murphey, Y. L., Stent, S., & Li, Y. (2018) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr.2018.8546255 · Cited by 17
Global Regularizer and Temporal-Aware Cross-Entropy for Skeleton-Based Early Action Recognition
Ke, Q., Liu, J., Bennamoun, M., Rahmani, H., An, S., Sohel, F. A., & Boussaïd, F. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-20870-7_45 · Cited by 13
Ma, Y., Ye, W., Cao, X., Abdelraouf, A., Han, K., Gupta, R., & Wang, Z. (2023) · IEEE International Conference on Intelligent Transportation Systems (ITSC) · doi:10.1109/itsc57777.2023.10421798 · Cited by 31
Karaduman, M., & Eren, H. (2017) · International Istanbul Smart Grids and Cities Congress and Fair (ICSG) · doi:10.1109/sgcf.2017.7947614 · Cited by 13
Sharma, S., & Ghose, A. (2018) · IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops) · doi:10.1109/percomw.2018.8480148 · Cited by 15
Deep Action- and Context-Aware Sequence Learning for Activity Recognition and AnticipationPreprint
Akbarian, M. S. A., Saleh, F., Fernando, B., Salzmann, M., Petersson, L., & Andersson, L. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.05520 · Cited by 13
Identifying Distracted and Drowsy Drivers Using Naturalistic Driving Data
Yadawadkar, S., Mayer, B., Lokegaonkar, S., Islam, M. R., Ramakrishnan, N., Song, M., & Mollenhauer, M. (2018) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata.2018.8622612 · Cited by 20
Confidence-Guided Self Refinement for Action Prediction in Untrimmed Videos
Hou, J., Wu, X., Wang, R., Luo, J., & Jia, Y. (2020) · IEEE Transactions on Image Processing · doi:10.1109/tip.2020.2987425 · Cited by 16
Driving Maneuvers Prediction Based on Cognition-driven and Data-driven Method
Zhou, D. H., Ma, H., & Dong, Y. (2018) · IEEE International Conference on Visual Communications and Image Processing (VCIP) · doi:10.1109/vcip.2018.8698695 · Cited by 15
RsSafe: Personalized Driver Behavior Prediction for Safe Driving
Bhumika, B., Das, D., & Das, S. K. (2022) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn55064.2022.9892982 · Cited by 15
A Deep Multichannel Network Model for Driving Behavior Risk Classification
Li, D., Wang, Y., & Xu, W. X. (2022) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2022.3201378 · Cited by 12
Tactile Perception for Teleoperated Robotic Exploration within Granular Media
Jia, S., & Santos, V. J. (2021) · ACM Transactions on Human-Robot Interaction · doi:10.1145/3459996 · Cited by 15
User-Aware Multilevel Cognitive Workload Estimation From Multimodal Physiological Signals
Amadori, P. V., & Demiris, Y. (2023) · IEEE Transactions on Cognitive and Developmental Systems · doi:10.1109/tcds.2023.3342139 · Cited by 12
Identifying Near-Miss Traffic Incidents in Event Recorder Data
Yamamoto, S., Kurashima, T., & Toda, H. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-47436-2_54 · Cited by 14
Multimodal Global Relation Knowledge Distillation for Egocentric Action Anticipation
Huang, Y., Yang, X., & Xu, C. (2021) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3474085.3475327 · Cited by 14
Towards Immediate Backchannel Generation Using Attention-Based Early Prediction Model
Adiba, A. I., Homma, T., & Miyoshi, T. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9414193 · Cited by 19
Decision Anticipation for Driving Assistance Systems
Amadori, P. V., Fischer, T., Wang, R., & Demiris, Y. (2020) · IEEE International Conference on Intelligent Transportation Systems (ITSC) · doi:10.1109/itsc45102.2020.9294216 · Cited by 12
Lane Change Prediction for Autonomous Driving With Transferred Trajectory Interaction
Lu, Y., Xu, P., Jiang, X., Bashir, A. K., Gadekallu, T. R., Wang, W. W., & Hu, X. (2025) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2025.3542517 · Cited by 26
Rethinking the Evaluation of Driver Behavior Analysis Approaches
Chai, W., Wang, J., Chen, J., Velipasalar, S., & Sharma, A. (2024) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2024.3354506 · Cited by 11
Neis, N., & Beyerer, J. (2023) · IEEE International Conference on Intelligent Transportation Systems (ITSC) · doi:10.1109/itsc57777.2023.10421975 · Cited by 10
Tian, Y., Zhao, X. K., Liu, R., et al. (2023) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2023.3330008 · Cited by 10
Action Anticipation with Goal Consistency
Zatsarynna, O., & Gall, J. (2023) · Proceedings - International Conference on Image Processing · doi:10.1109/icip49359.2023.10222914 · Cited by 10
Cao, C., Sun, Z., Lv, Q., Min, L., & Zhang, Y. (2024) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2024.3425598 · Cited by 12
Modeling Popularity in Asynchronous Social Media Streams with Recurrent Neural Networks
Mishra, S., Rizoiu, M.-A., & Xie, L. (2018) · arXiv · doi:10.48550/arxiv.1804.02101 · Cited by 10
Driving Maneuver Classification Using Domain Specific Knowledge and Transfer Learning
Sarker, S., Haque, M. M., & Dewan, M. A. A. (2021) · IEEE Access · doi:10.1109/access.2021.3089660 · Cited by 11
A Learning-Based Prediction Model for Baby Accidents
Wang, P., Lien, S.-F., & Lee, M. (2019) · Proceedings - International Conference on Image Processing · doi:10.1109/icip.2019.8803820 · Cited by 8
Konstantakopoulos, I. C., Das, H. P., Barkan, A. R., et al. (2019) · arXiv · doi:10.48550/arxiv.1910.07899 · Cited by 9
Uncertainty-Boosted Robust Video Activity Anticipation
Qi, Z., Wang, S., Zhang, W., & Huang, Q. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3393730 · Cited by 41
Morales-Alvarez, W., Certad, N., Roitberg, A., Stiefelhagen, R., & Olaverri-Monreal, C. (2023) · IEEE International Conference on Intelligent Transportation Systems (ITSC) · doi:10.1109/itsc57777.2023.10422678 · Cited by 10
Predicting Driver Lane Change Maneuvers Using Driver's Face
Moussaid, A., Berrada, I., Kamili, M. E., & Fardousse, K. (2019) · International Conference on Wireless Networks and Mobile Communications (WINCOM) · doi:10.1109/wincom47513.2019.8942531 · Cited by 9
Virmani, S., & Gite, S. (2017) · International Conference on Computing, Communication and Networking Technologies (ICCCNT) · doi:10.1109/icccnt.2017.8204039 · Cited by 10
RetractedBrooks, C., Atreya, M., & Szafir, D. J. (2018) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2018.8594180 · Cited by 10
Frenet Coordinate Based Driving Maneuver Prediction at Roundabouts Using LSTM Networks
Vogl, C., Sackmann, M., Kürzinger, L., & Hofmann, U. (2020) · ACM Computer Science in Cars Symposium (CSCS) · doi:10.1145/3385958.3430475 · Cited by 11
Adaptive Window Size Based Deep Neural Network for Driving Maneuver Prediction
Jun, G., Zhu, H., & Murphey, Y. L. (2020) · Chinese Control and Decision Conference (CCDC) · doi:10.1109/ccdc49329.2020.9163824 · Cited by 9
Canuto, C., Moreno, P., Samatelo, J. L. A., Vassallo, R. F., & Santos-Victor, J. (2020) · Neurocomputing · doi:10.1016/j.neucom.2020.07.135 · Cited by 12
Morawski, I., Lie, W., Aing, L., Chiang, J., & Chen, K. (2022) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2022.3229059 · Cited by 10
Unified Recurrence Modeling for Video Action Anticipation
Tai, T., Fiameni, G., Lee, C., See, S. C. W., & Lanz, O. (2022) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr56361.2022.9956467 · Cited by 12
Route and Stopping Intent Prediction at Intersections From Car Fleet Data
Gross, F., Jordan, J., Weninger, F. J., Klanner, F., & Schuller, B. W. (2016) · IEEE Transactions on Intelligent Vehicles · doi:10.1109/tiv.2016.2617625 · Cited by 10
Interpretable Deep Feature Propagation for Early Action RecognitionPreprint
Zhao, H., & Wildes, R. P. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.05122 · Cited by 13
Knowledge Distillation for Human Action Anticipation
Tran, V., Wang, Y., Zhang, Z., & Hoai, M. (2021) · Proceedings - International Conference on Image Processing · doi:10.1109/icip42928.2021.9506693 · Cited by 14
Back to the Future: Knowledge Distillation for Human Action AnticipationPreprint
Trần, V. C., Wang, Y., & Hoai, M. (2019) · arXiv (Cornell University) · Cited by 7
Deformable Morphing and Multivariable Stiffness in the Evolutionary Robotics
Feng, H., Xue, Y., Li, H., Tang, Z., Wang, W., Wei, Z., Zeng, G., Li, M., & Dai, J. S. (2023) · International Journal of Automotive Manufacturing and Materials · doi:10.53941/ijamm.2023.100013 · Cited by 9
RiskBench: A Scenario-based Benchmark for Risk Identification
Kung, C.-H., Yang, C.-C., Pao, P.-Y., Lu, S.-W., Chen, P.-L., Lu, H.-C., & Chen, Y. (2024) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra57147.2024.10610270 · Cited by 20
Predicting Human Errors from Gaze and Cursor Movements
Saboundji, R. R., & Rill, R. A. (2020) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn48605.2020.9207189 · Cited by 8
Classifying Near-Miss Traffic Incidents through Video, Sensor, and Object Features
Yamamoto, S., Kurashima, T., & Toda, H. (2022) · IEICE Transactions on Information and Systems · doi:10.1587/transinf.2021edp7017 · Cited by 6
Anticipating the Start of User Interaction for Service Robot in the Wild
Ito, K., Kong, Q., Horiguchi, S., Sumiyoshi, T., & Nagamatsu, K. (2020) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra40945.2020.9196548 · Cited by 9
Visual Forecasting by Imitating Dynamics in Natural Sequences
Zeng, K.-H., Shen, W. B., Huang, D.-A., et al. (2017) · arXiv · doi:10.48550/arxiv.1708.05827 · Cited by 6
He, B., Yu, N., Wang, Z., & Chen, X. (2024) · Applied Sciences · doi:10.3390/app14062460 · Cited by 7
Robotic Walker with High Maneuverability through Deep Learning for Sensor Fusion
Molano, C., Chen, L.-P., & Fu, L. (2019) · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/smc.2019.8914379 · Cited by 6
Xu, X., Li, Y., & Lu, C. (2023) · International Journal of Computer Vision · doi:10.1007/s11263-023-01850-6 · Cited by 9
Improve the LSTM Trajectory Prediction Accuracy through an Attention Mechanism
Zhang, T., & Wang, Z. (2022) · IEEE Transportation Electrification Conference and Expo · doi:10.1109/itec53557.2022.9813863 · Cited by 6
A Multimodal and Hybrid Framework for Human Navigational Intent Inference
Zhang, Z., Rhim, J., Lim, A., & Chen, M. (2021) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros51168.2021.9635900 · Cited by 8
Video Action Understanding: A Tutorial.Preprint
Hutchinson, M., & Gadepally, V. N. (2020) · arXiv (Cornell University) · Cited by 6
Towards Lifelong Self-Supervision: A Deep Learning Direction for RoboticsPreprint
Wong, J. M. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.00201 · Cited by 13
Tanama, C., Peng, K., Marinov, Z., Stiefelhagen, R., & Roitberg, A. (2023) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros55552.2023.10342203 · Cited by 12
Metric-Free Exploration for Topological Mapping by Task and Motion Imitation in Feature Space
He, Y., Fang, I., Li, Y., Shah, R., & Feng, C. (2023) · Robotics: Science and Systems · doi:10.15607/rss.2023.xix.099 · Cited by 14
Attention-Driven Driving Maneuver Detection System
Peng, X., Zhao, A., Wang, S., Murphey, Y. L., & Li, Y. (2019) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2019.8852404 · Cited by 6
Generative Hybrid Representations for Activity Forecasting with No-Regret LearningPreprint
Guan, J., Yuan, Y., Kitani, K., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.06250 · Cited by 5
Using Recurrent Neural Networks for Action and Intention Recognition of Car Drivers
Torstensson, M., Durán, B., & Englund, C. (2019) · International Conference on Pattern Recognition Applications and Methods · doi:10.5220/0007682502320242 · Cited by 6
Towards Practical Driver Cognitive Load Detection Based on Visual Attention Information
Liu, C. C. (2017) · TSpace (University of Toronto) · Cited by 9
Zhao, Z., Wei, Z., Tian, D., Reimer, B., Gershon, P., & Moradi‐Pari, E. (2023) · IEEE Intelligent Vehicles Symposium · doi:10.1109/iv55152.2023.10186602 · Cited by 7
TODO-Net: Temporally Observed Domain Contrastive Network for 3-D Early Action Prediction
Wang, W., Chang, F., Liu, C., Wang, B., & Liu, Z. (2024) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2024.3394254 · Cited by 7
Looking Inside Out: Anticipating Driver Intent From Videos
Kung, Y.-C., Zhang, A., Wang, J., & Biswas, J. (2024) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra57147.2024.10610257 · Cited by 5
Classifying Human Manual Control Behavior Using LSTM Recurrent Neural Networks
Versteeg, R. C. W., Pool, D. M., & Mulder, M. (2023) · IEEE Transactions on Human-Machine Systems · doi:10.1109/thms.2023.3327145 · Cited by 4
FedPRM: Federated Personalized Mixture Representation for Driver Intention Prediction
Zhu, Z., Zhao, S., Chu, C., Wang, C. H., Du, A., & He, B. (2024) · IEEE Transactions on Intelligent Vehicles · doi:10.1109/tiv.2024.3416846 · Cited by 7
Behavioral Research and Practical Models of Drivers' AttentionPreprint
Kotseruba, I., & Tsotsos, J. K. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.05677 · Cited by 48
Delay Mitigation for Backchannel Prediction in Spoken Dialog System
Adiba, A. I., Homma, T., Bertero, D., Sumiyoshi, T., & Nagamatsu, K. (2020) · Lecture notes in electrical engineering · doi:10.1007/978-981-15-8395-7_10 · Cited by 7
Temporal Multimodal Fusion for Driver Behavior Prediction Tasks using Gated Recurrent Fusion Units.
Narayanan, A., Siravuru, A., & Dariush, B. (2019) · Cited by 8
Spatial-temporal episodic memory modeling for ADLs: encoding, retrieval, and prediction
Song, X., Wang, D., Quek, C. H., Tan, A., & Wang, Y. (2023) · Complex & Intelligent Systems · doi:10.1007/s40747-023-01298-8 · Cited by 5
Patera, P., Chen, Y., & Fang, W. (2025) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2025.3552895 · Cited by 4
Adversarial Graph Convolutional Network for Skeleton-Based Early Action Prediction
Li, X.-S., Zhang, N., Cai, B.-Q., Kang, J., & Zhao, F. (2024) · Journal of Computer Science and Technology · doi:10.1007/s11390-023-2638-7 · Cited by 6
FutureHuman3D: Forecasting Complex Long-Term 3D Human Behavior from Video Observations
Diller, C., Funkhouser, T. A., & Dai, A. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01881 · Cited by 7
Anticipating Traffic Accidents with Adaptive Loss and Large-scale Incident DBPreprint
Suzuki, T., Kataoka, H., Aoki, Y., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1804.02675 · Cited by 4
Individualized Driving Intention Prediction With Inverse Reinforcement Learning
Liu, S., Li, X., Chen, J., Guo, C., Wu, J., Luo, Q., & Ma, H. (2025) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2025.3543553 · Cited by 9
Metadata
- Authors
- Jain, A., Singh, A., Koppula, H. S., Soh, S., & Saxena, A.
- Affiliations
- Cornell University (US)
- Stanford University (US)
- Venue
- 2016 IEEE International Conference on Robotics and Automation (ICRA)
- Publisher
- IEEE
- Published
- 2016-05
- Pages
- 3118-3125
- ISSN
- 1050-4729 (ISSN-L)
- OpenAlex
- W2174887554
- Cited by
- 271
- References
- 51
- Citations by year
- 2026: 2 · 2025: 16 · 2024: 24 · 2023: 34 · 2022: 19 · 2021: 42 · 2020: 33 · 2019: 41 · 2018: 31 · 2017: 21 · 2016: 8
- FWCI
- 43.54
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- Proceedings - IEEE International Conference on Robotics and Automation/Proceedings (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1426452)
- Nvidia (K40 GPU)
- Office of Naval Research (N00014)
- Topics
- Physical Sciences › Engineering › Automotive Engineering › Autonomous Vehicle Technology and Safety
- Social Sciences › Human Factors and Ergonomics › Human-Automation Interaction and Safety
- Social Sciences › Transportation › Traffic Prediction and Management Techniques
- Keywords
- sensory fusion, recurrent neural networks, long short-term memory, sequence-to-sequence prediction, driving data, spatio-temporal reasoning, automotive sensors
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.