The DOI matches its Crossref record; 48 of the 51 references it cites were found in a registry or an index; 12 of the 48 sources its claims cite were checked for support.
Recurrent Neural Networks for driver activity anticipation via sensory-fusion architecture
Open accessAbstract
Anticipating the future actions of a human is a widely studied problem in robotics that requires spatio-temporal reasoning. In this work we propose a deep learning approach for anticipation in sensory-rich robotics applications. We introduce a sensory-fusion architecture which jointly learns to anticipate and fuse information from multiple sensory streams. Our architecture consists of Recurrent Neural Networks (RNNs) that use Long Short-Term Memory (LSTM) units to capture long temporal dependencies. We train our architecture in a sequence-to-sequence prediction manner, and it explicitly learns to predict the future given only a partial temporal context. We further introduce a novel loss layer for anticipation which prevents over-fitting and encourages early anticipation. We use our architecture to anticipate driving maneuvers several seconds before they happen on a natural driving data set of 1180 miles. The context for maneuver anticipation comes from multiple sensors installed on the vehicle. Our approach shows significant improvement over the state-of-the-art in maneuver anticipation by increasing the precision from 77.4% to 90.5% and recall from 71.2% to 87.4%.
Overview
CoChat Summary
TL;DR
The paper proposes a sensory-fusion RNN-LSTM architecture for anticipating driver maneuvers from multiple vehicle sensors, trained sequence-to-sequence with an exponential loss layer. On 1180 miles of natural driving data, it improves maneuver anticipation over prior state-of-the-art, reaching 90.5% precision and 87.4% recall when using 3D head-pose features.
Key findings
- The Fusion-RNN with exponential loss (F-RNN-EL) improved all-maneuver anticipation over AIO-HMM on the same features, increasing precision from 77.4% to 84.5% and recall from 71.2% to 77.1%, with time-to-maneuver 3.58 s.
- Adding CLNF-tracked 68 facial landmarks and 3D head-pose features further increased F-RNN-EL to 90.5% precision and 87.4% recall, with time-to-maneuver 3.16 s.
- Fusing high-level representations from separate sensory RNNs outperformed simple concatenation: in the all-maneuvers setting, F-RNN-EL had about 6% higher precision and recall than S-RNN.
- The exponential loss improved over uniform loss by about 2% precision and recall in the all-maneuvers setting, while time-to-maneuver remained comparable.
Methods
- Architecture: each sensory stream is processed by an independent RNN with LSTM units; high-level representations are combined through a fully connected fusion layer and softmax output. Training is sequence-to-sequence, mapping partial observation sequences to the future event, with an exponential loss \(e^{-(T-t)}\log(y_t^k)\) that penalizes later mistakes more.
- Data: 1180 miles of natural freeway and city driving from 10 drivers, containing 700 annotated events (274 lane changes, 131 turns, 295 straight-driving instances), augmented to 2250 events. Context comes from a driver-facing camera, road-facing camera, GPS, and vehicle dynamics; improved vision uses CLNF for 68 facial landmarks and 3D head pose.
- Evaluation: 5-fold cross-validation with precision, recall, and time-to-maneuver, choosing a prediction threshold that maximizes F1. Baselines include Chance, Random-Forest, SVM, IOHMM, AIO-HMM, S-RNN, F-RNN-UL, and F-RNN-EL.
Limitations
- The paper does not include an explicit limitations section.
- Evident from the text: evaluation is on a single natural driving dataset of 1180 miles from 10 drivers with 5-fold cross-validation; no external or cross-dataset validation is reported.
- Evident from the text: precision and recall exclude the “driving straight” maneuver because anticipation algorithms default to it when not confident, and data augmentation only adds redundancy rather than new information.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures

![Fig. 2 Variations in the data set. Images from the data set [19] for a left lane change. ( Left ) Views from the road facing camera. ( Right ) Driving style of the drivers vary for the same maneuver.](https://figures.cochat.ai/paper-figures/f183cae4b17b4bccb552955c36acb9d6/recurrent-neural-networks-for-driver-activity-anticipation-via-sensory-fusion-figure-2.png)


![Fig. 5 Maneuver anticipation pipeline. Temporal context in maneuver anticipation comes from cameras facing the driver and the road, the GPS, and the vehicle's dynamics. ( a , b and c ) We improve upon the vision pipeline from Jain et al. [19] by tracking 68 landmark points on the driver's face and including the 3D head-pose features. ( d ) Using the Fusion-RNN we combine the sensory streams of features from inside the vehicle (driver facing camera), with the features from outside the vehicle (road camera, GPS, vehicle dynamics). ( e ) The model outputs the predicted probabilities of future maneuvers.](https://figures.cochat.ai/paper-figures/9617ccca82c44db4a16ed8745a46abd8/recurrent-neural-networks-for-driver-activity-anticipation-via-sensory-fusion-figure-5.png)
![Fig. 6 Improved features for maneuver anticipation. We track facial landmark points using the CLNF tracker [3] which results in more consistent 2D trajectories as compared to the KLT tracker [32] used by Jain et al. [19]. Furthermore, the CLNF also gives an estimate of the driver's 3D head pose.](https://figures.cochat.ai/paper-figures/c2c2ce8af37648d3918180e38c0abe7b/recurrent-neural-networks-for-driver-activity-anticipation-via-sensory-fusion-figure-6.png)


Verification
Claims the paper makes
4 Valid · 9 Not valid · 18 Not checked · 47 Own results and interpretations
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“Anticipating human activities has been shown to improve human-robot collaboration [40, 21, 25, 10].”
What the sources say
- Not checked — no full text of the sourcecites [40] [40] Z. Wang, K. M¨ ulling, M. Deisenroth, H. Amor, D. Vogt, B. Sch¨ olkopf, and J. Peters. Probabilistic movement modeling for intention inference in human-robot interaction. IJRR , 2013. - source
- Not matchedcites [21] [21] H. Koppula and A. Saxena. Anticipating human activities using object affordances for reactive robotic response. In RSS , 2013. -
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not checked — no full text of the sourcecites [25] [25] J. Mainprice and D. Berenson. Human-robot collaborative manipulation planning using early prediction of human motion. In IROS , 2013. - source
- Could not verifycites [10] [10] A. Dragan and S. Srinivasa. Formalizing assistive teleoperation. In RSS , 2012. - sourceExperimental — accuracy not yet measured
The Related Work section mentions human activity detection but does not say that anticipating activities improves human-robot collaboration. Searches for activity anticipation and collaborative outcomes returned passages about object manipulation planning instead, so our copy does not address the attributed claim. (LLM, conf=0.76)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“In another application, anticipating driving maneuvers several seconds in advance [19, 27, 38, 36] enables assistive cars to alert drivers before they make a dangerous maneuver.”
What the sources say
- Not checked — no full text of the sourcecites [19] [19] A. Jain, H. S. Koppula, B. Raghavan, S. Soh, and A. Saxena. Car that knows before you do: Anticipating maneuvers via learning temporal driving models. In ICCV (To appear) , 2015. - source
- Could not verifycites [27] [27] B. Morris, A. Doshi, and M. Trivedi. Lane change intent prediction for driver assistance: On-road design and evaluation. In IEEE IVS , 2011. - sourceExperimental — accuracy not yet measured
The passages from §2.2 (pp. 3–4) discuss recurrent-network dynamics, not driving. Searches for maneuver anticipation, driver alerts, and vehicle prediction found no passage in our copy addressing the claim. (LLM, conf=0.76)
- Not checked — no full text of the sourcecites [38] [38] R. Vasudevan, V. Shia, Y. Gao, R. Cervera-Navarro, R. Bajcsy, and F. Borrelli. Safe semi-autonomous control with enhanced driver modeling. In ACC , 2012. - source
- Not checked — no full text of the sourcecites [36] [36] A. Tawari, S. Sivaraman, M. Trivedi, T. Shannon, and M. Tippelhofer. Looking-in and looking-out vision for urban intelligent assistance: Estimation of driver attentive state and dynamic surround for safe merging and braking. In IEEE IVS , 2014. - source
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Furthermore, the time-to-maneuver remains comparable for F-RNN with and without exponential loss.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As shown in Table I, exponentially growing the loss improves performance.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The F1-score does not undergo large variations with changes to the prediction threshold.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
32 more claims
Create a free account to see all 78 claims and to verify the 1 reference still open.
- It may state two separate claims as one. 23
- Probably not a substantial claim. 25
- The quote only partly states this claim. 1
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ VII. CONCLUSION · p. 8
“It improved the precision from 78% to 84.5% and recall from 71.1% to 77.1%.”
§ I. INTRODUCTION · p. 1
“On this data set, our deep learning approach improves the state-ofthe-art in maneuver anticipation by increasing the precision from 77.4% to 84.5% and recall from 71.2% to 77.1% .”
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 4×
“A common remedy to vanishing gradients is to replace tanh non-linearities by Long ShortTerm Memory cells [16].”
BackgroundMethodInfluentialSequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766 · cited 1×
“previous works on RNNs and LSTM from the language [35], speech [15], and vision [8] communities.”
Are we ready for autonomous driving? The KITTI vision benchmark suite
Geiger, A., Lenz, P., & Urtasun, R. (2012) · 2012 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2012.6248074 · Cited by 14,897 · cited 1×
“cs applications is challenging because the contextual information can come from multiple sensors with different data modalities. Examples include autonomous vehicles that reason from multiple sensors [2] or robots that jointly reason over perception and language instructions [26]. In such applications the way informat…”
BackgroundShi, J., & Tomasi (1994) · Proceedings of IEEE Conference on Computer Vision and Pattern Recognition CVPR-94 · doi:10.1109/cvpr.1994.323794 · Cited by 7,724 · cited 2×
“[19], the driver facing camera detects discriminative points on the driver’s face and tracks the detected points across frames using the KLT tracker [32].”
MethodLong-term recurrent convolutional networks for visual recognition and description
Donahue, J., Hendricks, L. A., Guadarrama, S., Rohrbach, M., Venugopalan, S., Darrell, T., & Saenko, K. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298878 · Cited by 5,228 · cited 2×
“We consider RNNs in multimodal setting, which is related to the recent use of RNNs in image-captioning [8].”
MethodOn the difficulty of training recurrent neural networks
Pascanu, R., Mikolov, T., & Bengio, Y. (2012) · arXiv · doi:10.48550/arxiv.1211.5063 · Cited by 4,310 · cited 3×
“RNNs with this choice of f suffer from a well-studied problem of vanishing gradients [29], and hence are poor at capturing long temporal dependencies which are essential for anticipation.”
BackgroundMethodAction Recognition with Improved Trajectories
Wang, H., & Schmid, C. (2013) · 2013 IEEE International Conference on Computer Vision · doi:10.1109/iccv.2013.441 · Cited by 3,407
(2011) · Cited by 2,752
Hierarchical recurrent neural network for skeleton based action recognition
Du, Y., Wang, W., & Wang, L. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298714 · Cited by 1,895 · cited 14×
“The high level representations from both RNNs {(hx1 , ...,hxT ), (hz1, ...,hzT ) are then concatenated at each time step t and passed through a fully connected (fusion) layer which fuses the two representations (11), as shown in Figure 4.”
MethodInfluential
Research works printed but not linked: 2
- jain, 2015, Neural
- 0, Bosch urban
Cited by
Deep multimodal learning: a survey on recent advances and trends
Ramachandram, D., & Taylor, G. W. (2017) · IEEE Signal Processing Magazine · doi:10.1109/msp.2017.2738401 · Cited by 1,139
“To solve this problem, Jain et al. [35] incorporated a temporal term that grows exponentially in time into their cost function for a multimodal RNN with LSTM units.”
BackgroundDESIRE: Distant Future Prediction in Dynamic Scenes with Interacting Agents
Lee, N., Choi, W., Vernaza, P., Choy, C. B., Torr, P. H. S., & Chandraker, M. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.233 · Cited by 1,067
“• RNN ED-SI: An RNN ED augmented with our SCF unit into the decoder similar to [17].”
BackgroundMethodHuang, S.-C., Pareek, A., Seyyedi, S., Banerjee, I., & Lungren, M. P. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-00341-z · Cited by 934
“While selecting features known to be associated with disease is meaningful, future work may further bene fi t from utilizing large volumes of feature-rich data, as seen in fi elds outside medicine such as autonomous driving 44,45 .”
BackgroundA Bi-Hemisphere Domain Adversarial Neural Network Model for EEG Emotion Recognition
Li, Y., Zheng, W., Zong, Y., Cui, Z., Zhang, T., & Zhou, X. (2021) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2018.2885474 · Cited by 389
“In contrast to the traditional neural network methods, the recurrent neural networks (RNN) method [33] aims to map the input sequence into a series of hidden states through a complex dynamics transform with a chain of repeating modules of neural network and remember and forget the information in…”
MethodDriver behavior detection and classification using deep convolutional neural networks
Shahverdy, M., Fathy, M., Berangi, R., & Sabokrou, M. (2020) · Expert Systems with Applications · doi:10.1016/j.eswa.2020.113240 · Cited by 352
Metadata
- Authors
- Jain, A., Singh, A., Koppula, H. S., Soh, S., & Saxena, A.
- Affiliations
- Cornell University (US)
- Stanford University (US)
- Venue
- 2016 IEEE International Conference on Robotics and Automation (ICRA)
- Publisher
- IEEE
- Published
- 2016-05
- Pages
- 3118-3125
- ISSN
- 1050-4729 (ISSN-L)
- OpenAlex
- W2174887554
- Cited by
- 271
- References
- 51
- Citations by year
- 2026: 2 · 2025: 16 · 2024: 24 · 2023: 34 · 2022: 19 · 2021: 42 · 2020: 33 · 2019: 41 · 2018: 31 · 2017: 21 · 2016: 8
- FWCI
- 43.54
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- Proceedings - IEEE International Conference on Robotics and Automation/Proceedings (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1426452)
- Nvidia (K40 GPU)
- Office of Naval Research (N00014)
- Topics
- Physical Sciences › Engineering › Automotive Engineering › Autonomous Vehicle Technology and Safety
- Social Sciences › Human Factors and Ergonomics › Human-Automation Interaction and Safety
- Social Sciences › Transportation › Traffic Prediction and Management Techniques
- Keywords
- sensory fusion, recurrent neural networks, long short-term memory, sequence-to-sequence prediction, driving data, spatio-temporal reasoning, automotive sensors
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Recurrent Neural Networks for driver activity anticipation via sensory-fusion architecture. https://app.cochat.ai/papers/10.1109/icra.2016.7487478
- The paper
- Jain, A., Singh, A., Koppula, H. S., Soh, S., & Saxena, A. (2016). Recurrent Neural Networks for driver activity anticipation via sensory-fusion architecture. 2016 IEEE International Conference on Robotics and Automation (ICRA). https://doi.org/10.1109/icra.2016.7487478
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.