Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 48 of the 51 references it cites were found in a registry or an index; 12 of the 48 sources its claims cite were checked for support.

Recurrent Neural Networks for driver activity anticipation via sensory-fusion architecture

Open access
Jain, Ashesh, Singh, Avi, Koppula, Hema S, Soh, Shane, Saxena, Ashutosh · 2016 · 2016 IEEE International Conference on Robotics and Automation (ICRA)

Abstract

Anticipating the future actions of a human is a widely studied problem in robotics that requires spatio-temporal reasoning. In this work we propose a deep learning approach for anticipation in sensory-rich robotics applications. We introduce a sensory-fusion architecture which jointly learns to anticipate and fuse information from multiple sensory streams. Our architecture consists of Recurrent Neural Networks (RNNs) that use Long Short-Term Memory (LSTM) units to capture long temporal dependencies. We train our architecture in a sequence-to-sequence prediction manner, and it explicitly learns to predict the future given only a partial temporal context. We further introduce a novel loss layer for anticipation which prevents over-fitting and encourages early anticipation. We use our architecture to anticipate driving maneuvers several seconds before they happen on a natural driving data set of 1180 miles. The context for maneuver anticipation comes from multiple sensors installed on the vehicle. Our approach shows significant improvement over the state-of-the-art in maneuver anticipation by increasing the precision from 77.4% to 90.5% and recall from 71.2% to 87.4%.

Overview

CoChat Summary

TL;DR

The paper proposes a sensory-fusion RNN-LSTM architecture for anticipating driver maneuvers from multiple vehicle sensors, trained sequence-to-sequence with an exponential loss layer. On 1180 miles of natural driving data, it improves maneuver anticipation over prior state-of-the-art, reaching 90.5% precision and 87.4% recall when using 3D head-pose features.

Key findings

  • The Fusion-RNN with exponential loss (F-RNN-EL) improved all-maneuver anticipation over AIO-HMM on the same features, increasing precision from 77.4% to 84.5% and recall from 71.2% to 77.1%, with time-to-maneuver 3.58 s.
  • Adding CLNF-tracked 68 facial landmarks and 3D head-pose features further increased F-RNN-EL to 90.5% precision and 87.4% recall, with time-to-maneuver 3.16 s.
  • Fusing high-level representations from separate sensory RNNs outperformed simple concatenation: in the all-maneuvers setting, F-RNN-EL had about 6% higher precision and recall than S-RNN.
  • The exponential loss improved over uniform loss by about 2% precision and recall in the all-maneuvers setting, while time-to-maneuver remained comparable.

Methods

  • Architecture: each sensory stream is processed by an independent RNN with LSTM units; high-level representations are combined through a fully connected fusion layer and softmax output. Training is sequence-to-sequence, mapping partial observation sequences to the future event, with an exponential loss \(e^{-(T-t)}\log(y_t^k)\) that penalizes later mistakes more.
  • Data: 1180 miles of natural freeway and city driving from 10 drivers, containing 700 annotated events (274 lane changes, 131 turns, 295 straight-driving instances), augmented to 2250 events. Context comes from a driver-facing camera, road-facing camera, GPS, and vehicle dynamics; improved vision uses CLNF for 68 facial landmarks and 3D head pose.
  • Evaluation: 5-fold cross-validation with precision, recall, and time-to-maneuver, choosing a prediction threshold that maximizes F1. Baselines include Chance, Random-Forest, SVM, IOHMM, AIO-HMM, S-RNN, F-RNN-UL, and F-RNN-EL.

Limitations

  • The paper does not include an explicit limitations section.
  • Evident from the text: evaluation is on a single natural driving dataset of 1180 miles from 10 drivers with 5-fold cross-validation; no external or cross-dataset validation is reported.
  • Evident from the text: precision and recall exclude the “driving straight” maneuver because anticipation algorithms default to it when not confident, and data augmentation only adds redundancy rather than new information.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1109/icra.2016.7487478

Cites

48 of 48 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Fig. 1 ( Left ) Shows training RNN for anticipation in a sequence-tosequence prediction manner. The network explicitly learns to map the partial context ( x 1 , .., x t ) ∀ t to the future event y . ( Right ) At test time the network's goal is to anticipate the future event as soon as possible, i.e. by observing only a partial temporal context.
Fig. 1 ( Left ) Shows training RNN for anticipation in a sequence-tosequence prediction manner. The network explicitly learns to map the partial context ( x 1 , .., x t ) ∀ t to the future event y . ( Right ) At test time the network's goal is to anticipate the future event as soon as possible, i.e. by observing only a partial temporal context.From the PDF
Fig. 2 Variations in the data set. Images from the data set [19] for a left lane change. ( Left ) Views from the road facing camera. ( Right ) Driving style of the drivers vary for the same maneuver.
Fig. 2 Variations in the data set. Images from the data set [19] for a left lane change. ( Left ) Views from the road facing camera. ( Right ) Driving style of the drivers vary for the same maneuver.From the PDF
Fig. 3 Internal working of an LSTM unit.
Fig. 3 Internal working of an LSTM unit.From the PDF
Fig. 4 Sensory fusion RNN for anticipation. ( Bottom ) In the Fusion-RNN each sensory stream is passed through their independent RNN. ( Middle ) High-level representations from RNNs are then combined through a fusion layer. ( Top ) In order to prevent over-fitting early in time the loss exponentially increases with time. way our model explicitly learns to anticipate. We additionally use LSTM units which prevents the gradients from vanishing and allows our model to capture long temporal dependencies in human activities. 1
Fig. 4 Sensory fusion RNN for anticipation. ( Bottom ) In the Fusion-RNN each sensory stream is passed through their independent RNN. ( Middle ) High-level representations from RNNs are then combined through a fusion layer. ( Top ) In order to prevent over-fitting early in time the loss exponentially increases with time. way our model explicitly learns to anticipate. We additionally use LSTM units which prevents the gradients from vanishing and allows our model to capture long temporal dependencies in human activities. 1From the PDF
Fig. 5 Maneuver anticipation pipeline. Temporal context in maneuver anticipation comes from cameras facing the driver and the road, the GPS, and the vehicle's dynamics. ( a , b and c ) We improve upon the vision pipeline from Jain et al. [19] by tracking 68 landmark points on the driver's face and including the 3D head-pose features. ( d ) Using the Fusion-RNN we combine the sensory streams of features from inside the vehicle (driver facing camera), with the features from outside the vehicle (road camera, GPS, vehicle dynamics). ( e ) The model outputs the predicted probabilities of future maneuvers.
Fig. 5 Maneuver anticipation pipeline. Temporal context in maneuver anticipation comes from cameras facing the driver and the road, the GPS, and the vehicle's dynamics. ( a , b and c ) We improve upon the vision pipeline from Jain et al. [19] by tracking 68 landmark points on the driver's face and including the 3D head-pose features. ( d ) Using the Fusion-RNN we combine the sensory streams of features from inside the vehicle (driver facing camera), with the features from outside the vehicle (road camera, GPS, vehicle dynamics). ( e ) The model outputs the predicted probabilities of future maneuvers.From the PDF
Fig. 6 Improved features for maneuver anticipation. We track facial landmark points using the CLNF tracker [3] which results in more consistent 2D trajectories as compared to the KLT tracker [32] used by Jain et al. [19]. Furthermore, the CLNF also gives an estimate of the driver's 3D head pose.
Fig. 6 Improved features for maneuver anticipation. We track facial landmark points using the CLNF tracker [3] which results in more consistent 2D trajectories as compared to the KLT tracker [32] used by Jain et al. [19]. Furthermore, the CLNF also gives an estimate of the driver's 3D head pose.From the PDF
Fig. 8 Effect of prediction threshold p th . At test time an algorithm makes a prediction only when it is at least p th confident in its prediction. This plot shows how F1-score vary with change in prediction threshold.
Fig. 8 Effect of prediction threshold p th . At test time an algorithm makes a prediction only when it is at least p th confident in its prediction. This plot shows how F1-score vary with change in prediction threshold.From the PDF
Fig. 7 Confusion matrix of different algorithms when jointly predicting all the maneuvers. Predictions made by algorithms are represented by rows and actual maneuvers are represented by columns. Numbers on the diagonal represent precision.
Fig. 7 Confusion matrix of different algorithms when jointly predicting all the maneuvers. Predictions made by algorithms are represented by rows and actual maneuvers are represented by columns. Numbers on the diagonal represent precision.From the PDF

Verification

Claims the paper makes

4 Valid · 9 Not valid · 18 Not checked · 47 Own results and interpretations

32 more claims

Create a free account to see all 78 claims and to verify the 1 reference still open.

See all claims — free account
  • It may state two separate claims as one. 23
  • Probably not a substantial claim. 25
  • The quote only partly states this claim. 1

Create a free account to see these claims.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

1 of 1

Citations

Cites

Research works printed but not linked: 2

  1. jain, 2015, Neural
  2. 0, Bosch urban

Cited by

See all 272

Metadata

Authors
Jain, A., Singh, A., Koppula, H. S., Soh, S., & Saxena, A.
Affiliations
  • Cornell University (US)
  • Stanford University (US)
Venue
2016 IEEE International Conference on Robotics and Automation (ICRA)
Publisher
IEEE
Published
2016-05
Pages
3118-3125
ISSN
1050-4729 (ISSN-L)
OpenAlex
W2174887554
Cited by
271
References
51
Citations by year
2026: 2 · 2025: 16 · 2024: 24 · 2023: 34 · 2022: 19 · 2021: 42 · 2020: 33 · 2019: 41 · 2018: 31 · 2017: 21 · 2016: 8
FWCI
43.54
Type
conference paper
Language
en
Open access
Locations
  • Proceedings - IEEE International Conference on Robotics and Automation/Proceedings (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • National Science Foundation (1426452)
  • Nvidia (K40 GPU)
  • Office of Naval Research (N00014)
Topics
  • Physical Sciences › Engineering › Automotive Engineering › Autonomous Vehicle Technology and Safety
  • Social Sciences › Human Factors and Ergonomics › Human-Automation Interaction and Safety
  • Social Sciences › Transportation › Traffic Prediction and Management Techniques
Keywords
sensory fusion, recurrent neural networks, long short-term memory, sequence-to-sequence prediction, driving data, spatio-temporal reasoning, automotive sensors

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.