Recurrent Neural Networks for driver activity anticipation via sensory-fusion architecture
Open accessThe DOI matches its Crossref record; 48 of the 51 references it cites were found in a registry or an index; 12 of the 48 sources its claims cite were checked for support.
Abstract
Anticipating the future actions of a human is a widely studied problem in robotics that requires spatio-temporal reasoning. In this work we propose a deep learning approach for anticipation in sensory-rich robotics applications. We introduce a sensory-fusion architecture which jointly learns to anticipate and fuse information from multiple sensory streams. Our architecture consists of Recurrent Neural Networks (RNNs) that use Long Short-Term Memory (LSTM) units to capture long temporal dependencies. We train our architecture in a sequence-to-sequence prediction manner, and it explicitly learns to predict the future given only a partial temporal context. We further introduce a novel loss layer for anticipation which prevents over-fitting and encourages early anticipation. We use our architecture to anticipate driving maneuvers several seconds before they happen on a natural driving data set of 1180 miles. The context for maneuver anticipation comes from multiple sensors installed on the vehicle. Our approach shows significant improvement over the state-of-the-art in maneuver anticipation by increasing the precision from 77.4% to 90.5% and recall from 71.2% to 87.4%.
Overview
CoChat Summary
TL;DR
The paper proposes a sensory-fusion RNN-LSTM architecture for anticipating driver maneuvers from multiple vehicle sensors, trained sequence-to-sequence with an exponential loss layer. On 1180 miles of natural driving data, it improves maneuver anticipation over prior state-of-the-art, reaching 90.5% precision and 87.4% recall when using 3D head-pose features.
Key findings
- The Fusion-RNN with exponential loss (F-RNN-EL) improved all-maneuver anticipation over AIO-HMM on the same features, increasing precision from 77.4% to 84.5% and recall from 71.2% to 77.1%, with time-to-maneuver 3.58 s.
- Adding CLNF-tracked 68 facial landmarks and 3D head-pose features further increased F-RNN-EL to 90.5% precision and 87.4% recall, with time-to-maneuver 3.16 s.
- Fusing high-level representations from separate sensory RNNs outperformed simple concatenation: in the all-maneuvers setting, F-RNN-EL had about 6% higher precision and recall than S-RNN.
- The exponential loss improved over uniform loss by about 2% precision and recall in the all-maneuvers setting, while time-to-maneuver remained comparable.
Methods
- Architecture: each sensory stream is processed by an independent RNN with LSTM units; high-level representations are combined through a fully connected fusion layer and softmax output. Training is sequence-to-sequence, mapping partial observation sequences to the future event, with an exponential loss \(e^{-(T-t)}\log(y_t^k)\) that penalizes later mistakes more.
- Data: 1180 miles of natural freeway and city driving from 10 drivers, containing 700 annotated events (274 lane changes, 131 turns, 295 straight-driving instances), augmented to 2250 events. Context comes from a driver-facing camera, road-facing camera, GPS, and vehicle dynamics; improved vision uses CLNF for 68 facial landmarks and 3D head pose.
- Evaluation: 5-fold cross-validation with precision, recall, and time-to-maneuver, choosing a prediction threshold that maximizes F1. Baselines include Chance, Random-Forest, SVM, IOHMM, AIO-HMM, S-RNN, F-RNN-UL, and F-RNN-EL.
Limitations
- The paper does not include an explicit limitations section.
- Evident from the text: evaluation is on a single natural driving dataset of 1180 miles from 10 drivers with 5-fold cross-validation; no external or cross-dataset validation is reported.
- Evident from the text: precision and recall exclude the “driving straight” maneuver because anticipation algorithms default to it when not confident, and data augmentation only adds redundancy rather than new information.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures

![Fig. 2 Variations in the data set. Images from the data set [19] for a left lane change. ( Left ) Views from the road facing camera. ( Right ) Driving style of the drivers vary for the same maneuver.](https://figures.cochat.ai/paper-figures/f183cae4b17b4bccb552955c36acb9d6/recurrent-neural-networks-for-driver-activity-anticipation-via-sensory-fusion-figure-2.png)


![Fig. 5 Maneuver anticipation pipeline. Temporal context in maneuver anticipation comes from cameras facing the driver and the road, the GPS, and the vehicle's dynamics. ( a , b and c ) We improve upon the vision pipeline from Jain et al. [19] by tracking 68 landmark points on the driver's face and including the 3D head-pose features. ( d ) Using the Fusion-RNN we combine the sensory streams of features from inside the vehicle (driver facing camera), with the features from outside the vehicle (road camera, GPS, vehicle dynamics). ( e ) The model outputs the predicted probabilities of future maneuvers.](https://figures.cochat.ai/paper-figures/9617ccca82c44db4a16ed8745a46abd8/recurrent-neural-networks-for-driver-activity-anticipation-via-sensory-fusion-figure-5.png)
![Fig. 6 Improved features for maneuver anticipation. We track facial landmark points using the CLNF tracker [3] which results in more consistent 2D trajectories as compared to the KLT tracker [32] used by Jain et al. [19]. Furthermore, the CLNF also gives an estimate of the driver's 3D head pose.](https://figures.cochat.ai/paper-figures/c2c2ce8af37648d3918180e38c0abe7b/recurrent-neural-networks-for-driver-activity-anticipation-via-sensory-fusion-figure-6.png)


Verification
Claims the paper makes
4 Valid · 9 Not valid · 18 Not checked · 47 Own results and interpretations
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“Anticipating human activities has been shown to improve human-robot collaboration [40, 21, 25, 10].”
What the sources say
- Not checked — no full text of the sourcecites [40] [40] Z. Wang, K. M¨ ulling, M. Deisenroth, H. Amor, D. Vogt, B. Sch¨ olkopf, and J. Peters. Probabilistic movement modeling for intention inference in human-robot interaction. IJRR , 2013. - source
- Not matchedcites [21] [21] H. Koppula and A. Saxena. Anticipating human activities using object affordances for reactive robotic response. In RSS , 2013. -
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not checked — no full text of the sourcecites [25] [25] J. Mainprice and D. Berenson. Human-robot collaborative manipulation planning using early prediction of human motion. In IROS , 2013. - source
- Could not verifycites [10] [10] A. Dragan and S. Srinivasa. Formalizing assistive teleoperation. In RSS , 2012. - sourceExperimental — accuracy not yet measured
The Related Work section mentions human activity detection but does not say that anticipating activities improves human-robot collaboration. Searches for activity anticipation and collaborative outcomes returned passages about object manipulation planning instead, so our copy does not address the attributed claim. (LLM, conf=0.76)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“In another application, anticipating driving maneuvers several seconds in advance [19, 27, 38, 36] enables assistive cars to alert drivers before they make a dangerous maneuver.”
What the sources say
- Not checked — no full text of the sourcecites [19] [19] A. Jain, H. S. Koppula, B. Raghavan, S. Soh, and A. Saxena. Car that knows before you do: Anticipating maneuvers via learning temporal driving models. In ICCV (To appear) , 2015. - source
- Could not verifycites [27] [27] B. Morris, A. Doshi, and M. Trivedi. Lane change intent prediction for driver assistance: On-road design and evaluation. In IEEE IVS , 2011. - sourceExperimental — accuracy not yet measured
The passages from §2.2 (pp. 3–4) discuss recurrent-network dynamics, not driving. Searches for maneuver anticipation, driver alerts, and vehicle prediction found no passage in our copy addressing the claim. (LLM, conf=0.76)
- Not checked — no full text of the sourcecites [38] [38] R. Vasudevan, V. Shia, Y. Gao, R. Cervera-Navarro, R. Bajcsy, and F. Borrelli. Safe semi-autonomous control with enhanced driver modeling. In ACC , 2012. - source
- Not checked — no full text of the sourcecites [36] [36] A. Tawari, S. Sivaraman, M. Trivedi, T. Shannon, and M. Tippelhofer. Looking-in and looking-out vision for urban intelligent assistance: Estimation of driver attentive state and dynamic surround for safe merging and braking. In IEEE IVS , 2014. - source
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Furthermore, the time-to-maneuver remains comparable for F-RNN with and without exponential loss.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As shown in Table I, exponentially growing the loss improves performance.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The F1-score does not undergo large variations with changes to the prediction threshold.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
32 more claims
Create a free account to see all 78 claims and to verify the 1 reference still open.
- It may state two separate claims as one. 23
- Probably not a substantial claim. 25
- The quote only partly states this claim. 1
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ VII. CONCLUSION · p. 8
“It improved the precision from 78% to 84.5% and recall from 71.1% to 77.1%.”
§ I. INTRODUCTION · p. 1
“On this data set, our deep learning approach improves the state-ofthe-art in maneuver anticipation by increasing the precision from 77.4% to 84.5% and recall from 71.2% to 77.1% .”
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 4×
“A common remedy to vanishing gradients is to replace tanh non-linearities by Long ShortTerm Memory cells [16].”
BackgroundMethodInfluentialSequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766 · cited 1×
“previous works on RNNs and LSTM from the language [35], speech [15], and vision [8] communities.”
Are we ready for autonomous driving? The KITTI vision benchmark suite
Geiger, A., Lenz, P., & Urtasun, R. (2012) · 2012 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2012.6248074 · Cited by 14,897 · cited 1×
“cs applications is challenging because the contextual information can come from multiple sensors with different data modalities. Examples include autonomous vehicles that reason from multiple sensors [2] or robots that jointly reason over perception and language instructions [26]. In such applications the way informat…”
BackgroundShi, J., & Tomasi (1994) · Proceedings of IEEE Conference on Computer Vision and Pattern Recognition CVPR-94 · doi:10.1109/cvpr.1994.323794 · Cited by 7,724 · cited 2×
“[19], the driver facing camera detects discriminative points on the driver’s face and tracks the detected points across frames using the KLT tracker [32].”
MethodLong-term recurrent convolutional networks for visual recognition and description
Donahue, J., Hendricks, L. A., Guadarrama, S., Rohrbach, M., Venugopalan, S., Darrell, T., & Saenko, K. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298878 · Cited by 5,228 · cited 2×
“We consider RNNs in multimodal setting, which is related to the recent use of RNNs in image-captioning [8].”
MethodOn the difficulty of training recurrent neural networks
Pascanu, R., Mikolov, T., & Bengio, Y. (2012) · arXiv · doi:10.48550/arxiv.1211.5063 · Cited by 4,310 · cited 3×
“RNNs with this choice of f suffer from a well-studied problem of vanishing gradients [29], and hence are poor at capturing long temporal dependencies which are essential for anticipation.”
BackgroundMethodAction Recognition with Improved Trajectories
Wang, H., & Schmid, C. (2013) · 2013 IEEE International Conference on Computer Vision · doi:10.1109/iccv.2013.441 · Cited by 3,407
(2011) · Cited by 2,752
Hierarchical recurrent neural network for skeleton based action recognition
Du, Y., Wang, W., & Wang, L. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298714 · Cited by 1,895 · cited 14×
“The high level representations from both RNNs {(hx1 , ...,hxT ), (hz1, ...,hzT ) are then concatenated at each time step t and passed through a fully connected (fusion) layer which fuses the two representations (11), as shown in Figure 4.”
MethodInfluential
Research works printed but not linked: 2
- jain, 2015, Neural
- 0, Bosch urban
Cited by
Deep multimodal learning: a survey on recent advances and trends
Ramachandram, D., & Taylor, G. W. (2017) · IEEE Signal Processing Magazine · doi:10.1109/msp.2017.2738401 · Cited by 1,139
“To solve this problem, Jain et al. [35] incorporated a temporal term that grows exponentially in time into their cost function for a multimodal RNN with LSTM units.”
BackgroundDESIRE: Distant Future Prediction in Dynamic Scenes with Interacting Agents
Lee, N., Choi, W., Vernaza, P., Choy, C. B., Torr, P. H. S., & Chandraker, M. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.233 · Cited by 1,067
“• RNN ED-SI: An RNN ED augmented with our SCF unit into the decoder similar to [17].”
BackgroundMethodHuang, S.-C., Pareek, A., Seyyedi, S., Banerjee, I., & Lungren, M. P. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-00341-z · Cited by 934
“While selecting features known to be associated with disease is meaningful, future work may further bene fi t from utilizing large volumes of feature-rich data, as seen in fi elds outside medicine such as autonomous driving 44,45 .”
BackgroundA Bi-Hemisphere Domain Adversarial Neural Network Model for EEG Emotion Recognition
Li, Y., Zheng, W., Zong, Y., Cui, Z., Zhang, T., & Zhou, X. (2021) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2018.2885474 · Cited by 389
“In contrast to the traditional neural network methods, the recurrent neural networks (RNN) method [33] aims to map the input sequence into a series of hidden states through a complex dynamics transform with a chain of repeating modules of neural network and remember and forget the information in…”
MethodDriver behavior detection and classification using deep convolutional neural networks
Shahverdy, M., Fathy, M., Berangi, R., & Sabokrou, M. (2020) · Expert Systems with Applications · doi:10.1016/j.eswa.2020.113240 · Cited by 352
Li, Y., Zheng, W., Wang, L., Zong, Y., & Cui, Z. (2022) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2019.2922912 · Cited by 313
“The conventional single directional long short term memory (LSTM) network [26] is a special type of recurrent neural network (RNN) [27], which usually consists of three gates and a cell state for dealing with the long-term dependence of data sample sequence.”
BackgroundDriver Lane Change Intention Inference for Intelligent Vehicles: Framework, Survey, and Challenges
Xing, Y., Lv, C., Wang, H., Wang, H., Ai, Y., Cao, D., Velenis, E., & Wang, F.-Y. (2019) · IEEE Transactions on Vehicular Technology · doi:10.1109/tvt.2019.2903299 · Cited by 274
“as natural language processing and image captioning [73], [96].”
BackgroundAnticipating Accidents in Dashcam Videos
Chan, F.-H., Chen, Y.-T., Xiang, Y., & Sun, M. (2017) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-54190-7_9 · Cited by 236
“– Exponential-loss: Inspired by [7] on anticipating drivers’ maneuvers, we adopt the exponential-loss function as the loss for positive examples.”
MethodInfluentialr2p2: A ReparameteRized Pushforward Policy for Diverse, Precise Generative Path Forecasting
Rhinehart, N., Kitani, K. M., & Vernaza, P. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01261-8_47 · Cited by 211
“Deterministic vehicle predictions are produced in [18], and deterministic pedestrian trajectories are produced in [3,30,34].”
BackgroundAnticipative Video Transformer
Girdhar, R., & Grauman, K. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01325 · Cited by 211
“While well explored in thirdperson video [2, 26, 37, 38, 46, 48, 79, 87], it has recently gained in popularity for first-person (egocentric) videos [13, 14, 16, 24, 55, 62, 74], due to its applicability on wearable computing platforms.”
BackgroundModeling the Intensity Function of Point Process Via Recurrent Neural Networks
Xiao, S., Yan, J., Yang, X., Zha, H., & Chu, S. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.10724 · Cited by 207
“In a broader sense, video frames can also be treated as time series and RNN are widely used in recent visual analytics works (Jain et al. 2016; Tripathi et al. 2016) and so for speech (Graves, rahman Mohamed, and Hinton 2014).”
BackgroundMethodA Recurrent Neural Network Solution for Predicting Driver Intention at Unsignalized Intersections
Zyner, A., Worrall, S., & Nebot, E. (2018) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2018.2805314 · Cited by 191
“Data such as GPS, steering wheel encoding, [8], [9] or even internal cabin cameras to study driver’s gaze and head position [10], [11] is recorded to use as leading indicators to predict the next maneuver of the vehicle.”
MethodFurnari, A., & Farinella, G. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00635 · Cited by 182
“We also consider baselines obtained adapting approaches proposed for early action recognition to the problem of egocentric action anticipation: the Feedback Network LSTM (FN) proposed in [19], a single LSTM architecture (we use the same parameters of our R-LSTM) trained using the Ranking Loss on Detection Score propos…”
BackgroundMethodInfluentialEncouraging LSTMs to Anticipate Actions Very Early
Aliakbarian, M. S., Saleh, F. S., Salzmann, M., Fernando, B., Petersson, L., & Andersson, L. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.39 · Cited by 167
“The importance of this result is exemplified by research showing that a large proportion of vehicle accidents are due to mistakes/misinterpretations of the scene in the immediate time leading up to the crash [11, 23].”
BackgroundMethodResultInfluentialLong short term memory for driver intent prediction
Zyner, A., Worrall, S., Ward, J., & Nebot, E. (2017) · IEEE Intelligent Vehicles Symposium · doi:10.1109/ivs.2017.7995919 · Cited by 192
“[14] use a similar network to fuse data from a driver facing camera, a road facing camera, and GPS.”
MethodForecasting Interactive Dynamics of Pedestrians with Fictitious Play
Ma, W.-C., Huang, D.-A., Lee, N., & Kitani, K. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.493 · Cited by 178
“There has been growing interest in developing computational models of human activities that can extrapolate unseen information and predict future unobserved activities [34, 38, 22, 33, 18, 42, 40, 37, 15, 14, 16, 17, 35, 9, 49, 43, 36, 32].”
BackgroundKonstantakopoulos, I. C., Barkan, A. R., He, S., Veeravalli, T., Liu, H., & Spanos, C. (2019) · Applied Energy · doi:10.1016/j.apenergy.2018.12.065 · Cited by 145
Rolling-Unrolling LSTMs for Action Anticipation from First-Person Video
Furnari, A., & Farinella, G. M. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2020.2992889 · Cited by 142
“Jain et al. [14] proposed a learning architecture based on LSTMs for driver activity anticipation.”
BackgroundMethodInfluentialAnticipating Traffic Accidents with Adaptive Loss and Large-Scale Incident DB
Suzuki, T., Kataoka, H., Aoki, Y., & Satoh, Y. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00371 · Cited by 172
“In the experimental section, we show the effectiveness of them, in comparison with above mentioned algorithms [11, 4, 24].”
BackgroundMethodResultInfluentialSurround vehicles trajectory analysis with recurrent neural networks
Khosroshahi, A., Ohn-Bar, E., & Trivedi, M. M. (2016) · IEEE International Conference on Intelligent Transportation Systems (ITSC) · doi:10.1109/itsc.2016.7795922 · Cited by 139
“RNNs, and in particular LSTMs have recently emerged as powerful temporal data models [14], and this work discusses employing them for a trajectory classification task.”
MethodAffordances from Human Videos as a Versatile Representation for Robotics
Bahl, S., Mendonca, R., Chen, L., Jain, U., & Pathak, D. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01324 · Cited by 119
A survey on human-aware robot navigation
Möller, R., Furnari, A., Battiato, S., Härmä, A., & Farinella, G. M. (2021) · Robotics and Autonomous Systems · doi:10.1016/j.robot.2021.103837 · Cited by 140
“Being able to anticipate the next action of a human has a wide range of use cases, it is relevant for applications such as automotive safety [247] as well as observation tasks for healthcare, surveillance or social robots [248],[249].”
BackgroundA Survey on Driver Behavior Analysis From In-Vehicle Cameras
Wang, J., Chai, W., Venkatachalapathy, A., Tan, K. L., Haghighat, A. K., Velipasalar, S., Adu-Gyamfi, Y. O., & Sharma, A. (2021) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2021.3126231 · Cited by 128
(2019) · doi:10.1109/tvt.2019.2927242 · Cited by 102
“A sensor fusion architecture based on the RNN is proposed to predict the drivers’ lane-change intention by combining the driving information frommultiple sensors [12].”
BackgroundMethodHybrid machine learning for human action recognition and prediction in assembly
Zhang, J., Wang, P. E., & Gao, R. X. (2021) · Robotics and Computer-Integrated Manufacturing · doi:10.1016/j.rcim.2021.102184 · Cited by 112
Learning Time Series Associated Event Sequences With Recurrent Point Process Networks
Xiao, S., Yan, J., Farajtabar, M., Song, L., Yang, X., & Zha, H. (2019) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2018.2889776 · Cited by 116
“Empirically, recent study [35] has shown that using separate encoders for data with different properties, e.”
BackgroundReal-Time Driver Maneuver Prediction Using LSTM
Khairdoost, N., Shirpour, M., Bauer, M., & Beauchemin, S. S. (2020) · IEEE Transactions on Intelligent Vehicles · doi:10.1109/tiv.2020.3003889 · Cited by 106
“As mentioned before, LSTMs have been employed in different ADAS applications [13], [15], [16].”
BackgroundMethodInfluentialYang, H., Wu, J., Hu, Z., & Lv, C. (2023) · IEEE Transactions on Industrial Electronics · doi:10.1109/tie.2023.3288182 · Cited by 95
Diverse Trajectory Forecasting with Determinantal Point ProcessesPreprint
Yuan, Y., & Kitani, K. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1907.04967 · Cited by 136
“…work takes a deterministic approach and forecasts only one possible future motion from past 3D poses (Fragkiadaki et al., 2015; Butepage et al., 2017; Li et al., 2017; Jain et al., 2016b), static images (Chao et al., 2017; Kanazawa et al., 2018) or egocentric videos (Yuan and Kitani, 2019).”
BackgroundVision-Based Traffic Accident Detection and Anticipation: A Survey
Fang, J., Qiao, J., Xue, J., & Li, Z. G. (2023) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2023.3307655 · Cited by 134
Song, R., & Li, B. (2021) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2021.3076164 · Cited by 105
Predicting the Future: A Jointly Learnt Model for Action Anticipation
Gammulle, H., Denman, S., Sridharan, S., & Fookes, C. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00566 · Cited by 113
“To further demonstrate the proposed AA-GAN method, we conducted an ablation study by strategically removing Method Earliest Latest Context-aware + loss in [21] 30.”
MethodLooking at the Driver/Rider in Autonomous Vehicles to Predict Take-Over Readiness
Deo, N., & Trivedi, M. M. (2019) · IEEE Transactions on Intelligent Vehicles · doi:10.1109/tiv.2019.2955364 · Cited by 104
Exploiting the use of recurrent neural networks for driver behavior profiling
Carvalho, E., Ferreira, B. V., Junior, J. D. S. F., de Souza, C. R. B., Carvalho, H. V., Suhara, Y., Pentland, A., & Pessin, G. (2017) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2017.7966230 · Cited by 92
“Some of the noteworthy points found in the related work examined here can be summarized as follows: (i) all the papers use sensor-fusion data as input for the event detection algorithms; (ii) the employed machine learning algorithms are based on fuzzy logic [10], [12], [13] or variations of DTW [9], [11]; and (iii) mo…”
MethodAction Anticipation with RBF Kernelized Feature Mapping RNN
Shi, Y., Fernando, B., & Hartley, R. I. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01249-6_19 · Cited by 90
“Others ELSTM [39] 55% Within-class Loss [28] 33% DP-SVM [42] 5% S-SVM [42] 5% Where/What [43] 10% Context-fusion [16] 28% Table 2: Comparison of our model against state-of-the-arts on UT-Interaction dataset for action anticipation.”
BackgroundMethodInfluentialMiyajima, C., & Takeda, K. (2016) · IEEE Signal Processing Magazine · doi:10.1109/msp.2016.2602377 · Cited by 76
“Jain et al. applied a recurrent neural network (RNN) to predict driving behavior [28] .”
BackgroundDESIRE: Distant Future Prediction in Dynamic Scenes with Interacting AgentsPreprint
Lee, N., Choi, W., Vernaza, P., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1704.04394 · Cited by 74
“• RNN ED-SI: An RNN ED augmented with our SCF unit into the decoder similar to [17].”
BackgroundMethodLearning Latent Global Network for Skeleton-Based Action Prediction
Ke, Q., Bennamoun, M., Rahmani, H., An, S., Sohel, F. A., & Boussaïd, F. (2019) · IEEE Transactions on Image Processing · doi:10.1109/tip.2019.2937757 · Cited by 81
“We also compared the proposed method to recent state-ofthe-art action recognition and prediction methods [9], [36],”
BackgroundMethodResultInfluentialEnd-to-end Prediction of Driver Intention using 3D Convolutional Neural Networks
Gebert, P., Roitberg, A., Haurilet, M., & Stiefelhagen, R. (2019) · IEEE Intelligent Vehicles Symposium · doi:10.1109/ivs.2019.8814249 · Cited by 85
“Despite recent progress in end-to-end deep learning, the vast majority of previous work on driver maneuver anticipation has been based on manually designed feature descriptors, often employing eye gaze, head and body pose or context features, such as GPS and car speed [7], [5], [6].”
BackgroundMethodShared Cross-Modal Trajectory Prediction for Autonomous Driving
Choi, C., Choi, J. H., Li, J., & Malla, S. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00031 · Cited by 99
“Basically, the same set of input types should be provided at test time as in [20, 45].”
BackgroundAgent-Centric Risk Assessment: Accident Anticipation and Risky Region Localization
Zeng, K.-H., Chou, S.-H., Chan, F.-H., Niebles, J. C., & Sun, M. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.146 · Cited by 95
“The task of accident anticipation is related to early activity recognition [8, 25], and event anticipation [10, 15].”
BackgroundFelsen, P., Lucey, P., & Ganguly, S. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01252-6_45 · Cited by 71
“They demonstrated the approach on data from autonomous vehicles and aerial drones and outperformed other RNN-based methods [3]; however, in the absence of the refinement module, the predictions were poor.”
MethodHARD-Net: Hardness-AwaRe Discrimination Network for 3D Early Activity Prediction
Li, T., Liu, J., Zhang, W., & Duan, L. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58621-8_25 · Cited by 83
“Some of the existing methods [32, 26, 24, 12, 1, 46] used RNN/LSTM-based methods for 3D human activity recognition.”
MethodPredicting Human Decision-Making: From Prediction to Action
Rosenfeld, A., & Kraus, S. (2018) · Synthesis lectures on artificial intelligence and machine learning · doi:10.2200/s00820ed1v01y201712aim036 · Cited by 100
Anticipating human actions by correlating past with the future with Jaccard similarity measures
Fernando, B., & Herath, S. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01302 · Cited by 89
“There are methods that aim to learn video representations suitable for early action recognition by handling uncertainty using new loss functions [26, 10, 18].”
MethodLearning to Anticipate Egocentric Actions by Imagination
Wu, Y., Zhu, L., Wang, X., Yang, Y., & Wu, F. (2020) · IEEE Transactions on Image Processing · doi:10.1109/tip.2020.3040521 · Cited by 70
“We also compare with the early action recognition methods to the problem of egocentric action anticipation: Feedback Network LSTM (FN) [25], and an LSTM trained using the Exponential Anticipation Loss [58] (EL).”
MethodAttention for Vision-Based Assistive and Automated Driving: A Review of Algorithms and Datasets
Kotseruba, I., & Tsotsos, J. K. (2022) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2022.3186613 · Cited by 81
“Recurrent networks are also effective for multi-modal data [160] but lack the explainability of HMMs. 2) Evaluation and Limitations: Since action prediction is typically framed as classification, common metrics such as precision, recall, and F1 are used to evaluate the results.”
MethodAre all objects equal? Deep spatio-temporal importance prediction in driving videos
Ohn-Bar, E., & Trivedi, M. M. (2016) · Pattern Recognition · doi:10.1016/j.patcog.2016.08.029 · Cited by 63
“Further investigation of the costsensitive training procedure [49, 50, 43] may lead to additional insights in the future.”
BackgroundSchydlo, P., Raković, M., Jamone, L., & Santos-Victor, J. (2018) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2018.8460924 · Cited by 76
“This ability finds applications in areas such as cooperative robotics [1], [2], auto-mobile safety [3], elderly care [4], among many others [5].”
BackgroundVisual Forecasting by Imitating Dynamics in Natural Sequences
Zeng, K.-H., Shen, W. B., Huang, D.-A., Sun, M., & Niebles, J. C. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.326 · Cited by 53
“Such framework for forecasting can be widely useful in real world scenarios: robot motion planning [17, 20] or warning systems for autonomous vehicles [4].”
BackgroundChen, Y., Lu, C., & Chu, W. (2020) · IEEE Internet of Things Journal · doi:10.1109/jiot.2020.2969209 · Cited by 58
“To make the network trainable, the LSTM cell is introduced to avoid the gradient problem [24].”
MethodAction Anticipation by Predicting Future Dynamic Images
Rodríguez, C., Fernando, B., & Li, H. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-11015-4_10 · Cited by 64
“eep learning approaches for action anticipation [2,4,16]. Deep learning-based methods can be primarily categorised into two types; 1. methods that rely on novel loss functions for action anticipation [2,4,17] and 2. methods that try to generate future content by content prediction [16]. In this context, [2] propose to…”
MethodInfluentialLi, J., Ma, H., & Tomizuka, M. (2019) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2019.8793661 · Cited by 66
“…of autonomous vehicles with various models such as intelligent driver model [5], multi-layer perceptron [6] and recurrent neural network [7], [8] which are deterministic models as well as mixture model [9]–[11], probabilistic graphical models [6], [11], [12] and variational auto-encoder [13]…”
MethodTime-to-lane-change prediction with deep learning
Dang, H. Q., Fürnkranz, J., Biedermann, A., & Höpfl, M. (2017) · IEEE International Conference on Intelligent Transportation Systems (ITSC) · doi:10.1109/itsc.2017.8317674 · Cited by 67
“In [4], Jain et al.”
A Bayesian inference based adaptive lane change prediction model
Wang, J., Zhang, Z., & Lu, G. (2021) · Transportation Research Part C Emerging Technologies · doi:10.1016/j.trc.2021.103363 · Cited by 62
Deep Learning-Based Driving Maneuver Prediction System
Ou, C., & Karray, F. O. (2019) · IEEE Transactions on Vehicular Technology · doi:10.1109/tvt.2019.2958622 · Cited by 55
Self-Regulated Learning for Egocentric Video Activity Anticipation
Qi, Z., Wang, S., Su, C., Su, L., Huang, Q., & Tian, Q. C. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3059923 · Cited by 89
Attention-based Lane Change Prediction
Scheel, O., Nagaraja, N. S., Schwarz, L., Navab, N., & Tombari, F. (2019) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2019.8793648 · Cited by 62
“As proposed in [16], we employ an exponentially growing”
BackgroundMethodInfluentialModeling the intensity function of point process via recurrent neural networkssPreprint
Xiao, S., Yan, J., Yang, X., Zha, H., & Chu, S. M. (2017) · arXiv (Cornell University) · Cited by 52
Xu, W., Wang, J., Fu, T., Gong, H., & Sobhani, A. (2021) · Accident Analysis & Prevention · doi:10.1016/j.aap.2021.106477 · Cited by 50
Driver Authentication for Smart Car Using Wireless Sensing
Regani, S. D., Xu, Q., Wang, B., Wu, M. H., & Liu, K. J. R. (2019) · IEEE Internet of Things Journal · doi:10.1109/jiot.2019.2958692 · Cited by 48
“[4] used vision-based techniques to detect head, eye, and hand movements to predict driver behavior for accident prevention, although cameras, in”
MethodMC-LSTM: Real-Time 3D Human Action Detection System for Intelligent Healthcare Applications
Yin, J., Han, J. S., Xie, R., Wang, C., Duan, X., Rong, Y., Zeng, X., & Tao, J. (2021) · IEEE Transactions on Biomedical Circuits and Systems · doi:10.1109/tbcas.2021.3064841 · Cited by 43
Unified Pose Sequence Modeling
Foo, L. G., Li, T., Rahmani, H., Ke, Q., & Liu, J. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01251 · Cited by 67
A Review of Abnormal Behavior Detection in Activities of Daily Living
Tay, N. C., Connie, T., Ong, T. S., Teoh, A. B. J., & Teh, P. S. (2023) · IEEE Access · doi:10.1109/access.2023.3234974 · Cited by 55
Tonutti, M., Ruffaldi, E., Cattaneo, A., & Avizzano, C. A. (2019) · Robotics and Autonomous Systems · doi:10.1016/j.robot.2019.02.007 · Cited by 54
“Samples from all test subjects are often shuffled together before the training-test split [13]: while this enhances a classifier’s performance, it will likely cause it not to generalise and scale well in real-world applications.”
BackgroundMethodInfluentialVIENA $$^2$$ : A Driving Anticipation Dataset
Aliakbarian, M. S., Saleh, F. S., Salzmann, M., Fernando, B., Petersson, L., & Andersson, L. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-20887-5_28 · Cited by 53
“In this context, when consulting the main actors in the field, may they be from the computer vision community, the intelligent vehicle one or the automotive industry, the consensus is that predicting the intentions of a car’s own driver, for Advanced Driver Assistance Systems (ADAS), remains a challenging task for a c…”
BackgroundDriving Behavior Prediction Considering Cognitive Prior and Driving Context
Zhou, D. H., Liu, H., Ma, H., Wang, X., Zhang, X., & Dong, Y. (2020) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2020.2973751 · Cited by 42
Computing with oscillators from theoretical underpinnings to applications and demonstrators
Todri‐Sanial, A., Delacour, C., Abernot, M., & Sabo, F. (2024) · npj Unconventional Computing · doi:10.1038/s44335-024-00015-z · Cited by 73
GSC: A Graph and Spatio-Temporal Continuity Based Framework for Accident Anticipation
Wang, T., Chen, K., Chen, G., Li, B., Li, Z., Liu, Z., & Jiang, C. (2023) · IEEE Transactions on Intelligent Vehicles · doi:10.1109/tiv.2023.3257169 · Cited by 59
Future-State Predicting LSTM for Early Surgery Type Recognition
Kannan, S., Yengera, G., Mutter, D., Marescaux, J., & Padoy, N. (2019) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2019.2931158 · Cited by 53
“We hope to overcome this potential challenge by encouraging our model to gather discriminative information from the initial frames of laparoscopic videos and also by leveraging the novel loss functions proposed in previous works on action anticipation [5], [19] to encourage a model to learn the early recognition of th…”
BackgroundMethodSpatiotemporal Feature Residual Propagation for Action Prediction
Zhao, H., & Wildes, R. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00710 · Cited by 44
“Method Accuracy (%) ELSTM [35] 55 Within-class Loss [28] 33 DP-SVM [41] 5 S-SVM [41] 5 Where/What [40] 10 Context-fusion [15] 28”
Hutchinson, M., & Gadepally, V. N. (2021) · IEEE Access · doi:10.1109/access.2021.3115476 · Cited by 55
“Examples include F-RNN-EL [225] which uses an exponential loss to bias a multi-modal CNN+LSTM fusion strategy towards the most recent predictions,”
MethodAbductive Ego-View Accident Video Understanding for Safe Driving Perception
Fang, J., Li, L., Zhou, J., Xiao, J., Yu, H., Lv, C., Xue, J., & Chua, T. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02080 · Cited by 91
“The earliness is maintained by taking the exponential loss [3, 5, 22, 51, 59] to penalize the positive occurrence of the accident.”
BackgroundZhang, H., & Fu, R. (2020) · Sensors · doi:10.3390/s20174887 · Cited by 40
“Since an LSTM can be trained in a sequence-to-sequence prediction manner, it can predict the trend of a period of time in the time series, enabling to predict the future activities of drivers [41].”
BackgroundNg, Y. B., & Fernando, B. (2020) · IEEE Transactions on Image Processing · doi:10.1109/tip.2020.3021497 · Cited by 45
Predicting Secondary Task Performance: A Directly Actionable Metric for Cognitive Overload Detection
Amadori, P. V., Fischer, T., Wang, R., & Demiris, Y. (2021) · IEEE Transactions on Cognitive and Developmental Systems · doi:10.1109/tcds.2021.3114162 · Cited by 30
“Accurate inference of a user’s cognitive state in real-time could lead to disruptive benefits toward optimized interface designs and adaptive user interfaces [4], [6], more effective and situational-aware robots [7], [8], as well as safer and smarter vehicles [9]–[11].”
BackgroundMethodSSRL: Self-Supervised Spatial-Temporal Representation Learning for 3D Action Recognition
Jin, Z., Wang, Y., Wang, Q., Shen, Y., & Meng, H. (2023) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2023.3284493 · Cited by 31
“From existed works on action prediction [53], [54], [55], [56], it can be concluded that about 80% of the sequences can achieve the same classification accuracy as the complete sequences.”
BackgroundPredicting Future Lane Changes of Other Highway Vehicles using RNN-based Deep ModelsPreprint
Patel, S., Griffin, B., Kusano, K. D., & Corso, J. J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1801.04340 · Cited by 46
“Recent work in predicting driver maneuvers has primarily focused on the intent of the target vehicle’s human driver; intent is based on tracking the driver’s face with an inwardfacing camera along with features outside of and in front of the vehicle using a camera, velocity sensor, and GPS [11], [10], [9].”
BackgroundMethodInfluentialA data-driven lane-changing behavior detection system based on sequence learning
Jun, G., Murphey, Y. L., Yi, J., & Zhu, H. (2020) · Transportmetrica B Transport Dynamics · doi:10.1080/21680566.2020.1782786 · Cited by 32
“In general, the proposed DLCD model achieves the best performance on all 3 driving actions with the average accuracy of 88.26%, which is an improvement of 10% over the NN model, 4% over the SFA model, 3% over the DBRNN model, and 2% over the HRC-LSTM model.”
BackgroundMethodInfluentialEarly Gesture Recognition With Reliable Accuracy Based on High-Resolution IoT Radar Sensors
Min, R., Wang, X., Zou, J., Gao, J., Wang, L., & Cao, Z. (2021) · IEEE Internet of Things Journal · doi:10.1109/jiot.2021.3072169 · Cited by 30
Dilated Convolutional Neural Network for Predicting Driver's Activity
Rekabdar, B., & Mousas, C. (2018) · IEEE International Conference on Intelligent Transportation Systems (ITSC) · doi:10.1109/itsc.2018.8569818 · Cited by 37
“Unlike the other approaches for driver activity prediction [8][4][5], we use raw pixel values as our input features.”
BackgroundMethodInfluentialAn Ensemble Learning–Online Semi-Supervised Approach for Vehicle Behavior Recognition
Zhang, H., & Fu, R. (2021) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2021.3095053 · Cited by 33
Crowd behavior analysis from fixed and moving cameras
Bour, P., Cribelier, E., & Argyriou, V. (2018) · Elsevier eBooks · doi:10.1016/b978-0-12-814601-9.00023-7 · Cited by 30
Driving maneuver early detection via sequence learning from vehicle signals and video images
Peng, X., Murphey, Y. L., Liu, R., & Li, Y. (2020) · Pattern Recognition · doi:10.1016/j.patcog.2020.107276 · Cited by 32
Multi-Modal Temporal Convolutional Network for Anticipating Actions in Egocentric Videos
Zatsarynna, O., Farha, Y. A., & Gall, J. (2021) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw53098.2021.00254 · Cited by 40
“While some works try to anticipate several actions into the future (long-term anticipation) [1,11,15,25], others aim at anticipating only the next action at a fixed anticipation time based on the recent observations preceding it (short-term anticipation) [14, 20, 23, 45].”
BackgroundMethodJoint Modeling of Event Sequence and Time Series with Attentional Twin Recurrent Neural NetworksPreprint
Xiao, S., Yan, J., Farajtabar, M., Song, L., Yang, X., & Hongyuan Zha, B. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.08524 · Cited by 42
“Em-pirically, it has been shown in some recent study [34] that using separate RNN for each time series data e.g. , video and time series sensors can lead to better prediction accuracy than a single RNN fed with the combination of the multiple time series data.”
BackgroundDIPNet: Driver intention prediction for a safe takeover transition in autonomous vehicles
Bonyani, M., Rahmanian, M., Jahangard, S., & Rezaei, M. (2023) · IET Intelligent Transport Systems · doi:10.1049/itr2.12370 · Cited by 33
Artificial Intelligence Methods in In-Cabin Use Cases: A Survey
Rong, Y., Han, C., Hellert, C., Loyal, A., & Kasneci, E. (2021) · IEEE Intelligent Transportation Systems Magazine · doi:10.1109/mits.2021.3050883 · Cited by 29
“In [48], [49], more driving behaviors are included in addition to the three lane changing classes, i.e. right/left turn.”
BackgroundMethodGaze-based Intention Anticipation over Driving Manoeuvres in Semi-Autonomous Vehicles
Wu, M., Louw, T., Lahijanian, M., Ruan, W., Huang, X., Merat, N., & Kwiatkowska, M. (2019) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros40897.2019.8967779 · Cited by 40
“Recent works [2], [3] showed that by relying on multiple data sources including inside-vehicle features, e.”
BackgroundLiu, T., & Lam, K. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01353 · Cited by 36
(2019) · doi:10.1145/3343031.3351073 · Cited by 24
Adaptive Graph Convolutional Network With Adversarial Learning for Skeleton-Based Action Prediction
Li, G., Li, N., Chang, F., & Liu, C. (2021) · IEEE Transactions on Cognitive and Developmental Systems · doi:10.1109/tcds.2021.3103960 · Cited by 27
“The accuracy of [2] with an observation ratio of [3], [43], and [44] with an observation ratio of 0.4 is 53.37%, 71.01%, and 74.21%.”
BackgroundMethodResultInfluentialGenerative Hybrid Representations for Activity Forecasting With No-Regret Learning
Guan, J., Yuan, Y., Kitani, K., & Rhinehart, N. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00025 · Cited by 34
“Deterministic trajectory modeling has been used for vehicle [17] and pedestrian [1, 38, 52] trajectory prediction.”
MethodA Survey on Self-Evolving Autonomous Driving: A Perspective on Data Closed-Loop Technology
Li, X., Wang, Z., Huang, Y., & Chen, H. (2023) · IEEE Transactions on Intelligent Vehicles · doi:10.1109/tiv.2023.3319689 · Cited by 32
“Some datasets have acquisition devices mounted on [18].”
BackgroundGated Recurrent Fusion to Learn Driving Behavior from Temporal Multimodal Data
Narayanan, A., Siravuru, A., & Dariush, B. (2020) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2020.2967738 · Cited by 27
“Typical driving automation tasks of interest are learning driver behavior [3] and intent [12], motion forecasting [28], [29], object detection [16], learning affordances [30],”
BackgroundModeling Popularity in Asynchronous Social Media Streams with Recurrent Neural Networks
Mishra, S., Rizoiu, M.-A., & Xie, L. (2018) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v12i1.15030 · Cited by 26
“2015), video (Jain et al. 2016) and time-series data (Chandra and Zhang 2012).”
BackgroundEarly event detection based on dynamic images of surveillance videos
Fan, Y., Wen, G., Li, D., et al. (2018) · Journal of Visual Communication and Image Representation · doi:10.1016/j.jvcir.2018.01.002 · Cited by 22
“[28] use Long Short-Term Memory (LSTM) units to capture long temporal dependencies.”
BackgroundMethodInfluentialBONSEYES: Platform for Open Development of Systems of Artificial Intelligence: Invited paper
Llewellynn, T., Fernández-Carrobles, M. M., Déniz, Ó., Fricker, S., Storkey, A., Pazos, N., Velikic, G. S., Leufgen, K., Dahyot, R., Koller, S., et al. (2017) · ACM International Conference on Computing Frontiers (CF) · doi:10.1145/3075564.3076259 · Cited by 30
“Consumers, with a plethora of electronic devices that inform them [5], entertain them and keep them safe [9], [6], will nd themselves enjoying the overall experience of their vehicles.”
BackgroundSpatial temporal GRU convnets for vision-based real time epileptic seizure detection
Fang, Z., Leung, H., & Choy, C. S. (2018) · 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018) · doi:10.1109/isbi.2018.8363746 · Cited by 22
“2, epileptic seizure score (ES) is generated by FC for each frame to indicate whether there is seizure happening [7].”
BackgroundMethodEvaluation of recurrent neural networks as epileptic seizure predictor
de Freitas, L. B. A., & Balbinot, A. (2020) · Array · doi:10.1016/j.array.2020.100038 · Cited by 24
Metadata
- Authors
- Jain, A., Singh, A., Koppula, H. S., Soh, S., & Saxena, A.
- Affiliations
- Cornell University (US)
- Stanford University (US)
- Venue
- 2016 IEEE International Conference on Robotics and Automation (ICRA)
- Publisher
- IEEE
- Published
- 2016-05
- Pages
- 3118-3125
- ISSN
- 1050-4729 (ISSN-L)
- OpenAlex
- W2174887554
- Cited by
- 271
- References
- 51
- Citations by year
- 2026: 2 · 2025: 16 · 2024: 24 · 2023: 34 · 2022: 19 · 2021: 42 · 2020: 33 · 2019: 41 · 2018: 31 · 2017: 21 · 2016: 8
- FWCI
- 43.54
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- Proceedings - IEEE International Conference on Robotics and Automation/Proceedings (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1426452)
- Nvidia (K40 GPU)
- Office of Naval Research (N00014)
- Topics
- Physical Sciences › Engineering › Automotive Engineering › Autonomous Vehicle Technology and Safety
- Social Sciences › Human Factors and Ergonomics › Human-Automation Interaction and Safety
- Social Sciences › Transportation › Traffic Prediction and Management Techniques
- Keywords
- sensory fusion, recurrent neural networks, long short-term memory, sequence-to-sequence prediction, driving data, spatio-temporal reasoning, automotive sensors
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.