CoChat
Sign Up
Cited by 572Autonomous Vehicle Technology and SafetyEvacuation and Crowd DynamicsData Management and Algorithms

Transformer Networks for Trajectory Forecasting

Open access
Giuliari, Francesco, Hasan, Irtiza, Cristani, Marco, Galasso, Fabio·2021·2020 25th International Conference on Pattern Recognition (ICPR)
DOI 10.1109/icpr48806.2021.9412190
Open this paper in CoChat — free accountOpen PDF

Abstract

Most recent successes on forecasting the people motion are based on LSTM models and all most recent progress has been achieved by modelling the social interaction among people and the people interaction with the scene. We question the use of the LSTM models and propose the novel use of Transformer Networks for trajectory forecasting. This is a fundamental switch from the sequential step-by-step processing of LSTMs to the only-attention-based memory mechanisms of Transformers. In particular, we consider both the original Transformer Network (TF) and the larger Bidirectional Transformer (BERT), state-of-the-art on all natural language processing tasks. Our proposed Transformers predict the trajectories of the individual people in the scene. These are “simple” models because each person is modelled separately without any complex human-human nor scene interaction terms. In particular, the TF modelwithoutbellsandwhistlesyields the best score on the largest and most challenging trajectory forecasting benchmark of TrajNet [1]. Additionally, its extension which predicts multiple plausible future trajectories performs on par with more engineered techniques on the 5 datasets of ETH [2]+UCY [3]. Finally, we show that Transformers may deal with missing observations, as it may be the case with real sensor data. Code is available at github.com/FGiuliari/Trajectory-Transformer.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/icpr48806.2021.9412190

Cites

3 of 3 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The Stanley controller is particularly effective because it accounts for both lateral displacement and heading errors.

    “It is particularly effective because it accounts for both lateral displacement and heading errors.”

    From Ethics-Aware Safe Reinforcement Learning for Rare-Event Risk Control in Interactive Urban Driving (2026) · cites this paper as [31]

    The Metrics and Results sections discuss pedestrian trajectory forecasting, not the Stanley controller. Searches for the Stanley controller, lateral displacement, heading error, and vehicle path tracking found no passage addressing the claimed reason for its effectiveness. Page numbers were not available in our copy. (LLM, conf=0.76)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 42

  1. A. Sadeghian, V. Kosaraju, A. Gupta, S. Savarese, and A. Alahi, "Trajnet: Towards a benchmark for human trajectory prediction," arXiv preprint, 2018.
  2. A. Alahi, K. Goel, V. Ramanathan, A. Robicquet, L. Fei-Fei, and S. Savarese, "Social LSTM: Human trajectory prediction in crowded spaces," in CVPR, 2016.
  3. A. Gupta, J. Johnson, L. Fei-Fei, S. Savarese, and A. Alahi, "Social gan: Socially acceptable trajectories with generative adversarial networks," in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018, cONF.
  4. V. Kosaraju, A. Sadeghian, R. Martín-Martín, I. Reid, H. Rezatofighi, and S. Savarese, "Social-bigat: Multimodal trajectory forecasting using bicycle-gan and graph attention networks," in Advances in Neural Information Processing Systems, 2019, pp. 137-146.
  5. T. Salzmann, B. Ivanovic, P. Chakravarty, and M. Pavone, "Trajectron++: Multi-agent generative trajectory forecasting with heterogeneous data for control," arXiv preprint arXiv:2001.03093, 2020. · arXiv
  6. B. Ivanovic and M. Pavone, "The trajectron: Probabilistic multi-agent trajectory modeling with dynamic spatiotemporal graphs," in Proceed- ings of the IEEE International Conference on Computer Vision, 2019, pp. 2375-2384.
  7. A. Sadeghian, V. Kosaraju, A. Sadeghian, N. Hirose, H. Rezatofighi, and S. Savarese, "Sophie: An attentive gan for predicting paths compliant to social and physical constraints," in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019, pp. 1349-1358.
  8. S. Bai, J. Z. Kolter, and V. Koltun, "An empirical evaluation of generic convolutional and recurrent networks for sequence modeling," arXiv preprint arXiv:1803.01271, 2018. · arXiv
  9. W. Luo, B. Yang, and R. Urtasun, "Fast and furious: Real time end- to-end 3d detection, tracking and motion forecasting with a single convolutional net," in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 3569-3577.
  10. C. Schöller, V. Aravantinos, F. Lay, and A. Knoll, "What the constant velocity model can teach us about pedestrian motion prediction," IEEE Robotics and Automation Letters, 2020.
  11. S. Becker, R. Hug, W. Hubner, and M. Arens, "Red: A simple but effective baseline predictor for the trajnet benchmark," in Proceedings of the European Conference on Computer Vision (ECCV), 2018, pp. 0-0.
  12. S. Becker, R. Hug, W. Hübner, and M. Arens, "An evaluation of trajectory prediction approaches and notes on the trajnet benchmark," arXiv preprint arXiv:1805.07663, 2018. · arXiv
  13. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, "Transformer attention is all you need," in NIPS, 2017.
  14. R. Child, S. Gray, A. Radford, and I. Sutskever, "Generating long sequences with sparse transformers," arXiv preprint:1904.10509, 2019. · arXiv
  15. B. T. Morris and M. M. Trivedi, "A survey of vision-based trajectory learning and analysis for surveillance," IEEE Trans. on Circuits and Systems for Video Technology, vol. 18, no. 8, pp. 1114-1127, 2008.
  16. J. Quiñonero-Candela and C. E. Rasmussen, "A unifying view of sparse approximate gaussian process regression," Journal of Machine Learning Research, vol. 6, no. 12, pp. 1939-1959, 2005.
  17. C. K. I. Williams, "Prediction with gaussian processes: From linear regression to linear prediction and beyond," in Learning in graphical models. Springer, 1998, pp. 599-621.
  18. M. B. Priestley, Spectral analysis and time series. Academic press, 1981.
  19. H. Akaike, "Fitting autoregressive models for prediction," Annals of the institute of Statistical Mathematics, vol. 21, no. 1, pp. 243-247, 1969.
  20. A. Vemula, K. Muelling, and J. Oh, "Social attention: Modeling attention in human crowds," in 2018 IEEE international Conference on Robotics and Automation (ICRA). IEEE, 2018, pp. 1-7.
  21. A. Sadeghian, A. Alahi, and S. Savarese, "Tracking the untrackable: Learning to track multiple cues with long-term dependencies," arXiv preprint arXiv:1701.01909, 2017. · arXiv
  22. H. Su, J. Zhu, Y. Dong, and B. Zhang, "Forecast the plausible paths in crowd scenes," in Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI-17, 2017, pp. 2772-2778. [Online]. Available: https://doi.org/10.24963/ijcai.2017/386 · DOI
  23. H. Su, Y. Dong, J. Zhu, H. Ling, and B. Zhang, "Crowd scene understanding with coherent recurrent neural networks," in IJCAI, 2016.
  24. P. Zhang, W. Ouyang, P. Zhang, J. Xue, and N. Zheng, "Sr-lstm: State refinement for lstm towards pedestrian trajectory prediction," in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019, pp. 12 085-12 094.
  25. A. Graves, "Generating sequences with recurrent neural networks," arXiv preprint arXiv:1308.0850, 2013. · arXiv
  26. A. Rudenko, L. Palmieri, M. Herman, K. M. Kitani, D. M. Gavrila, and K. O. Arras, "Human motion trajectory prediction: A survey," arXiv preprint arXiv:1905.06113, 2019. · arXiv
  27. J. Ferryman and A. Shahrokni, "Pets2009: Dataset and challenge," in 2009 Twelfth IEEE International Workshop on Performance Evaluation of Tracking and Surveillance, Dec 2009, pp. 1-6.
  28. A. Robicquet, A. Sadeghian, A. Alahi, and S. Savarese, "Learning social etiquette: Human trajectory understanding in crowded scenes," in European conference on computer vision. Springer, 2016, pp. 549- 565.
  29. N. Deo and M. M. Trivedi, "Trajectory forecasts in unknown environments conditioned on grid-based plans," arXiv preprint arXiv:2001.00735, 2020. · arXiv
  30. S. Haddad and S.-K. Lam, "Self-growing spatial graph networks for pedestrian trajectory prediction," in The IEEE Winter Conference on Applications of Computer Vision, 2020, pp. 1151-1159.
  31. D. Ridel, N. Deo, D. Wolf, and M. Trivedi, "Scene compliant trajectory forecast with agent-centric spatio-temporal grids," IEEE Robotics and Automation Letters, vol. 5, no. 2, pp. 2816-2823, 2020.
  32. T. van der Heiden, N. S. Nagaraja, C. Weiss, and E. Gavves, "Safecritic: Collision-aware trajectory prediction," arXiv preprint arXiv:1910.06673, 2019. · arXiv
  33. D. Helbing and P. Molnar, "Social force model for," Physical review E, vol. 51, no. 5, p. 4282, 1995.
  34. I. Hasan, F. Setti, T. Tsesmelis, A. Del Bue, F. Galasso, and M. Cristani, "Mx-lstm: mixing tracklets and vislets to jointly forecast trajectories and head poses," in CVPR, 2018.
  35. P. Trautman and A. Krause, "Unfreezing the robot: Navigation in dense, interacting crowds," in IROS, 2010.
  36. K. Kitani, B. Ziebart, J. Bagnell, and M. Hebert, "Activity forecasting," in ECCV, 2012.
  37. S.-H. Bae and K.-J. Yoon, "Confidence-based data association and discriminative deep appearance learning for robust online multi-object tracking," IEEE transactions on pattern analysis and machine intelli- gence, vol. 40, no. 3, pp. 595-610, 2017.
  38. O. Anava, E. Hazan, and A. Zeevi, "Online time series prediction with missing data," in International Conference on Machine Learning, 2015, pp. 2191-2199.
  39. H. Chen, S. Grant-Muller, L. Mussone, and F. Montgomery, "A study of hybrid neural network approaches and the effects of missing data on traffic forecasting," Neural Computing & Applications, vol. 10, no. 3, pp. 277-286, 2001.
  40. P. C. Rodrigues and M. De Carvalho, "Spectral modeling of time series with missing data," Applied Mathematical Modelling, vol. 37, no. 7, pp. 4676-4684, 2013.
  41. M. M. Ghazi, M. Nielsen, A. Pai, M. J. Cardoso, M. Modat, S. Ourselin, and L. Sørensen, "Robust training of recurrent neural networks to handle missing data for disease progression modeling," arXiv preprint arXiv:1808.05500, 2018. · arXiv
  42. N. Golyandina and E. Osipov, "The "caterpillar"-ssa method for analysis of time series with missing values," Journal of Statistical planning and Inference, vol. 137, no. 8, pp. 2642-2653, 2007.

Cited by

Sorted by citations

Authors
Giuliari, F., Hasan, I., Cristani, M., & Galasso, F.
Affiliations
  • University of Verona (IT)
  • Inception Institute of Artificial Intelligence (AE)
  • Sapienza University of Rome (IT)
ORCID
Venue
2020 25th International Conference on Pattern Recognition (ICPR)
Publisher
IEEE
Published
2021-01-10
Pages
10335-10342
ISSN
1041-3278 (ISSN-L)
DOI
10.1109/icpr48806.2021.9412190
OpenAlex
W3011344527
Cited by
572
References
3
Citations by year
2026: 81 · 2025: 125 · 2024: 119 · 2023: 119 · 2022: 72 · 2021: 51 · 2020: 5
FWCI
87.29
Type
conference paper
Language
en
Open access
Licence
Locations
  • Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • IRIS Research product catalog (Sapienza University of Rome) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • Dipartimenti di Eccellenza (2018–2022)
  • Ministero dell’Istruzione, dell’Università e della Ricerca (20182022, PRIN_2017, 20172BH297)
Topics
  • Physical Sciences › Engineering › Automotive Engineering › Autonomous Vehicle Technology and Safety
  • Physical Sciences › Engineering › Ocean Engineering › Evacuation and Crowd Dynamics
  • Physical Sciences › Computer Science › Signal Processing › Data Management and Algorithms
Keywords
trajectory prediction, transformer models, bidirectional transformer, BERT, human motion prediction, attention mechanism, missing data, ETH/UCY dataset, LSTM comparison

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review