CoChat
Sign Up
Cited by 584Autonomous Vehicle Technology and SafetyTraffic Prediction and Management TechniquesTraffic and Road Safety

Large Scale Interactive Motion Forecasting for Autonomous Driving : The Waymo Open Motion Dataset

Open access
Ettinger, Scott, Cheng, Shuyang, Caine, Benjamin, Liu, Chenxi, Zhao, Hang, Pradhan, Sabeek, Chai, Yuning, Sapp, Ben +10 more·2021·2021 IEEE/CVF International Conference on Computer Vision (ICCV)
DOI 10.1109/iccv48922.2021.00957
Open this paper in CoChat — free accountOpen PDF

Abstract

As autonomous driving systems mature, motion forecasting has received increasing attention as a critical requirement for planning. Of particular importance are interactive situations such as merges, unprotected turns, etc., where predicting individual object motion is not sufficient. Joint predictions of multiple objects are required for effective route planning. There has been a critical need for high-quality motion data that is rich in both interactions and annotation to develop motion planning models. In this work, we introduce the most diverse interactive motion dataset to our knowledge, and provide specific labels for interacting objects suitable for developing joint prediction models. With over 100,000 scenes, each 20 seconds long at 10 Hz, our new dataset contains more than 570 hours of unique data over 1750 km of roadways. It was collected by mining for interesting interactions between vehicles, pedestrians, and cyclists across six cities within the United States. We use a high-accuracy 3D auto-labeling system to generate high quality 3D bounding boxes for each road agent, and provide corresponding high definition 3D maps for each scene. Furthermore, we introduce a new set of metrics that provides a comprehensive evaluation of both single agent and joint agent interaction motion forecasting models. Finally, we provide strong baseline models for individual-agent prediction and joint-prediction. We hope that this new large-scale interactive motion dataset will provide new opportunities for advancing motion forecasting models.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/iccv48922.2021.00957

Cites

4 of 4 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Traffic light statuses are crucial for realistic traffic simulation, but only Waymo and Lyft contain these and only from an online perception system, rather than developing offline traffic light status inference as in nuPlan.

    “Likewise, the availability of traffic light statuses is crucial for realistic traffic simulation, but only Waymo [17] and Lyft [19] contain these and only from an online perception system, rather than developing offline traffic light status inference as in nuPlan.”

    From Towards learning-based planning: The nuPlan benchmark for real-world autonomous driving (2024) · cites this paper as [17]

    Section 3 (Dataset; page unavailable) explicitly says the Waymo Open Motion Dataset includes traffic signal states, supporting the part attached to the Waymo citation. Section 3.3 discusses offboard perception for object tracks, but does not establish that the traffic signal states came only from an online perception system. The copy also does not establish the broader 'only Waymo and Lyft' comparison or address nuPlan's inference method. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • Offline perception is crucial to train and evaluate planners using high-quality object tracks, but only present in Waymo, MONA and nuPlan.

    “Offline perception [16] is crucial to train and evaluate planners using high-quality object tracks, but only present in Waymo [17], MONA [18] and nuPlan.”

    From Towards learning-based planning: The nuPlan benchmark for real-world autonomous driving (2024) · cites this paper as [17]

  • Only the Lyft, Shifts and nuPlan datasets provide more than 1000h of driving data, and nuPlan was the first such dataset that provides lidar and camera sensor data (128h), although Waymo later also released compressed lidar data.

    “Only the Lyft [19], Shifts [15] and nuPlan datasets provide more than 1000h of driving data and nuPlan was the first such dataset that provides lidar and camera sensor data (128h), although Waymo [17] later also released compressed lidar data.”

    From Towards learning-based planning: The nuPlan benchmark for real-world autonomous driving (2024) · cites this paper as [17]

    The Dataset section describes releasing object tracks and map features, while the Offboard perception system section describes using LiDAR frames to generate tracks. Neither passage says Waymo later released compressed LiDAR data. Searches for “compressed LiDAR data,” “sensor data,” and “compress” found no passage establishing the part attributed to Waymo [17]. (LLM, conf=0.75)

    Experimental — accuracy not yet measured

  • Most other datasets use a combination of manual and automated driving, which may lead the planner to imitate less desirable driving behavior.

    “Since our focus is on planning, it is crucial that we drive manually, while most other datasets [13], [14], [17], [19] use a combination of manual and automated driving, which may lead the planner to imitate less desirable driving behavior.”

    From Towards learning-based planning: The nuPlan benchmark for real-world autonomous driving (2024) · cites this paper as [17]

    The Related Work section compares datasets by their coverage, interactions, and labeling, but does not say that most combine manual and automated driving. Searches for driving mode and imitation found no passage linking such a combination to less desirable behavior for a planner. (LLM, conf=0.73)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 32

  1. Alexandre Alahi, Kratarth Goel, Vignesh Ramanathan, Alexandre Robicquet, Li Fei-Fei, and Silvio Savarese. So- cial lstm: Human trajectory prediction in crowded spaces. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 961-971, 2016. 3, 6, 12
  2. Ben Benfold and Ian Reid. Stable multi-target tracking in real-time surveillance video. In CVPR 2011, pages 3457- 3464. IEEE, 2011. 3
  3. Thibault Buhet, Emilie Wirbel, and Xavier Perrotton. Plop: Probabilistic polynomial objects trajectory planning for au- tonomous driving. arXiv preprint arXiv:2003.08744, 2020. · arXiv
  4. Sergio Casas, Cole Gulino, Renjie Liao, and Raquel Urta- sun. Spagnn: Spatially-aware graph neural networks for re- lational behavior forecasting from sensor data. In 2020 IEEE International Conference on Robotics and Automation, ICRA 2020, Paris, France, May 31 -August 31, 2020, pages 9491- 9497. IEEE, 2020. 3
  5. Sergio Casas, Cole Gulino, Simon Suo, Katie Luo, Renjie Liao, and Raquel Urtasun. Implicit latent variable model for scene-consistent motion forecasting. In Proceedings of the European Conference on Computer Vision (ECCV). Springer, 2020. 2, 3, 5, 7, 12
  6. Sergio Casas, Wenjie Luo, and Raquel Urtasun. Intentnet: Learning to predict intention from raw sensor data. In Con- ference on Robot Learning, pages 947-956. PMLR, 2018.
  7. Yuning Chai, Benjamin Sapp, Mayank Bansal, and Dragomir Anguelov. Multipath: Multiple probabilistic anchor tra- jectory hypotheses for behavior prediction. arXiv preprint arXiv:1910.05449, 2019. 1, 3, 8, 12 · arXiv
  8. Ming-Fang Chang, John Lambert, Patsorn Sangkloy, Jag- jeet Singh, Slawomir Bak, Andrew Hartnett, De Wang, Peter Carr, Simon Lucey, Deva Ramanan, et al. Argoverse: 3d tracking and forecasting with rich maps. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8748-8757, 2019. 2, 4, 8, 12
  9. Henggang Cui, Thi Nguyen, Fang-Chieh Chou, Tsung-Han Lin, Jeff Schneider, David Bradley, and Nemanja Djuric. Deep kinematic models for kinematically feasible vehicle trajectory predictions. In 2020 IEEE International Con- ference on Robotics and Automation (ICRA), pages 10563- 10569. IEEE, 2020. 3
  10. Henggang Cui, Vladan Radosavljevic, Fang-Chieh Chou, Tsung-Han Lin, Thi Nguyen, Tzu-Kuo Huang, Jeff Schnei- der, and Nemanja Djuric. Multimodal trajectory predictions for autonomous driving using deep convolutional networks. In 2019 International Conference on Robotics and Automa- tion (ICRA), pages 2090-2096. IEEE, 2019. 3, 6
  11. M. Everingham, L. Gool, C. K. Williams, J. Winn, and An- drew Zisserman. The pascal visual object classes (voc) chal- lenge. International Journal of Computer Vision, 88:303- 338, 2009. 5
  12. Jiyang Gao, Chen Sun, Hang Zhao, Yi Shen, Dragomir Anguelov, Congcong Li, and Cordelia Schmid. VectorNet: Encoding hd maps and agent dynamics from vectorized rep- resentation. In CVPR, 2020. 1, 3, 6, 8, 13
  13. Joey Hong, Benjamin Sapp, and James Philbin. Rules of the road: Predicting driving behavior with a convolutional model of semantic interactions. In CVPR, 2019. 3
  14. John Houston, Guido Zuidhof, Luca Bergamini, Yawei Ye, Ashesh Jain, Sammy Omari, Vladimir Iglovikov, and Peter Ondruska. One thousand and one hours: Self-driving motion prediction dataset. arXiv preprint arXiv:2006.14480, 2020. 1, 4 · arXiv
  15. Namhoon Lee, Wongun Choi, Paul Vernaza, Christopher B Choy, Philip HS Torr, and Manmohan Chandraker. Desire: Distant future prediction in dynamic scenes with interacting agents. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 336-345, 2017. 3
  16. Alon Lerner, Yiorgos Chrysanthou, and Dani Lischinski. Crowds by example. In Computer graphics forum, vol- ume 26, pages 655-664. Wiley Online Library, 2007. 3
  17. Ming Liang, Bin Yang, Rui Hu, Yun Chen, Renjie Liao, Song Feng, and Raquel Urtasun. Learning lane graph representations for motion forecasting. arXiv preprint arXiv:2007.13732, 2020. 3 · arXiv
  18. Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740-755. Springer, 2014. 5
  19. Tung Phan-Minh, Elena Corina Grigore, Freddy A Boulton, Oscar Beijbom, and Eric M Wolff. CoverNet: Multimodal behavior prediction using trajectory sets. arXiv:1911.10298, 2019. 3 · arXiv
  20. Nicholas Rhinehart, Kris M Kitani, and Paul Vernaza. R2p2: A reparameterized pushforward policy for diverse, precise generative path forecasting. In Proceedings of the European Conference on Computer Vision (ECCV), pages 772-788, 2018. 12
  21. Nicholas Rhinehart, Rowan McAllister, Kris Kitani, and Sergey Levine. Precog: Prediction conditioned on goals in visual multi-agent settings. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2821- 2830, 2019. 1, 2, 3, 5, 12
  22. Alexandre Robicquet, Amir Sadeghian, Alexandre Alahi, and Silvio Savarese. Learning social etiquette: Human tra- jectory understanding in crowded scenes. In European con- ference on computer vision, pages 549-565, 2016. 3
  23. Tim Salzmann, Boris Ivanovic, Punarjay Chakravarty, and Marco Pavone. Trajectron++: Dynamically-feasible trajec- tory forecasting with heterogeneous data. arXiv preprint arXiv:2001.03093, 2020. 12 · arXiv
  24. Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, et al. Scalability in perception for autonomous driving: Waymo open dataset. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2446-2454, 2020. 1, 3, 4
  25. Simon Suo, Sebastian Regalado, Sergio Casas, and Raquel Urtasun. Trafficsim: Learning to simulate realistic multi- agent behaviors. In Conference on Computer Vision and Pat- tern Recognition (CVPR), 2021. 2, 3
  26. Charlie Tang and Russ R Salakhutdinov. Multiple futures prediction. In NeurIPS, 2019. 1, 2, 3, 5
  27. Luca Anthony Thiede and Pratik Prabhanjan Brahma. An- alyzing the variety loss in the context of probabilistic tra- jectory prediction. In Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision, pages 9954-9963, 2019. 6
  28. Ekaterina Tolstaya, Reza Mahjourian, Carlton Downey, Balakrishnan Vadarajan, Benjamin Sapp, and Dragomir Anguelov. Identifying driver interactions via conditional be- havior prediction. 2021 IEEE International Conference on Robotics and Automation (ICRA), 2021. 1, 7, 13
  29. Fisher Yu, Haofeng Chen, Xin Wang, Wenqi Xian, Yingying Chen, Fangchen Liu, Vashisht Madhavan, and Trevor Dar- rell. Bdd100k: A diverse driving dataset for heterogeneous multitask learning. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 2636-2645, 2020. 1
  30. Wei Zhan, Liting Sun, Di Wang, Haojie Shi, Aubrey Clausse, Maximilian Naumann, Julius Kümmerle, Hendrik Königshof, Christoph Stiller, Arnaud de La Fortelle, and Masayoshi Tomizuka. INTERACTION Dataset: An IN- TERnational, Adversarial and Cooperative moTION Dataset in Interactive Driving Scenarios with Semantic Maps. arXiv:1910.03088 [cs, eess], 2019. 2, 3, 4 · arXiv
  31. Hang Zhao, Jiyang Gao, Tian Lan, Chen Sun, Benjamin Sapp, Balakrishnan Varadarajan, Yue Shen, Yi Shen, Yuning Chai, Cordelia Schmid, et al. Tnt: Target-driven trajectory prediction. arXiv preprint arXiv:2008.08294, 2020. 1, 3, 8 · arXiv
  32. Yin Zhou, Pei Sun, Yu Zhang, Dragomir Anguelov, Jiyang Gao, Tom Ouyang, James Guo, Jiquan Ngiam, and Vijay Va- sudevan. End-to-end multi-view fusion for 3d object detec- tion in lidar point clouds. In Conference on Robot Learning, pages 923-932, 2020.

Cited by

Sorted by citations

Authors
Ettinger, S., Cheng, S., Caine, B., Liu, C., Zhao, H., Pradhan, S., Chai, Y., Sapp, B., Qi, C., Zhou, Y., Yang, Z., Chouard, A., Sun, P., Ngiam, J., Vasudevan, V., McCauley, A., Shlens, J., & Anguelov, D.
Affiliations
  • Waymo LLC
  • Google (United States) (US)
ORCID
Venue
2021 IEEE/CVF International Conference on Computer Vision (ICCV)
Publisher
IEEE
Published
2021-10
Pages
9690-9699
DOI
10.1109/iccv48922.2021.00957
OpenAlex
W3156216502
Cited by
584
References
4
Citations by year
2026: 113 · 2025: 160 · 2024: 120 · 2023: 116 · 2022: 66 · 2021: 9
FWCI
77.03
Type
conference paper
Language
en
Open access
Licence
Locations
  • IEEE/CVF International Conference on Computer Vision (ICCV) (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Engineering › Automotive Engineering › Autonomous Vehicle Technology and Safety
  • Physical Sciences › Engineering › Building and Construction › Traffic Prediction and Management Techniques
  • Physical Sciences › Engineering › Safety, Risk, Reliability and Quality › Traffic and Road Safety
Keywords
motion prediction, autonomous driving, Waymo Open Dataset, 3D bounding boxes, 3D map, motion planning, evaluation metrics, data annotation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review