CoChat
Sign Up
Cited by 1,474Autonomous Vehicle Technology and SafetyRobotics and Sensor-Based LocalizationAdvanced Vision and Imaging

Argoverse: 3D Tracking and Forecasting With Rich Maps

Open access
Chang, Ming-Fang, Lambert, John, Sangkloy, Patsorn, Singh, Jagjeet, Bak, Slawomir, Hartnett, Andrew, Wang, De, Carr, Peter +3 more·2019·2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
DOI 10.1109/cvpr.2019.00895
Open this paper in CoChat — free accountOpen PDF

Abstract

We present Argoverse, a dataset designed to support autonomous vehicle perception tasks including 3D tracking and motion forecasting. Argoverse includes sensor data collected by a fleet of autonomous vehicles in Pittsburgh and Miami as well as 3D tracking annotations, 300k extracted interesting vehicle trajectories, and rich semantic maps. The sensor data consists of 360 degree images from 7 cameras with overlapping fields of view, forward-facing stereo imagery, 3D point clouds from long range LiDAR, and 6-DOF pose. Our 290km of mapped lanes contain rich geometric and semantic metadata which are not currently available in any public dataset. All data is released under a Creative Commons license at Argoverse.org. In baseline experiments, we use map information such as lane direction, driveable area, and ground height to improve the accuracy of 3D object tracking. We use 3D object tracking to mine for more than 300k interesting vehicle trajectories to create a trajectory forecasting benchmark. Motion forecasting experiments ranging in complexity from classical methods (k-NN) to LSTMs demonstrate that using detailed vector maps with lane-level information substantially reduces prediction error. Our tracking and forecasting experiments represent only a superficial exploration of the potential of rich maps in robotic perception. We hope that Argoverse will enable the research community to explore these problems in greater depth.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/cvpr.2019.00895

Cites

7 of 7 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The offline perception system used for auto-labeling is not constrained by latency and causality, so the fidelity of generated tracks is drastically higher than online perception systems, enabling evaluation of planning under very limited perception noise.

    “Compared with the online perception systems used in many other datasets [19], [13], the offline version is not constrained by latency and causality.Therefore, the fidelity of the generated tracks is drastically higher, enabling us to evaluate planning performance under very limited perception noise.”

    From Towards learning-based planning: The nuPlan benchmark for real-world autonomous driving (2024) · cites this paper as [13]

    Split into 2 atoms — 0 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 1 unverifiable. Expand the claim to see per-atom verdicts.

    Experimental — accuracy not yet measured

  • The Argoverse dataset is a real (not simulated) dataset.

    “Argoverse Real”

    From Vehicle-to-Everything Communication in Intelligent Connected Vehicles: A Survey and Taxonomy (2025) · cites this paper as [124]

  • Most other datasets use a combination of manual and automated driving, which may lead the planner to imitate less desirable driving behavior.

    “Since our focus is on planning, it is crucial that we drive manually, while most other datasets [13], [14], [17], [19] use a combination of manual and automated driving, which may lead the planner to imitate less desirable driving behavior.”

    From Towards learning-based planning: The nuPlan benchmark for real-world autonomous driving (2024) · cites this paper as [13]

    The Related Work section compares autonomous-driving datasets, and §3.3 describes trajectories mined from Argoverse driving logs. Neither passage says whether those logs combine manual and automated driving or whether such a combination could teach a planner less desirable behavior. Searches for those driving modes and planner imitation found no passage addressing the claim. (LLM, conf=0.72)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 51

  1. Alexandre Alahi, Kratarth Goel, Vignesh Ramanathan, Alexandre Robicquet, Li Fei-Fei, and Silvio Savarese. So- cial lstm: Human trajectory prediction in crowded spaces. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
  2. Mykhaylo Andriluka, Stefan Roth, and Bernt Schiele. People-tracking-by-detection and people-detection-by- tracking. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2008.
  3. Andrew Bacha, Cheryl Bauman, Ruel Faruque, Michael Fleming, Chris Terwelp, Charles Reinholtz, Dennis Hong, Al Wicks, Thomas Alberi, David Anderson, Stephen Cacci- ola, Patrick Currier, Aaron Dalton, Jesse Farmer, Jesse Hur- dus, Shawn Kimmel, Peter King, Andrew Taylor, David Van Covern, and Mike Webster. Odin: Team victortango's entry in the darpa urban challenge. J. Field Robot., 25(8):467-492, Aug. 2008.
  4. Min Bai, Gellért Máttyus, Namdar Homayounfar, Shenlong Wang, Shrinidhi Kowshika Lakshmikanth, and Raquel Urta- sun. Deep multi-sensor lane detection. In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2018, Madrid, Spain, October 1-5, 2018, pages 3102- 3109, 2018.
  5. Sergio Casas, Wenjie Luo, and Raquel Urtasun. Intentnet: Learning to predict intention from raw sensor data. In Aude Billard, Anca Dragan, Jan Peters, and Jun Morimoto, edi- tors, Proceedings of The 2nd Conference on Robot Learning, volume 87 of Proceedings of Machine Learning Research, pages 947-956. PMLR, 29-31 Oct 2018.
  6. Florian Chabot, Mohamed Chaouch, Jaonary Rabarisoa, Ce- line Teulière, and Thierry Chateau. Deep manta: A coarse- to-fine many-task network for joint 2d and 3d vehicle analy- sis from monocular image. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
  7. Chenyi Chen, Ari Seff, Alain Kornhauser, and Jianxiong Xiao. Deepdriving: Learning affordance for direct percep- tion in autonomous driving. In The IEEE International Con- ference on Computer Vision (ICCV), 2015.
  8. Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
  9. Nachiket Deo and Mohan M Trivedi. Convolutional so- cial pooling for vehicle trajectory prediction. arXiv preprint arXiv:1805.06771, 2018. · arXiv
  10. Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. Carla: An open urban driving simulator. arXiv preprint arXiv:1711.03938, 2017. · arXiv
  11. Martin Ester, Hans peter Kriegel, Jörg Sander, and Xiaowei Xu. A density-based algorithm for discovering clusters in large spatial databases with noise. In KDD Proceedings of the Second International Conference on Knowledge Discov- ery and Data Mining, pages 226-231. AAAI Press, 1996.
  12. Alex Graves. Generating sequences with recurrent neural networks. CoRR, abs/1308.0850, 2013. · arXiv
  13. Junyao Guo, Unmesh Kurup, and Mohak Shah. Is it safe to drive? an overview of factors, challenges, and datasets for driveability assessment in autonomous driving. IEEE Trans- actions on Intelligent Transportation Systems, 2019.
  14. Agrim Gupta, Justin Johnson, Li Fei-Fei, Silvio Savarese, and Alexandre Alahi. Social gan: Socially acceptable trajec- tories with generative adversarial networks. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
  15. K. He, G. Gkioxari, P. Dollár, and R. Girshick. Mask r-cnn. In 2017 IEEE International Conference on Computer Vision (ICCV), pages 2980-2988, Oct 2017.
  16. Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Gir- shick. Mask R-CNN. In Proceedings of the International Conference on Computer Vision (ICCV), 2017.
  17. Simon Hecker, Dengxin Dai, and Luc Van Gool. End-to-end learning of driving models with surround-view cameras and route planners. In European Conference on Computer Vision (ECCV), 2018.
  18. David Held, Devin Guillory, Brice Rebsamen, Sebastian Thrun, and Silvio Savarese. A probabilistic framework for real-time 3d segmentation using spatial, temporal, and se- mantic cues. In Proceedings of Robotics: Science and Sys- tems, 2016.
  19. David Held, Jesse Levinson, and Sebastian Thrun. Precision tracking with sparse 3d and dense color 2d data. In ICRA, 2013.
  20. David Held, Jesse Levinson, Sebastian Thrun, and Silvio Savarese. Combining 3d shape, color, and motion for ro- bust anytime tracking. In Proceedings of Robotics: Science and Systems, Berkeley, USA, July 2014.
  21. Michael Himmelsbach and Hans-Joachim Wünsche. Lidar- based 3d object perception. In Proceedings of 1st Interna- tional Workshop on Cognition for Technical Systems, 2008.
  22. Namdar Homayounfar, Wei-Chiu Ma, Shrinidhi Kow- shika Lakshmikanth, and Raquel Urtasun. Hierarchical re- current attention networks for structured online maps. In The IEEE Conference on Computer Vision and Pattern Recogni- tion (CVPR), June 2018.
  23. Simon Julier, Jeffrey Uhlmann, and Hugh F Durrant-Whyte. A new method for the nonlinear transformation of means and covariances in filters and estimators. IEEE Transactions on automatic control, 45(3):477-482, 2000.
  24. Simon J Julier, Jeffrey K Uhlmann, and Hugh F Durrant- Whyte. A new approach for filtering nonlinear systems. In Proceedings of 1995 American Control Conference-ACC'95, volume 3, pages 1628-1632. IEEE, 1995.
  25. Abhijit Kundu, Yin Li, and James M. Rehg. 3d-rcnn: Instance-level 3d object reconstruction via render-and- compare. In IEEE Conference on Computer Vision and Pat- tern Recognition (CVPR), 2018.
  26. Alex H. Lang, Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang, and Oscar Beijbom. Pointpillars: Fast encoders for object detection from point clouds. In CVPR, 2018.
  27. Namhoon Lee, Wongun Choi, Paul Vernaza, Christo- pher Bongsoo Choy, Philip H. S. Torr, and Manmo- han Krishna Chandraker. DESIRE: distant future predic- tion in dynamic scenes with interacting agents. CoRR, abs/1704.04394, 2017. · arXiv
  28. Jesse Levinson, Jake Askeland, Jan Becker, Jennifer Dolson, David Held, Sören Kammel, J. Zico Kolter, Dirk Langer, Oliver Pink, Vaughan R. Pratt, Michael Sokolsky, Ganymed Stanek, David Michael Stavens, Alex Teichman, Moritz Werling, and Sebastian Thrun. Towards fully autonomous driving: Systems and algorithms. In IEEE Intelligent Ve- hicles Symposium (IV), 2011, Baden-Baden, Germany, June 5-9, 2011, pages 163-168, 2011.
  29. Justin Liang, Namdar Homayounfar, Wei-Chiu Ma, Shen- long Wang, and Raquel Urtasun. Convolutional recurrent network for road boundary extraction. In The IEEE Confer- ence on Computer Vision and Pattern Recognition (CVPR), June 2019.
  30. Justin Liang and Raquel Urtasun. End-to-end deep structured models for drawing crosswalks. In The European Conference on Computer Vision (ECCV), September 2018.
  31. Ming Liang, Bin Yang, Yun Chen, Rui Hu, and Raquel Urta- sun. Multi-task multi-sensor fusion for 3d object detection. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
  32. Xingyu Liu, Charles R Qi, and Leonidas J Guibas. Flownet3d: Learning scene flow in 3d point clouds. arXiv preprint arXiv:1806.01411, 2019. · arXiv
  33. Wenjie Luo, Bin Yang, and Raquel Urtasun. Fast and furious: Real time end-to-end 3d detection, tracking and motion fore- casting with a single convolutional net. In The IEEE Confer- ence on Computer Vision and Pattern Recognition (CVPR), June 2018.
  34. Suraj M S, Hugo Grimmett, Lukas Platinsky, and Peter On- druska. Visual vehicle tracking through noise and occlusions using crowd-sourced maps. In Intelligent Robots and Sys- tems (IROS), 2018 IEEE international conference on, pages 4531-4538. IEEE, 2018.
  35. Yuexin Ma, Xinge Zhu, Sibo Zhang, Ruigang Yang, Wen- ping Wang, and Dinesh Manocha. Trafficpredict: Trajectory prediction for heterogeneous traffic-agents. In Proceedings of the 33rd National Conference on Artifical Intelligence, AAAI'19. AAAI Press, 2019.
  36. Michael Montemerlo, Jan Becker, Suhrid Bhat, Hendrik Dahlkamp, Dmitri Dolgov, Scott Ettinger, Dirk Haehnel, Tim Hilden, Gabe Hoffmann, Burkhard Huhnke, Doug John- ston, Stefan Klumpp, Dirk Langer, Anthony Levandowski, Jesse Levinson, Julien Marcil, David Orenstein, Johannes Paefgen, Isaac Penny, Anna Petrovskaya, Mike Pflueger, Ganymed Stanek, David Stavens, Antone Vogt, and Sebas- tian Thrun. Junior: The stanford entry in the urban challenge. J. Field Robot., 25(9):569-597, Sept. 2008.
  37. Gaurav Pandey, James R Mcbride, and Ryan M Eustice. Ford campus vision and lidar data set. Int. J. Rob. Res., 30(13):1543-1552, Nov. 2011.
  38. SeongHyeon Park, Byeongdo Kim, Chang Mook Kang, Chung Choo Chung, and Jun Won Choi. Sequence-to- sequence prediction of vehicle trajectory via LSTM encoder- decoder architecture. In Intelligent Vehicles Symposium, pages 1672-1678. IEEE, 2018.
  39. Abhishek Patil, Srikanth Malla, Haiming Gang, and Yi-Ting Chen. The h3d dataset for full-surround 3d multi-object de- tection and tracking in crowded urban scenes. In Interna- tional Conference on Robotics and Automation, 2019.
  40. Luis Patino, Tom Cane, Alain Vallee, and James Ferryman. Pets 2016: Dataset and challenge. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recog- nition Workshops, pages 1-8, 2016.
  41. Akshay Rangesh, Kevan Yuen, Ravi Kumar Satzoda, Rakesh Nattoji Rajaram, Pujitha Gunaratne, and Mohan M. Trivedi. A multimodal, full-surround vehicular testbed for naturalistic studies and benchmarking: Design, calibration and deployment. CoRR, abs/1709.07502, 2017. · arXiv
  42. Nicholas Rhinehart, Kris M Kitani, and Paul Vernaza. R2p2: A reparameterized pushforward policy for diverse, precise generative path forecasting. In Proceedings of the European Conference on Computer Vision (ECCV), pages 772-788, 2018.
  43. Nicholas Rhinehart, Rowan McAllister, Kris Kitani, and Sergey Levine. Precog: Prediction conditioned on goals in visual multi-agent settings. arXiv preprint arXiv:1905.01296, 2019. · arXiv
  44. Stephan R. Richter, Zeeshan Hayder, and Vladlen Koltun. Playing for benchmarks. In IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22- 29, 2017, pages 2232-2241, 2017.
  45. R.B. Rusu and S. Cousins. 3d is here: Point cloud library (pcl). In Robotics and Automation (ICRA), 2011 IEEE Inter- national Conference on, pages 1 -4, may 2011.
  46. Xibin Song, Peng Wang, Dingfu Zhou, Rui Zhu, Chenye Guan, Yuchao Dai, Hao Su, Hongdong Li, and Ruigang Yang. Apollocar3d: A large 3d car instance understanding benchmark for autonomous driving. CoRR, abs/1811.12222, 2018. · arXiv
  47. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. In Z. Ghahra- mani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger, editors, Advances in Neural Information Pro- cessing Systems 27, pages 3104-3112.
  48. Weiyue Wang, Ronald Yu, Qiangui Huang, and Ulrich Neu- mann. Sgpn: Similarity group proposal network for 3d point cloud instance segmentation. In CVPR, 2018.
  49. Bin Yang, Ming Liang, and Raquel Urtasun. Hdnet: Exploit- ing hd maps for 3d object detection. In Aude Billard, Anca Dragan, Jan Peters, and Jun Morimoto, editors, Proceedings of The 2nd Conference on Robot Learning, volume 87 of Proceedings of Machine Learning Research, pages 146-155. PMLR, 29-31 Oct 2018.
  50. Raymond A Yeh, Alexander G Schwing, Jonathan Huang, and Kevin Murphy. Diverse generation for multi-agent sports games. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4610-4619, 2019.
  51. Yin Zhou and Oncel Tuzel. Voxelnet: End-to-end learn- ing for point cloud based 3d object detection. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.

Cited by

Sorted by citations

Authors
Chang, M.-F., Lambert, J., Sangkloy, P., Singh, J., Bak, S., Hartnett, A., Wang, D., Carr, P., Lucey, S., Ramanan, D., & Hays, J.
Affiliations
  • Georgia Institute of Technology (US)
  • CMU
  • Carnegie Mellon University (US)
  • Georgia Institute of Technology
  • Argo AI
ORCID
Venue
2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
Publisher
IEEE
Published
2019-06
Pages
8740-8749
ISSN
1063-6919 (ISSN-L)
DOI
10.1109/cvpr.2019.00895
OpenAlex
W2955189650
Cited by
1,474
References
7
Citations by year
2026: 129 · 2025: 251 · 2024: 287 · 2023: 265 · 2022: 230 · 2021: 180 · 2020: 117 · 2019: 15
FWCI
134.87
Type
conference paper
Language
en
Open access
Licence
Locations
  • IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Engineering › Automotive Engineering › Autonomous Vehicle Technology and Safety
  • Physical Sciences › Engineering › Aerospace Engineering › Robotics and Sensor-Based Localization
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Vision and Imaging
Keywords
autonomous vehicles, 3D object tracking, motion prediction, trajectory prediction, semantic maps, vehicle trajectories, LiDAR point cloud, stereoscopic images, robot perception

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review