CoChat
Sign Up
PreprintCited by 1Advanced Neural Network ApplicationsAutonomous Vehicle Technology and SafetyAdvanced Vision and Imaging

Understanding Bird's-Eye View of Road Semantics using an Onboard Camera

Open access
Can, Yigit Baran, Liniger, Alexander, Unal, Ozan, Paudel, Danda, Van Gool, Luc·2020·arXiv
DOI 10.48550/arxiv.2012.03040arXiv 2012.03040
Open this paper in CoChat — free accountOpen PDF

Abstract

Autonomous navigation requires scene understanding of the action-space to move or anticipate events. For planner agents moving on the ground plane, such as autonomous vehicles, this translates to scene understanding in the bird's-eye view (BEV). However, the onboard cameras of autonomous cars are customarily mounted horizontally for a better view of the surrounding. In this work, we study scene understanding in the form of online estimation of semantic BEV maps using the video input from a single onboard camera. We study three key aspects of this task, image-level understanding, BEV level understanding, and the aggregation of temporal information. Based on these three pillars we propose a novel architecture that combines these three aspects. In our extensive experiments, we demonstrate that the considered aspects are complementary to each other for BEV understanding. Furthermore, the proposed architecture significantly surpasses the current state-of-the-art. Code: https://github.com/ybarancan/BEV_feat_stitch.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2012.03040

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • BEVFormer's temporal self-attention can more effectively model long temporal dependency than simply stacking BEV features.

    “Compared to simply stacking BEV in [18, 38, 6], our temporal self-attention can more effectively model long temporal dependency.”

    From BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformers (2022) · cites this paper as [6]

    The citation is attached to the characterization of this paper as ‘simply stacking BEV,’ not to the citing authors’ claim about their own self-attention. The source’s ‘Aggregation’ section (page unavailable) instead describes warping features into BEV, combining frames with symmetric max or mean aggregation, and then concatenating the resulting maps. It does not simply stack BEV features across frames. (LLM, conf=0.85)

    Checked by a reasoning model — may be wrong.

    Experimental — accuracy not yet measured

  • Previous works consider temporal information by stacking BEV features from several timestamps.

    “In addition to the spatial information, previous works [18, 38, 6] also consider the temporal information by stacking BEV features from several timestamps.”

    From BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformers (2022) · cites this paper as [6]

    The Temporal Aggregation section describes transforming features from multiple frames into BEV grids. But the Aggregation subsection says those grids are combined across frames using max or mean aggregation; the feature maps subsequently concatenated are not simply BEV features stacked from several timestamps. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 34

  1. Holger Caesar, Varun Bankiti, Alex H. Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. arXiv preprint arXiv:1903.11027, 2019. · arXiv
  2. Yigit Baran Can, Alexander Liniger, Danda Pani Paudel, and Luc Van Gool. Structured bird's-eye-view traffic scene understanding from onboard images. In Proceedings of the IEEE/CVF International Con- ference on Computer Vision, pages 15661-15670, 2021.
  3. Yigit Baran Can, Alexander Liniger, Danda Pani Paudel, and Luc Van Gool. Topology preserving local road network estimation from single onboard camera image. arXiv preprint arXiv:2112.10155, 2021. · arXiv
  4. Sergio Casas, Wenjie Luo, and Raquel Urtasun. Intentnet: Learning to predict intention from raw sensor data. In Conference on Robot Learning, pages 947-956, 2018.
  5. Ming-Fang Chang, John Lambert, Patsorn Sangkloy, Jagjeet Singh, Slawomir Bak, Andrew Hartnett, De Wang, Peter Carr, Simon Lucey, Deva Ramanan, and James Hays. Argoverse: 3d tracking and forecasting with rich maps. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 8748-8757. Computer Vision Foundation / IEEE, 2019. Component IoU per class [%] mIoU
  6. Dian Chen, Brady Zhou, Vladlen Koltun, and Philipp Krähenbühl. Learning by cheating. In Conference on Robot Learning (CoRL), 2020.
  7. Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder-decoder with atrous separable convolution for semantic image segmentation. arXiv:1802.02611, 2018. · arXiv
  8. Bowen Cheng, Maxwell D Collins, Yukun Zhu, Ting Liu, Thomas S Huang, Hartwig Adam, and Liang-Chieh Chen. Panoptic-deeplab: A simple, strong, and fast baseline for bottom-up panoptic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12475-12485, 2020.
  9. Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene under- standing. In Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
  10. Henggang Cui, Vladan Radosavljevic, Fang-Chieh Chou, Tsung-Han Lin, Thi Nguyen, Tzu-Kuo Huang, Jeff Schneider, and Nemanja Djuric. Multimodal trajectory predictions for autonomous driving using deep convolutional networks. In International Conference on Robotics and Automation (ICRA), pages 2090-2096, 2019.
  11. Hugh Durrant-Whyte and Tim Bailey. Simultaneous localization and mapping: part i. IEEE robotics & automation magazine, 13(2):99-110, 2006.
  12. Noureldin Hendy, Cooper Sloan, Feng Tian, Pengfei Duan, Nick Charchut, Yuesong Xie, Chuang Wang, and James Philbin. Fishing net: Future inference of semantic heatmaps in grids. arXiv preprint arXiv:2006.09917, 2020. · arXiv
  13. Joey Hong, Benjamin Sapp, and James Philbin. Rules of the road: Predicting driving behavior with a convolutional model of semantic interactions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8454-8462, 2019.
  14. Wonje Jang, Jhonghyun An, Sangyun Lee, Minho Cho, Myungki Sun, and Euntai Kim. Road lane semantic segmentation for high definition map. In 2018 IEEE Intelligent Vehicles Symposium (IV), pages 1001- 1006. IEEE, 2018.
  15. Maximilian Jaritz. 2D-3D scene understanding for autonomous driving. PhD thesis, PSL Research University, 2020.
  16. Tae-young Ko and Seung-ho Lee. Novel method of semantic segmen- tation applicable to augmented reality. Sensors, 20(6):1737, 2020.
  17. Tsung-Yi Lin, Priya Goyal, Ross B. Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. IEEE Trans. Pattern Anal. Mach. Intell., 42(2):318-327, 2020.
  18. Rosanne Liu, Joel Lehman, Piero Molino, Felipe Petroski Such, Eric Frank, Alex Sergeev, and Jason Yosinski. An intriguing failing of convolutional neural networks and the coordconv solution. In Advances in Neural Information Processing Systems, NeurIPS 2018, Montréal, Canada, pages 9628-9639, 2018.
  19. Chenyang Lu, Marinus Jacobus Gerardus van de Molengraft, and Gijs Dubbelman. Monocular semantic occupancy grid mapping with convo- lutional variational encoder-decoder networks. IEEE Robotics Autom. Lett., 4(2):445-452, 2019.
  20. Kaustubh Mani, Swapnil Daga, Shubhika Garg, N. Sai Shankar, Kr- ishna Murthy Jatavallabhula, and K. Madhava Krishna. Monolayout: Amodal scene layout from a single image. CoRR, abs/2002.08394, 2020. · arXiv
  21. David Nistér, Oleg Naroditsky, and James Bergen. Visual odometry. In Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004., volume 1, pages I-I. Ieee, 2004.
  22. Bowen Pan, Jiankai Sun, Ho Yin Tiga Leung, Alex Andonian, and Bolei Zhou. Cross-view semantic segmentation for sensing surroundings. IEEE Robotics Autom. Lett., 5(3):4867-4873, 2020.
  23. David Paz, Hengyuan Zhang, Qinru Li, Hao Xiang, and Henrik Chris- tensen. Probabilistic semantic mapping for urban autonomous driving applications. arXiv preprint arXiv:2006.04894, 2020. · arXiv
  24. Jonah Philion and Sanja Fidler. Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting to 3d. In Proceedings of the European Conference on Computer Vision, 2020.
  25. Lennart Reiher, Bastian Lampe, and Lutz Eckstein. A sim2real deep learning approach for the transformation of images from multiple vehicle-mounted cameras to a semantically segmented image in bird's eye view. arXiv preprint arXiv:2005.04078, 2020. · arXiv
  26. Hayko Riemenschneider, András Bódis-Szomorú, Julien Weissenberg, and Luc Van Gool. Learning where to classify in multi-view semantic segmentation. In European Conference on Computer Vision, pages 516- 532. Springer, 2014.
  27. Thomas Roddick and Roberto Cipolla. Predicting semantic map rep- resentations from images using pyramid occupancy networks. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pages 11135-11144.
  28. Samuel Schulter, Menghua Zhai, Nathan Jacobs, and Manmohan Chan- draker. Learning to look around objects for top-view representations of outdoor scenes. In Proceedings of the European Conference on Computer Vision (ECCV), pages 787-802, 2018.
  29. Muhammad Sualeh and Gon-Woo Kim. Simultaneous localization and mapping in the epoch of semantics: a survey. International Journal of Control, Automation and Systems, 17(3):729-742, 2019.
  30. Shinya Sumikura, Mikiya Shibuya, and Ken Sakurada. Openvslam: a versatile visual slam framework. In Proceedings of the 27th ACM International Conference on Multimedia, pages 2292-2295, 2019.
  31. Pengxiang Wu, Siheng Chen, and Dimitris N Metaxas. Motionnet: Joint perception and motion prediction for autonomous driving based on bird's eye view maps. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 11385-11395, 2020.
  32. Kai Xu, Longyin Wen, Guorong Li, Liefeng Bo, and Qingming Huang. Spatiotemporal cnn for video object segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1379-1388, 2019.
  33. Bin Yang, Ming Liang, and Raquel Urtasun. Hdnet: Exploiting hd maps for 3d object detection. In Conference on Robot Learning, pages 146- 155, 2018.
  34. Jan-Nico Zaech, Dengxin Dai, Alexander Liniger, and Luc Van Gool. Action sequence predictions of vehicles in urban environments using map and social context. In International Conference on Intelligent Robots and Systems (IROS), 2020.

Cited by

Sorted by citations

Authors
Can, Y. B., Liniger, A., Unal, O., Paudel, D., & Van Gool, L.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2020
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2012.03040
arXiv
2012.03040
OpenAlex
W4287556977
Cited by
0
References
0
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Engineering › Automotive Engineering › Autonomous Vehicle Technology and Safety
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Vision and Imaging
Keywords
bird's-eye view, semantic scene understanding, autonomous driving, onboard camera, online estimation, temporal aggregation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review