Translating Images into Maps
Open accessAbstract
We approach instantaneous mapping, converting images to a top-down view of the world, as a translation problem. We show how a novel form of transformer network can be used to map from images and video directly to an overhead map or bird's-eye-view (BEV) of the world, in a single end-to-end network. We assume a 1-1 correspondence between a vertical scanline in the image, and rays passing through the camera location in an overhead map. This lets us formulate map generation from an image as a set of sequence-to-sequence translations. Posing the problem as translation allows the network to use the context of the image when interpreting the role of each pixel. This constrained formulation, based upon a strong physical grounding of the problem, leads to a restricted transformer network that is convolutional in the horizontal direction only. The structure allows us to make efficient use of data when training, and obtains state-of-the-art results for instantaneous mapping of three large-scale datasets, including a 15% and 30% relative gain against existing best performing methods on the nuScenes and Argoverse datasets, respectively. We make our code available on https://github.com/avishkarsaha/translating-images-into-maps.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
BEVFormer's temporal self-attention can more effectively model long temporal dependency than simply stacking BEV features.
“Compared to simply stacking BEV in [18, 38, 6], our temporal self-attention can more effectively model long temporal dependency.”
From BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformers (2022) · cites this paper as [38]
The citation is attached to the description of prior work as ‘simply stacking BEV.’ In ‘C. Model architecture’ (page unavailable), the cited paper instead says it applies axial attention across spatial and temporal axes to BEV features from multiple timesteps. Its temporal method is therefore not simply stacking features; the paper does not assess BEVFormer’s comparative claim. (LLM, conf=0.86)
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredPrevious works consider temporal information by stacking BEV features from several timestamps.
“In addition to the spatial information, previous works [18, 38, 6] also consider the temporal information by stacking BEV features from several timestamps.”
From BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformers (2022) · cites this paper as [38]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
No citation lists for this paper
No citation lists for this paper yet — checked again daily.
References 41 · Cited by 142
Research works printed but not linked: 35
- S. Sengupta, P. Sturgess, L. Ladickỳ, and P. H. Torr, "Automatic dense visual semantic mapping from street-level imagery," in 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems. IEEE, 2012, pp. 857-862.
- B. Pan, J. Sun, H. Y. T. Leung, A. Andonian, and B. Zhou, "Cross-view semantic segmentation for sensing surroundings," IEEE Robotics and Automation Letters, 2020.
- B. Liu, B. Zhuang, S. Schulter, P. Ji, and M. Chandraker, "Understanding road layout from videos as a whole," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 4414-4423.
- Z. Wang, B. Liu, S. Schulter, and M. Chandraker, "A parametric top- view representation of complex road scenes," in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019, pp. 10 325-10 333.
- S. Schulter, M. Zhai, N. Jacobs, and M. Chandraker, "Learning to look around objects for top-view representations of outdoor scenes," in Proceedings of the European Conference on Computer Vision (ECCV), 2018, pp. 787-802.
- C. Lu, M. J. G. van de Molengraft, and G. Dubbelman, "Monocular semantic occupancy grid mapping with convolutional variational encoder-decoder networks," IEEE Robotics and Automation Letters, vol. 4, no. 2, pp. 445-452, 2019.
- K. Mani, S. Daga, S. Garg, S. S. Narasimhan, M. Krishna, and K. M. Jatavallabhula, "Monolayout: Amodal scene layout from a single image," in The IEEE Winter Conference on Applications of Computer Vision, 2020, pp. 1689-1697.
- T. Roddick and R. Cipolla, "Predicting semantic map representations from images using pyramid occupancy networks," in The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
- J. Philion and S. Fidler, "Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting to 3d," in Proceedings of the European Conference on Computer Vision, 2020.
- A. Saha, O. Mendez, C. Russell, and R. Bowden, "Enabling spatio- temporal aggregation in birds-eye-view vehicle estimation," in Pro- ceedings of the International Conference on Robotics and Automation, 2021.
- A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, "Attention is all you need," in Advances in neural information processing systems, 2017, pp. 5998-6008.
- D. Wang, C. Devin, Q.-Z. Cai, P. Krähenbühl, and T. Darrell, "Monocular plan view networks for autonomous driving," in 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2019, pp. 2876-2883.
- A. Mousavian, D. Anguelov, J. Flynn, and J. Kosecka, "3d bounding box estimation using deep learning and geometry," in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 7074-7082.
- W. Kehl, F. Manhardt, F. Tombari, S. Ilic, and N. Navab, "Ssd-6d: Making rgb-based 3d detection and 6d pose estimation great again," in Proceedings of the IEEE International Conference on Computer Vision, 2017, pp. 1521-1529.
- A. Simonelli, S. R. Bulo, L. Porzi, M. López-Antequera, and P. Kontschieder, "Disentangling monocular 3d object detection," in Proceedings of the IEEE International Conference on Computer Vision, 2019, pp. 1991-1999.
- P. Poirson, P. Ammirato, C.-Y. Fu, W. Liu, J. Kosecka, and A. C. Berg, "Fast single shot detection and pose estimation," in 2016 Fourth International Conference on 3D Vision (3DV). IEEE, 2016, pp. 676- 684.
- A. Palazzi, G. Borghi, D. Abati, S. Calderara, and R. Cucchiara, "Learning to map vehicles into bird's eye view," in International Conference on Image Analysis and Processing. Springer, 2017, pp. 233-243.
- X. Chen, K. Kundu, Z. Zhang, H. Ma, S. Fidler, and R. Urtasun, "Monocular 3d object detection for autonomous driving," in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 2147-2156.
- T. Roddick, A. Kendall, and R. Cipolla, "Orthographic feature transform for monocular 3d object detection," in Proceedings of the British Machine Vision Conference (BMVC), 2019.
- A. Hu, Z. Murez, N. Mohan, S. Dudas, J. Hawke, V. Badrinarayanan, R. Cipolla, and A. Kendall, "FIERY: Future instance segmentation in bird's-eye view from surround monocular cameras," in Proceedings of the International Conference on Computer Vision (ICCV), 2021.
- D. Bahdanau, K. Cho, and Y. Bengio, "Neural machine translation by jointly learning to align and translate," in 3rd International Conference on Learning Representations, ICLR 2015, 2015.
- J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, "Bert: Pre-training of deep bidirectional transformers for language understanding," in NAACL-HLT (1), 2019.
- N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko, "End-to-end object detection with transformers," in Proceedings of the European Conference on Computer Vision, 2020.
- N. Arivazhagan, C. Cherry, W. Macherey, C.-C. Chiu, S. Yavuz, R. Pang, W. Li, and C. Raffel, "Monotonic infinite lookback attention for simultaneous machine translation," in Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019, pp. 1313-1323.
- C. Raffel, M.-T. Luong, P. J. Liu, R. J. Weiss, and D. Eck, "Online and linear-time attention by enforcing monotonic alignments," in International Conference on Machine Learning, 2017, pp. 2837-2846.
- C.-C. Chiu and C. Raffel, "Monotonic chunkwise attention," in International Conference on Learning Representations, 2018.
- X. Ma, J. M. Pino, J. Cross, L. Puzon, and J. Gu, "Monotonic multihead attention," in International Conference on Learning Representations, 2019.
- Z. Tian, C. Shen, H. Chen, and T. He, "Fcos: Fully convolutional one- stage object detection," in Proceedings of the IEEE/CVF international conference on computer vision, 2019, pp. 9627-9636.
- H. Badino, U. Franke, and D. Pfeiffer, "The stixel world-a compact medium level representation of the 3d-world," in Joint Pattern Recognition Symposium. Springer, 2009, pp. 51-60.
- D. Pfeiffer and U. Franke, "Modeling dynamic 3d environments by means of the stixel world," IEEE Intelligent Transportation Systems Magazine, vol. 3, no. 3, pp. 24-36, 2011.
- F. Yu, D. Wang, E. Shelhamer, and T. Darrell, "Deep layer aggregation," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 2403-2412.
- H. Caesar, V. Bankiti, A. H. Lang, S. Vora, V. E. Liong, Q. Xu, A. Krishnan, Y. Pan, G. Baldan, and O. Beijbom, "nuscenes: A multimodal dataset for autonomous driving," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 11 621-11 631.
- M.-F. Chang, J. Lambert, P. Sangkloy, J. Singh, S. Bak, A. Hartnett, D. Wang, P. Carr, S. Lucey, D. Ramanan et al., "Argoverse: 3d tracking and forecasting with rich maps," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 8748-8757.
- K. He, X. Zhang, S. Ren, and J. Sun, "Deep residual learning for image recognition," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770-778.
- T.-Y. Lin, P. Dollár, R. Girshick, K. He, B. Hariharan, and S. Belongie, "Feature pyramid networks for object detection," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 2117-2125.
- Authors
- Saha, A., Maldonado, O. M., Russell, C., & Bowden, R.
- Affiliations
- University of Surrey (GB)
- Amazon (Germany) (DE)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2021
- Pages
- 9200-9206
- ISSN
- 1050-4729 (ISSN-L)
- arXiv
- 2110.00966
- OpenAlex
- W3203158837
- Cited by
- 142
- References
- 41
- Citations by year
- 2026: 8 · 2025: 24 · 2024: 60 · 2023: 43 · 2022: 7
- FWCI
- 58.55
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- Proceedings - IEEE International Conference on Robotics and Automation/Proceedings (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by-nc-sa)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Engineering › Aerospace Engineering › Robotics and Sensor-Based Localization
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Vision and Imaging
- Keywords
- transformer models, sequence-to-sequence translation, physical grounding, autonomous driving, nuScenes, Argoverse
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
