Real-Time Semantic Segmentation With Fast Attention
Open accessThe DOI matches its Crossref record; 23 of the 23 references it cites were found in a registry or an index.
Abstract
In deep CNN based models for semantic segmentation, high accuracy relies on rich spatial context (large receptive fields) and fine spatial details (high resolution), both of which incur high computational costs. In this letter, we propose a novel architecture that addresses both challenges and achieves state-of-the-art performance for semantic segmentation of high-resolution images and videos in real-time. The proposed architecture relies on our fast spatial attention, which is a simple yet efficient modification of the popular self-attention mechanism and captures the same rich spatial context at a small fraction of the computational cost, by changing the order of operations. Moreover, to efficiently process high-resolution input, we apply an additional spatial reduction to intermediate feature stages of the network with minimal loss in accuracy thanks to the use of the fast attention module to fuse features. We validate our method with a series of experiments, and show that results on multiple datasets demonstrate superior performance with better accuracy and speed compared to existing approaches for real-time semantic segmentation. On Cityscapes, our network achieves 74.4% mIoU at 72 FPS and 75.5% mIoU at 58 FPS on a single Titan X GPU, which is ~50% faster than the state-of-the-art while retaining the same accuracy.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
U-Net: Convolutional Networks for Biomedical Image Segmentation
Ronneberger, O., Fischer, P., & Brox, T. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24574-4_28 · Cited by 99,404
Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation
Chen, L.-C., Zhu, Y., Papandreou, G., Schroff, F., & Adam, H. (2018) · arXiv · doi:10.48550/arxiv.1802.02611 · Cited by 20,063 · cited 2×
“Recently, Deeplab-v3+ [39] integrates dilated convolution and spatial pyramid pooling into an encoder-decoder network to further boost the effectiveness.”
BackgroundEncoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation
Chen, L.-C., Zhu, Y., Papandreou, G., Schroff, F., & Adam, H. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01234-2_49 · Cited by 20,054
SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation
Badrinarayanan, V., Kendall, A., & Cipolla, R. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2644615 · Cited by 19,631
Zhao, H., Shi, J., Qi, X., Wang, X., & Jia, J. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.660 · Cited by 16,438
The Cityscapes Dataset for Semantic Urban Scene Understanding
Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., & Schiele, B. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.350 · Cited by 12,433 · cited 8×
“To further analyze our cosine-similarity based fast attention, we also train without the L2-normalization for both Query and Key features (denoted as “FA w/o L2-norm”) and achieve 74.1 % mIoU on the Cityscapes val , which is lower than 75.0 % mIoU of our full model.”
BackgroundMethodInfluentialWang, X., Girshick, R., Gupta, A., & He, K. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00813 · Cited by 11,596
Vision meets robotics: The KITTI dataset
Geiger, A., Lenz, P., Stiller, C., & Urtasun, R. (2013) · The International Journal of Robotics Research · doi:10.1177/0278364913491297 · Cited by 9,913 · cited 1×
“S EMANTIC segmentation is a fundamental task in robotic sensing and computer vision, aiming to predict dense semantic labels for given images [1]–[8].”
BackgroundMulti-scale context aggregation by dilated convolutions
Yu, F., & Koltun, V. (2015) · arXiv · doi:10.48550/arxiv.1511.07122 · Cited by 7,738 · cited 1×
“To this end, dilated convo-lutions [31], [32] are proposed as an effective tool to enlarge receptive field without shrinking spatial resolution [28], [33].”
Method
Cited by
Wang, L., Li, R., Zhang, C., Fang, S., Duan, C., Meng, X., & Atkinson, P. M. (2022) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2022.06.008 · Cited by 1,349
Review the state-of-the-art technologies of semantic segmentation based on deep learning
Mo, Y., Wu, Y., Yang, X., Liu, F., & Liao, Y. (2022) · Neurocomputing · doi:10.1016/j.neucom.2022.01.005 · Cited by 632
Deep Dual-Resolution Networks for Real-Time and Accurate Semantic Segmentation of Traffic Scenes
Pan, H., Hong, Y., Sun, W., & Jia, Y. (2023) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2022.3228042 · Cited by 495
Li, R., Zheng, S., Zhang, C., Duan, C., Wang, L., & Atkinson, P. M. (2021) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2021.09.005 · Cited by 443
Tripathy, K. P., & Mishra, A. K. (2024) · Journal of Hydrology · doi:10.1016/j.jhydrol.2023.130458 · Cited by 336
Wang, L., Li, R., Wang, D., Duan, C., Wang, T., & Meng, X. (2021) · Remote Sensing · doi:10.3390/rs13163065 · Cited by 281
Deep Dual-resolution Networks for Real-time and Accurate Semantic Segmentation of Road Scenes
Hong, Y., Pan, H., Sun, W., & Jia, Y. (2021) · arXiv · doi:10.48550/arxiv.2101.06085 · Cited by 261
A Survey on Deep Learning-based Architectures for Semantic Segmentation on 2D Images
Ulku, I., & Akagündüz, E. (2022) · Applied Artificial Intelligence · doi:10.1080/08839514.2022.2032924 · Cited by 237
“Some other strategies have also been proposed for the retrieval of rich contextual information in real-time applications including attention mechanisms (Ding et al. 2021; Hu et al. 2020), depth-wise separable convolutions (Chollet 2017; Howard et al.”
BackgroundAdaShare: Learning What To Share For Efficient Deep Multi-Task Learning
Sun, X., Panda, R., Feris, R., & Saenko, K. (2019) · arXiv · doi:10.48550/arxiv.1911.12423 · Cited by 169
“Following [47], we sampled 5 representative tasks out of 26 tasks for our experiments, namely Semantic Segmentation, Surface Normal Prediction, Depth Prediction, Keypoint Detection and Edge Detection.”
MethodInfluentialSelf-citation
Metadata
- Authors
- Hu, P., Perazzi, F., Heilbron, F. C., Wang, O., Lin, Z., Saenko, K., & Sclaroff, S.
- Venue
- IEEE Robotics and Automation Letters
- Published
- 2021-01
- Volume
- 6
- Issue
- 1
- Pages
- 263-270
- ISSN
- 2377-3766 (ISSN-L)
- 2377-3774
- OpenAlex
- W3108450508
- Cited by
- 159
- References
- 23
- FWCI
- 8.37
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.