The DOI matches its Crossref record; 104 of the 104 references it cites were found in a registry or an index; 8 of the 23 sources its claims cite were checked for support.
Vision Transformers: From Semantic Segmentation to Dense Prediction
Open accessAbstract
The emergence of vision transformers (ViTs) in image classification has shifted the methodologies for visual representation learning. In particular, ViTs learn visual representation at full receptive field per layer across all the image patches, in comparison to the increasing receptive fields of CNNs across layers and other alternatives (e.g., large kernels and atrous convolution). In this work, for the first time we explore the global context learning potentials of ViTs for dense visual prediction (e.g., semantic segmentation). Our motivation is that through learning global context at full receptive field layer by layer, ViTs may capture stronger long-range dependency information, critical for dense prediction tasks. We first demonstrate that encoding an image as a sequence of patches, a vanilla ViT without local convolution and resolution reduction can yield stronger visual representation for semantic segmentation. For example, our model, termed as SEgmentation TRansformer (SETR), excels on ADE20K (50.28% mIoU, the first position in the test leaderboard on the day of submission) and performs competitively on Cityscapes. However, the basic ViT architecture falls short in broader dense prediction applications, such as object detection and instance segmentation, due to its lack of a pyramidal structure, high computational demand, and insufficient local context. For tackling general dense visual prediction tasks in a cost-effective manner, we further formulate a family of Hier…
Overview
CoChat Summary
TL;DR
This paper extends Vision Transformers from image classification to dense prediction, first introducing SETR for semantic segmentation and then a Hierarchical Local-Global (HLG) Transformer backbone for general dense tasks. It reports state-of-the-art or competitive results on ADE20K, Cityscapes, ImageNet, and COCO while noting high computational and pretraining costs.
Key findings
- SETR replaces the FCN with a ViT encoder using global self-attention at every layer; the paper reports new state of the art on ADE20K (50.28%) and Pascal Context (55.83%), competitive Cityscapes results, and first place on the ADE20K test server leaderboard at submission day.
- On ADE20K, SETR-MLA achieves 48.64% mIoU single-scale and 50.3% multi-scale; with BEiT pretraining, SETR-MLA-BEiT reaches 53.1% mIoU multi-scale. On Cityscapes validation, SETR-PUP reaches 79.34% mIoU at 80k; on Cityscapes test, SETR-PUP reaches 81.08% and 81.64% with coarse data.
- HLG Transformers achieve ImageNet-1K Top-1 of 81.1% (HLG-Tiny), 82.3% (HLG-Small), 83.6% (HLG-Medium), and 84.1% (HLG-Large). On COCO, HLG-Large reaches 47.7 AP with RetinaNet detection and 49.1 APb / 44.2 APm with Mask R-CNN instance segmentation.
- Ablations show pretraining is critical: randomly initialized SETR-PUP gives only 42.27% mIoU on Cityscapes. Hierarchical local attention brings a 0.64% increment, and step-by-step HLG components improve ImageNet Top-1 from 79.2% to 82.3%.
Methods
- Design: SETR treats an image as a sequence of patches, uses a standard Transformer encoder with global self-attention, and introduces three decoders—Naive upsampling, Progressive UPsampling (PUP), and Multi-Level feature Aggregation (MLA). HLG adds a four-stage pyramidal backbone with local window attention, dilated local attention, global attention, parameter/computation sharing, and DWMLP.
- Data: semantic segmentation on Cityscapes (5000 finely annotated images, 19,998 coarse images, 19 classes), ADE20K (20,210/2,000/3,352 images, 150 classes), and Pascal Context; classification on ImageNet-1K (1.28M training, 50K validation, 1,000 classes) and ImageNet-1K V2; detection and instance segmentation on COCO 2017 (118K training, 5K validation).
- Analysis: mean IoU and pixel accuracy for segmentation, Top-1 accuracy for classification, AP/APb/APm for detection and instance segmentation; comparisons against FCN, Semantic FPN, ViT/DeiT/BEiT, Swin, PVT, and others, plus ablations on decoders, pretraining, auxiliary loss, window embedding, and local attention.
Limitations
- The paper states that single-stage Transformers suffer from high computational and pretraining costs, and that Vision Transformers incur higher FLOPs, time, and GPU memory usage than convolutional models; randomly initialized SETR-PUP gives only 42.27% mIoU, showing pretraining is critical.
- The paper notes that all its models use output stride 16 due to GPU memory constraints, whereas the dilated FCN baseline uses output stride 8.
- No dedicated limitations section is present; evident from reported results, HLG-Large has lower FPS (221) and higher memory (31.6GB) than Swin-B (278 FPS, 25.2GB) in Table 8.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
7 Valid · 3 Not valid · 9 Not checked · 98 Own results and interpretations
Every source the paper cites for this claim was checked and supports it.
In the paper
“Even with DeiT Touvron et al. (2022) to reduce computation costs, the results of SETR paired with DeiT show a significant decline.”
What the sources say
- Supportedcites Touvron et al. (2022) source
The citation is attached to DeiT, identifying the method used in the citing authors’ SETR experiment. The source discusses DeiT in Related Work (p. 4) and reports a DeiT-S segmentation comparison in Section 4.4.3 (p. 19). The reported decline in SETR results is the citing authors’ own observation, not a finding attributed to this source. (LLM, conf=0.84)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Nonetheless, none of these can fully eliminate the limitation of FCN architecture in receptive field.”
What the sources say
- Not checked — no full text of the sourcecites Peng et al. (2017) source
- Partly supportedcites Chen et al. (2017) sourceExperimental — accuracy not yet measured
Section 3.1 (p. 5) describes a trade-off between a small field of view for accurate localization and a large one for context, which supports the idea that enlarging it does not solve every problem. But the paper says atrous convolution can arbitrarily enlarge a filter’s field of view; it does not conclude that none of the listed remedies can fully eliminate an FCN receptive-field limitation, and it does not assess the attention methods in the claim. (LLM, conf=0.76)
- Not checked — no full text of the sourcecites Zhao et al. (2017) source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“However, employing superior pre-training techniques on 'TBase' such as DeiT Touvron et al. (2022) yields even better performance for these smaller Vision Transformer models on Cityscapes compared to their larger counterparts.”
What the sources say
- Partly supportedcites Touvron et al. (2022) sourceExperimental — accuracy not yet measured
Section 4.4.2 (p. 18) reports that the paper's ViT training recipe improves the FLOPs–accuracy trade-off in semantic segmentation, but those experiments use ADE20k, not Cityscapes. Table 11 (p. 19) does not establish that a T-Base model outperforms larger ViTs; its reported ViT-L segmentation scores exceed those of ViT-B. (LLM, conf=0.86)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“their model still follows the conventional FCN design in that spatial resolution of feature maps is reduced progressively.”
What the sources say
- Supportedcites Wang et al. (2020) source
Section 3.2 (p. 7) describes a ResNet-based architecture with a strided stem and striding between stages, yielding feature maps at one-sixteenth of the input resolution. The authors remove the stride from the final stage, but the backbone still progressively reduces spatial resolution. (LLM, conf=0.89)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“and its performance is on par with the best results reported so far.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
72 more claims
Create a free account to see all 117 claims and to verify the 2 references still open.
- It may state two separate claims as one. 28
- The quote only partly states this claim. 2
- Probably not a substantial claim. 14
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ 4.1 Single-stage Transformer Setup · p. 9
“| SETR- PUP | 21K | T-Large | 318M | 1083G | 18 | 3582M | 78.39 | 79.34 |”
§ 4.1 Single-stage Transformer Setup · p. 9
“| SETR- PUP | 21K | T-Large | 426G | 70 | 48.58 | 1083G | 6 | 78.39 |”
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ Image classification details · p. 11
“The number of training iteration is set to 40K for Cityscapes.”
§ Image classification details · p. 11
“| SETR- PUP-HLG-Large (80k, MS) | 1K | HLG-Large | 100M | 791G | 82.9 |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ SETR variants · p. 10
“| SETR- MLA-BEiT (160k, SS) | 21K | ViT-Large | 311M | 318G | 53.0 |”
§ 4.1 Single-stage Transformer Setup · p. 9
“| SETR- MLA-BEiT | 21K | T-Large | 368G | 80 | 53.0 | 972G | 7 | 80.20 |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 4.3 Vision Transformer Pretraining · p. 12
“HLG-Small with 34.2M parameters achieves 82.3% accuracy”
§ 4.4 Results of Semantic Segmentation · p. 13
“| HLG-Small | 224 2 | 24.2M | 4.7G | 82.3 |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 4.1 Single-stage Transformer Setup · p. 9
“| SETR- MLA | 21K | T-Large | 311M | 972G | 20 | 3353M | 76.65 | 77.24 |”
§ 4.1 Single-stage Transformer Setup · p. 9
“| SETR- MLA | 21K | T-Large | 368G | 80 | 48.64 | 972G | 7 | 76.65 |”
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 13×
“Method #Params AP AP50 AP75 APS APM APL ResNet18 [4] 21.”
InfluentialVaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148 · cited 2×
“In this work, we alternatively rethink the semantic segmentation task from a different perspective.”
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · cited 3×
“S INCE the introduction of Vision Transformers (ViTs) to image classification [1], the landscape of visual representation learning has gradually shifted away from CNNs [2], [3], [4], [5], [6].”
U-Net: Convolutional Networks for Biomedical Image Segmentation
Ronneberger, O., Fischer, P., & Brox, T. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24574-4_28 · Cited by 99,404 · cited 1×
“Many recent efforts have been focused on addressing the limited receptive field/context modeling problem in FCN.”
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Densely Connected Convolutional Networks
Huang, G., Liu, Z., Van Der Maaten, L., & Weinberger, K. Q. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.243 · Cited by 48,566 · cited 2×
“Inspired by this phenomenal success, for the first time we expand the application of ViTs from image classification [1] to more challenging dense prediction ( e.g. , semantic segmentation).”
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562 · cited 3×
“AANet [41] mixes convolution and self-attention for backbone training.”
InfluentialAn Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.11929 · Cited by 47,472 · cited 8×
“Inspired by this phenomenal success, for the first time we expand the application of ViTs from image classification [1] to more challenging dense prediction (e.”
Influential
Research works printed but not linked: 6
- Bao, H., Dong, L., Piao, S., & Wei, F. (2022) Beit: Bert pre-training of image transformers. In International conference on learning representations.
- Chen, L. -C., Papandreou, G., Kokkinos, I., Murphy, K., & Yuille, A. L. (2015). Semantic image segmentation with deep convolutional nets and fully connected CRFs. In International conference on learning representations.
- Larsson, G., Maire, M., & Shakhnarovich, G. (2016) Fractalnet: Ultra-deep neural networks without residuals. In International conference on learning representations.
- Recht, B., Roelofs, R., Schmidt, L., & Shankar, V. (2019). Do imagenet classifiers generalize to imagenet? In International conference on machine learning, (pp. 5389–5400). PMLR.
- Veličković, P., Cucurull, G., Casanova, A., Romero, A., Lio, P., & Bengio, Y. (2018). Graph attention networks. In International conference on learning representations.
- Yang, C., Qiao, S., Yu, Q., Yuan, X., Zhu, Y., Yuille, A., Adam, H., & Chen, L.-C. (2022) Moat: Alternating mobile convolution and attention brings strong vision models. In International conference on learning representations.
Cited by
Hybrid Quantum–Classical Neural Networks for Efficient MNIST Binary Image Classification
Ranga, D., Prajapat, S., Akhtar, Z., Kumar, P., & Vasilakos, A. V. (2024) · Mathematics · doi:10.3390/math12233684 · Cited by 51
“ViT is computationally expensive and lacks specialization to smaller datasets [55,56].”
Ghribi, F., & Hamdaoui, F. (2025) · The Visual Computer · doi:10.1007/s00371-025-04122-1 · Cited by 25
SATR: A Structure-Affinity Attention-Based Transformer Encoder for Spine Segmentation
Xie, H., Huang, Z., Leung, F. H. F., et al. (2024) · IEEE International Symposium on Biomedical Imaging (ISBI) · doi:10.1109/isbi56570.2024.10635612 · Cited by 7
“This first pure transformer was applied directly to sequences of image patches and obtained comparable and even better re-sults in some tasks than CNNs, such as semantic segmentation [19, 20].”
MethodNoroozi, M., Arabi, H., & Karimian, A. (2024) · National and International Iranian Conference on Biomedical Engineering (ICBME) · doi:10.1109/icbme64381.2024.10895718 · Cited by 7
“• SETR [20] has a relatively lower dice coefficient of 0.9745 ± 2.215 e -05 but an improved accuracy value of 0.9649 ± 1.967 e -05 , making it meaningful for medical image segmentation, including handling different complexities related to cardiac anatomy.”
InfluentialKewenig, V. N., Lampinen, A. K., Nastase, S. A., Edwards, C., D'Elascombe, Q., Rechardt, A., Skipper, J. I., & Vigliocco, G. (2024) · Research Square · doi:10.21203/rs.3.rs-3913308/v1 · Cited by 6
Metadata
- Authors
- Zhang, L., Lu, J., Zheng, S., Zhao, X., Zhu, X., Fu, Y., Xiang, T., Feng, J., & Torr, P. H. S.
- Affiliations
- Fudan University (CN)
- University of Surrey (GB)
- University of Oxford (GB)
- Corresponding author
- Zhang, Li
- ORCID
- Venue
- International Journal of Computer Vision
- Published
- 2024-07-16
- Volume
- 132
- Issue
- 12
- Pages
- 6142-6162
- ISSN
- 0920-5691 (ISSN-L)
- 1573-1405
- OpenAlex
- W4400677548
- Cited by
- 23
- References
- 89
- Citations by year
- 2026: 8 · 2025: 11 · 2024: 4
- FWCI
- 3.20
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- www.springernature.com/gp/researchers/text-and-data-mining (tdm)
- www.springernature.com/gp/researchers/text-and-data-mining (vor)
- Locations
- International Journal of Computer Vision (publishedVersion)
- arXiv (Cornell University) (publishedVersion)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Visual Attention and Saliency Detection
- Keywords
- vision transformer, semantic segmentation, long-range dependencies, object detection, instance segmentation, image classification, visual representation learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-30, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-30 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Vision Transformers: From Semantic Segmentation to Dense Prediction. https://app.cochat.ai/papers/10.1007/s11263-024-02173-w
- The paper
- Zhang, L., Lu, J., Zheng, S., Zhao, X., Zhu, X., Fu, Y., Xiang, T., Feng, J., & Torr, P. H. S. (2024). Vision Transformers: From Semantic Segmentation to Dense Prediction. International Journal of Computer Vision. https://doi.org/10.1007/s11263-024-02173-w
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.