Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 104 of the 104 references it cites were found in a registry or an index; 8 of the 23 sources its claims cite were checked for support.

Vision Transformers: From Semantic Segmentation to Dense Prediction

Open access

Abstract

The emergence of vision transformers (ViTs) in image classification has shifted the methodologies for visual representation learning. In particular, ViTs learn visual representation at full receptive field per layer across all the image patches, in comparison to the increasing receptive fields of CNNs across layers and other alternatives (e.g., large kernels and atrous convolution). In this work, for the first time we explore the global context learning potentials of ViTs for dense visual prediction (e.g., semantic segmentation). Our motivation is that through learning global context at full receptive field layer by layer, ViTs may capture stronger long-range dependency information, critical for dense prediction tasks. We first demonstrate that encoding an image as a sequence of patches, a vanilla ViT without local convolution and resolution reduction can yield stronger visual representation for semantic segmentation. For example, our model, termed as SEgmentation TRansformer (SETR), excels on ADE20K (50.28% mIoU, the first position in the test leaderboard on the day of submission) and performs competitively on Cityscapes. However, the basic ViT architecture falls short in broader dense prediction applications, such as object detection and instance segmentation, due to its lack of a pyramidal structure, high computational demand, and insufficient local context. For tackling general dense visual prediction tasks in a cost-effective manner, we further formulate a family of Hier…

Overview

CoChat Summary

TL;DR

This paper extends Vision Transformers from image classification to dense prediction, first introducing SETR for semantic segmentation and then a Hierarchical Local-Global (HLG) Transformer backbone for general dense tasks. It reports state-of-the-art or competitive results on ADE20K, Cityscapes, ImageNet, and COCO while noting high computational and pretraining costs.

Key findings

  • SETR replaces the FCN with a ViT encoder using global self-attention at every layer; the paper reports new state of the art on ADE20K (50.28%) and Pascal Context (55.83%), competitive Cityscapes results, and first place on the ADE20K test server leaderboard at submission day.
  • On ADE20K, SETR-MLA achieves 48.64% mIoU single-scale and 50.3% multi-scale; with BEiT pretraining, SETR-MLA-BEiT reaches 53.1% mIoU multi-scale. On Cityscapes validation, SETR-PUP reaches 79.34% mIoU at 80k; on Cityscapes test, SETR-PUP reaches 81.08% and 81.64% with coarse data.
  • HLG Transformers achieve ImageNet-1K Top-1 of 81.1% (HLG-Tiny), 82.3% (HLG-Small), 83.6% (HLG-Medium), and 84.1% (HLG-Large). On COCO, HLG-Large reaches 47.7 AP with RetinaNet detection and 49.1 APb / 44.2 APm with Mask R-CNN instance segmentation.
  • Ablations show pretraining is critical: randomly initialized SETR-PUP gives only 42.27% mIoU on Cityscapes. Hierarchical local attention brings a 0.64% increment, and step-by-step HLG components improve ImageNet Top-1 from 79.2% to 82.3%.

Methods

  • Design: SETR treats an image as a sequence of patches, uses a standard Transformer encoder with global self-attention, and introduces three decoders—Naive upsampling, Progressive UPsampling (PUP), and Multi-Level feature Aggregation (MLA). HLG adds a four-stage pyramidal backbone with local window attention, dilated local attention, global attention, parameter/computation sharing, and DWMLP.
  • Data: semantic segmentation on Cityscapes (5000 finely annotated images, 19,998 coarse images, 19 classes), ADE20K (20,210/2,000/3,352 images, 150 classes), and Pascal Context; classification on ImageNet-1K (1.28M training, 50K validation, 1,000 classes) and ImageNet-1K V2; detection and instance segmentation on COCO 2017 (118K training, 5K validation).
  • Analysis: mean IoU and pixel accuracy for segmentation, Top-1 accuracy for classification, AP/APb/APm for detection and instance segmentation; comparisons against FCN, Semantic FPN, ViT/DeiT/BEiT, Swin, PVT, and others, plus ablations on decoders, pretraining, auxiliary loss, window embedding, and local attention.

Limitations

  • The paper states that single-stage Transformers suffer from high computational and pretraining costs, and that Vision Transformers incur higher FLOPs, time, and GPU memory usage than convolutional models; randomly initialized SETR-PUP gives only 42.27% mIoU, showing pretraining is critical.
  • The paper notes that all its models use output stride 16 due to GPU memory constraints, whereas the dilated FCN baseline uses output stride 8.
  • No dedicated limitations section is present; evident from reported results, HLG-Large has lower FPS (221) and higher memory (31.6GB) than Swin-B (278 FPS, 25.2GB) in Table 8.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1007/s11263-024-02173-w

Cites

104 of 104 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

7 Valid · 3 Not valid · 9 Not checked · 98 Own results and interpretations

72 more claims

Create a free account to see all 117 claims and to verify the 2 references still open.

See all claims — free account
  • It may state two separate claims as one. 28
  • The quote only partly states this claim. 2
  • Probably not a substantial claim. 14

Create a free account to see these claims.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

5 of 5

Citations

Cites

Show all 89 references

Research works printed but not linked: 6

  1. Bao, H., Dong, L., Piao, S., & Wei, F. (2022) Beit: Bert pre-training of image transformers. In International conference on learning representations.
  2. Chen, L. -C., Papandreou, G., Kokkinos, I., Murphy, K., & Yuille, A. L. (2015). Semantic image segmentation with deep convolutional nets and fully connected CRFs. In International conference on learning representations.
  3. Larsson, G., Maire, M., & Shakhnarovich, G. (2016) Fractalnet: Ultra-deep neural networks without residuals. In International conference on learning representations.
  4. Recht, B., Roelofs, R., Schmidt, L., & Shankar, V. (2019). Do imagenet classifiers generalize to imagenet? In International conference on machine learning, (pp. 5389–5400). PMLR.
  5. Veličković, P., Cucurull, G., Casanova, A., Romero, A., Lio, P., & Bengio, Y. (2018). Graph attention networks. In International conference on learning representations.
  6. Yang, C., Qiao, S., Yu, Q., Yuan, X., Zhu, Y., Yuille, A., Adam, H., & Chen, L.-C. (2022) Moat: Alternating mobile convolution and attention brings strong vision models. In International conference on learning representations.

Cited by

See all 23

Metadata

Authors
Zhang, L., Lu, J., Zheng, S., Zhao, X., Zhu, X., Fu, Y., Xiang, T., Feng, J., & Torr, P. H. S.
Affiliations
  • Fudan University (CN)
  • University of Surrey (GB)
  • University of Oxford (GB)
Corresponding author
Zhang, Li
Venue
International Journal of Computer Vision
Published
2024-07-16
Volume
132
Issue
12
Pages
6142-6162
ISSN
  • 0920-5691 (ISSN-L)
  • 1573-1405
OpenAlex
W4400677548
Cited by
23
References
89
Citations by year
2026: 8 · 2025: 11 · 2024: 4
FWCI
3.20
Type
journal article
Language
en
Open access
Licence
Locations
  • International Journal of Computer Vision (publishedVersion)
  • arXiv (Cornell University) (publishedVersion)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Visual Attention and Saliency Detection
Keywords
vision transformer, semantic segmentation, long-range dependencies, object detection, instance segmentation, image classification, visual representation learning

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-30, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-30 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Vision Transformers: From Semantic Segmentation to Dense Prediction. https://app.cochat.ai/papers/10.1007/s11263-024-02173-w
The paper
Zhang, L., Lu, J., Zheng, S., Zhao, X., Zhu, X., Fu, Y., Xiang, T., Feng, J., & Torr, P. H. S. (2024). Vision Transformers: From Semantic Segmentation to Dense Prediction. International Journal of Computer Vision. https://doi.org/10.1007/s11263-024-02173-w

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.