CoChat
Sign Up
Cited by 2,373Advanced Neural Network ApplicationsDomain Adaptation and Few-Shot LearningAdvanced Vision and Imaging

Vision Transformers for Dense Prediction

Open access
Ranftl, Rene, Bochkovskiy, Alexey, Koltun, Vladlen·2021·2021 IEEE/CVF International Conference on Computer Vision (ICCV)
DOI 10.1109/iccv48922.2021.01196
Open this paper in CoChat — free accountOpen PDF

Abstract

We introduce dense prediction transformers, an architecture that leverages vision transformers in place of convolutional networks as a backbone for dense prediction tasks. We assemble tokens from various stages of the vision transformer into image-like representations at various resolutions and progressively combine them into full-resolution predictions using a convolutional decoder. The transformer backbone processes representations at a constant and relatively high resolution and has a global receptive field at every stage. These properties allow the dense prediction transformer to provide finer-grained and more globally coherent predictions when compared to fully-convolutional networks. Our experiments show that this architecture yields substantial improvements on dense prediction tasks, especially when a large amount of training data is available. For monocular depth estimation, we observe an improvement of up to 28% in relative performance when compared to a state-of-the-art fully-convolutional network. When applied to semantic segmentation, dense prediction transformers set a new state of the art on ADE20K with 49.02% mIoU. We further show that the architecture can be fine-tuned on smaller datasets such as NYUv2, KITTI, and Pascal Context where it also sets the new state of the art. Our models are available at https://github.com/intel-isl/DPT.

Integrity signals

Registry

Crossref · 10.1109/iccv48922.2021.01196

Cites

68 of 68 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

8 of 27 cited sources checked · 5 supported · 0 flagged · accuracy not yet measured

  • Premisecitation

    DPT uses a vision transformer (ViT) as a backbone architecture.

    “Specifically, we use the recently proposed vision transformer (ViT) [11] as a backbone architecture.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [11] 11. Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020. source
  • Premisecitation

    Virtually all existing architectures for dense prediction are based on convolutional networks.

    “Virtually all existing architectures for dense prediction are based on convolutional networks [6,31,34,42,49,50,53].”
    Quote in the paperSubstantial claim3 of 7 sources checked
    • cites [6] 6. Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L. Yuille. DeepLab: Semantic im- age segmentation with deep convolutional nets, atrous con- volution, and fully connected crfs. TPAMI, 40(4):834-848, 2018. source

      The Introduction (p. 1) says convolutional models such as PixelCNN had received much more attention for pixel-by-pixel image generation. It does not establish that virtually all architectures across the broader category of dense prediction are convolutional; it also discusses recurrent models and proposes a self-attention alternative (Introduction, p. 2). (LLM, conf=0.78)

      Experimental — accuracy not yet measured
    • cites [31] 31. Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U- Net: Convolutional networks for biomedical image segmen- tation. In MICCAI, 2015. source
    • cites [34] 34. Evan Shelhamer, Jonathan Long, and Trevor Darrell. Fully convolutional networks for semantic segmentation. CVPR, 2015. source
    • cites [42] 42. Jingdong Wang, Ke Sun, Tianheng Cheng, Borui Jiang, Chaorui Deng, Yang Zhao, Dong Liu, Yadong Mu, Mingkui Tan, Xinggang Wang, Wenyu Liu, and Bin Xiao. Deep high-resolution representation learning for visual recogni- tion. TPAMI, 2020. source

      Section 2.1 (p. 8) describes transformer language-model architectures, not architectures for dense prediction. Searches for dense prediction, convolutional networks, and pixel-wise segmentation found no passage addressing whether dense-prediction architectures are predominantly convolutional. (LLM, conf=0.78)

      Experimental — accuracy not yet measured
    • cites [49] 49. Fisher Yu and Vladlen Koltun. Multi-scale context aggrega- tion by dilated convolutions. In ICLR, 2016.

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [50] 50. Yuhui Yuan, Xilin Chen, and Jingdong Wang. Object- contextual representations for semantic segmentation. In ECCV, 2020. source
    • cites [53] 53. Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In CVPR, 2017. source
  • Findingreasoning

    The biggest gains in performance for zero-shot transfer were achieved for datasets that feature dense, high-resolution evaluations.

    “We notice that the biggest gains in performance for zero-shot transfer were achieved for datasets that feature dense, high-resolution evaluations [15,55,59].”
    Quote in the paperNot a substantial claim1 of 3 sources checked
    • cites [15] 15. Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? The KITTI vision benchmark suite. In CVPR, 2012. source
    • cites [55] 55. D. J. Butler, J. Wulff, G. B. Stanley, and M. J. Black. A naturalistic open source movie for optical flow evaluation. In ECCV, 2012. source
    • cites [59] 59. Thomas Schöps, Johannes L. Schönberger, Silvano Galliani, Torsten Sattler, Konrad Schindler, Marc Pollefeys, and An- dreas Geiger. A multi-view stereo benchmark with high- resolution images and multi-camera videos. In CVPR, 2017. source
  • Premisecitation

    ResNext101-WSL was pretrained on a billion-scale corpus of weakly supervised data in addition to ImageNet pretraining.

    “Note that ResNext101-WSL was pretrained on a billion-scale corpus of weakly supervised data [25] in addition to ImageNet pretraining.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [25] 25. Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens van der Maaten. Exploring the limits of weakly supervised pretraining. In ECCV, 2018. source

      Section 2.3 (p. 4) identifies the models as ResNeXt-101, and Section A.5 (p. 23) says they were pretrained on billions of Instagram images using hashtags as labels. But it describes subsequent training on ImageNet as fine-tuning, not additional ImageNet pretraining; Section 3.2 (p. 12) also distinguishes Instagram-pretrained models from ImageNet-pretrained baselines. (LLM, conf=0.88)

      Experimental — accuracy not yet measured
  • Findingown data

    Projection provides slightly better performance on average than ignoring the readout token.

    “While ignoring the token yields good performance, projection provides slightly better performance on average.”
    Quote in the paperNot a substantial claim
  • Classified as probably not a substantial claim. 11
  • May combine two claims. 13
  • The quote only partly states this claim. 1

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    A direct application of token-based transformer architectures that have been successful in NLP can yield competitive performance on image classification.

    “In particular, it has recently been demonstrated that a direct application of token-based transformer architectures that have been successful in NLP can yield competitive performance on image classification [11].”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [11] 11. Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020. source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Vision transformers need to be paired with a sufficient amount of training data to realize their potential.

    “A key insight of this work was that, like transformer models in NLP, vision transformers need to be paired with a sufficient amount of training data to realize their potential.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [11] 11. Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020. source

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Findingnone given

    The ViT-Hybrid architecture extracts features at 1/16 the input resolution, which is twice as high as the lowest-resolution features that are commonly used with convolutional backbones.

    “Similarly, the ViT-Hybrid architecture extracts features at 1 16 the input resolution, which is twice as high as the lowest-resolution features that are commonly used with convolutional backbones.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingnone given

    MIX 6 contains about 1.4 million images and is the largest training set for monocular depth estimation that has ever been compiled.

    “It contains about 1.4 million images and is, to the best of our knowledge, the largest training set for monocular depth estimation that has ever been compiled.”
    Quote in the paperNot a substantial claim

  • The quote only partly states this claim.

    Findingown data

    DPT sets a new state of the art on ADE20K with 49.02% mIoU.

    “When applied to semantic segmentation, dense vision transformers set a new state of the art on ADE20K with 49.02% mIoU.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    DPT tends to produce cleaner and finer-grained delineations of object boundaries.

    “We observe that the DPT tends to produce cleaner and finer-grained delineations of object boundaries and that the predictions are also in some cases less cluttered.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    DPT can better reconstruct fine details while also improving global coherence in areas that are challenging for the convolutional architecture.

    “DPT can better reconstruct fine details while also improving global coherence in areas that are challenging for the convolutional architecture (for example, large homogeneous regions or relative depth arrangement across the image).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The average relative improvement over the best published architecture, MiDaS, is more than 23% for DPT-Hybrid and 28% for DPT-Large.

    “The average relative improvement over the best published architecture, MiDaS, is more than 23% for DPT-Hybrid and 28% for DPT-Large.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    DPT-Hybrid achieves this with a comparable network capacity, while DPT-Large is about 3 times larger than MiDaS.

    “DPT-Hybrid achieves this with a comparable network capacity (Table 9), while DPT-Large is about 3 times larger than MiDaS.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Attention is spatially more localized close to the reference token in shallow layers, whereas deeper layers frequently attend across the whole image.

    “We observe the tendency that attention is spatially more localized close to the reference token in shallow layers (leftmost columns), whereas deeper layers (rightmost columns) frequently attend across the whole image.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingnone given

    ViT-Hybrid and ViT-Large improve performance over ResNext101-WSL even though they have been pretrained on significantly less data.

    “ViT-Base has comparable performance to ResNext101-WSL, while ViT-Hybrid and ViT-Large improve performance even though they have been pretrained on significantly less data.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    There is a general trend of improvement in per-class IoU in comparison to the baseline on ADE20K.

    “While we observe a general trend of an improvement in per-class IoU in comparison to the baseline [51], we do not observe a strong pattern across classes.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [51] 51. Hang Zhang, Chongruo Wu, Zhongyue Zhang, Yi Zhu, Zhi Zhang, Haibin Lin, Yue Sun, Tong He, Jonas Muller, R. Manmatha, Mu Li, and Alexander Smola. ResNeSt: Split- attention networks. arXiv preprint arXiv:2004.08955, 2020.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    There is no strong pattern across classes in per-class IoU improvement.

    “While we observe a general trend of an improvement in per-class IoU in comparison to the baseline [51], we do not observe a strong pattern across classes.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [51] 51. Hang Zhang, Chongruo Wu, Zhongyue Zhang, Yi Zhu, Zhi Zhang, Haibin Lin, Yue Sun, Tong He, Jonas Muller, R. Manmatha, Mu Li, and Alexander Smola. ResNeSt: Split- attention networks. arXiv preprint arXiv:2004.08955, 2020.

  • May combine two claims.

    Findingown data

    DPT predictions show more details and better global depth arrangement compared to the fully-convolutional baseline.

    “We observe more details and also better global depth arrangement in DPT predictions when compared to the fullyconvolutional baseline.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    The transformer backbone processes representations at a constant and relatively high resolution and has a global receptive field at every stage.

    “The transformer backbone processes representations at a constant and relatively high resolution and has a global receptive field at every stage.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    As long as the image size is divisible by p, the embedding procedure can be applied and will produce a varying number of image tokens N_p.

    “As long as the image size is divisible by p, the embedding procedure can be applied and will produce a varying number of image tokens N p .”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationnone given

    Unlike fully-convolutional networks, the vision transformer backbone foregoes explicit downsampling operations after an initial image embedding and maintains a representation with constant dimensionality throughout all processing stages.

    “Unlike fully-convolutional networks, the vision transformer backbone foregoes explicit downsampling operations after an initial image embedding has been computed and maintains a representation with constant dimensionality throughout all processing stages.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    Monocular depth estimation is an ideal task to test the capabilities of DPT because transformers realize their full potential only when an abundance of training data is available.

    “Since transformers are known to realize their full potential only when an abundance of training data is available, monocular depth estimation is an ideal task to test the capabilities of DPT.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Premisenone given

    Convolutional networks progressively increase their receptive field as features pass through consecutive convolution and downsampling layers.

    “This is in stark contrast to convolutional networks, which progressively increase their receptive field as features pass through consecutive convolution and downsampling layers.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationnone given

    The slightly worse performance of DPT-Large on ADE20K is likely due to the significantly smaller dataset compared to previous experiments.

    “DPT-Large performs slightly worse, likely because of the significantly smaller dataset compared to our previous experiments.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    DeIT trains the ViT architecture with a more data-efficient pretraining procedure.

    “DeIT trains the ViT architecture with a more data-efficient pretraining procedure.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [38] 38. Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through at- tention. arXiv preprint arXiv:2012.12877, 2020. source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    DeIT-Base-Dist introduces an additional distillation token.

    “Note that the DeIT-Base architecture is identical to ViT-Base, while DeIT-Base-Dist introduces an additional distillation token, which we ignore in the Reassemble operation.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [38] 38. Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through at- tention. arXiv preprint arXiv:2012.12877, 2020. source

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    DPT unfolds its full potential when trained on large-scale datasets.

    “Similar to prior work on transformers, DPT unfolds its full potential when trained on large-scale datasets.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    Results for DPT and MiDaS are computed at the same input resolution of 384 pixels.

    “Note that results for DPT and Mi-DaS are computed at the same input resolution (384 pixels).”
    Quote in the paperNot a substantial claim
24 of 24

  • The paper relies on this without support.

    The choice of backbone architecture has a large influence on the capabilities of the overall model, because information lost in the encoder cannot be recovered in the decoder.

    “However, it is widely recognized that the choice of backbone architecture has a large influence on the capabilities of the overall model, as any information that is lost in the encoder is impossible to recover in the decoder.”

  • The paper relies on this without support.

    Feature resolution and granularity are critical for dense prediction, where the architecture should ideally resolve features at or close to the input resolution.

    “While feature resolution and granularity may not matter for some tasks, such as image classification, they are critical for dense prediction, where the architecture should ideally be able to resolve features at or close to the resolution of the input image.”

  • The paper relies on this without support.

    Convolutions are linear operators with a limited receptive field.

    “Convolutions, by definition, are linear operators that have a limited receptive field.”

  • The paper relies on this without support.

    Downsampling intermediate representations is necessary to keep memory consumption at levels feasible with existing computer architectures.

    “Downsampling the intermediate representations is necessary to keep memory consumption at levels that are feasible with existing computer architectures.”

  • The paper relies on this without support.

    The transformer backbone maintains a constant feature resolution.

    “Since the transformer backbone maintains a constant feature resolution, it is not clear at which points in the backbone features should be tapped.”

  • The paper relies on this without support.

    Performance of fully-convolutional architectures suffers heavily when performing inference at an input resolution significantly different from the training resolution.

    “Performance thus suffers heavily when performing inference at an input resolution that is significantly different from the training resolution.”

  • The paper relies on this without support.

    Transformer encoders have a global receptive field in every layer.

    “Transformer encoders, on the other hand, have a global receptive field in every layer.”

  • The paper relies on this without support.

    The reduced meta-dataset datasets provide high-quality ground truth.

    “We choose these datasets since they provide high-quality ground truth.”

  • The paper relies on this without support.

    Convolutional architectures offer natural points of interest for passing features from the encoder to the decoder, namely before or after downsampling of the representation.

    “Convolutional architectures offer natural points of interest for passing features from the encoder to the decoder, namely before or after downsampling of the representation.”
9 of 9

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The Transformer framework does not involve dimensionality reduction, thus keeping all attention layers traceable to the original input features, making the models interpretable.

    “However, the Transformer framework does not involve dimensionality reduction10,12, thus keeping all attention layer traceable to the original input features13, thus making the models interpretable.”

    From Transformer for one stop interpretable cell type annotation (2023) · cites this paper as [10]

  • The change of dimensions and non-linear aggregation of features throughout the autoencoder's deep processing stages leads to untraceable and uninterpretable latent space and loss of information and feature resolution.

    “For example, the change of dimensions and non-linear aggregation of features throughout the autoencoder’s deep processing stages leads to untraceable and uninterpretable latent space and loss of information and feature resolution9,10.”

    From Transformer for one stop interpretable cell type annotation (2023) · cites this paper as [10]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • KITTI absolute relative error is worse than a listed baseline

    Minor

    The claim of matching or improving every metric on both datasets conflicts with the KITTI absolute relative error: DPT-Hybrid reports 0.062, while BTS reports a better 0.059.

1 of 1

Cites

Flagged first, then by citations

20 of the 88 references couldn’t be matched to a known paper.

Research works printed but not linked: 3

  1. yuan, 2020, Objectcontextual representations for semantic segmentation, ECCV
  2. godard, 2017, Unsupervised monocular depth estimation with leftright consistency, CVPR
  3. casser, 2019, Unsupervised learning of depth and ego-motion: A structured approach, AAAI

Cited by

Sorted by citations

Related

Authors
Ranftl, R., Bochkovskiy, A., & Koltun, V.
Affiliations
  • Intel Labs
  • Intel (United States) (US)
ORCID
Venue
2021 IEEE/CVF International Conference on Computer Vision (ICCV)
Publisher
IEEE
Published
2021-10
Pages
12159-12168
DOI
10.1109/iccv48922.2021.01196
OpenAlex
W3136635488
Cited by
2,373
References
88
Citations by year
2026: 480 · 2025: 607 · 2024: 586 · 2023: 431 · 2022: 225 · 2021: 44
FWCI
104.94
Type
conference paper
Language
en
Open access
Licence
Locations
  • IEEE/CVF International Conference on Computer Vision (ICCV) (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
  • TIB Data Manager
  • IEEE/CVF International Conference on Computer Vision (ICCV)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Vision and Imaging
Keywords
vision transformer, dense prediction, monocular depth estimation, semantic segmentation, convolutional decoder, global receptive field, Pascal Context, NYUv2, KITTI, ADE20K

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-11.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-11.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review