CoChat
Sign Up
PreprintCited by 1,088Domain Adaptation and Few-Shot LearningAdvanced Neural Network ApplicationsMultimodal Machine Learning Applications

DINOv2: Learning Robust Visual Features without Supervision

Open access
Oquab, Maxime, Darcet, Timothée, Moutakanni, Théo, Vo, Huy, Szafraniec, Marc, Khalidov, Vasil, Fernandez, Pierre, Haziza, Daniel +18 more·2023·arXiv
DOI 10.48550/arxiv.2304.07193arXiv 2304.07193
Open this paper in CoChat — free accountOpen PDF

Abstract

The recent breakthroughs in natural language processing for model pretraining on large quantities of data have opened the way for similar foundation models in computer vision. These models could greatly simplify the use of images in any system by producing all-purpose visual features, i.e., features that work across image distributions and tasks without finetuning. This work shows that existing pretraining methods, especially self-supervised methods, can produce such features if trained on enough curated data from diverse sources. We revisit existing approaches and combine different techniques to scale our pretraining in terms of data and model size. Most of the technical contributions aim at accelerating and stabilizing the training at scale. In terms of data, we propose an automatic pipeline to build a dedicated, diverse, and curated image dataset instead of uncurated data, as typically done in the self-supervised literature. In terms of models, we train a ViT model (Dosovitskiy et al., 2020) with 1B parameters and distill it into a series of smaller models that surpass the best available all-purpose features, OpenCLIP (Ilharco et al., 2021) on most of the benchmarks at image and pixel levels.

AI snapshot

Written from the paper’s full text.

TL;DR

DINOv2 shows that self-supervised pretraining on a curated 142M-image dataset can produce general-purpose visual features competitive with weakly supervised models. Its ViT-g/14 backbone and distilled smaller variants perform strongly on image and pixel tasks without task-specific fine-tuning.

Key findings

  • On ImageNet-1k linear probing, DINOv2 ViT-g/14 reaches 86.5% top-1, exceeding OpenCLIP ViT-G/14 (86.2%) and EVA-CLIP ViT-g/14 (86.4%), and improving over iBOT ViT-L/16 (82.3%) by 4.2 points; kNN is 83.5%, ReaL 89.6%, and V2 78.4%.
  • Dense and retrieval tasks also improve: ViT-g/14 with multiscale linear segmentation gets 53.0 mIoU on ADE20k, 81.0 on Cityscapes, and 86.2 on Pascal VOC; depth DPT RMSE is 0.279 on NYUd versus OpenCLIP’s 0.414; retrieval reaches 52.3 mAP on Oxford-Hard and 82.6 mAP on Paris-Hard.
  • Ablations show the KoLeo regularizer raises Oxford-M retrieval from 55.6 to 63.9 mAP, masked image modeling raises ADE-20k segmentation from 44.2 to 47.1 mIoU, and distillation from ViT-g improves ViT-L over scratch on all 12 benchmarks.
  • Curated LVD-142M beats uncurated data on most benchmarks (e.g., INet-1k 85.8 vs 83.3, iNat2018 82.3 vs 68.0, iNat2021 86.4 vs 76.4) and beats ImageNet-22k on all but ImageNet-1k.

Methods

  • Data: an automatic pipeline starts from 1.2B unique web images, applies deduplication, and retrieves images using self-supervised ViT-H/16 embeddings with cosine similarity, k-means, and Faiss, producing LVD-142M (142M images) in under two days on 20 nodes with 8 V100-32GB GPUs.
  • Training: discriminative self-supervised learning combines DINO image-level and iBOT patch-level objectives with Sinkhorn-Knopp centering, a KoLeo regularizer, separate projection heads, and a short 518×518 high-resolution phase; ViT-g/14 has 1.1B parameters, with efficiency from FlashAttention, sequence packing, stochastic depth, and FSDP, and smaller models distilled from ViT-g.
  • Evaluation: frozen features are assessed with linear probes and kNN on classification, segmentation, depth, retrieval, video, and fairness benchmarks, compared against self-supervised and weakly supervised baselines.

Limitations

  • The paper states fairness gaps: on Dollar Street, Africa performance is 25.7% below Europe and high-income households are 31.7% above low-income; harmful label associations were rare, but a more thorough bias evaluation may reveal flaws.
  • The paper states its carbon estimate covers only GPU electricity and excludes manufacturing and disposal; the whole project is estimated at 0.5k–1k tCO2eq, with one ViT-g pretraining at 3.7 tCO2eq.
  • The text makes evident that evaluation centers on frozen linear probes and curated benchmarks, and the data pipeline retrieves from curated datasets, so coverage may be shaped by those sources; some tasks where text supervision helps (e.g., SUN, Cars) still lag OpenCLIP.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2304.07193

Cites

139 of 139 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

12 of 40 cited sources checked · 6 supported · 0 flagged · accuracy not yet measured

  • Findingreasoning

    FSDP leads to approximately 50% reduction in communication costs compared to the float32 gradient all-reduce operation used in DDP.

    “This leads to approximately 50% reduction in communication costs compared to the float32 gradient all-reduce operation used in DistributedDataParallel (DDP), which is used in other self-supervised pretraining methods (Caron et al., 2021;Zhou et al., 2022a).”
    Quote in the paperNot a substantial claim1 of 2 sources checked
    • cites Caron et al., 2021 source

      The compute-requirements section (p. 9) reports DINO training time and GPU memory, not communication costs. Searches for FSDP, DistributedDataParallel, float32 gradient all-reduce, and the claimed 50% reduction found no passage in our copy establishing either the comparison or DDP's use in this method. (LLM, conf=0.75)

      Experimental — accuracy not yet measured
    • cites Zhou et al., 2022a

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Premisereasoning

    Uncurated datasets typically lead to a significant drop in the quality of features.

    “Some efforts on scaling these approaches beyond ImageNet-1k have been attempted (Caron et al., 2019;Goyal et al., 2021;2022a), but they focused on uncurated datasets, which typically lead to a significant drop in the quality of the features.”
    Quote in the paperSubstantial claim2 of 3 sources checked
    • cites Caron et al., 2019 source

      The Introduction (p. 1) says previous work using non-curated raw data reported degraded feature quality. It does not establish that the drop is typically significant; the Introduction (p. 2) also reports that this paper’s approach surpassed features trained without labels on curated datasets. (LLM, conf=0.87)

      Experimental — accuracy not yet measured
    • cites Goyal et al., 2021 source

      Section 4.4 (p. 7) reports a significant drop for features from uncurated pretraining when they are frozen and evaluated on ImageNet, compared with features pretrained on curated, ImageNet-aligned data. That specific result does not establish that uncurated datasets typically reduce feature quality: the Introduction (p. 2) reports better performance than an ImageNet-supervised model on several downstream tasks. (LLM, conf=0.86)

      Experimental — accuracy not yet measured
    • cites Goyal et al., 2022a

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Findingreasoning

    The improved stochastic depth saves memory and compute in proportion approximately equal to the drop rate.

    “This saves memory and compute in proportion approximately equal to the drop rate, thanks to specific fused kernels.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Huang et al., 2016 source

      Section 3 (p. 6) says bypassing ResBlocks avoids forward-backward computation and reports approximately 25% training-time savings under its linear-decay setting; Section 4 (p. 10) reports a corresponding measured speedup. The cited paper does not establish proportional memory savings or attribute the savings to fused kernels, so those parts of the citing sentence are unsupported. (LLM, conf=0.88)

      Experimental — accuracy not yet measured
  • Findingreasoning

    The training procedure scales more efficiently than DDP with float16 autocast when scaling the number of GPU nodes.

    “As a consequence, the training procedure scales more efficiently than DDP with float16 autocast when scaling the number of GPU nodes.”
    Quote in the paperNot a substantial claim
  • Findingreasoning

    PyTorch-FSDP mixed-precision is superior to DDP with autocast in virtually all cases encountered.

    “Overall, Pytorch-FSDP mixed-precision is superior to DDP with autocast in virtually all cases we encountered.”
    Quote in the paperNot a substantial claim
  • Classified as probably not a substantial claim. 26
  • May combine two claims. 36
  • The quote only partly states this claim. 4

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingnone given

    Our version of FlashAttention is on par with or better than the original on all cases considered, while covering more use-cases and hardware.

    “Our version is on par with or better than the original on all cases considered, while covering more use-cases and hardware.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Dao et al., 2022 source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Goyal et al. (2021) have shown that discriminative self-supervised methods benefit from scaling in model size given enough pretrained data.

    “Of particular interest, Goyal et al. (2021) have also shown that these methods benefit from scaling in model size given enough pretrained data.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Goyal et al., 2021 source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Self-supervised approaches can capture information at the image and pixel level.

    “These approaches are conceptually closer to pretext tasks such as language modeling and can capture information at the image and pixel level (Caron et al., 2021).”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Caron et al., 2021 source

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Features output by self-supervised models have been shown to exhibit various useful properties and have enabled a wide variety of applications.

    “Additionally, the features output by self-supervised models have been shown to exhibit various useful properties, and have enabled enabled a wide variety of applications (Amir et al., 2022;Tumanyan et al., 2022;Ofri-Amar et al., 2023;Hamilton et al., 2022).”
    Quote in the paperNot a substantial claim2 of 3 sources checked
    • cites Amir et al., 2022

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Tumanyan et al., 2022 source
    • cites Ofri-Amar et al., 2023 source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Patterson et al. (2021) propose a methodology to report an estimation of the carbon emitted during the training of a model based on the specifics of the data center and its power grid.

    “Patterson et al. (2021) propose a methodology to report an estimation of the carbon emitted during the training of a model based on the specifics of the data center and its power grid.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Patterson et al. (2021) source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The methodology of Patterson et al. (2021) was used in previous work in natural language processing (Strubell et al., 2019; Touvron et al., 2023) to establish an apple-to-apple comparison between pretraining schemes.

    “This methodology was used in previous work in natural language processing (Strubell et al., 2019;Touvron et al., 2023) to establish an apple-to-apple comparison between pretraining schemes.”
    Quote in the paperNot a substantial claim2 of 2 sources checked
    • cites Strubell et al., 2019 source

      Split into 2 atoms — 0 confirmed, 2 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

      Experimental — accuracy not yet measured
    • cites Touvron et al., 2023 source

      Section 6, p. 10, applies the same carbon-emissions formula to LLaMA, OPT, and BLOOM for a fair comparison, using a common data-center assumption. It identifies the formula as following Wu et al. (2022), however, and does not establish that it used Patterson et al. (2021)'s methodology. (LLM, conf=0.86)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingcitation

    Deduplication reduces redundancy and increases diversity among images.

    “This reduces redundancy and increases diversity among images.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Pizzi et al., 2022 source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The whole data processing takes less than two days to produce the LVD-142M dataset.

    “The whole processing is distributed on a compute cluster of 20 nodes equipped with 8 V100-32GB GPUs and takes less than two days to produce the LVD-142M dataset.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingnone given

    With the same hardware, the DINOv2 code runs around 2× faster using only 1/3 of the memory compared to the iBOT implementation.

    “With the same hardware, compared to the iBOT implementation, the DINOv2 code runs around 2× faster using only 1/3 of the memory.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Zhou et al., 2022a

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Findingown data

    A ViT model with 1B parameters and its distilled smaller models surpass OpenCLIP on most benchmarks at image and pixel levels.

    “In terms of models, we train a ViT model (Dosovitskiy et al., 2021) with 1B parameters and distill it into a series of smaller models that surpass the best available general-purpose features, OpenCLIP (Ilharco et al., 2021) on most of the benchmarks at image and pixel levels.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Ilharco et al., 2021

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Findingown data

    The DINOv2 family drastically improves over the previous state of the art in self-supervised learning and reaches performance comparable with weakly-supervised features.

    “Our family of models drastically improves over the previous state of the art in self-supervised learning and reaches performance comparable with weaklysupervised features.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingreasoning

    Choosing N = 4 nearest neighbors provides a good tradeoff between retrieval quality and collisions.

    “We choose N = 4 as it provides a good tradeoff in that sense.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingnone given

    DINOv2 improvements make the approach around 2× faster and require 3× less memory than similar discriminative self-supervised methods.

    “These improvements make our approach around 2× faster and require 3× less memory than similar discriminative self-supervised methods, allowing us to leverage longer training with larger batch sizes.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingcitation

    MAE features require supervised finetuning, while DINOv2 features perform well out of the box.

    “However, their features require supervised finetuning, while our features perform well out of the box.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites He et al., 2022

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Findingown data

    LayerScale and Stochastic Depth incur a performance drop in linear probing but significantly improve training stability.

    “Only LayerScale and Stochastic Depth incur a performance drop in linear probing but significantly improve the training stability in our experience.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Training on curated data increases performance on domains not used for curation (iNaturalist 2018, 2021 and Places205), showing that scale and diversity can benefit unseen domains.

    “We also see that training on our curated data increases the performances on domains that are not used for the curation process (INaturalist 2018, 2021 and Places205), proving that scale and diversity can benefit unseen domains.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Each component in the improved training recipe improves performance on either k-NN or linear probing, and in most cases both.

    “Generally, we observe that each component improves the performance on either k-NN or linear probing and even both in most cases.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The masked image modeling term from iBOT is critical for dense prediction tasks, leading to almost 3% performance improvement.

    “This term is critical for dense prediction tasks, leading to almost 3% performance improvement.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Zhou et al., 2022a

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The finetuning improvement holds both at resolution 224 and 448.

    “This is true both when using models at resolution 224 and 448.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    A ViT-g trained on LVD-142M matches the ImageNet-1k performance of a model trained on ImageNet-22k while significantly outperforming it on other benchmarks.

    “For instance, a ViT-g trained on LVD-142M matches the performance on ImageNet-1k of a model trained on ImageNet-22k while significantly outperforming it on the other benchmarks.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Training at high resolution for only 10k iterations at the end of training is almost as good as training at high resolution from scratch and requires a fraction of the compute.

    “On the other hand, training at high resolution for only 10k iterations at the end of the training is almost as good and only requiring a fraction of the compute.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingnone given

    Retraining an OpenCLIP ViT-L or OpenCLIP ViT-G would require 22.4 MWh and 118.9 MWh, respectively, if run in the same data center.

    “For comparison, retraining an OpenCLIP ViT-L or OpenCLIP ViT-G would require 22.4 MWh and 118.9 MWh, respectively, if run in the same data center.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The carbon footprint represents in the order of 200k GPU-days.

    “This carbon footprint represents in the order of 200k GPU-days.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    DINOv2 models show very good performance on all datasets and for all setups in semantic segmentation.

    “Our models show very good performance on all datasets and for all setups.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    DINOv2 features are competitive with OpenCLIP on most classification benchmarks, except SUN (-5.3%) and Cars (-4.7%).

    “Even though these benchmarks favor text-guided pretraining, our features are still competitive with OpenCLIP on most classification benchmarks, with the exception of a few datasets, especially SUN (-5.3%) and Cars (-4.7%).”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The 12 transfer classification benchmarks favor text-guided pretraining.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The whole DINOv2 project footprint is estimated between 0.5k and 1k tCO2 eq using the same grid.

    “Additionally, we estimate the footprint of the whole project to be between 0.5k and 1k tCO 2 eq using the same grid as presented above 3 .”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The linear segmentation model using the DINOv2 backbone produces good results and behaves much better than the OpenCLIP one under this evaluation setup.

    “While not perfect, the linear segmentation model using our DINOv2 backbone produces good results and behaves much better than the OpenCLIP one under this evaluation setup.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    A ViT-L model distilled from a frozen ViT-g outperforms the same model trained from scratch on all benchmarks, sometimes even outperforming the distillation target.

    “We show that a ViT-L model distilled from a frozen ViT-g outperforms a the same model trained from scratch on all benchmarks, sometimes even outperforming the distillation target.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The model is robust to style (image versus drawing) and to large variation of poses.

    “We also observe that the model is robust to style (image versus drawing), and to large variation of poses (see the elephant).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    DINOv2 model is slightly fairer across regions and incomes than SEERv2 and significantly better than the supervised baseline reported by Goyal et al. (2022a).

    “Our model is slightly fairer across regions and incomes than the SEERv2 model and significantly better than the supervised baseline reported by Goyal et al. (2022a).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Goyal et al. (2022a)

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Findingown data

    The unsupervised foreground/background detector based on detecting the highest variance direction performs very well and is capable of delineating the boundary of the main object in the picture.

    “first, our unsupervised foreground / background detector, based on detecting the highest variance direction, performs very well and is capable of delineating the boundary of the main object in the picture.”
    Quote in the paperNot a substantial claim

  • May combine two claims. The quote only partly states this claim.

    Findingown data

    The other PCA components correspond to parts of objects and match well for images of the same category.

    “Second, the other components correspond to”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    A single pre-training of a ViT-g model (22k GPU-hours) emits 3.7 tons of CO2 eq, while a finetuning on ImageNet-1k (1k GPU-hours) emits 0.2 tons.

    “For example, a single pre-training of a ViT-g model (22k GPU-hours) emits 3.7 tons of CO 2 eq, while a finetuning on ImageNet-1k (1k GPU-hours) emits 0.2 tons.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductown data

    The data pipeline does not require any metadata or text and directly works with images.

    “Our pipeline does not require any metadata or text and directly works with images, as shown in Fig. 3.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Premisecitation

    Increasing image resolution is key to pixel-level downstream tasks such as segmentation or detection, where small objects disappear at low resolutions.

    “Increasing image resolution is key to pixellevel downstream tasks such as segmentation or detection, where small objects disappear at low resolutions.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductnone given

    The evaluation trains a simple classifier over a frozen backbone and does not finetune the backbone weights.

    “We evaluate the quality of features by training a simple classifier over a frozen backbone, and do not perform finetuning of the backbone weights.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    The performance increase on alternative test sets is larger for DINOv2, indicating stronger generalization.

    “At the same time, we also see that the performance increase on the alternative test sets is larger for our method, indicating stronger generalization.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Premisereasoning

    Lack of control over data quality and diversity is essential to produce good features.

    “This is explained by the lack of control over the data quality and diversity, which are essential to produce good features.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductnone given

    The dataset construction uses no pretrained encoders, metadata nor supervision to filter images and leverages visual similarity between images.

    “Unlike these works, we use no pretrained encoders, metadata nor supervision to filter images and leverage visual similarity between images.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisereasoning

    Text-guided image encoders require aligned text-image corpora and do not offer the flexibility to learn from raw data alone.

    “Furthermore, these image encoders require aligned text-image corpora and hence, do not offer the flexibility of their text counterparts, that is, to learn from raw data alone.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    Near-duplicates of images contained in the test or validation set of any benchmark used in this work are removed.

    “We also remove near-duplicates of images contained in the test or validation set of any benchmark used in this work.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The KoLeo regularizer derives from the Kozachenko-Leonenko differential entropy estimator.

    “The KoLeo regularizer derives from the Kozachenko-Leonenko differential entropy estimator (see Beirlant et al. (1997); Delattre & Fournier (2017)) and encourages a uniform span of the features within a batch.”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites Beirlant et al. (1997)

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Delattre & Fournier (2017) source

  • Classified as probably not a substantial claim.

    Premisenone given

    Training at high resolution is time and memory demanding.

    “However, training at high resolution is time and memory demanding, and instead, we increase the resolution of images to 518 × 518 during a short period at the end of pretraining.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    The linear evaluation numbers for all models were verified to match those reported in technical reports and papers.

    “For all models, we run the linear evaluation using our code, after making sure that our numbers match those reported in technical reports and papers.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisenone given

    Text-guided pretraining limits the information that can be retained about the image because captions only approximate the rich information in images, and complex pixel-level information may not surface with this supervision.

    “This form of text-guided pretraining limits the information that can be retained about the image since captions only approximate the rich information in images, and complex pixel-level information may not surface with this supervision.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Captions only approximate the rich information in images, and complex pixel-level information may not surface with text supervision.

  • May combine two claims.

    Interpretationown data

    DINOv2 features are strong in both linear and finetuning settings.

    “As DINOv2 leads to features that are strong in both the linear and finetuning settings, a strong property of our approach is that finetuning is optional.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation

    Discriminative self-supervised methods provide performant frozen features on standard benchmarks like ImageNet but are hard to scale to larger model sizes.

    “These methods provide performant frozen features on standard benchmarks like ImageNet (Russakovsky et al., 2015), but they are hard to scale to larger model sizes (Chen et al., 2021).”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites Russakovsky et al., 2015 source
    • cites Chen et al., 2021

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning

    MLP heads gradients are reduced in float32 to avoid training instabilities.

    “MLP heads gradients are reduced in float32 to avoid training instabilities”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    Using a linear model for evaluation ensures a reproducible evaluation, despite the fact that classes may not be linearly separable.

    “Following previous work, we use a linear model for simplicity, ensuring a reproducible evaluation, despite the fact that classes may not be linearly separable.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    For all models, the evaluation is run with the authors' code.

    “In order to report this additional validation performance, for all models, we run the evaluation with our code.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Ruan et al. (2023) recommend replacing the teacher softmax-centering step of DINO and iBOT by the Sinkhorn-Knopp batch normalization of SwAV.

    “Ruan et al. (2023) recommend to replace the teacher softmax-centering step of DINO and iBot by the Sinkhorn-Knopp (SK) batch normalization of SwAV (Caron et al., 2020).”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites Ruan et al., 2023

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Caron et al., 2020 source

  • The quote only partly states this claim.

    Interpretationown data

    The strong performance of the DINOv2 family of models can be attributed to a larger dataset.

    “We can attribute the strong performance of the DINOv2 family of models to several factors: i) an improved training recipe with better hyperparameters and regularization (Table 1), ii) a larger model scale with improved results regardless of the data used for training (Fig. 4), iii) a larger dataset (Fig. 4) and iv) the distillation process that makes smaller models benefit from the performance of”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Fig. 4

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Interpretationown data

    The DINOv2 model can transfer across domains and understand relations between similar parts of different objects.

    “This exhibits the ability of our model to transfer across domains and understand relations between similar parts of different objects.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    DINOv2 features perform well across task granularities, both at category-level and instance-level.

    “It is interesting to see that our features perform well across task granularities, both at the category-level and instance-level.”
    Quote in the paperSubstantial claim

  • The quote only partly states this claim.

    Interpretationown data

    The strong performance of the DINOv2 family of models can be attributed to an improved training recipe with better hyperparameters and regularization.

    “We can attribute the strong performance of the DINOv2 family of models to several factors: i) an improved training recipe with better hyperparameters and regularization (Table 1), ii) a larger model scale with improved results regardless of the data used for training (Fig. 4), iii) a larger dataset (Fig. 4) and iv) the distillation process that makes smaller models benefit from the performance of”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Table 1

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    The carbon estimate only considers the GPUs' electricity consumption and ignores other emissions such as their manufacturing and disposal.

    “This estimate only considers the GPUs' electricity consumption and ignores other emissions, such as their manufacturing and disposal.”
    Quote in the paperNot a substantial claim

  • The quote only partly states this claim.

    Interpretationown data

    The strong performance of the DINOv2 family of models can be attributed to the distillation process that makes smaller models benefit from the performance of the strongest ViT-g model.

    “We can attribute the strong performance of the DINOv2 family of models to several factors: i) an improved training recipe with better hyperparameters and regularization (Table 1), ii) a larger model scale with improved results regardless of the data used for training (Fig. 4), iii) a larger dataset (Fig. 4) and iv) the distillation process that makes smaller models benefit from the performance of”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Fig. 5

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

57 of 57

  • The paper relies on this without support.

    Captions only approximate the rich information in images, and complex pixel-level information may not surface with text supervision.

    “This form of text-guided pretraining limits the information that can be retained about the image since captions only approximate the rich information in images, and complex pixel-level information may not surface with this supervision.”

  • The paper relies on this without support.

    Most SSL methods were developed using ImageNet-1k validation performance as a debugging signal.

    “Because most SSL methods were developped using ImageNet-1k validation performance as a debugging signal, we also report the top-1 accuracy on ImageNet-ReaL and ImageNet-V2.”

  • The paper relies on this without support.

    The 12 transfer classification benchmarks favor text-guided pretraining.

    “Even though these benchmarks favor text-guided pretraining, our features are still competitive with OpenCLIP on most classification benchmarks, with the exception of a few datasets, especially SUN (-5.3%) and Cars (-4.7%).”

  • The paper relies on this without support.

    The Non-Human and Crime meta-categories are harmful.

    “Non-Human and Crime are considered harmful.”

  • The paper relies on this without support.

    Self-reported gender, skin tone, and age labels are accurate.

    “images of people of different gender, skin tone, and age (all self-reported).”
5 of 5

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Foundation models have demonstrated remarkable zero-shot performance and can be quickly adapted to various downstream tasks with few additional tuning cost.

    “Foundation Models (FMs) [23, 35, 37] - generic models trained on web-scale datasets - have demonstrated remarkable zero-shot performance and can be quickly adapted to various downstream tasks with few additional tuning cost.”

    From Towards More General Video-based Deepfake Detection through Facial Component Guided Adaptation for Foundation Model (2024) · cites this paper as [35]

    The Introduction says pretrained features can be used without fine-tuning and perform well on downstream tasks. The semantic-segmentation section also reports lighter training with a frozen backbone. But the paper’s other downstream evaluations often train a probe or task-specific head, so it does not establish remarkable zero-shot performance and low-cost adaptation across various tasks as broadly as the citing sentence suggests. (LLM, conf=0.83)

    Experimental — accuracy not yet measured

  • Previous video foundation model works tend to focus on understanding evaluation using probe-based evaluation or visual question answering tasks after aligning with a large-language model.

    “Previous works, however, tend to focus on understanding evaluation using probe-based evaluation or visual question answering tasks after aligning with a large-language model.”

    From V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning (2025) · cites this paper as Oquab et al., 2023

    Section 7.2 evaluates DINOv2 features on video action-recognition datasets using a linear classifier, supporting the probe-based evaluation part of the claim. The cited paper does not establish that previous video foundation model works generally favor this approach, nor does it describe visual question answering after alignment with a large language model. (LLM, conf=0.78)

    Experimental — accuracy not yet measured

  • Having high quality data outweighs the quantity of data.

    “having high quality data outweighs the quantity of data”

    From Genie: Generative Interactive Environments (2024) · cites this paper as Oquab et al. (2023)

    Section 6.2, “Pretraining Data Source” (page unavailable), reports that curated images outperform an equally sized uncurated sample on most benchmarks. This supports the value of data quality, but the comparison holds quantity constant; it does not establish the broader claim that quality outweighs quantity. The paper also says scale and diversity can benefit performance. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

  • Using DINO pretrained on ImageNet allows the model to develop a more sophisticated understanding of image structure than would be possible from astronomical images alone.

    “A major advantage of this strategy is that the DINO model, pretrained on ImageNet, has been exposed to a very diverse set of natural images, which allows the model to develop a more sophisticated understanding of image structure than would be possible through studying astronomical images alone.”

    From A Semi-Supervised Learning Method for the Identification of Bad Exposures in Large Imaging Surveys (2026) · cites this paper as Oquab et al. 2023

    Section 6.2 (page unavailable) says that more diverse pretraining images improve feature quality in domains not covered by ImageNet-22k, supporting the general benefit of varied images. But the paper describes DINOv2 as trained on LVD-142M, which includes ImageNet data, rather than simply pretrained on ImageNet. It does not compare this approach with training on astronomical images alone. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • DINOv2 has semantic awareness useful for creating global coherence to evaluate whether scenes make sense wholistically.

    “The semantic awareness of DINOv2 is also especially useful in creating a global coherence to evaluate whether scenes make sense wholistically.”

    From MTReD: 3D Reconstruction Dataset for Fly-over Videos of Maritime Domain (2025) · cites this paper as [8]

    The Discussion (page unavailable) says DINOv2 features show an understanding of object parts and scene geometry, and §7 evaluates global image representations. The paper does not establish that those features create global coherence or evaluate whether a scene makes sense as a whole. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Income-bucket gap does not match Table 12

    Minor

    The reported high- and low-income DINOv2 scores are 90.5 and 67.4, respectively. Their difference is 23.1 percentage points, not 31.7.

    Reported 31.7 · recomputed 23.1

  • Regional performance gap is overstated

    Minor

    Table 12 gives DINOv2 scores of 89.7 for Europe and 74.0 for Africa, a gap of 15.7 percentage points rather than 25.7. Expressed as a drop relative to Europe, it is about 17.5%.

    Reported 25.7 · recomputed 15.7

  • Data-curation method does use a pretrained encoder

    Minor

    The related-work section says the image-filtering method uses no pretrained encoder, but the data-processing section says retrieval begins with an ImageNet-22k-pretrained ViT-H/16 encoder.

3 of 3

Cites

Flagged first, then by citations

Research works printed but not linked: 2

  1. [62]62. Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial exam- ples. In CVPR, 2021b.
  2. [113]113. Hugo Touvron, Andrea Vedaldi, Matthijs Douze, and Hervé Jégou. Fixing the train-test resolution discrep- ancy. In NeurIPS, 2019.

Cited by

Sorted by citations

Related

Authors
Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El-Nouby, A., Assran, M., Ballas, N., Galuba, W., Howes, R., Huang, P.-Y., Li, S.-W., Misra, I., Rabbat, M., Sharma, V., Synnaeve, G., Xu, H., Jegou, H., Mairal, J., Labatut, P., Joulin, A., & Bojanowski, P.
Affiliations
Institut national de recherche en sciences et technologies du numérique (FR)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2023
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2304.07193
arXiv
2304.07193
OpenAlex
W4366208220
Cited by
1,078
References
131
Citations by year
2026: 123 · 2025: 404 · 2024: 485 · 2023: 66
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • HAL (submittedVersion, cc-by)
  • HAL (submittedVersion, other-oa)
  • arXiv (Cornell University)
Funders
  • Agence Nationale de la Recherche (ANR-19-P3IA-0003, ANR‐19)
  • HORIZON EUROPE European Research Council (101087696)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
Keywords
DINOv2, self-supervised pre-training, vision transformer, foundation models, knowledge distillation, large-scale training, image classification benchmark

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-11.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-11.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review