PreprintCited by 1,090Domain Adaptation and Few-Shot LearningAdvanced Neural Network ApplicationsMultimodal Machine Learning Applications

The DOI matches its DataCite record; 139 of the 139 references it cites were found in a registry or an index; 12 of the 40 sources its claims cite were checked for support.

DINOv2: Learning Robust Visual Features without Supervision

Open access

Abstract

The recent breakthroughs in natural language processing for model pretraining on large quantities of data have opened the way for similar foundation models in computer vision. These models could greatly simplify the use of images in any system by producing all-purpose visual features, i.e., features that work across image distributions and tasks without finetuning. This work shows that existing pretraining methods, especially self-supervised methods, can produce such features if trained on enough curated data from diverse sources. We revisit existing approaches and combine different techniques to scale our pretraining in terms of data and model size. Most of the technical contributions aim at accelerating and stabilizing the training at scale. In terms of data, we propose an automatic pipeline to build a dedicated, diverse, and curated image dataset instead of uncurated data, as typically done in the self-supervised literature. In terms of models, we train a ViT model (Dosovitskiy et al., 2020) with 1B parameters and distill it into a series of smaller models that surpass the best available all-purpose features, OpenCLIP (Ilharco et al., 2021) on most of the benchmarks at image and pixel levels.

Overview

AI snapshot

Written from the paper’s full text.

TL;DR

DINOv2 shows that self-supervised pretraining on a curated 142M-image dataset can produce general-purpose visual features competitive with weakly supervised models. Its ViT-g/14 backbone and distilled smaller variants perform strongly on image and pixel tasks without task-specific fine-tuning.

Key findings

  • On ImageNet-1k linear probing, DINOv2 ViT-g/14 reaches 86.5% top-1, exceeding OpenCLIP ViT-G/14 (86.2%) and EVA-CLIP ViT-g/14 (86.4%), and improving over iBOT ViT-L/16 (82.3%) by 4.2 points; kNN is 83.5%, ReaL 89.6%, and V2 78.4%.
  • Dense and retrieval tasks also improve: ViT-g/14 with multiscale linear segmentation gets 53.0 mIoU on ADE20k, 81.0 on Cityscapes, and 86.2 on Pascal VOC; depth DPT RMSE is 0.279 on NYUd versus OpenCLIP’s 0.414; retrieval reaches 52.3 mAP on Oxford-Hard and 82.6 mAP on Paris-Hard.
  • Ablations show the KoLeo regularizer raises Oxford-M retrieval from 55.6 to 63.9 mAP, masked image modeling raises ADE-20k segmentation from 44.2 to 47.1 mIoU, and distillation from ViT-g improves ViT-L over scratch on all 12 benchmarks.
  • Curated LVD-142M beats uncurated data on most benchmarks (e.g., INet-1k 85.8 vs 83.3, iNat2018 82.3 vs 68.0, iNat2021 86.4 vs 76.4) and beats ImageNet-22k on all but ImageNet-1k.

Methods

  • Data: an automatic pipeline starts from 1.2B unique web images, applies deduplication, and retrieves images using self-supervised ViT-H/16 embeddings with cosine similarity, k-means, and Faiss, producing LVD-142M (142M images) in under two days on 20 nodes with 8 V100-32GB GPUs.
  • Training: discriminative self-supervised learning combines DINO image-level and iBOT patch-level objectives with Sinkhorn-Knopp centering, a KoLeo regularizer, separate projection heads, and a short 518×518 high-resolution phase; ViT-g/14 has 1.1B parameters, with efficiency from FlashAttention, sequence packing, stochastic depth, and FSDP, and smaller models distilled from ViT-g.
  • Evaluation: frozen features are assessed with linear probes and kNN on classification, segmentation, depth, retrieval, video, and fairness benchmarks, compared against self-supervised and weakly supervised baselines.

Limitations

  • The paper states fairness gaps: on Dollar Street, Africa performance is 25.7% below Europe and high-income households are 31.7% above low-income; harmful label associations were rare, but a more thorough bias evaluation may reveal flaws.
  • The paper states its carbon estimate covers only GPU electricity and excludes manufacturing and disposal; the whole project is estimated at 0.5k–1k tCO2eq, with one ViT-g pretraining at 3.7 tCO2eq.
  • The text makes evident that evaluation centers on frozen linear probes and curated benchmarks, and the data pipeline retrieves from curated datasets, so coverage may be shaped by those sources; some tasks where text supervision helps (e.g., SUN, Cars) still lag OpenCLIP.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2304.07193

Cites

139 of 139 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

63 Own result · 24 Own interpretation · 18 No basis given · 13 Sources not found · 10 Own method · 5 Not checked yet · 4 Supported · 4 Partly supported · 3 No full text of the sources · 1 Couldn’t confirm · accuracy not yet measured

Verify all 1 references — free account

Show all 88 claims

  • Probably not a substantial claim. 26
  • It may state two separate claims as one. 36
  • The quote only partly states this claim. 4
57 of 57
  • The paper relies on this without support.

    Captions only approximate the rich information in images, and complex pixel-level information may not surface with text supervision.

    “This form of text-guided pretraining limits the information that can be retained about the image since captions only approximate the rich information in images, and complex pixel-level information may not surface with this supervision.”
  • The paper relies on this without support.

    Most SSL methods were developed using ImageNet-1k validation performance as a debugging signal.

    “Because most SSL methods were developped using ImageNet-1k validation performance as a debugging signal, we also report the top-1 accuracy on ImageNet-ReaL and ImageNet-V2.”
  • The paper relies on this without support.

    The 12 transfer classification benchmarks favor text-guided pretraining.

    “Even though these benchmarks favor text-guided pretraining, our features are still competitive with OpenCLIP on most classification benchmarks, with the exception of a few datasets, especially SUN (-5.3%) and Cars (-4.7%).”
  • The paper relies on this without support.

    The Non-Human and Crime meta-categories are harmful.

    “Non-Human and Crime are considered harmful.”
  • The paper relies on this without support.

    Self-reported gender, skin tone, and age labels are accurate.

    “images of people of different gender, skin tone, and age (all self-reported).”
5 of 5

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

3 of 3

Citations

Cites

Show all 131 references

Research works printed but not linked: 2

  1. [62]62. Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial exam- ples. In CVPR, 2021b.
  2. [113]113. Hugo Touvron, Andrea Vedaldi, Matthijs Douze, and Hervé Jégou. Fixing the train-test resolution discrep- ancy. In NeurIPS, 2019.

Cited by

Sorted by citations

Items 301–400 of 1,090

Metadata

Authors
Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El-Nouby, A., Assran, M., Ballas, N., Galuba, W., Howes, R., Huang, P.-Y., Li, S.-W., Misra, I., Rabbat, M., Sharma, V., Synnaeve, G., Xu, H., Jegou, H., Mairal, J., Labatut, P., Joulin, A., & Bojanowski, P.
Affiliations
Institut national de recherche en sciences et technologies du numérique (FR)
Venue
arXiv
Publisher
arXiv
Published
2023
ISSN
2331-8422 (ISSN-L)
OpenAlex
W4366208220
Cited by
1,078
References
131
Citations by year
2026: 123 · 2025: 404 · 2024: 485 · 2023: 66
Type
posted content
Language
en
Open access
Locations
  • arXiv (Cornell University) (submittedVersion)
  • HAL (submittedVersion, cc-by)
  • HAL (submittedVersion, other-oa)
  • arXiv (Cornell University)
Funders
  • Agence Nationale de la Recherche (ANR-19-P3IA-0003, ANR‐19)
  • HORIZON EUROPE European Research Council (101087696)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
Keywords
DINOv2, self-supervised pre-training, vision transformer, foundation models, knowledge distillation, large-scale training, image classification benchmark

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review