The DOI matches its DataCite record; 139 of the 139 references it cites were found in a registry or an index; 12 of the 40 sources its claims cite were checked for support.
DINOv2: Learning Robust Visual Features without Supervision
Open accessAbstract
The recent breakthroughs in natural language processing for model pretraining on large quantities of data have opened the way for similar foundation models in computer vision. These models could greatly simplify the use of images in any system by producing all-purpose visual features, i.e., features that work across image distributions and tasks without finetuning. This work shows that existing pretraining methods, especially self-supervised methods, can produce such features if trained on enough curated data from diverse sources. We revisit existing approaches and combine different techniques to scale our pretraining in terms of data and model size. Most of the technical contributions aim at accelerating and stabilizing the training at scale. In terms of data, we propose an automatic pipeline to build a dedicated, diverse, and curated image dataset instead of uncurated data, as typically done in the self-supervised literature. In terms of models, we train a ViT model (Dosovitskiy et al., 2020) with 1B parameters and distill it into a series of smaller models that surpass the best available all-purpose features, OpenCLIP (Ilharco et al., 2021) on most of the benchmarks at image and pixel levels.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
DINOv2 shows that self-supervised pretraining on a curated 142M-image dataset can produce general-purpose visual features competitive with weakly supervised models. Its ViT-g/14 backbone and distilled smaller variants perform strongly on image and pixel tasks without task-specific fine-tuning.
Key findings
- On ImageNet-1k linear probing, DINOv2 ViT-g/14 reaches 86.5% top-1, exceeding OpenCLIP ViT-G/14 (86.2%) and EVA-CLIP ViT-g/14 (86.4%), and improving over iBOT ViT-L/16 (82.3%) by 4.2 points; kNN is 83.5%, ReaL 89.6%, and V2 78.4%.
- Dense and retrieval tasks also improve: ViT-g/14 with multiscale linear segmentation gets 53.0 mIoU on ADE20k, 81.0 on Cityscapes, and 86.2 on Pascal VOC; depth DPT RMSE is 0.279 on NYUd versus OpenCLIP’s 0.414; retrieval reaches 52.3 mAP on Oxford-Hard and 82.6 mAP on Paris-Hard.
- Ablations show the KoLeo regularizer raises Oxford-M retrieval from 55.6 to 63.9 mAP, masked image modeling raises ADE-20k segmentation from 44.2 to 47.1 mIoU, and distillation from ViT-g improves ViT-L over scratch on all 12 benchmarks.
- Curated LVD-142M beats uncurated data on most benchmarks (e.g., INet-1k 85.8 vs 83.3, iNat2018 82.3 vs 68.0, iNat2021 86.4 vs 76.4) and beats ImageNet-22k on all but ImageNet-1k.
Methods
- Data: an automatic pipeline starts from 1.2B unique web images, applies deduplication, and retrieves images using self-supervised ViT-H/16 embeddings with cosine similarity, k-means, and Faiss, producing LVD-142M (142M images) in under two days on 20 nodes with 8 V100-32GB GPUs.
- Training: discriminative self-supervised learning combines DINO image-level and iBOT patch-level objectives with Sinkhorn-Knopp centering, a KoLeo regularizer, separate projection heads, and a short 518×518 high-resolution phase; ViT-g/14 has 1.1B parameters, with efficiency from FlashAttention, sequence packing, stochastic depth, and FSDP, and smaller models distilled from ViT-g.
- Evaluation: frozen features are assessed with linear probes and kNN on classification, segmentation, depth, retrieval, video, and fairness benchmarks, compared against self-supervised and weakly supervised baselines.
Limitations
- The paper states fairness gaps: on Dollar Street, Africa performance is 25.7% below Europe and high-income households are 31.7% above low-income; harmful label associations were rare, but a more thorough bias evaluation may reveal flaws.
- The paper states its carbon estimate covers only GPU electricity and excludes manufacturing and disposal; the whole project is estimated at 0.5k–1k tCO2eq, with one ViT-g pretraining at 3.7 tCO2eq.
- The text makes evident that evaluation centers on frozen linear probes and curated benchmarks, and the data pipeline retrieves from curated datasets, so coverage may be shaped by those sources; some tasks where text supervision helps (e.g., SUN, Cars) still lag OpenCLIP.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
63 Own result · 24 Own interpretation · 18 No basis given · 13 Sources not found · 10 Own method · 5 Not checked yet · 4 Supported · 4 Partly supported · 3 No full text of the sources · 1 Couldn’t confirm · accuracy not yet measured
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“This leads to approximately 50% reduction in communication costs compared to the float32 gradient all-reduce operation used in DistributedDataParallel (DDP), which is used in other self-supervised pretraining methods (Caron et al., 2021;Zhou et al., 2022a).”
§ DINOv2: Learning Robust Visual Features without Supervision > Fully-Sharded Data Parallel (FSDP).
What the sources say
- Could not verifycites Caron et al., 2021 sourceExperimental — accuracy not yet measured
The compute-requirements section (p. 9) reports DINO training time and GPU memory, not communication costs. Searches for FSDP, DistributedDataParallel, float32 gradient all-reduce, and the claimed 50% reduction found no passage in our copy establishing either the comparison or DDP's use in this method. (LLM, conf=0.75)
- Not matchedcites Zhou et al., 2022a
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Some efforts on scaling these approaches beyond ImageNet-1k have been attempted (Caron et al., 2019;Goyal et al., 2021;2022a), but they focused on uncurated datasets, which typically lead to a significant drop in the quality of the features.”
§ DINOv2: Learning Robust Visual Features without Supervision > 1 Introduction
What the sources say
- Partly supportedcites Caron et al., 2019 sourceExperimental — accuracy not yet measured
The Introduction (p. 1) says previous work using non-curated raw data reported degraded feature quality. It does not establish that the drop is typically significant; the Introduction (p. 2) also reports that this paper’s approach surpassed features trained without labels on curated datasets. (LLM, conf=0.87)
- Partly supportedcites Goyal et al., 2021 sourceExperimental — accuracy not yet measured
Section 4.4 (p. 7) reports a significant drop for features from uncurated pretraining when they are frozen and evaluated on ImageNet, compared with features pretrained on curated, ImageNet-aligned data. That specific result does not establish that uncurated datasets typically reduce feature quality: the Introduction (p. 2) reports better performance than an ImageNet-supervised model on several downstream tasks. (LLM, conf=0.86)
- Not matchedcites Goyal et al., 2022a
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“This saves memory and compute in proportion approximately equal to the drop rate, thanks to specific fused kernels.”
§ DINOv2: Learning Robust Visual Features without Supervision > Efficient stochastic depth.
What the sources say
- Partly supportedcites Huang et al., 2016 sourceExperimental — accuracy not yet measured
Section 3 (p. 6) says bypassing ResBlocks avoids forward-backward computation and reports approximately 25% training-time savings under its linear-decay setting; Section 4 (p. 10) reports a corresponding measured speedup. The cited paper does not establish proportional memory savings or attribute the savings to fused kernels, so those parts of the citing sentence are unsupported. (LLM, conf=0.88)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“As a consequence, the training procedure scales more efficiently than DDP with float16 autocast when scaling the number of GPU nodes.”
§ DINOv2: Learning Robust Visual Features without Supervision > Fully-Sharded Data Parallel (FSDP).
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Overall, Pytorch-FSDP mixed-precision is superior to DDP with autocast in virtually all cases we encountered.”
§ DINOv2: Learning Robust Visual Features without Supervision > Fully-Sharded Data Parallel (FSDP).
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Our experiments did not show significant differences in final accuracy, and our ViT-g backbone counts 1.1B parameters.”
§ DINOv2: Learning Robust Visual Features without Supervision > Fast and memory-efficient attention.
What the sources say
- Not checked — no full text of the sourcecites Zhai et al., 2022 source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“At scale, we observed that the opposite is true, and we therefore use two separate heads in all our experiments.”
§ DINOv2: Learning Robust Visual Features without Supervision > 4 Discriminative Self-supervised Pre-training
What the sources say
- Not matchedcites Zhou et al., 2022a
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“In this work, a naive clustering approach works reasonably well to resolve this issue.”
§ DINOv2: Learning Robust Visual Features without Supervision > 1 Introduction
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The KoLeo loss term improves nearest-neighbor search tasks (e.g.retrieval), and the MIM loss improves patch-level tasks (e.g.segmentation).”
§ DINOv2: Learning Robust Visual Features without Supervision > 6.2 Pretraining Data Source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As the size of models grow, training on LVD-142M becomes more beneficial than training on ImageNet-22k.”
§ DINOv2: Learning Robust Visual Features without Supervision > 6.3 Model Size and Data
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Probably not a substantial claim. 26
- It may state two separate claims as one. 36
- The quote only partly states this claim. 4
Every source the paper cites for this claim was checked and supports it.
In the paper
“Our version is on par with or better than the original on all cases considered, while covering more use-cases and hardware.”
§ DINOv2: Learning Robust Visual Features without Supervision > Fast and memory-efficient attention.
What the sources say
- Supportedcites Dao et al., 2022 source
The source identifies and describes the original FlashAttention in the Introduction (§1). The citing authors’ comparison of their own version with that original—including performance, use-cases, and hardware—is their own claim, not a finding attributed to this source. (LLM, conf=0.84)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Of particular interest, Goyal et al. (2021) have also shown that these methods benefit from scaling in model size given enough pretrained data.”
§ DINOv2: Learning Robust Visual Features without Supervision > Scaling self-supervised pretraining.
What the sources say
- Supportedcites Goyal et al., 2021 source
In §4.1 (p. 4), the authors report that increasing model capacity has a greater effect for self-supervised pretrained models than for models trained from scratch, and confirm the gain with uncurated pretraining data. The Related Work (§2, p. 2) explains why large architectures matter for discriminative self-supervised learning on large datasets. (LLM, conf=0.90)
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“These approaches are conceptually closer to pretext tasks such as language modeling and can capture information at the image and pixel level (Caron et al., 2021).”
§ DINOv2: Learning Robust Visual Features without Supervision > 1 Introduction
What the sources say
- Supportedcites Caron et al., 2021 source
The Introduction (p. 1) compares self-supervised vision pretraining with language-modeling pretext tasks. It reports image-classification performance, while §4.2.2 (p. 7) shows that the learned features retain spatial information and support segmentation, consistent with capturing information at both the image and pixel level. (LLM, conf=0.90)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Additionally, the features output by self-supervised models have been shown to exhibit various useful properties, and have enabled enabled a wide variety of applications (Amir et al., 2022;Tumanyan et al., 2022;Ofri-Amar et al., 2023;Hamilton et al., 2022).”
§ DINOv2: Learning Robust Visual Features without Supervision > 1 Introduction
What the sources say
- Not matchedcites Amir et al., 2022
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Supportedcites Tumanyan et al., 2022 source
The Introduction (p. 1) describes useful properties of self-supervised DINO-ViT features and lists applications including image retrieval, object segmentation, and copy detection. The Conclusions (p. 8) add semantic appearance transfer as an application of those features. (LLM, conf=0.90)
- Supportedcites Ofri-Amar et al., 2023 source
The Introduction (p. 2) describes features from a self-supervised DINO-ViT as effective visual descriptors that capture localized semantic information and underpin the paper’s image-alignment method. The DINO-ViT Features section (p. 3) also describes their use across applications including segmentation, object discovery, and transfer learning. (LLM, conf=0.92)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Not matched
Every source the paper cites for this claim was checked and supports it.
In the paper
“Patterson et al. (2021) propose a methodology to report an estimation of the carbon emitted during the training of a model based on the specifics of the data center and its power grid.”
§ DINOv2: Learning Robust Visual Features without Supervision > 9 Estimating the Environmental Impact of Training our Models
What the sources say
- Supportedcites Patterson et al. (2021) source
Section 2.5 (p. 5) gives formulas for estimating training energy use and CO₂e emissions using datacenter PUE and emissions per kWh. It also explains how to obtain the datacenter’s emissions figure or estimate the regional energy mix, supporting the claim that the methodology accounts for the datacenter and its power grid. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“This methodology was used in previous work in natural language processing (Strubell et al., 2019;Touvron et al., 2023) to establish an apple-to-apple comparison between pretraining schemes.”
§ DINOv2: Learning Robust Visual Features without Supervision > 9 Estimating the Environmental Impact of Training our Models
What the sources say
- Partly supportedcites Strubell et al., 2019 sourceExperimental — accuracy not yet measured
Split into 2 atoms — 0 confirmed, 2 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.
- Partly supportedcites Touvron et al., 2023 sourceExperimental — accuracy not yet measured
Section 6, p. 10, applies the same carbon-emissions formula to LLaMA, OPT, and BLOOM for a fair comparison, using a common data-center assumption. It identifies the formula as following Wu et al. (2022), however, and does not establish that it used Patterson et al. (2021)'s methodology. (LLM, conf=0.86)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Partly supported
In the paper
“This reduces redundancy and increases diversity among images.”
§ DINOv2: Learning Robust Visual Features without Supervision > Data sources.
What the sources say
- Not checked — no full text of the sourcecites Pizzi et al., 2022 source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The whole processing is distributed on a compute cluster of 20 nodes equipped with 8 V100-32GB GPUs and takes less than two days to produce the LVD-142M dataset.”
§ DINOv2: Learning Robust Visual Features without Supervision > Self-supervised image retrieval.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“With the same hardware, compared to the iBOT implementation, the DINOv2 code runs around 2× faster using only 1/3 of the memory.”
§ DINOv2: Learning Robust Visual Features without Supervision > 5 Efficient implementation
What the sources say
- Not matchedcites Zhou et al., 2022a
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
In the paper
“In terms of models, we train a ViT model (Dosovitskiy et al., 2021) with 1B parameters and distill it into a series of smaller models that surpass the best available general-purpose features, OpenCLIP (Ilharco et al., 2021) on most of the benchmarks at image and pixel levels.”
§ DINOv2: Learning Robust Visual Features without Supervision > Abstract
What the sources say
- Not matchedcites Ilharco et al., 2021
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our family of models drastically improves over the previous state of the art in self-supervised learning and reaches performance comparable with weaklysupervised features.”
§ DINOv2: Learning Robust Visual Features without Supervision > 1 Introduction
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“We choose N = 4 as it provides a good tradeoff in that sense.”
§ DINOv2: Learning Robust Visual Features without Supervision > Self-supervised image retrieval.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“These improvements make our approach around 2× faster and require 3× less memory than similar discriminative self-supervised methods, allowing us to leverage longer training with larger batch sizes.”
§ DINOv2: Learning Robust Visual Features without Supervision > 1 Introduction
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“However, their features require supervised finetuning, while our features perform well out of the box.”
§ DINOv2: Learning Robust Visual Features without Supervision > Intra-image self-supervised training.
What the sources say
- Not matchedcites He et al., 2022
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Only LayerScale and Stochastic Depth incur a performance drop in linear probing but significantly improve the training stability in our experience.”
§ DINOv2: Learning Robust Visual Features without Supervision > 6.1 Improved Training Recipe
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We also see that training on our curated data increases the performances on domains that are not used for the curation process (INaturalist 2018, 2021 and Places205), proving that scale and diversity can benefit unseen domains.”
§ DINOv2: Learning Robust Visual Features without Supervision > 6.2 Pretraining Data Source
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Generally, we observe that each component improves the performance on either k-NN or linear probing and even both in most cases.”
§ DINOv2: Learning Robust Visual Features without Supervision > 6.1 Improved Training Recipe
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“This term is critical for dense prediction tasks, leading to almost 3% performance improvement.”
§ DINOv2: Learning Robust Visual Features without Supervision > 6.4 Loss Components
What the sources say
- Not matchedcites Zhou et al., 2022a
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This is true both when using models at resolution 224 and 448.”
§ DINOv2: Learning Robust Visual Features without Supervision > 7.1 ImageNet Classification
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For instance, a ViT-g trained on LVD-142M matches the performance on ImageNet-1k of a model trained on ImageNet-22k while significantly outperforming it on the other benchmarks.”
§ DINOv2: Learning Robust Visual Features without Supervision > 6.3 Model Size and Data
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On the other hand, training at high resolution for only 10k iterations at the end of the training is almost as good and only requiring a fraction of the compute.”
§ DINOv2: Learning Robust Visual Features without Supervision > 6.6 Impact of Resolution
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“For comparison, retraining an OpenCLIP ViT-L or OpenCLIP ViT-G would require 22.4 MWh and 118.9 MWh, respectively, if run in the same data center.”
§ DINOv2: Learning Robust Visual Features without Supervision > 9 Estimating the Environmental Impact of Training our Models
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This carbon footprint represents in the order of 200k GPU-days.”
§ DINOv2: Learning Robust Visual Features without Supervision > 9 Estimating the Environmental Impact of Training our Models
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our models show very good performance on all datasets and for all setups.”
§ DINOv2: Learning Robust Visual Features without Supervision > Semantic segmentation.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Even though these benchmarks favor text-guided pretraining, our features are still competitive with OpenCLIP on most classification benchmarks, with the exception of a few datasets, especially SUN (-5.3%) and Cars (-4.7%).”
§ DINOv2: Learning Robust Visual Features without Supervision > 7.2 Additional Image and Video classification Benchmarks
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The 12 transfer classification benchmarks favor text-guided pretraining.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Additionally, we estimate the footprint of the whole project to be between 0.5k and 1k tCO 2 eq using the same grid as presented above 3 .”
§ DINOv2: Learning Robust Visual Features without Supervision > 9 Estimating the Environmental Impact of Training our Models
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“While not perfect, the linear segmentation model using our DINOv2 backbone produces good results and behaves much better than the OpenCLIP one under this evaluation setup.”
§ DINOv2: Learning Robust Visual Features without Supervision > Semantic Segmentation and Depth Estimation.
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We show that a ViT-L model distilled from a frozen ViT-g outperforms a the same model trained from scratch on all benchmarks, sometimes even outperforming the distillation target.”
§ DINOv2: Learning Robust Visual Features without Supervision > 10 Future work and Discussion
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We also observe that the model is robust to style (image versus drawing), and to large variation of poses (see the elephant).”
§ DINOv2: Learning Robust Visual Features without Supervision > Out-of-distribution generalization.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“Our model is slightly fairer across regions and incomes than the SEERv2 model and significantly better than the supervised baseline reported by Goyal et al. (2022a).”
§ DINOv2: Learning Robust Visual Features without Supervision > 8.1 Geographical Fairness
What the sources say
- Not matchedcites Goyal et al. (2022a)
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“first, our unsupervised foreground / background detector, based on detecting the highest variance direction, performs very well and is capable of delineating the boundary of the main object in the picture.”
§ DINOv2: Learning Robust Visual Features without Supervision > Out-of-distribution generalization.
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Second, the other components correspond to”
§ DINOv2: Learning Robust Visual Features without Supervision > Out-of-distribution generalization.
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one. The quote only partly states this claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For example, a single pre-training of a ViT-g model (22k GPU-hours) emits 3.7 tons of CO 2 eq, while a finetuning on ImageNet-1k (1k GPU-hours) emits 0.2 tons.”
§ DINOv2: Learning Robust Visual Features without Supervision > 9 Estimating the Environmental Impact of Training our Models
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Our pipeline does not require any metadata or text and directly works with images, as shown in Fig. 3.”
§ DINOv2: Learning Robust Visual Features without Supervision > 3 Data Processing
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“Increasing image resolution is key to pixellevel downstream tasks such as segmentation or detection, where small objects disappear at low resolutions.”
§ DINOv2: Learning Robust Visual Features without Supervision > 4 Discriminative Self-supervised Pre-training
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“We evaluate the quality of features by training a simple classifier over a frozen backbone, and do not perform finetuning of the backbone weights.”
§ DINOv2: Learning Robust Visual Features without Supervision > 7.1 ImageNet Classification
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“At the same time, we also see that the performance increase on the alternative test sets is larger for our method, indicating stronger generalization.”
§ DINOv2: Learning Robust Visual Features without Supervision > 7.1 ImageNet Classification
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“This is explained by the lack of control over the data quality and diversity, which are essential to produce good features.”
§ DINOv2: Learning Robust Visual Features without Supervision > 1 Introduction
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Unlike these works, we use no pretrained encoders, metadata nor supervision to filter images and leverage visual similarity between images.”
§ DINOv2: Learning Robust Visual Features without Supervision > Automatic data curation.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Furthermore, these image encoders require aligned text-image corpora and hence, do not offer the flexibility of their text counterparts, that is, to learn from raw data alone.”
§ DINOv2: Learning Robust Visual Features without Supervision > 1 Introduction
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“We also remove near-duplicates of images contained in the test or validation set of any benchmark used in this work.”
§ DINOv2: Learning Robust Visual Features without Supervision > Data sources.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim have been checked yet.
In the paper
“The KoLeo regularizer derives from the Kozachenko-Leonenko differential entropy estimator (see Beirlant et al. (1997); Delattre & Fournier (2017)) and encourages a uniform span of the features within a batch.”
§ DINOv2: Learning Robust Visual Features without Supervision > 4 Discriminative Self-supervised Pre-training
What the sources say
- Not matchedcites Beirlant et al. (1997)
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not checked — no full text of the sourcecites Delattre & Fournier (2017) source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“However, training at high resolution is time and memory demanding, and instead, we increase the resolution of images to 518 × 518 during a short period at the end of pretraining.”
§ DINOv2: Learning Robust Visual Features without Supervision > 4 Discriminative Self-supervised Pre-training
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“For all models, we run the linear evaluation using our code, after making sure that our numbers match those reported in technical reports and papers.”
§ DINOv2: Learning Robust Visual Features without Supervision > 7.1 ImageNet Classification
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“This form of text-guided pretraining limits the information that can be retained about the image since captions only approximate the rich information in images, and complex pixel-level information may not surface with this supervision.”
§ DINOv2: Learning Robust Visual Features without Supervision > 1 Introduction
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
It may state two separate claims as one.
- Rests on an assumption: Captions only approximate the rich information in images, and complex pixel-level information may not surface with text supervision.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As DINOv2 leads to features that are strong in both the linear and finetuning settings, a strong property of our approach is that finetuning is optional.”
§ DINOv2: Learning Robust Visual Features without Supervision > 7.1 ImageNet Classification
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“These methods provide performant frozen features on standard benchmarks like ImageNet (Russakovsky et al., 2015), but they are hard to scale to larger model sizes (Chen et al., 2021).”
§ DINOv2: Learning Robust Visual Features without Supervision > Discriminative self-supervised learning.
What the sources say
- Not checked — no full text of the sourcecites Russakovsky et al., 2015 source
- Not matchedcites Chen et al., 2021
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“MLP heads gradients are reduced in float32 to avoid training instabilities”
§ DINOv2: Learning Robust Visual Features without Supervision > Fully-Sharded Data Parallel (FSDP).
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Following previous work, we use a linear model for simplicity, ensuring a reproducible evaluation, despite the fact that classes may not be linearly separable.”
§ DINOv2: Learning Robust Visual Features without Supervision > 7.1 ImageNet Classification
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“In order to report this additional validation performance, for all models, we run the evaluation with our code.”
§ DINOv2: Learning Robust Visual Features without Supervision > 7.1 ImageNet Classification
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim have been checked yet.
In the paper
“Ruan et al. (2023) recommend to replace the teacher softmax-centering step of DINO and iBot by the Sinkhorn-Knopp (SK) batch normalization of SwAV (Caron et al., 2020).”
§ DINOv2: Learning Robust Visual Features without Supervision > 4 Discriminative Self-supervised Pre-training
What the sources say
- Not matchedcites Ruan et al., 2023
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not checked yetcites Caron et al., 2020 source
Checked before — its earlier verdict is held back until a new check runs.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
In the paper
“We can attribute the strong performance of the DINOv2 family of models to several factors: i) an improved training recipe with better hyperparameters and regularization (Table 1), ii) a larger model scale with improved results regardless of the data used for training (Fig. 4), iii) a larger dataset (Fig. 4) and iv) the distillation process that makes smaller models benefit from the performance of”
§ DINOv2: Learning Robust Visual Features without Supervision > 10 Future work and Discussion
What the sources say
- Not matchedcites Fig. 4
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The quote only partly states this claim.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This exhibits the ability of our model to transfer across domains and understand relations between similar parts of different objects.”
§ DINOv2: Learning Robust Visual Features without Supervision > 10 Future work and Discussion
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“It is interesting to see that our features perform well across task granularities, both at the category-level and instance-level.”
§ DINOv2: Learning Robust Visual Features without Supervision > 7.3 Instance Recognition
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“We can attribute the strong performance of the DINOv2 family of models to several factors: i) an improved training recipe with better hyperparameters and regularization (Table 1), ii) a larger model scale with improved results regardless of the data used for training (Fig. 4), iii) a larger dataset (Fig. 4) and iv) the distillation process that makes smaller models benefit from the performance of”
§ DINOv2: Learning Robust Visual Features without Supervision > 10 Future work and Discussion
What the sources say
- Not matchedcites Table 1
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The quote only partly states this claim.
- Not matched
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“This estimate only considers the GPUs' electricity consumption and ignores other emissions, such as their manufacturing and disposal.”
§ DINOv2: Learning Robust Visual Features without Supervision > 9 Estimating the Environmental Impact of Training our Models
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
In the paper
“We can attribute the strong performance of the DINOv2 family of models to several factors: i) an improved training recipe with better hyperparameters and regularization (Table 1), ii) a larger model scale with improved results regardless of the data used for training (Fig. 4), iii) a larger dataset (Fig. 4) and iv) the distillation process that makes smaller models benefit from the performance of”
§ DINOv2: Learning Robust Visual Features without Supervision > 10 Future work and Discussion
What the sources say
- Not matchedcites Fig. 5
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The quote only partly states this claim.
- Not matched
The paper relies on this without support.
Captions only approximate the rich information in images, and complex pixel-level information may not surface with text supervision.
“This form of text-guided pretraining limits the information that can be retained about the image since captions only approximate the rich information in images, and complex pixel-level information may not surface with this supervision.”
The paper relies on this without support.
Most SSL methods were developed using ImageNet-1k validation performance as a debugging signal.
“Because most SSL methods were developped using ImageNet-1k validation performance as a debugging signal, we also report the top-1 accuracy on ImageNet-ReaL and ImageNet-V2.”
The paper relies on this without support.
The 12 transfer classification benchmarks favor text-guided pretraining.
“Even though these benchmarks favor text-guided pretraining, our features are still competitive with OpenCLIP on most classification benchmarks, with the exception of a few datasets, especially SUN (-5.3%) and Cars (-4.7%).”
The paper relies on this without support.
The Non-Human and Crime meta-categories are harmful.
“Non-Human and Crime are considered harmful.”
The paper relies on this without support.
Self-reported gender, skin tone, and age labels are accurate.
“images of people of different gender, skin tone, and age (all self-reported).”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Foundation Models (FMs) [23, 35, 37] - generic models trained on web-scale datasets - have demonstrated remarkable zero-shot performance and can be quickly adapted to various downstream tasks with few additional tuning cost.”
From Towards More General Video-based Deepfake Detection through Facial Component Guided Adaptation for Foundation Model (2024) · cites this paper as [35]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Previous works, however, tend to focus on understanding evaluation using probe-based evaluation or visual question answering tasks after aligning with a large-language model.”
From V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning (2025) · cites this paper as Oquab et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“having high quality data outweighs the quantity of data”
From Genie: Generative Interactive Environments (2024) · cites this paper as Oquab et al. (2023)
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“A major advantage of this strategy is that the DINO model, pretrained on ImageNet, has been exposed to a very diverse set of natural images, which allows the model to develop a more sophisticated understanding of image structure than would be possible through studying astronomical images alone.”
From A Semi-Supervised Learning Method for the Identification of Bad Exposures in Large Imaging Surveys (2026) · cites this paper as Oquab et al. 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The semantic awareness of DINOv2 is also especially useful in creating a global coherence to evaluate whether scenes make sense wholistically.”
From MTReD: 3D Reconstruction Dataset for Fly-over Videos of Maritime Domain (2025) · cites this paper as [8]
Experimental — accuracy not yet measured
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“While DINOv2 was not explicitly trained for the medical domain, it has shown strong generalization, making it effective for general tasks, including those in medical imaging.”
From ConfIC-RCA: Statistically Grounded Efficient Estimation of Segmentation Quality (2025) · cites this paper as [54]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Because uncurated and unbalanced data can hinder model performance (Assran et al., 2022; Oquab et al., 2023), we filter YT1B by adapting an existing retrieval-based curation pipeline to handle videos.”
From V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning (2025) · cites this paper as Oquab et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“The latter is a much larger (300 million parameters) Vision Transformer21 trained at 0.5 mpp resolution on over 100,000 WSIs from 22 major tissue types using self-distillation and masked modeling22.”
From Generating dermatopathology reports from gigapixel whole slide images with HistoGPT (2025) · cites this paper as [22]
The cited source says what the claim attributes to it.
In the citing paper
“Similar to Oquab et al. (2023), we ensure that none of the videos from the target validation sets are contained in the initial, uncurated data pool.”
From V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning (2025) · cites this paper as Oquab et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“The OpenVLA model architecture consists of a fused vision backbone that combines the SigLIP [9] and DinoV2 [25] encoders.”
From OpenVLA: An Open-Source Vision-Language-Action Model (2024) · cites this paper as [25]
The cited source says what the claim attributes to it.
In the citing paper
“Models such as DINOv2 [7], Vision Transformers (ViT) [8], and Swin Transformers [9] have demonstrated outstanding transfer learning performance across a wide range of domains, often achieving state-of-the-art results with only minimal fine-tuning.”
From Trustworthy Celestial Eye: Calibrated and Robust Planetary Classification via Self-Supervised Vision Transformers (2026) · cites this paper as [7]
The cited source says what the claim attributes to it.
In the citing paper
“Oquab et al. [7] proposed DINOv2, a self-supervised vision foundation model trained on 142 million curated natural images that demonstrates strong transfer learning capabilities across various downstream tasks.”
From Trustworthy Celestial Eye: Calibrated and Robust Planetary Classification via Self-Supervised Vision Transformers (2026) · cites this paper as [7]
The cited source says what the claim attributes to it.
In the citing paper
“we propose that a DINOv2 [8] based perception score be used, and argue that as DINOv2 is trained using internet-scale levels of data, additional training data is not required.”
From MTReD: 3D Reconstruction Dataset for Fly-over Videos of Maritime Domain (2025) · cites this paper as [8]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“but the model's effectiveness on scientific imaging domains with distinct statistical properties remains underexplored.”
From Trustworthy Celestial Eye: Calibrated and Robust Planetary Classification via Self-Supervised Vision Transformers (2026) · cites this paper as [7]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“When compared to LPIPS, it demonstrates superior ability in evaluating scene completion while retaining equal sensitivity to scene accuracy”
From MTReD: 3D Reconstruction Dataset for Fly-over Videos of Maritime Domain (2025) · cites this paper as [8]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“significantly better on high-income households than low-income ones, with a difference of 31.7%.”
§ DINOv2: Learning Robust Visual Features without Supervision > Income buckets Regions
Reported 31.7 · recomputed 23.1
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“However, we still observe a significant difference between regions, particularly in Africa, where our model performance drops by 25.7% compared to Europe.”
§ DINOv2: Learning Robust Visual Features without Supervision > 8.1 Geographical Fairness
Reported 25.7 · recomputed 15.7
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ DINOv2: Learning Robust Visual Features without Supervision > Automatic data curation.
“Unlike these works, we use no pretrained encoders, metadata nor supervision to filter images and leverage visual similarity between images.”
§ DINOv2: Learning Robust Visual Features without Supervision > Self-supervised image retrieval.
“In order to do this, we first compute an image embedding using a self-supervised ViT-H/16 network pretrained on ImageNet-22k, and use cosine-similarity as a distance measure between images.”
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.11929 · Cited by 47,472
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380 · cited 2×
“However, despite their potential to learn general-purpose features, most of the advances in self-supervised learning were made in the context of pretraining on a small curated dataset, ImageNet-1k (Russakovsky et al., 2015).”
BackgroundMethodLearning Multiple Layers of Features from Tiny Images
Krizhevsky, A. (2024) · doi:10.57702/zp44cu3g · Cited by 23,889
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation
Chen, L.-C., Zhu, Y., Papandreou, G., Schroff, F., & Adam, H. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01234-2_49 · Cited by 20,054
The pascal visual object classes (VOC) challenge
Everingham, M., Van Gool, L., Williams, C. K. I., Winn, J., & Zisserman, A. (2010) · International Journal of Computer Vision · doi:10.1007/s11263-009-0275-4 · Cited by 19,599
Learning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293
Distilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204 · cited 1×
“7.”
Momentum Contrast for Unsupervised Visual Representation Learning
He, K., Fan, H., Wu, Y., Xie, S., & Girshick, R. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00975 · Cited by 13,347 · cited 2×
“The DINO loss term corresponds to: We learn the parameters of the student and build the teacher head with an exponential moving average of past iterates (He et al., 2020).”
BackgroundMethodThe Cityscapes Dataset for Semantic Urban Scene Understanding
Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., & Schiele, B. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.350 · Cited by 12,433
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · cited 1×
“Learning task-agnostic pretrained representations have become the standard in Natural Language Processing (NLP) (Radford et al., 2019; Raffel et al., 2020; Chowdhery et al., 2022; Hoffmann et al., 2022; Touvron et al., 2023).”
BackgroundA simple framework for contrastive learning of visual representations
Chen, T., Kornblith, S., Norouzi, M., & Hinton, G. (2020) · arXiv · doi:10.48550/arxiv.2002.05709 · Cited by 10,857 · cited 4×
“For our semantic segmentation evaluation, we consider two different setups.”
BackgroundMethodInfluentialVision meets robotics: The KITTI dataset
Geiger, A., Lenz, P., Stiller, C., & Urtasun, R. (2013) · The International Journal of Robotics Research · doi:10.1177/0278364913491297 · Cited by 9,913
Masked Autoencoders Are Scalable Vision Learners
He, K., Chen, X., Xie, S., Li, Y., Dollar, P., & Girshick, R. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01553 · Cited by 8,489 · cited 6×
“Interestingly, our evaluation using +ms is on par with fully finetuning MAE with an Upernet decoder (53 .”
BackgroundMethodInfluentialIndoor segmentation and support inference from rgbd images
Silberman, N., Hoiem, D., Kohli, P., & Fergus, R. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-33715-4_54 · Cited by 5,760
Context Encoders: Feature Learning by Inpainting
Pathak, D., Krahenbuhl, P., Donahue, J., Darrell, T., & Efros, A. A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.278 · Cited by 5,691 · cited 1×
“Many other pretext tasks were introduced based on, for example, re-colorizing images (Zhang et al., 2016), predicting transformations (Gidaris et al., 2018), inpainting (Pathak et al., 2016) or patch re-ordering (Noroozi & Favaro, 2016; Misra & Maaten, 2020).”
MethodEmerging Properties in Self-Supervised Vision Transformers
Caron, M., Touvron, H., Misra, I., Jegou, H., Mairal, J., Bojanowski, P., & Joulin, A. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00951 · Cited by 5,638 · cited 5×
“First, we run our evaluations for MAE (He et al., 2022), DINO (Caron et al., 2021), SEERv2 (Goyal et al., 2022a), MSN (Assran et al., 2022), EsViT (Li et al., 2022a), Mugs (Zhou et al., 2022b) and iBOT (Zhou et al., 2022a).”
BackgroundMethodInfluentialSelf-citationDimensionality Reduction by Learning an Invariant Mapping
Hadsell, R., Chopra, S., & LeCun, Y. (2006) · 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Volume 2 (CVPR'06) · doi:10.1109/cvpr.2006.100 · Cited by 5,336 · cited 1×
“This family of methods has roots in early deep learning work (Hadsell et al., 2006) but became popular with the emergence of instance classification methods (Dosovitskiy et al., 2016; Bojanowski & Joulin, 2017; Wu et al., 2018).”
MethodThe Kinetics Human Action Video DatasetPreprint
Kay, W., Carreira, J., Simonyan, K., Zhang, B., Hillier, C., Vijayanarasimhan, S., Viola, F., Green, T., Back, T., Natsev, P., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.06950 · Cited by 5,004 · cited 1×
“We report the performance of our model variants as well as the baselines on three datasets under the two setups in Table 10.”
MethodLLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602 · cited 3×
“Learning task-agnostic pretrained representations have become the standard in Natural Language Processing (NLP) (Radford et al., 2019; Raffel et al., 2020; Chowdhery et al., 2022; Hoffmann et al., 2022; Touvron et al., 2023).”
BackgroundInfluentialSelf-citationUcf101: A dataset of 101 human actions classes from videos in the wild
Soomro, K., Zamir, A. R., & Shah, M. (2012) · arXiv · doi:10.48550/arxiv.1212.0402 · Cited by 4,575 · cited 1×
“We report the performance of our model variants as well as the baselines on three datasets under the two setups in Table 10.”
MethodBillion-scale similarity search with GPUs
Johnson, J., Douze, M., & Jegou, H. (2021) · IEEE Transactions on Big Data · doi:10.1109/tbdata.2019.2921572 · Cited by 4,228 · cited 1×
“Our approach is inspired by text curation pipelines (Wenzek et al., 2020), where a language model is trained on Wikipedia to score texts extracted from an uncurated source.”
MethodBootstrap Your Own Latent: A New Approach to Self-Supervised Learning
Grill, J.-B., Strub, F., Altché, F., Tallec, C., Richemond, P. H., Buchatskaya, E., Doersch, C., Pires, B. A., Guo, Z. D., Azar, M. G., et al. (2020) · arXiv · doi:10.48550/arxiv.2006.07733 · Cited by 3,846 · cited 2×
“…dataset through a self-supervised retrieval system. were made based either on instance-level objectives (Hénaff et al., 2019; He et al., 2020; Chen & He, 2021; Chen et al., 2020; Grill et al., 2020; Caron et al., 2021) or clustering (Caron et al., 2018; Asano et al., 2020; Caron et al., 2020).”
MethodExploring Simple Siamese Representation Learning
Chen, X., & He, K. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01549 · Cited by 3,725
Energy and Policy Considerations for Deep Learning in NLP
Strubell, E., Ganesh, A., & McCallum, A. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1355 · Cited by 3,684 · cited 1×
“This methodology was used in previous work in natural language processing (Strubell et al., 2019; Touvron et al., 2023) to establish an apple-to-apple comparison between pretraining schemes.”
Method3D Object Representations for Fine-Grained Categorization
Krause, J., Stark, M. M., Deng, J., & Fei-Fei, L. (2013) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw.2013.77 · Cited by 3,666
Unsupervised Feature Learning via Non-parametric Instance Discrimination
Wu, Z., Xiong, Y., Yu, S. X., & Lin, D. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00393 · Cited by 3,664 · cited 2×
“, 2006) but became popular with the emergence of instance classification methods (Dosovitskiy et al., 2014; Bojanowski & Joulin, 2017; Wu et al., 2018).”
MethodMasked-attention Mask Transformer for Universal Image Segmentation
Cheng, B., Misra, I., Schwing, A. G., Kirillov, A., & Girdhar, R. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00135 · Cited by 3,565 · cited 1×
“5, the segmentation training in this experiment took 28 hours on 16 V100 GPUs.”
MethodSelf-citationAutomated Flower Classification over a Large Number of Classes
Nilsback, M.-E., & Zisserman, A. (2008) · 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing · doi:10.1109/icvgip.2008.47 · Cited by 3,442
Zhang, R., Isola, P., & Efros, A. A. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46487-9_40 · Cited by 3,417 · cited 1×
“Many other pretext tasks were introduced based on, for example, re-colorizing images (Zhang et al., 2016), predicting transformations (Gidaris et al., 2018), inpainting (Pathak et al., 2016) or patch re-ordering (Noroozi & Favaro, 2016; Misra & Maaten, 2020).”
MethodScene Parsing through ADE20K Dataset
Zhou, B., Zhao, H., Puig, X., Fidler, S., Barriuso, A., & Torralba, A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.544 · Cited by 3,326
SUN database: Large-scale scene recognition from abbey to zoo
Xiao, J., Hays, J., Ehinger, K. A., Oliva, A., & Torralba, A. (2010) · 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2010.5539970 · Cited by 3,283
Product Quantization for Nearest Neighbor Search
Jégou, H., Douze, M., & Schmid, C. (2011) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2010.57 · Cited by 2,956 · cited 1×
“In particular, we heavily leverage its support for GPU-accelerated indices, using inverted file indices with product quantization codes (Jegou et al., 2010).”
MethodSelf-citationUnsupervised Learning of Visual Representations by Solving Jigsaw Puzzles
Noroozi, M., & Favaro, P. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46466-4_5 · Cited by 2,775 · cited 1×
“Many other pretext tasks were introduced based on, for example, re-colorizing images (Zhang et al., 2016), predicting transformations (Gidaris et al., 2018), inpainting (Pathak et al., 2016) or patch re-ordering (Noroozi & Favaro, 2016; Misra & Maaten, 2020).”
MethodImageNet: A large-scale hierarchical image database
Deng, J., Dong, W. F., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvprw.2009.5206848 · Cited by 2,714
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676 · cited 1×
“Following this paradigm shift in NLP, we expect similar “foundation” models to appear in computer vision (Bommasani et al., 2021).”
MethodInfluentialDeep Clustering for Unsupervised Learning of Visual Features
Caron, M., Bojanowski, P., Joulin, A., & Douze, M. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01264-9_9 · Cited by 2,674 · cited 2×
“An alternative to text-guided pretraining is self-supervised learning (Caron et al., 2018; Chen et al., 2020; He et al., 2022) where features are learned from images alone.”
BackgroundMethodSelf-citationUnsupervised Visual Representation Learning by Context Prediction
Doersch, C., Gupta, A., & Efros, A. A. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.167 · Cited by 2,629 · cited 1×
“This idea has become prevalent with the work of Doersch et al. (2015), where they train by predicting the context of a given patch.”
BackgroundPaLM: Scaling Language Modeling with Pathways
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.02311 · Cited by 2,590 · cited 1×
“Learning task-agnostic pretrained representations have become the standard in Natural Language Processing (NLP) (Radford et al., 2019; Raffel et al., 2020; Chowdhery et al., 2022; Hoffmann et al., 2022; Touvron et al., 2023).”
Background(2014) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 2,534
Fei-Fei, L., Fergus, R., & Perona, P. (2005) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvpr.2004.383 · Cited by 2,513
Unsupervised Learning of Visual Features by Contrasting Cluster Assignments
Caron, M., Misra, I., Mairal, J., Goyal, P., Bojanowski, P., & Joulin, A. (2020) · arXiv · doi:10.48550/arxiv.2006.09882 · Cited by 2,499 · cited 5×
“• Sinkhorn-Knopp centering (Caron et al., 2020).”
MethodInfluentialSelf-citationUnsupervised Representation Learning by Predicting Image Rotations
Gidaris, S., Singh, P., & Komodakis, N. (2018) · arXiv · doi:10.48550/arxiv.1803.07728 · Cited by 2,477 · cited 2×
“Many other pretext tasks were introduced based on, for example, re-colorizing images (Zhang et al., 2016), predicting transformations (Gidaris et al., 2018), inpainting (Pathak et al., 2016) or patch re-ordering (Noroozi & Favaro, 2016; Misra & Maaten, 2020).”
BackgroundMethodDescribing Textures in the Wild
Cimpoi, M., Maji, S., Kokkinos, I., Mohamed, S., & Vedaldi, A. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.461 · Cited by 2,438
Vision Transformers for Dense Prediction
Ranftl, R., Bochkovskiy, A., & Koltun, V. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01196 · Cited by 2,373 · cited 1×
“5, the segmentation training in this experiment took 28 hours on 16 V100 GPUs.”
MethodFood-101 – Mining Discriminative Components with Random Forests
Bossard, L., Guillaumin, M., & Gool, L. V. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-10599-4_29 · Cited by 2,105
Deep Networks with Stochastic Depth
Huang, G., Sun, Y., Liu, Z., Sedra, D., & Weinberger, K. Q. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46493-0_39 · Cited by 2,104 · cited 1×
“We implement an improved version of stochastic depth (Huang et al., 2016) that skips the computation of the dropped residuals rather than masking the result.”
MethodSUN RGB-D: A RGB-D scene understanding benchmark suite
Song, S., Lichtenberg, S. P., & Xiao, J. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298655 · Cited by 1,943
Parkhi, O., Vedaldi, A., Zisserman, A., & Jawahar, C. V. (2012) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2012.6248092 · Cited by 1,702
An Empirical Study of Training Self-Supervised Vision Transformers
Chen, X., Xie, S., & He, K. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00950 · Cited by 1,648 · cited 2×
“These methods provide performant frozen features on standard benchmarks like ImageNet (Russakovsky et al., 2015), but they are hard to scale to larger model sizes (Chen et al., 2021).”
BackgroundMethodFine-Tuning CNN Image Retrieval with No Human Annotation
Radenović, F., Tolias, G., & Chum, O. (2018) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2846566 · Cited by 1,515 · cited 1×
“A first family of self-supervised methods focuses on pretext tasks built from the image, i.e., extracting a signal from the image to be predicted from the rest of the image.”
BackgroundThe “Something Something” Video Database for Learning and Evaluating Visual Common Sense
Goyal, R., Kahou, S. E., Michalski, V., Materzynska, J., Westphal, S., Kim, H., Haenel, V., Fruend, I., Yianilos, P., Mueller-Freitag, M., et al. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.622 · Cited by 1,471 · cited 1×
“We report the performance of our model variants as well as the baselines on three datasets under the two setups in Table 10.”
MethodThe iNaturalist Species Classification and Detection Dataset
Van Horn, G., Mac Aodha, O., Song, Y., Cui, Y., Sun, C., Shepard, A., Adam, H., Perona, P., & Belongie, S. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00914 · Cited by 1,451
Self-Supervised Learning of Pretext-Invariant Representations
Misra, I., & van der Maaten, L. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00674 · Cited by 1,435 · cited 1×
“Many other pretext tasks were introduced based on, for example, re-colorizing images (Zhang et al., 2016), predicting transformations (Gidaris et al., 2018), inpainting (Pathak et al., 2016) or patch re-ordering (Noroozi & Favaro, 2016; Misra & Maaten, 2020).”
MethodSelf-citationBEiT: BERT Pre-Training of Image Transformers
Bao, H., Dong, L., Piao, S., & Wei, F. (2021) · arXiv · doi:10.48550/arxiv.2106.08254 · Cited by 1,392 · cited 2×
“Recently, the emergence of patch-based architectures, like ViTs, has led to a revisit of inpainting for pre-training (He et al., 2022; Bao et al., 2021; El-Nouby et al., 2021), potentially in feature space (Assran et al., 2023; Baevski et al., 2022).”
BackgroundBenchmarking Neural Network Robustness to Common Corruptions and Perturbations
Hendrycks, D., & Dietterich, T. (2019) · arXiv · doi:10.48550/arxiv.1903.12261 · Cited by 1,335
data2vec: A General Framework for Self-supervised Learning in Speech, Vision and LanguagePreprint
Baevski, A., Hsu, W.-N., Xu, Q., Babu, A., Gu, J., & Auli, M. (2022) · International Conference on Machine Learning · doi:10.48550/arxiv.2202.03555 · Cited by 1,303 · cited 2×
“Recently, the emergence of patch-based architectures, like ViTs, has led to a revisit of inpainting for pre-training (He et al., 2022; Bao et al., 2021; El-Nouby et al., 2021), potentially in feature space (Assran et al., 2023; Baevski et al., 2022).”
BackgroundiBOT: Image BERT Pre-Training with Online TokenizerPreprint
Zhou, J., Wei, C., Wang, H., Shen, W., Xie, C., Yuille, A., & Kong, T. (2021) · arXiv.org · doi:10.48550/arxiv.2111.07832 · Cited by 1,278 · cited 5×
“For the uncurated data source, we collect a raw unfiltered dataset of images from a publicly available repository of crawled web data.”
BackgroundMethodInfluentialFine-Grained Visual Classification of AircraftPreprint
Maji, S., Rahtu, E., Kannala, J., Blaschko, M. B., & Vedaldi, A. (2013) · HAL · doi:10.48550/arxiv.1306.5151 · Cited by 1,250
The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization
Hendrycks, D., Basart, S., Mu, N., Kadavath, S., Wang, F., Dorundo, E., Desai, R., Zhu, T., Parajuli, S., Guo, M., et al. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00823 · Cited by 1,250 · cited 2×
“1% on R (Hendrycks et al., 2021a) and +23 .”
Exploring the Limits of Weakly Supervised Pretraining
Mahajan, D., Girshick, R., Ramanathan, V., He, K., Paluri, M., Li, Y., Bharambe, A., & van der Maaten, L. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01216-8_12 · Cited by 1,224 · cited 1×
“Most promising efforts towards these foundation models focus on text-guided pretraining, i.e., using a form of textual supervision to guide the training of the features (Joulin et al., 2016; Mahajan et al., 2018; Radford et al., Figure 1: Visualization of the first PCA components.”
BackgroundFlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
Dao, T., Fu, D. Y., Ermon, S., Rudra, A., & Ré, C. (2022) · arXiv · doi:10.48550/arxiv.2205.14135 · Cited by 1,214 · cited 1×
“We implemented our own version of FlashAttention (Dao et al., 2022) to improve memory usage and speed on the self-attention layers.”
MethodHendrycks, D., Zhao, K., Basart, S., Steinhardt, J., & Song, D. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01501 · Cited by 1,090 · cited 2×
“1% on R (Hendrycks et al., 2021a) and +23 .”
InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutions
Wang, W., Dai, J., Chen, Z., Huang, Z., Li, Z., Zhu, X., Hu, X., Lu, T., Lu, L., Li, H., et al. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01385 · Cited by 1,080
LAION-5B: An open large-scale dataset for training next generation image-text models
Schuhmann, C., Beaumont, R., Vencu, R., Gordon, C., Wightman, R., Cherti, M., Coombes, T., Katta, A., Mullis, C., Wortsman, M., et al. (2022) · arXiv · doi:10.48550/arxiv.2210.08402 · Cited by 1,022
Data-Efficient Image Recognition with Contrastive Predictive Coding
Hénaff, O. J., Srinivas, A., De Fauw, J., Razavi, A., Doersch, C., Eslami, S. M. A., & Oord, A. V. D. (2019) · arXiv · doi:10.48550/arxiv.1905.09272 · Cited by 1,014
Vision Transformer Adapter for Dense PredictionsPreprint
Chen, Z., Duan, Y., Wang, W., He, J.-J., Lu, T., Dai, J.-F., & Qiao, Y. (2022) · International Conference on Learning Representations · doi:10.48550/arxiv.2205.08534 · Cited by 959 · cited 2×
“For reference, using the Mask2Former pipeline (Steiner et al., 2021) with a ViT-Adapter (Chen et al., 2023b) on top of our frozen ViT-g/14 backbone gives 60.2 mIoU on ADE-20k.”
MethodInfluentialTraining Compute-Optimal Large Language Models
Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. D. L., Hendricks, L. A., Welbl, J., Clark, A., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2176 · Cited by 932
Training Compute-Optimal Large Language Models
Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. D. L., Hendricks, L. A., Welbl, J., Clark, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.15556 · Cited by 929 · cited 1×
“Learning task-agnostic pretrained representations have become the standard in Natural Language Processing (NLP) (Radford et al., 2019; Raffel et al., 2020; Chowdhery et al., 2022; Hoffmann et al., 2022; Touvron et al., 2023).”
BackgroundVideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training
Tong, Z., Song, Y., Wang, J., & Wang, L. (2022) · arXiv · doi:10.48550/arxiv.2203.12602 · Cited by 909 · cited 2×
“This property of MAEs has been further validated on video (Tong et al., 2022), audio (Xu et al., 2022), and across other modalities (Girdhar et al., 2023).”
BackgroundZhai, X., Kolesnikov, A., Houlsby, N., & Beyer, L. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01179 · Cited by 891 · cited 1×
“As a consequence, our ViT-g architecture slightly differs from the architecture proposed by Zhai et al. (2022) in order to maximize compute efficiency, and we use an embedding dimension of 1536 with 24 heads (64 dim/head), rather than 1408 with 16 heads (88 dim/head).”
MethodAdaBins: Depth Estimation Using Adaptive Bins
Bhat, S. F., Alhashim, I., & Wonka, P. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00400 · Cited by 862 · cited 1×
“(2021). lin.”
Do ImageNet Classifiers Generalize to ImageNet?
Recht, B., Roelofs, R., Schmidt, L., & Shankar, V. (2019) · arXiv · doi:10.48550/arxiv.1902.10811 · Cited by 790
Discriminative Unsupervised Feature Learning with Exemplar Convolutional Neural Networks
Dosovitskiy, A., Fischer, P., Springenberg, J. T., Riedmiller, M., & Brox, T. (2016) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2015.2496141 · Cited by 762
Particular object retrieval with integral max-pooling of CNN activations
Tolias, G., Sicre, R., & Jégou, H. (2015) · arXiv · doi:10.48550/arxiv.1511.05879 · Cited by 744 · cited 1×
“Our dataset construction borrows from the image retrieval community (Wein-zaepfel et al., 2021; Radenović et al., 2018b; Berman et al., 2019; Douze et al., 2009; Tolias et al., 2016; Revaud et al., 2019).”
MethodSelf-citationReproducible Scaling Laws for Contrastive Language-Image Learning
Cherti, M., Beaumont, R., Wightman, R., Wortsman, M., Ilharco, G., Gordon, C., Schuhmann, C., Schmidt, L., & Jitsev, J. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.00276 · Cited by 673 · cited 1×
“Second, we report performance of open-source weakly-supervised models such as CLIP (Radford et al., 2021), OpenCLIP (Ilharco et al., 2021; Cherti et al., 2023), and SWAG (Singh et al., 2022).”
MethodEVA: Exploring the Limits of Masked Visual Representation Learning at Scale
Fang, Y., Wang, W., Xie, B., Sun, Q., Wu, L., Wang, X., Huang, T., Wang, X., & Cao, Y. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01855 · Cited by 655 · cited 2×
“The dataset used for training EVA-CLIP is a custom mixture, see paper for details (Fang et al., 2023).”
Method(1997) · SZTAKI Publication Repository (Hungarian Academy of Sciences) · Cited by 598
SELF-LABELLING VIA SIMULTANEOUS CLUSTERING AND REPRESENTATION LEARNINGPreprint
Asano, Y. M., Rupprecht, C., & Vedaldi, A. (2019) · Oxford University Research Archive (ORA) (University of Oxford) · doi:10.48550/arxiv.1911.05371 · Cited by 523
Carbon Emissions and Large Neural Network Training
Patterson, D., Gonzalez, J., Le, Q., Liang, C., Munguia, L.-M., Rothchild, D., So, D., Texier, M., & Dean, J. (2021) · arXiv · doi:10.48550/arxiv.2104.10350 · Cited by 486
Billion-scale semi-supervised learning for image classificationPreprint
Yalnız, İ. Z., Jeǵou, H., Chen, K., Paluri, M., & Mahajan, D. K. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.00546 · Cited by 478 · cited 2×
“This idea has become prevalent with the work of Doersch et al. (2015), where they train by predicting the context of a given patch.”
BackgroundMethodSelf-citationCCNet: Extracting High Quality Monolingual Datasets from Web Crawl DataPreprint
Wenzek, G., Lachaux, M.-A., Conneau, A., Chaudhary, V., Guzmán, F., Joulin, A., & Grave, É. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.00359 · Cited by 473 · cited 5×
“Our approach is inspired by text curation pipelines (Wenzek et al., 2020), where a language model is trained on Wikipedia to score texts extracted from an uncurated source.”
MethodInfluentialSelf-citationSelf-Supervised Learning from Images with a Joint-Embedding Predictive Architecture
Assran, M., Duval, Q., Misra, I., Bojanowski, P., Vincent, P., Rabbat, M., LeCun, Y., & Ballas, N. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01499 · Cited by 456 · cited 2×
“Our distillation method ends up close to the one described by Duval et al. (2023), except we do not modify the loss terms for distillation and evaluate the EMA of the student.”
BackgroundMethodSelf-citationLearning Correspondence From the Cycle-Consistency of Time
Wang, X., Jabri, A., & Efros, A. A. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00267 · Cited by 452 · cited 1×
“0% on Sketch (Wang et al., 2019) (2018) respectively – are mentioned at the top of the Table.”
BackgroundRevisiting Oxford and Paris: Large-Scale Image Retrieval Benchmarking
Radenović, F., İşcen, A., Tolias, G., Avrithis, Y., & Chum, O. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00598 · Cited by 426 · cited 1×
“Our dataset construction borrows from the image retrieval community (Wein-zaepfel et al., 2021; Radenović et al., 2018b; Berman et al., 2019; Douze et al., 2009; Tolias et al., 2016; Revaud et al., 2019).”
MethodTouvron, H., Cord, M., & Jégou, H. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-20053-3_30 · Cited by 409
Learning With Average Precision: Training Image Retrieval With a Listwise Loss
Revaud, J., Almazan, J., Rezende, R., & Souza, C. D. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00521 · Cited by 408 · cited 1×
“Our dataset construction borrows from the image retrieval community (Wein-zaepfel et al., 2021; Radenović et al., 2018b; Berman et al., 2019; Douze et al., 2009; Tolias et al., 2016; Revaud et al., 2019).”
MethodEvaluation of GIST descriptors for web-scale image search
Douze, M., Jeǵou, H., Sandhawalia, H., Amsaleg, L., & Schmid, C. (2009) · ACM International Conference on Image and Video Retrieval (CIVR) · doi:10.1145/1646396.1646421 · Cited by 394 · cited 1×
“Our dataset construction borrows from the image retrieval community (Wein-zaepfel et al., 2021; Radenović et al., 2018b; Berman et al., 2019; Douze et al., 2009; Tolias et al., 2016; Revaud et al., 2019).”
MethodSelf-citationScaling and Benchmarking Self-Supervised Visual Representation Learning
Goyal, P., Mahajan, D. K., Gupta, A., & Misra, I. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00649 · Cited by 361 · cited 1×
“A growing body of work has focused on the scaling abilities of self-supervised learning in terms of data and model size (Caron et al., 2019; Goyal et al., 2019; Tian et al., 2021; Goyal et al., 2022a).”
BackgroundSelf-citationLAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs
Schuhmann, C., Vencu, R., Beaumont, R., Kaczmarczyk, R., Mullis, C., Katta, A., Coombes, T., Jitsev, J., & Komatsuzaki, A. (2021) · arXiv · doi:10.48550/arxiv.2111.02114 · Cited by 358 · cited 1×
“Similarly, others have used hashtags or other metadata (Ma-hajan et al., 2018; Radford et al., 2021) or pretrained vision encoders (Schuhmann et al., 2021; 2022) to filter uncurated datasets.”
BackgroundGoogle Landmarks Dataset v2 – A Large-Scale Benchmark for Instance-Level Recognition and Retrieval
Weyand, T., Araújo, A. D. M., Cao, B., & Sim, J. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00265 · Cited by 341
Learning to Generate Reviews and Discovering Sentiment
Radford, A., Jozefowicz, R., & Sutskever, I. (2017) · arXiv · doi:10.48550/arxiv.1704.01444 · Cited by 330 · cited 1×
“This success has been fueled by pretraining on large quantities of raw text using pretext objectives, such as language modeling (Radford et al., 2017) or word vectors (Devlin et al., 2019), that require no supervision.”
BackgroundLAION-5B: An Open Large-Scale Dataset for Training Next Generation Image-Text Models
Schuhmann, C., Beaumont, R., Vencu, R., Gordon, C., Wightman, R., Cherti, M., Coombes, T., Katta, A., Mullis, C., Wortsman, M., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1833 · Cited by 325
Learning Visual Features from Large Weakly Supervised Data
Joulin, A., van der Maaten, L. J. P., Jabri, A., & Vasilache, N. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46478-7_5 · Cited by 316 · cited 1×
“Most promising efforts towards these foundation models focus on text-guided pretraining, i.e., using a form of textual supervision to guide the training of the features (Joulin et al., 2016; Mahajan et al., 2018; Radford et al., Figure 1: Visualization of the first PCA components.”
BackgroundSelf-citationMasked Siamese Networks for Label-Efficient Learning
Assran, M., Caron, M., Misra, I., Bojanowski, P., Bordes, F., Vincent, P., Joulin, A., Rabbat, M., & Ballas, N. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19821-2_26 · Cited by 303
Self-citationSymbolic Discovery of Optimization Algorithms
Chen, X., Liang, C., Huang, D., Real, E., Wang, K., Liu, Y., Pham, H., Dong, X., Luong, T., Hsieh, C.-J., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.06675 · Cited by 303 · cited 1×
“1%), obtained by Chen et al. (2023a).”
GLU Variants Improve Transformer
Shazeer, N. (2020) · arXiv · doi:10.48550/arxiv.2002.05202 · Cited by 303
BinsFormer: Revisiting Adaptive Bins for Monocular Depth Estimation
Li, Z., Wang, X., Liu, X., & Jiang, J. (2024) · IEEE Transactions on Image Processing · doi:10.1109/tip.2024.3416065 · Cited by 301
How to train your ViT? Data, Augmentation, and Regularization in Vision Transformers
Steiner, A., Kolesnikov, A., Zhai, X., Wightman, R., Uszkoreit, J., & Beyer, L. (2021) · arXiv · doi:10.48550/arxiv.2106.10270 · Cited by 292 · cited 1×
“For reference, using the Mask2Former pipeline (Steiner et al., 2021) with a ViT-Adapter (Chen et al., 2023b) on top of our frozen ViT-g/14 backbone gives 60.2 mIoU on ADE-20k.”
MethodInfluentialUnsupervised Pre-Training of Image Features on Non-Curated Data
Caron, M., Bojanowski, P., Mairal, J., & Joulin, A. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00305 · Cited by 258 · cited 2×
“A growing body of work has focused on the scaling abilities of self-supervised learning in terms of data and model size (Caron et al., 2019; Goyal et al., 2019; Tian et al., 2021; Goyal et al., 2022a).”
BackgroundSelf-citationUnsupervised Semantic Segmentation by Distilling Feature CorrespondencesPreprint
Hamilton, M., Zhang, Z., Hariharan, B., Snavely, N., & Freeman, W. T. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2203.08414 · Cited by 252 · cited 1×
“Additionally, the features output by self-supervised models have been shown to exhibit various useful properties, and have enabled enabled a wide variety of applications (Amir et al., 2022; Tumanyan et al., 2022; Ofri-Amar et al., 2023; Hamilton et al., 2022).”
BackgroundMapillary Street-Level Sequences: A Dataset for Lifelong Place Recognition
Warburg, F., Hauberg, S., López-Antequera, M., Gargallo, P., Kuang, Y., & Civera, J. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00270 · Cited by 235
Unsupervised Learning by Predicting Noise
Bojanowski, P., & Joulin, A. (2017) · arXiv · doi:10.48550/arxiv.1704.05310 · Cited by 228 · cited 1×
“This family of methods has roots in early deep learning work (Hadsell et al., 2006) but became popular with the emergence of instance classification methods (Dosovitskiy et al., 2016; Bojanowski & Joulin, 2017; Wu et al., 2018).”
MethodSelf-citationMasked Autoencoders that Listen
Huang, P.-Y., Xu, H., Li, J., Baevski, A., Auli, M., Galuba, W., Metze, F., & Feichtenhofer, C. (2022) · arXiv · doi:10.48550/arxiv.2207.06405 · Cited by 206 · cited 1×
“, 2022), audio (Xu et al., 2022), and across other modalities (Girdhar et al.”
BackgroundSelf-citationMasked Autoencoders that Listen
Huang, P.-Y., Xu, H., Li, J., Baevski, A., Auli, M., Galuba, W., Metze, F., & Feichtenhofer, C. (2022) · neural information processing systems · doi:10.52202/068431-2081 · Cited by 206
Self-citationBeyer, L., Hénaff, O. J., Kolesnikov, A., Zhai, X., & Oord, A. V. D. (2020) · arXiv · doi:10.48550/arxiv.2006.07159 · Cited by 176
Polarized Self-Attention: Towards High-quality Pixel-wise RegressionPreprint
Liu, H., Liu, F., Fan, X., & Huang, D. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.00782 · Cited by 172
Efficient Self-supervised Vision Transformers for Representation LearningPreprint
Li, C., Yang, J., Zhang, P., Gao, M., Xiao, B., Dai, X., Yuan, L., & Gao, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.09785 · Cited by 164 · cited 3×
“First, we run our evaluations for MAE (He et al., 2022), DINO (Caron et al., 2021), SEERv2 (Goyal et al., 2022a), MSN (Assran et al., 2022), EsViT (Li et al., 2022a), Mugs (Zhou et al., 2022b) and iBOT (Zhou et al., 2022a).”
MethodResultSplicing ViT Features for Semantic Appearance Transfer
Tumanyan, N., Bar-Tal, O., Bagon, S., & Dekel, T. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01048 · Cited by 153 · cited 1×
“Additionally, the features output by self-supervised models have been shown to exhibit various useful properties, and have enabled enabled a wide variety of applications (Amir et al., 2022; Tumanyan et al., 2022; Ofri-Amar et al., 2023; Hamilton et al., 2022).”
BackgroundAre Large-scale Datasets Necessary for Self-Supervised Pre-training?Preprint
El-Nouby, A., Izacard, G., Touvron, H., Laptev, I., Jeǵou, H., & Grave, É. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2112.10740 · Cited by 150
Self-citationSelf-supervised Pretraining of Visual Features in the Wild
Goyal, P., Caron, M., Lefaudeux, B., Xu, M., Wang, P., Pai, V., Singh, M., Liptchinsky, V., Misra, I., Joulin, A., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.01988 · Cited by 141 · cited 2×
“Of particular interest, Goyal et al. (2021) have also shown that these methods benefit from scaling in model size given enough pretrained data.”
BackgroundSelf-citationDoes Object Recognition Work for Everyone?Preprint
DeVries, T., Misra, I., Wang, C., & van der Maaten, L. J. P. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.02659 · Cited by 141 · cited 1×
“We probe for geographical fairness and potential harmful label associations.”
BackgroundSelf-citationFlexiViT: One Model for All Patch Sizes
Beyer, L., Izmailov, P., Kolesnikov, A., Caron, M., Kornblith, S., Zhai, X., Minderer, M., Tschannen, M., Alabdulmohsin, I., & Pavetic, F. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01393 · Cited by 132 · cited 1×
“This is also similar to UniViT training from Likhomanenko et al. (2021) and FlexiViT training from Beyer et al. (2023).”
MethodKrell, M. M., Kosec, M., Perez, S. P., & Fitzgibbon, A. (2021) · arXiv · doi:10.48550/arxiv.2107.02027 · Cited by 122
A Self-Supervised Descriptor for Image Copy Detection
Pizzi, E., Roy, S. D., Ravindra, S. N., Goyal, P., & Douze, M. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01413 · Cited by 101
OmniMAE: Single Model Masked Pretraining on Images and Videos
Girdhar, R., El-Nouby, A., Singh, M., Alwala, K. V., Joulin, A., & Misra, I. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01003 · Cited by 101 · cited 1×
“This property of MAEs has been further validated on video (Tong et al., 2022), audio (Xu et al., 2022), and across other modalities (Girdhar et al., 2023).”
BackgroundSelf-citationRevisiting Weakly Supervised Pre-Training of Visual Perception Models
Singh, M., Gustafson, L., Adcock, A., De Freitas Reis, V., Gedik, B., Kosaraju, R. P., Mahajan, D., Girshick, R., Dollar, P., & Van Der Maaten, L. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00088 · Cited by 99
Towards Measuring Fairness in AI: The Casual Conversations Dataset
Hazirbas, C., Bitton, J., Dolhansky, B., Pan, J., Gordo, A., & Ferrer, C. C. (2022) · IEEE Transactions on Biometrics, Behavior, and Identity Science · doi:10.1109/tbiom.2021.3132237 · Cited by 79 · cited 2×
“Using this classifier, we run inference on 2955 images from the Casual Conversations dataset (Hazirbas et al., 2021) and keep all labels in the top-5 that are assigned a probability of 0.1 or more.”
BackgroundMethodInfluentialMugs: A Multi-Granular Self-Supervised Learning FrameworkPreprint
Zhou, P., Zhou, Y., Si, C.-Y., Yu, W., Ng, T. K., & Yan, S. (2022) · arXiv.org · doi:10.48550/arxiv.2203.14415 · Cited by 78 · cited 7×
“• Patch-level objective (Zhou et al., 2022a).”
BackgroundMethodInfluentialBinsFormer: Revisiting Adaptive Bins for Monocular Depth EstimationPreprint
Li, Z., Wang, X., Liu, X., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2204.00987 · Cited by 71 · cited 3×
“First, we run our evaluations for MAE (He et al., 2022), DINO (Caron et al., 2021), SEERv2 (Goyal et al., 2022a), MSN (Assran et al., 2022), EsViT (Li et al., 2022a), Mugs (Zhou et al., 2022b) and iBOT (Zhou et al., 2022a).”
MethodResultDivide and Contrast: Self-supervised Learning from Uncurated Data
Tian, Y., Hénaff, O. J., & van den Oord, A. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00991 · Cited by 69 · cited 1×
“A growing body of work has focused on the scaling abilities of self-supervised learning in terms of data and model size (Caron et al., 2019; Goyal et al., 2019; Tian et al., 2021; Goyal et al., 2022a).”
BackgroundMultiGrain: a unified image embedding for classes and instances
Berman, M., Jégou, H., Vedaldi, A., Kokkinos, I., & Douze, M. (2019) · arXiv · doi:10.48550/arxiv.1902.05509 · Cited by 69
Spreading vectors for similarity search
Sablayrolles, A., Douze, M., Schmid, C., & Jégou, H. (2018) · arXiv · doi:10.48550/arxiv.1806.03198 · Cited by 66 · cited 1×
“• KoLeo regularizer (Sablayrolles et al., 2019).”
Self-citationOn the Kozachenko–Leonenko entropy estimator
Delattre, S., & Fournier, N. (2017) · Journal of Statistical Planning and Inference · doi:10.1016/j.jspi.2017.01.004 · Cited by 50 · cited 1×
“The KoLeo regularizer derives from the Kozachenko-Leonenko differential entropy estimator (see Beirlant et al. (1997); Delattre & Fournier (2017)) and encourages a uniform span of the features within a batch.”
MethodVision Models Are More Robust And Fair When Pretrained On Uncurated Images Without SupervisionPreprint
Goyal, P., Duval, Q., Seessel, I., Caron, M., Misra, I., Sagun, L., Joulin, A., & Bojanowski, P. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2202.08360 · Cited by 47 · cited 6×
“We compare our model to SEERv2 in Table 13.”
BackgroundMethodInfluentialSelf-citationAmsterTime: A Visual Place Recognition Benchmark Dataset for Severe Domain Shift
Yildiz, B., Khademi, S., Siebes, R., & van Gemert, J. C. (2022) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr56361.2022.9956049 · Cited by 38
Learning Super-Features for Image Retrieval
Weinzaepfel, P., Lucas, T., Larlus, D., & Kalantidis, Y. (2022) · arXiv · doi:10.48550/arxiv.2201.13182 · Cited by 29 · cited 1×
“Our dataset construction borrows from the image retrieval community (Weinzaepfel et al., 2021; Radenović et al., 2018b; Berman et al., 2019; Douze et al., 2009; Tolias et al., 2015; Revaud et al., 2019).”
MethodFairness indicators for systematic assessments of visual feature extractors
Goyal, P., Soriano, A. R., Hazirbas, C., Sagun, L., & Usunier, N. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533074 · Cited by 24 · cited 9×
“We follow the protocol of Goyal et al. (2022b), where we train a multiclass classifier on a subset of 619 classes of ImageNet-22k.”
BackgroundMethodResultInfluentialNeural Congealing: Aligning Images to a Joint Semantic Atlas
Ofri-Amar, D., Geyer, M., Kasten, Y., & Dekel, T. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01859 · Cited by 23
Benchmarking Representation Learning for Natural World Image Collections
Horn, G. V., Cole, E., Beery, S., Wilber, M. J., Belongie, S., & Aodha, O. M. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01269 · Cited by 22
CAPE: Encoding Relative Positions with Continuous Augmented Positional EmbeddingsPreprint
Likhomanenko, T., Xu, Q., Synnaeve, G., Collobert, R., & Rogozhnikov, A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.03143 · Cited by 20 · cited 1×
“This is also similar to UniViT training from Likhomanenko et al. (2021) and FlexiViT training from Beyer et al. (2023).”
MethodSelf-citationThe Met Dataset: Instance-level Recognition for ArtworksPreprint
Ypsilantis, N.-A., García, N., Han, G., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2202.01747 · Cited by 15
Ji, Y., & Gao, S. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.giscience.2023.43 · Cited by 10
Weighted Ensemble Self-Supervised Learning
Ruan, Y., Singh, S., Morningstar, W., Alemi, A. A., Ioffe, S., Fischer, I., & Dillon, J. V. (2022) · arXiv · doi:10.48550/arxiv.2211.09981 · Cited by 7 · cited 1×
“We also remove near-duplicates of images contained in the test or validation set of any benchmark used in this work.”
MethodA Simple Recipe for Competitive Low-compute Self supervised Vision ModelsPreprint
Duval, Q., Misra, I., & Ballas, N. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2301.09451 · Cited by 3
Self-citationDeep ViT Features as Dense Visual Descriptors
Amir, S., Gandelsman, Y., Bagon, S., & Dekel, T. (2021) · arXiv · doi:10.48550/arxiv.2112.05814
Research works printed but not linked: 2
- [62]62. Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial exam- ples. In CVPR, 2021b.
- [113]113. Hugo Touvron, Andrea Vedaldi, Matthijs Douze, and Hervé Jégou. Fixing the train-test resolution discrep- ancy. In NeurIPS, 2019.
Cited by
Sorted by citations
Towards a general-purpose foundation model for computational pathology
Chen, R. J., Ding, T., Lu, M. Y., Williamson, D. F. K., Jaume, G., Song, A. H., Chen, B., Zhang, A., Shao, D., Shaban, M., et al. (2024) · Nature Medicine · doi:10.1038/s41591-024-02857-3 · Cited by 1,560
Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data
Yang, L., Kang, B., Huang, Z., Xu, X., Feng, J., & Zhao, H. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.00987 · Cited by 1,296
“Furthermore, to strengthen the teacher model T learned from these labeled images, we adopt the DINOv2 [42] pre-trained weights to initialize our encoder.”
BackgroundMethodInfluentialA whole-slide foundation model for digital pathology from real-world data
Xu, H., Usuyama, N., Bagga, J., Zhang, S., Rao, R., Naumann, T., Wong, C., Gero, Z., González, J., Gu, Y., et al. (2024) · Nature · doi:10.1038/s41586-024-07441-w · Cited by 921
“Prov-GigaPath tile encoder used the ViT model architecture with standard DINOv2 settings 24 .”
InfluentialSelf-citationEnd-to-End Autonomous Driving: Challenges and Frontiers
Chen, L., Wu, P., Chitta, K., Jaeger, B., Geiger, A., & Li, H. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3435937 · Cited by 843
“Recent advancements in foundation models in both language [166], [167], [274] and vision [273], [275], [276] have proved that large-scale data and model capacity can unleash the immense potential of AI in high-level reasoning tasks.”
BackgroundA foundation model for clinical-grade computational pathology and rare cancers detection
Vorontsov, E., Bozkurt, A., Casson, A., Shaikovski, G., Zelechowski, M., Severson, K., Zimmermann, E., Hall, J., Tenenholtz, N., Fusi, N., et al. (2024) · Nature Medicine · doi:10.1038/s41591-024-03141-0 · Cited by 572
“33 with the following changes: a teacher temperature schedule of 0.04–0.07 in 186,000 iterations and a reciprocal square root learning rate schedule with a warmup of 495,000 iterations (instead of 100,000) and linear cooldown to 0.0 for the last 819,200 iterations 29 .”
Influential
Metadata
- Authors
- Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El-Nouby, A., Assran, M., Ballas, N., Galuba, W., Howes, R., Huang, P.-Y., Li, S.-W., Misra, I., Rabbat, M., Sharma, V., Synnaeve, G., Xu, H., Jegou, H., Mairal, J., Labatut, P., Joulin, A., & Bojanowski, P.
- Affiliations
- Institut national de recherche en sciences et technologies du numérique (FR)
- ORCID
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2304.07193
- OpenAlex
- W4366208220
- Cited by
- 1,078
- References
- 131
- Citations by year
- 2026: 123 · 2025: 404 · 2024: 485 · 2023: 66
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- HAL (submittedVersion, cc-by)
- HAL (submittedVersion, other-oa)
- arXiv (Cornell University)
- Funders
- Agence Nationale de la Recherche (ANR-19-P3IA-0003, ANR‐19)
- HORIZON EUROPE European Research Council (101087696)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- DINOv2, self-supervised pre-training, vision transformer, foundation models, knowledge distillation, large-scale training, image classification benchmark
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
