Big Transfer (BiT): General Visual Representation Learning
Open accessThe claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The absolute numbers for ResNets pre-trained on JFT for 7 epochs are lower than those reported by Kolesnikov et al. (2020) because they pre-trained for 30 epochs.
“Note that the absolute numbers are lower than those reported by Kolesnikov et al. (2020), since we pre-train only for 7 epochs, not 30 .”
From An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (2020) · cites this paper as Kolesnikov et al. (2020)
Section 3.1 (p. 4) identifies BiT-L as the model pre-trained on JFT-300M. Section 3.3 (p. 5) says BiT-L was pre-trained for 40 epochs, not 30; 30 was a learning-rate decay point. (LLM, conf=0.90)
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredNoisy Student is currently the state of the art on ImageNet and BiT-L on the other datasets reported here.
“Currently, Noisy Student is the state of the art on ImageNet and BiT-L on the other datasets reported here.”
From An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (2020) · cites this paper as Kolesnikov et al., 2020
Table 1 (section 3.2, p. 4) lists a specialist result of 88.4% on ImageNet (ILSVRC-2012), above BiT-L’s 87.54%; reference [61] (p. 18) identifies that work as Noisy Student. BiT-L exceeds the listed results on CIFAR-10, CIFAR-100, Flowers, and VTAB, but not Pets, where the listed specialist result is 97.1% versus BiT-L’s 96.62%. Thus the claim about all the other reported datasets overstates the table. (LLM, conf=0.90)
Experimental — accuracy not yet measuredRunning fine-tuning at a different resolution than training is common practice.
“(running fine-tuning at different resolution than training is common practice (Kolesnikov et al., 2020)).”
From An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (2020) · cites this paper as Kolesnikov et al., 2020
Section 2.2 (p. 3) says the authors change resolution during fine-tuning. It calls scaling up resolution at test time common practice, but describes fine-tuning to the test resolution as an alternative; it does not say that fine-tuning at a different resolution is itself common practice. (LLM, conf=0.87)
Experimental — accuracy not yet measuredUsing a larger dataset during pre-training may be acting as a regularizer for ViT and BiT's low sensitivity.
“Using a larger dataset during pre-training may be acting as a regularizer here (Kolesnikov et al. 2020).”
From Vision Transformers Are Robust Learners (2022) · cites this paper as Kolesnikov et al. 2020
Section 2.2 (p. 3) says MixUp was not useful during BiT pre-training, likely because data were abundant. That is consistent with the tentative idea that more pre-training data can reduce the need for regularization, but the paper does not say that dataset size acts as a regularizer or explains low sensitivity in ViT and BiT. (LLM, conf=0.78)
Experimental — accuracy not yet measuredThe modifications to ResNet (Group Normalization and standardized convolutions) improve transfer.
“These modifications improve transfer (Kolesnikov et al., 2020), and we denote the modified model 'ResNet (BiT)'.”
From An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (2020) · cites this paper as Kolesnikov et al., 2020
In large-scale image recognition, classic ResNet-like architectures are still state of the art.
“Therefore, in large-scale image recognition, classic ResNetlike architectures are still state of the art (Mahajan et al., 2018; Xie et al., 2020; Kolesnikov et al., 2020).”
From An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (2020) · cites this paper as Kolesnikov et al., 2020
The pre-training datasets were de-duplicated with respect to the test sets of the downstream tasks following Kolesnikov et al. (2020).
“We de-duplicate the pre-training datasets w.r.t. the test sets of the downstream tasks following Kolesnikov et al. (2020).”
From An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (2020) · cites this paper as Kolesnikov et al. (2020)
Azizi et al. found that using generic and large-scale supervised pre-trained models, such as BigTransfer, can benefit subsequent domain-specific self-supervised pre-training and ultimately improve model performance and robustness for different medical imaging modalities.
“More recently, Azizi et al. found that using generic and large-scale supervised pre-trained models, such as BigTransfer94, can also benefit subsequent domain-specific self-supervised pre-training, and ultimately improve model performance and robustness for different medical imaging modalities95.”
From Self-supervised learning for medical image classification: a systematic review and implementation guidelines (2023) · cites this paper as [94]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · see citing works
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · see citing works
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 74,248 · see citing works
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380 · see citing works
Focal Loss for Dense Object Detection
Lin, T.-Y., Goyal, P., Girshick, R., He, K., & Dollar, P. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.324 · Cited by 30,581 · see citing works
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S., & Szegedy, C. (2015) · arXiv · doi:10.48550/arxiv.1502.03167 · Cited by 27,818 · see citing works
Pan, S. J., & Yang, Q. (2010) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2009.191 · Cited by 23,387 · see citing works
Xception: Deep Learning with Depthwise Separable Convolutions
Chollet, F. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.195 · Cited by 20,380 · see citing works
EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
Tan, M., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1905.11946 · Cited by 17,072 · see citing works
Momentum Contrast for Unsupervised Visual Representation LearningPreprint
He, K., Fan, H., Wu, Y., Xie, S., & Girshick, R. B. (2019) · Computer Vision and Pattern Recognition · doi:10.48550/arxiv.1911.05722 · Cited by 16,250 · see citing works
Distilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204 · see citing works
CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features
Yun, S., Han, D., Oh, S. J., Chun, S., Choe, J., & Yoo, Y. (2019) · arXiv · doi:10.48550/arxiv.1905.04899 · Cited by 5,439 · see citing works
SGDR: Stochastic Gradient Descent with Warm Restarts
Loshchilov, I., & Hutter, F. (2016) · arXiv · doi:10.48550/arxiv.1608.03983 · Cited by 5,309 · see citing works
Automated Flower Classification over a Large Number of Classes
Nilsback, M.-E., & Zisserman, A. (2008) · 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing · doi:10.1109/icvgip.2008.47 · Cited by 3,442 · see citing works
FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence
Sohn, K., Berthelot, D., Li, C.-L., Zhang, Z., Carlini, N., Cubuk, E. D., Kurakin, A., Zhang, H., & Raffel, C. (2020) · arXiv · doi:10.48550/arxiv.2001.07685 · Cited by 2,892 · see citing works
Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
Goyal, P., Dollár, P., Girshick, R., Noordhuis, P., Wesolowski, L., Kyrola, A., Tulloch, A., Jia, Y., & He, K. (2017) · arXiv · doi:10.48550/arxiv.1706.02677 · Cited by 2,294 · see citing works
Kuznetsova, A., Rom, H., Alldrin, N., Uijlings, J., Krasin, I., Pont-Tuset, J., Kamali, S., Popov, S., Malloci, M., Kolesnikov, A., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.00982 · Cited by 2,194 · see citing works
Self-citationScaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · Cited by 1,749 · cited 2× · see citing works
“Further, it is recognized that larger datasets require larger architectures to realize benefits, and vice versa [25,45].”
BackgroundGPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism
Huang, Y., Cheng, Y., Bapna, A., Firat, O., Chen, M. X., Chen, D., Lee, H., Ngiam, J., Le, Q. V., Wu, Y., et al. (2018) · arXiv · doi:10.48550/arxiv.1811.06965 · Cited by 1,180 · see citing works
Data-Efficient Image Recognition with Contrastive Predictive Coding
Hénaff, O. J., Srinivas, A., De Fauw, J., Razavi, A., Doersch, C., Eslami, S. M. A., & Oord, A. V. D. (2019) · arXiv · doi:10.48550/arxiv.1905.09272 · Cited by 1,014 · see citing works
Transfusion: Understanding Transfer Learning for Medical ImagingPreprint
Raghu, M., Zhang, C., Kleinberg, J. M., & Samy Bengio, A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.07208 · Cited by 845 · see citing works
ReMixMatch: Semi-Supervised Learning with Distribution Alignment and Augmentation AnchoringPreprint
Berthelot, D., Carlini, N., Cubuk, E., Kurakin, A., Sohn, K., Zhang, H., & Raffel, C. A. (2019) · arXiv.org · doi:10.48550/arxiv.1911.09785 · Cited by 823 · see citing works
Averaging Weights Leads to Wider Optima and Better GeneralizationPreprint
Izmailov, P., Podoprikhin, D. A., Garipov, T., Vetrov, D., & Wilson, A. G. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1803.05407 · Cited by 818 · see citing works
Billion-scale semi-supervised learning for image classificationPreprint
Yalnız, İ. Z., Jeǵou, H., Chen, K., Paluri, M., & Mahajan, D. K. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.00546 · Cited by 387 · see citing works
A Large-scale Study of Representation Learning with the Visual Task Adaptation BenchmarkPreprint
Zhai, X., Puigcerver, J., Kolesnikov, A., Ruyssen, P., Riquelme, C., Lučić, M., Djolonga, J., Pinto, A. S., Neumann, M., Dosovitskiy, A., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.04867 · Cited by 249 · see citing works
Self-citationSelf-training with Noisy Student improves ImageNet classificationPreprint
Xie, Q., Luong, M.-T., Hovy, E. H., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.04252 · Cited by 237 · see citing works
Micro-Batch Training with Batch-Channel Normalization and Weight Standardization
Qiao, S., Wang, H., Liu, C., Shen, W., & Yuille, A. (2019) · arXiv · doi:10.48550/arxiv.1903.10520 · Cited by 126 · see citing works
Domain Adaptive Transfer Learning with Specialist ModelsPreprint
Ngiam, J., Peng, D., Vasudevan, V., Kornblith, S., Le, Q. V., & Pang, R. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.07056 · Cited by 106 · see citing works
Do We Train on Test Data? Purging CIFAR of Near-Duplicates
Barz, B., & Denzler, J. (2020) · Journal of Imaging · doi:10.3390/jimaging6060041 · Cited by 95 · see citing works
YFCC100M: The New Data in Multimedia Research
Thomee, B., Shamma, D. A., Friedland, G., Elizalde, B., Ni, K., Poland, D., Borth, D., & Li, L.-J. (2015) · arXiv · doi:10.48550/arxiv.1503.01817 · Cited by 69 · see citing works
Revisiting Fine-tuning for Few-shot LearningPreprint
Nakamura, A., & Harada, T. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.00216 · Cited by 47 · see citing works
An Exponential Learning Rate Schedule for Deep LearningPreprint
Li, Z., & Arora, S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.07454 · Cited by 41 · see citing works
Recognition in Terra IncognitaPreprint
Beery, S., Horn, G. V., & Perona, P. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1807.04975 · Cited by 31 · see citing works
Weakly-supervised learning of visual relationsPreprint
Peyre, J., Ivan Laptev, A., Schmid, C., & Šivic, J. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1707.09472 · Cited by 15 · see citing works
ObjectNet Dataset: Reanalysis and CorrectionPreprint
Borji, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.02042 · Cited by 10 · see citing works
Shetty, R., Schiele, B., & Fritz, M. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1812.06707 · Cited by 3 · see citing works
Flagged first, then by citations
Research works printed but not linked: 29
- [1]1. Athiwaratkun, B., Finzi, M., Izmailov, P., Wilson, A.G.: There are many consistent explanations of unlabeled data: Why you should average. In: ICLR (2019)
- [2]2. Barbu, A., Mayo, D., Alverio, J., Luo, W., Wang, C., Gutfreund, D., Tenenbaum, J., Katz, B.: Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. In: Advances in Neural Information Processing Systems (2019)
- [7]7. Chen, W., Liu, Y., Kira, Z., Wang, Y.F., Huang, J.: A closer look at few-shot classification. In: ICLR (2019)
- [9]9. De, S., Smith, S.L.: Batch normalization has multiple benefits: An empirical study on residual networks (2020), https://openreview.net/forum?id=BJeVklHtPr
- [10]10. Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: A large-scale hierarchical image database. In: CVPR (2009)
- [12]12. Goyal, P., Doll´ ar, P., Girshick, R.B., Noordhuis, P., Wesolowski, L., Kyrola, A., Tulloch, A., Jia, Y., He, K.: Accurate, large minibatch sgd: Training imagenet in 1 hour. ArXiv abs/1706.02677 (2017) · arXiv
- [14]14. He, K., Girshick, R., Doll´ ar, P.: Rethinking imagenet pre-training. In: ICCV (2019)
- [16]16. He, K., Zhang, X., Ren, S., Sun, J.: Identity mappings in deep residual networks. In: ECCV (2016)
- [20]20. Ioffe, S.: Batch renormalization: Towards reducing minibatch dependence in batchnormalized models. In: NIPS (2017)
- [23]23. Joulin, A., van der Maaten, L., Jabri, A., Vasilache, N.: Learning visual features from large weakly supervised data. In: ECCV (2016)
- [24]24. Jouppi, N.P., Young, C., Patil, N., Patterson, D., Agrawal, G., Bajwa, R., Bates, S., Bhatia, S., Boden, N., Borchers, A., et al.: In-datacenter performance analysis of a tensor processing unit. In: International Symposium on Computer Architecture (ISCA) (2017)
- [29]29. van Laarhoven, T.: L2 regularization versus batch and weight normalization. CoRR (2017)
- [30]30. Li, A., Jabri, A., Joulin, A., van der Maaten, L.: Learning visual n-grams from web data. In: ICCV (2017)
- [31]31. Li, X., Grandvalet, Y., Davoine, F.: Explicit inductive bias for transfer learning with convolutional networks. In: ICML (2018)
- [34]34. Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Doll´ ar, P., Zitnick, C.L.: Microsoft COCO: Common objects in context. In: ECCV (2014)
- [36]36. Mahajan, D., Girshick, R., Ramanathan, V., He, K., Paluri, M., Li, Y., Bharambe, A., van der Maaten, L.: Exploring the limits of weakly supervised pretraining. In: ECCV (2018)
- [41]41. Parkhi, O.M., Vedaldi, A., Zisserman, A., Jawahar, C.V.: Cats and dogs. In: CVPR (2012)
- [45]45. Rosenfeld, J.S., Rosenfeld, A., Belinkov, Y., Shavit, N.: A constructive prediction of the generalization error across scales. In: ICLR (2020)
- [49]49. Snell, J., Swersky, K., Zemel, R.: Prototypical networks for few-shot learning. In: NIPS (2017)
- [51]51. Sun, C., Shrivastava, A., Singh, S., Gupta, A.: Revisiting unreasonable effectiveness of data in deep learning era. In: ICCV (2017)
- [52]52. Sung, F., Yang, Y., Zhang, L., Xiang, T., Torr, P.H., Hospedales, T.M.: Learning to compare: Relation network for few-shot learning. In: CVPR (2018)
- [53]53. Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., Rabinovich, A.: Going deeper with convolutions. In: CVPR (2015)
- [54]54. Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z.: Rethinking the inception architecture for computer vision. In: CVPR (2016)
- [57]57. Touvron, H., Vedaldi, A., Douze, M., J´ egou, H.: Fixing the train-test resolution discrepancy. In: NeurIPS (2019)
- [59]59. Vinyals, O., Blundell, C., Lillicrap, T., Wierstra, D., et al.: Matching networks for one shot learning. In: NIPS (2016)
- [60]60. Wu, Y., He, K.: Group normalization. In: ECCV (2018)
- [62]62. Xie, S., Girshick, R., Doll´ ar, P., Tu, Z., He, K.: Aggregated residual transformations for deep neural networks. In: CVPR (2017)
- [65]65. Zhai, X., Oliver, A., Kolesnikov, A., Beyer, L.: S 4 L: Self-Supervised SemiSupervised Learning. In: ICCV (2019)
- [67]67. Zhang, H., Cisse, M., Dauphin, Y.N., Lopez-Paz, D.: mixup: Beyond empirical risk minimization. In: ICLR (2017) Fig. 9: Blue curves display VTAB-1k score (mean accuracy across tasks) depending on the total number of random hyperparameters tested. Reported VTAB-1k scores are averaged over 100 random hyperparameter orderings, the shaded blue area indicates the standard error. Dashed gray line displays the performance on the small hold-out validation split with 200 examples. <!-- image -->
Cited by
Sorted by citations
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., & Guo, B. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00986 · Cited by 35,534 · see citing works
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.11929 · Cited by 21,793 · see citing works
“We de-duplicate the pre-training datasets w.r.t. the test sets of the downstream tasks following Kolesnikov et al. (2020).”
InfluentialSelf-citationLiu, Z., Mao, H., Wu, C.-Y., Feichtenhofer, C., Darrell, T., & Xie, S. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01167 · Cited by 8,494 · see citing works
Swin Transformer V2: Scaling Up Capacity and Resolution
Liu, Z., Hu, H., Lin, Y., Yao, Z., Xie, Z., Wei, Y., Ning, J., Cao, Y., Zhang, Z., Dong, L., et al. (2021) · arXiv · doi:10.48550/arxiv.2111.09883 · Cited by 2,679 · see citing works
Vision Transformers for Dense Prediction
Ranftl, R., Bochkovskiy, A., & Koltun, V. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01196 · Cited by 2,373 · see citing works
Pre-Trained Image Processing Transformer
Chen, H., Wang, Y., Guo, T., Xu, C., Deng, Y., Liu, Z., Ma, S., Xu, C., Xu, C., & Gao, W. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01212 · Cited by 2,308 · see citing works
Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
Jia, C., Yang, Y., Xia, Y., Chen, Y.-T., Parekh, Z., Pham, H., Le, Q. V., Sung, Y., Li, Z., & Duerig, T. (2021) · arXiv · doi:10.48550/arxiv.2102.05918 · Cited by 1,977 · see citing works
“Table 6 reports the mean accuracy (including the breakdown results on each subgroup) with standard deviation from three fine-tuning runs and shows that ALIGN outperforms BiT-L (Kolesnikov et al., 2020) with similar hyper-parameter selection method applied.”
BackgroundMethodInfluentialTowards a general-purpose foundation model for computational pathology
Chen, R. J., Ding, T., Lu, M. Y., Williamson, D. F. K., Jaume, G., Song, A. H., Chen, B., Zhang, A., Shao, D., Shaban, M., et al. (2024) · Nature Medicine · doi:10.1038/s41591-024-02857-3 · Cited by 1,560 · see citing works
InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutions
Wang, W., Dai, J., Chen, Z., Huang, Z., Li, Z., Zhu, X., Hu, X., Lu, T., Lu, L., Li, H., et al. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01385 · Cited by 1,080 · see citing works
Zhai, X., Kolesnikov, A., Houlsby, N., & Beyer, L. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01179 · Cited by 891 · see citing works
Self-citationReproducible Scaling Laws for Contrastive Language-Image Learning
Cherti, M., Beaumont, R., Wightman, R., Wortsman, M., Ilharco, G., Gordon, C., Schuhmann, C., Schmidt, L., & Jitsev, J. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.00276 · Cited by 673 · see citing works
Big Self-Supervised Models Advance Medical Image Classification
Azizi, S., Mustafa, B., Ryan, F., Beaver, Z., Freyberg, J., Deaton, J., Loh, A., Karthikesalingam, A., Kornblith, S., Chen, T., et al. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00346 · Cited by 685 · see citing works
Huang, S.-C., Pareek, A., Jensen, M., Lungren, M. P., Yeung, S., & Chaudhari, A. S. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00811-0 · Cited by 479 · see citing works
“More recently, Azizi et al. found that using generic and large-scale supervised pre-trained models, such as BigTransfer 94 , can also bene fi t subsequent domain-speci fi c self-supervised pre-training, and ultimately improve model performance and robustness for different medical imaging modalities 95 .”
MethodUnderstanding Robustness of Transformers for Image Classification
Bhojanapalli, S., Chakrabarti, A., Glasner, D., Li, D., Unterthiner, T., & Veit, A. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01007 · Cited by 440 · see citing works
“The behavior of our baseline ResNet models is in line with those observed in Appendix G of [28], where they are evaluated on objects out-of-context.”
BackgroundMethodResultInfluentialRobust fine-tuning of zero-shot models
Wortsman, M., Ilharco, G., Kim, J. W., Li, M., Kornblith, S., Roelofs, R., Lopes, R. G., Hajishirzi, H., Farhadi, A., Namkoong, H., et al. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00780 · Cited by 428 · see citing works
With a Little Help from My Friends: Nearest-Neighbor Contrastive Learning of Visual Representations
Dwibedi, D., Aytar, Y., Tompson, J., Sermanet, P., & Zisserman, A. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00945 · Cited by 404 · see citing works
Sub-Image Anomaly Detection with Deep Pyramid Correspondences
Cohen, N., & Hoshen, Y. (2020) · arXiv · doi:10.48550/arxiv.2005.02357 · Cited by 372 · see citing works
Wortsman, M., Ilharco, G., Gadre, S. Y., Roelofs, R., Gontijo-Lopes, R., Morcos, A. S., Namkoong, H., Farhadi, A., Carmon, Y., Kornblith, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.05482 · Cited by 307 · see citing works
Robust and data-efficient generalization of self-supervised machine learning for diagnostic imaging
Azizi, S., Culp, L., Freyberg, J., Mustafa, B., Baur, S., Kornblith, S., Chen, T., Tomasev, N., Mitrović, J., Strachan, P., et al. (2023) · Nature Biomedical Engineering · doi:10.1038/s41551-023-01049-7 · Cited by 250 · see citing works
Self-citationVision Transformers Are Robust Learners
Paul, S., & Chen, P.-Y. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i2.20103 · Cited by 213 · see citing works
“As pointed out in [43, 2, 44], a longer pre-training schedule on a larger pre-training dataset is transferring effectively to downstream tasks including few-shot learning.”
BackgroundMethodInfluentialExplaining Neural Scaling Laws
Bahri, Y., Dyer, E., Kaplan, J., Lee, J., & Sharma, U. (2021) · arXiv · doi:10.48550/arxiv.2102.06701 · Cited by 182
Improving chest X-ray report generation by leveraging warm starting
Nicolson, A., Dowling, J., & Koopman, B. (2023) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2023.102633 · Cited by 161 · see citing works
Generative models improve fairness of medical classifiers under distribution shifts
Ktena, I., Wiles, O., Albuquerque, I., Rebuffi, S.-A., Tanno, R., Roy, A. G., Azizi, S., Belgrave, D., Kohli, P., Cemgil, T., et al. (2024) · Nature Medicine · doi:10.1038/s41591-024-02838-6 · Cited by 165 · see citing works
Denoising Pretraining for Semantic Segmentation
Brempong, E. A., Kornblith, S., Chen, T., Parmar, N., Minderer, M., & Norouzi, M. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw56347.2022.00462 · Cited by 120 · see citing works
Do Vision Transformers See Like Convolutional Neural Networks?
Raghu, M., Unterthiner, T., Kornblith, S., Zhang, C., & Dosovitskiy, A. (2021) · arXiv · doi:10.48550/arxiv.2108.08810 · Cited by 663 · see citing works
Delving Deep into the Generalization of Vision Transformers under Distribution Shifts
Zhang, C., Zhang, M., Zhang, S., Jin, D., Zhou, Q., Cai, Z., Zhao, H., Liu, X., & Liu, Z. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00713 · Cited by 107 · see citing works
Guha Roy, A., Ren, J., Azizi, S., Loh, A., Natarajan, V., Mustafa, B., Pawlowski, N., Freyberg, J., Liu, Y., Beaver, Z., et al. (2022) · Medical Image Analysis · doi:10.1016/j.media.2021.102274 · Cited by 77 · see citing works
Are Convolutional Neural Networks or Transformers more like human vision?
Tuli, S., Dasgupta, I., Grant, E., & Griffiths, T. L. (2021) · arXiv · doi:10.48550/arxiv.2105.07197 · Cited by 119 · see citing works
Mitigating Adversarial Attacks in Federated Learning with Trusted Execution Environments
Queyrut, S., Schiavoni, V., & Felber, P. (2023) · 2023 IEEE 43rd International Conference on Distributed Computing Systems (ICDCS) · doi:10.1109/icdcs57875.2023.00069 · Cited by 18 · see citing works
- Authors
- Kolesnikov, A., Beyer, L., Zhai, X., Puigcerver, J., Yung, J., Gelly, S., & Houlsby, N.
- Affiliations
- Google (United States) (US)
- Corresponding author
- Zhai, Xiaohua
- Venue
- Lecture Notes in Computer Science
- Publisher
- Springer International Publishing
- Published
- 2020
- Pages
- 491-507
- ISSN
- 0302-9743 (ISSN-L)
- 1611-3349
- OpenAlex
- W3015146382
- Cited by
- 1,039
- References
- 36
- Citations by year
- 2026: 60 · 2025: 132 · 2024: 144 · 2023: 226 · 2022: 166 · 2021: 250 · 2020: 61
- FWCI
- 67.14
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- Lecture notes in computer science (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Keywords
- visual representation learning, pre-training, fine-tuning, transfer learning, small sample learning, image classification, CIFAR-10
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
