The DOI matches its OpenAlex record; 22 of the 22 references it cites were found in a registry or an index.
Big Self-Supervised Models are Strong Semi-Supervised Learners
Open accessOverview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380
Rethinking the Inception Architecture for Computer Vision
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.308 · Cited by 32,138
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Auto-Encoding Variational Bayes
Kingma, D. P., & Welling, M. (2013) · arXiv · doi:10.48550/arxiv.1312.6114 · Cited by 20,067
A fast learning algorithm for deep belief nets
Hinton, G. E., Osindero, S., & Teh, Y.-W. (2006) · Neural Computation · doi:10.1162/neco.2006.18.7.1527 · Cited by 16,336
Self-citationDistilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204
Self-citationMomentum Contrast for Unsupervised Visual Representation Learning
He, K., Fan, H., Wu, Y., Xie, S., & Girshick, R. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00975 · Cited by 13,347
Cited by
Learning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293
Emerging Properties in Self-Supervised Vision Transformers
Caron, M., Touvron, H., Misra, I., Jegou, H., Mairal, J., Bojanowski, P., & Joulin, A. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00951 · Cited by 5,638
Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning
Grill, J.-B., Strub, F., Altché, F., Tallec, C., Richemond, P. H., Buchatskaya, E., Doersch, C., Pires, B. A., Guo, Z. D., Azar, M. G., et al. (2020) · arXiv · doi:10.48550/arxiv.2006.07733 · Cited by 3,846
ProteinBERT: a universal deep-learning model of protein sequence and function
Brandes, N., Ofer, D., Peleg, Y., Rappoport, N., & Linial, M. (2022) · Bioinformatics · doi:10.1093/bioinformatics/btac020 · Cited by 1,009
Contrastive Representation Learning: A Framework and Review
Le-Khac, P. H., Healy, G., & Smeaton, A. F. (2020) · IEEE Access · doi:10.1109/access.2020.3031549 · Cited by 862
Zhai, X., Kolesnikov, A., Houlsby, N., & Beyer, L. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01179 · Cited by 891
Big Self-Supervised Models Advance Medical Image Classification
Azizi, S., Mustafa, B., Ryan, F., Beaver, Z., Freyberg, J., Deaton, J., Loh, A., Karthikesalingam, A., Kornblith, S., Chen, T., et al. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00346 · Cited by 685
Self-citationRobust fine-tuning of zero-shot models
Wortsman, M., Ilharco, G., Kim, J. W., Li, M., Kornblith, S., Roelofs, R., Lopes, R. G., Hajishirzi, H., Farhadi, A., Namkoong, H., et al. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00780 · Cited by 428
Self-citationWith a Little Help from My Friends: Nearest-Neighbor Contrastive Learning of Visual Representations
Dwibedi, D., Aytar, Y., Tompson, J., Sermanet, P., & Zisserman, A. (2021) · arXiv · doi:10.48550/arxiv.2104.14548 · Cited by 405
With a Little Help from My Friends: Nearest-Neighbor Contrastive Learning of Visual Representations
Dwibedi, D., Aytar, Y., Tompson, J., Sermanet, P., & Zisserman, A. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00945 · Cited by 404
High-resolution de novo structure prediction from primary sequence
Wu, R., Ding, F., Wang, R., Shen, R., Zhang, X., Luo, S., Su, C., Wu, Z., Xie, Q., Berger, B., et al. (2022) · doi:10.1101/2022.07.21.500999 · Cited by 392
Florence: A New Foundation Model for Computer Vision
Yuan, L., Chen, D., Chen, Y.-L., Codella, N., Dai, X., Gao, J., Hu, H., Huang, X., Li, B., Li, C., et al. (2021) · arXiv · doi:10.48550/arxiv.2111.11432 · Cited by 389
Self-Supervised Learning in Remote Sensing: A review
Wang, Y., Albrecht, C. M., Braham, N. A. A., Mou, L., & Zhu, X. X. (2022) · IEEE Geoscience and Remote Sensing Magazine · doi:10.1109/mgrs.2022.3198244 · Cited by 364
DocFormer: End-to-End Transformer for Document Understanding
Appalaraju, S., Jasani, B., Kota, B. U., Xie, Y., & Manmatha, R. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00103 · Cited by 363
Embodied intelligence via learning and evolution
Gupta, A., Savarese, S., Ganguli, S., & Fei-Fei, L. (2021) · Nature Communications · doi:10.1038/s41467-021-25874-z · Cited by 352
RingMo: A Remote Sensing Foundation Model With Masked Image Modeling
Sun, X., Wang, P., Lu, W., Zhu, Z., Lu, X., He, Q., Li, J., Rong, X., Yang, Z., Chang, H., et al. (2023) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2022.3194732 · Cited by 323
Foundation Models Defining a New Era in Vision: A Survey and Outlook
Awais, M., Naseer, M., Khan, S., Anwer, R. M., Cholakkal, H., Shah, M., Yang, M.-H., & Khan, F. S. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3506283 · Cited by 304
Self supervised contrastive learning for digital histopathology
Ciga, O., Xu, T., & Martel, A. L. (2022) · Machine Learning with Applications · doi:10.1016/j.mlwa.2021.100198 · Cited by 294
Inner Monologue: Embodied Reasoning through Planning with Language Models
Huang, W., Xia, F., Xiao, T., Chan, H., Liang, J., Florence, P., Zeng, A., Tompson, J., Mordatch, I., Chebotar, Y., et al. (2022) · arXiv · doi:10.48550/arxiv.2207.05608 · Cited by 287
“, CLIP [71]) have been trained on joint image(s) and corresponding text captions via variants of a masked language modeling objective [72, 73, 74, 75], a contrastive loss [76, 77, 71] or other supervised objectives[78, 79].”
A Survey on Semi-, Self- and Unsupervised Learning for Image Classification
Schmarje, L., Santarossa, M., Schroder, S.-M., & Koch, R. (2021) · IEEE Access · doi:10.1109/access.2021.3084358 · Cited by 246
Vision Transformers Are Robust Learners
Paul, S., & Chen, P.-Y. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i2.20103 · Cited by 244
Rao, H., Xu, S., Hu, X., Cheng, J., & Hu, B. (2021) · Information Sciences · doi:10.1016/j.ins.2021.04.023 · Cited by 235
Unsupervised Semantic Segmentation by Contrasting Object Mask Proposals
Van Gansbeke, W., Vandenhende, S., Georgoulis, S., & Van Gool, L. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00990 · Cited by 231
A large-scale study on unsupervised spatiotemporal representation learning
Feichtenhofer, C., Fan, H., Xiong, B., Girshick, R., & He, K. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00331 · Cited by 222
Review on self-supervised image recognition using deep neural networks
Ohri, K., & Kumar, M. (2021) · Knowledge-Based Systems · doi:10.1016/j.knosys.2021.107090 · Cited by 179
A Cookbook of Self-Supervised Learning
Balestriero, R., Ibrahim, M., Sobal, V., Morcos, A., Shekhar, S., Goldstein, T., Bordes, F., Bardes, A., Mialon, G., Tian, Y., et al. (2023) · arXiv · doi:10.48550/arxiv.2304.12210 · Cited by 182
“This key element of data-augmentation had also been largely used in the following SSL works [Chen et al., 2020d, Bardes et al., 2021, Zbontar et al., 2021] without significant changes.”
BackgroundMethodInfluentialPiCIE: Unsupervised Semantic Segmentation using Invariance and Equivariance in Clustering
Hyun Cho, J., Mall, U., Bala, K., & Hariharan, B. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01652 · Cited by 159
When Does Contrastive Visual Representation Learning Work?
Cole, E., Yang, X., Wilber, K., Aodha, O. M., & Belongie, S. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01434 · Cited by 105
Revisiting Weakly Supervised Pre-Training of Visual Perception Models
Singh, M., Gustafson, L., Adcock, A., De Freitas Reis, V., Gedik, B., Kosaraju, R. P., Mahajan, D., Girshick, R., Dollar, P., & Van Der Maaten, L. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00088 · Cited by 99
A Good Image Generator Is What You Need for High-Resolution Video Synthesis
Tian, Y., Ren, J., Chai, M., Olszewski, K., Peng, X., Metaxas, D. N., & Tulyakov, S. (2021) · arXiv · doi:10.48550/arxiv.2104.15069 · Cited by 88
The Surprising Effectiveness of Representation Learning for Visual Imitation
Pari, J., Shafiullah, N. M. (., Arunachalam, S. P., & Pinto, L. (2022) · Robotics: Science and Systems XVIII · doi:10.15607/rss.2022.xviii.010 · Cited by 95
Spatially Consistent Representation Learning
Roh, B., Shin, W., Kim, I., & Kim, S. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00120 · Cited by 85
Self-supervised Representation Learning for Astronomical Images
Hayat, M. A., Stein, G., Harrington, P., Lukić, Z., & Mustafa, M. (2021) · The Astrophysical Journal Letters · doi:10.3847/2041-8213/abf2c7 · Cited by 79
SpeechStew: Simply Mix All Available Speech Recognition Data to Train One Large Neural Network
Chan, W., Park, D., Lee, C., Zhang, Y., Le, Q., & Norouzi, M. (2021) · arXiv · doi:10.48550/arxiv.2104.02133 · Cited by 79
Self-citationMiller, J., Taori, R., Raghunathan, A., Sagawa, S., Koh, P. W., Shankar, V., Liang, P., Carmon, Y., & Schmidt, L. (2021) · arXiv · doi:10.48550/arxiv.2107.04649 · Cited by 74
Semi-Supervised Contrastive Learning With Similarity Co-Calibration
Zhang, Y., Zhang, X., Li, J., Qiu, R. C., Xu, H., & Tian, Q. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2022.3158069 · Cited by 67
Cheng, D., Zhang, L., Bu, C., Wu, H., & Song, A. (2023) · Knowledge-Based Systems · doi:10.1016/j.knosys.2023.110789 · Cited by 55
Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Loss
HaoChen, J. Z., Wei, C., Gaidon, A., & Ma, T. (2021) · arXiv · doi:10.48550/arxiv.2106.04156 · Cited by 53
Contrastive Learning Inverts the Data Generating Process
Zimmermann, R. S., Sharma, Y., Schneider, S., Bethge, M., & Brendel, W. (2021) · arXiv · doi:10.48550/arxiv.2102.08850 · Cited by 52
Mining for Strong Gravitational Lenses with Self-supervised Learning
Stein, G., Blaum, J., Harrington, P., Medan, T., & Lukić, Z. (2022) · The Astrophysical Journal · doi:10.3847/1538-4357/ac6d63 · Cited by 52
BYOL works even without batch statistics
Richemond, P. H., Grill, J.-B., Altché, F., Tallec, C., Strub, F., Brock, A., Smith, S., De, S., Pascanu, R., Piot, B., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.10241 · Cited by 51
An Improved Inter-Intra Contrastive Learning Framework on Self-Supervised Video Representation
Tao, L., Wang, X., & Yamasaki, T. (2022) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2022.3141051 · Cited by 50
Cheng, D., Zhang, L., Qin, L., Wang, S., Wu, H., & Song, A. (2024) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2024.3373019 · Cited by 31
Pre-training in Medical Data: A Survey
Qiu, Y., Lin, F., Chen, W., & Xu, M. (2023) · Machine Intelligence Research · doi:10.1007/s11633-022-1382-8 · Cited by 30
Contrasting Contrastive Self-Supervised Representation Learning Pipelines
Kotar, K., Ilharco, G., Schmidt, L., Ehsani, K., & Mottaghi, R. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00980 · Cited by 28
Shrivastava, A. D., & Kell, D. B. (2021) · Molecules · doi:10.3390/molecules26072065 · Cited by 25
Detecting galaxy tidal features using self-supervised representation learning
Desmons, A., Brough, S., & Lanusse, F. (2024) · Monthly Notices of the Royal Astronomical Society · doi:10.1093/mnras/stae1402 · Cited by 15
Detecting Online Grooming By Simple Contrastive Chat Embeddings
Rezaee Borj, P., Raja, K., & Bours, P. (2023) · Proceedings of the 9th ACM International Workshop on Security and Privacy Analytics · doi:10.1145/3579987.3586564 · Cited by 11
Kranz, D. D., Kahriman, O., Dischl, D., Treskatsch, S., Sander, A., Brachmann, J., Park, J.-W., & Wessel, N. (2025) · doi:10.64898/2025.11.30.25341301 · Cited by 1
Improving robustness against common corruptions by covariate shift adaptation
Schneider, S., Rusak, E., Eck, L., Bringmann, O., Brendel, W., & Bethge, M. (2020) · arXiv · doi:10.48550/arxiv.2006.16971
Metadata
- Authors
- Chen, T., Kornblith, S., Swersky, K., Norouzi, M., & Hinton, G. E.
- Venue
- arXiv (Cornell University)
- Publisher
- Cornell University
- Published
- 2020-06-17
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2006.10029
- OpenAlex
- W3036982689
- Cited by
- 421
- References
- 22
- Type
- posted content
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
