Big Self-Supervised Models Advance Medical Image Classification
Abstract
Self-supervised pretraining followed by supervised fine-tuning has seen success in image recognition, especially when labeled examples are scarce, but has received limited attention in medical image analysis. This paper studies the effectiveness of self-supervised learning as a pre-training strategy for medical image classification. We conduct experiments on two distinct tasks: dermatology condition classification from digital camera images and multi-label chest X-ray classification, and demonstrate that self-supervised learning on ImageNet, followed by additional self-supervised learning on unlabeled domain-specific medical images significantly improves the accuracy of medical image classifiers. We introduce a novel Multi-Instance Contrastive Learning (MICLe) method that uses multiple images of the underlying pathology per patient case, when available, to construct more informative positive pairs for self-supervised learning. Combining our contributions, we achieve an improvement of 6.7% in top-1 accuracy and an improvement of 1.1% in mean AUC on dermatology and chest X-ray classification respectively, outperforming strong supervised baselines pretrained on ImageNet. In addition, we show that big self-supervised models are robust to distribution shift and can learn efficiently with a small number of labeled medical images.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Foundation models' strong feature representations enable data-efficient applications, requiring fewer labeled samples.
“Their strong feature representations also enable data-efficient applications26,27, requiring fewer labeled samples, which is particularly crucial for medical domains in which expert-annotated data are often limited.”
From A Multimodal Vision Foundation Model for Clinical Dermatology (2025) · cites this paper as [27]
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · see citing works
Momentum Contrast for Unsupervised Visual Representation Learning
He, K., Fan, H., Wu, Y., Xie, S., & Girshick, R. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00975 · Cited by 13,347 · see citing works
A simple framework for contrastive learning of visual representations
Chen, T., Kornblith, S., Norouzi, M., & Hinton, G. (2020) · arXiv · doi:10.48550/arxiv.2002.05709 · Cited by 10,857 · see citing works
Self-citationRepresentation Learning with Contrastive Predictive Coding
Oord, A. V. D., Li, Y., & Vinyals, O. (2018) · arXiv · doi:10.48550/arxiv.1807.03748 · Cited by 6,242 · see citing works
Wang, X., Peng, Y., Lu, L., Lu, Z., Bagheri, M., & Summers, R. M. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.369 · Cited by 4,616 · see citing works
Unsupervised Feature Learning via Non-parametric Instance Discrimination
Wu, Z., Xiong, Y., Yu, S. X., & Lin, D. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00393 · Cited by 3,664 · see citing works
Zhang, R., Isola, P., & Efros, A. A. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46487-9_40 · Cited by 3,361 · see citing works
International evaluation of an AI system for breast cancer screening
McKinney, S. M., Sieniek, M., Godbole, V., Godwin, J., Antropova, N., Ashrafian, H., Back, T., Chesus, M., Corrado, G. S., Darzi, A., et al. (2020) · Nature · doi:10.1038/s41586-019-1799-6 · Cited by 3,287 · see citing works
Self-citationUnsupervised Learning of Visual Representations by Solving Jigsaw Puzzles
Noroozi, M., & Favaro, P. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46466-4_5 · Cited by 2,775 · see citing works
Unsupervised Visual Representation Learning by Context Prediction
Doersch, C., Gupta, A., & Efros, A. A. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.167 · Cited by 2,629 · see citing works
Improved Baselines with Momentum Contrastive LearningPreprint
Chen, X., Fan, H., Girshick, R., & He, K. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.04297 · Cited by 2,579 · see citing works
Unsupervised Representation Learning by Predicting Image Rotations
Gidaris, S., Singh, P., & Komodakis, N. (2018) · arXiv · doi:10.48550/arxiv.1803.07728 · Cited by 2,477 · see citing works
Tian, Y., Krishnan, D., & Isola, P. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58621-8_45 · Cited by 1,997 · see citing works
Learning deep representations by mutual information estimation and maximization
Hjelm, R. D., Fedorov, A., Lavoie-Marchildon, S., Grewal, K., Bachman, P., Trischler, A., & Bengio, Y. (2018) · arXiv · doi:10.48550/arxiv.1808.06670 · Cited by 1,991 · see citing works
Big Self-Supervised Models are Strong Semi-Supervised LearnersPreprint
Chen, T., Kornblith, S., Swersky, K., Norouzi, M., & Hinton, G. E. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.10029 · Cited by 1,480 · see citing works
Self-citationSelf-Supervised Learning of Pretext-Invariant Representations
Misra, I., & van der Maaten, L. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00674 · Cited by 1,435 · see citing works
Exploring the Limits of Weakly Supervised Pretraining
Mahajan, D., Girshick, R., Ramanathan, V., He, K., Paluri, M., Li, Y., Bharambe, A., & van der Maaten, L. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01216-8_12 · Cited by 1,224 · see citing works
Big Transfer (BiT): General Visual Representation Learning
Kolesnikov, A., Beyer, L., Zhai, X., Puigcerver, J., Yung, J., Gelly, S., & Houlsby, N. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58558-7_29 · Cited by 1,039 · see citing works
Data-Efficient Image Recognition with Contrastive Predictive Coding
Hénaff, O. J., Srinivas, A., De Fauw, J., Razavi, A., Doersch, C., Eslami, S. M. A., & Oord, A. V. D. (2019) · arXiv · doi:10.48550/arxiv.1905.09272 · Cited by 1,014 · see citing works
Large Batch Training of Convolutional NetworksPreprint
You, Y., Gitman, I., & Ginsburg, B. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1708.03888 · Cited by 992 · see citing works
A deep learning system for differential diagnosis of skin diseases
Liu, Y., Jain, A., Eng, C., Way, D. H., Lee, K., Bui, P., Kanada, K., de Oliveira Marinho, G., Gallegos, J., Gabriele, S., et al. (2020) · Nature Medicine · doi:10.1038/s41591-020-0842-3 · Cited by 853 · see citing works
Self-citationTransfusion: Understanding Transfer Learning for Medical ImagingPreprint
Raghu, M., Zhang, C., Kleinberg, J. M., & Samy Bengio, A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.07208 · Cited by 845 · see citing works
Unsupervised Embedding Learning via Invariant and Spreading Instance Feature
Ye, M., Zhang, X., Yuen, P. C., & Chang, S. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00637 · Cited by 629 · see citing works
Time-Contrastive Networks: Self-Supervised Learning from Video
Sermanet, P., Lynch, C., Chebotar, Y., Hsu, J., Jang, E., Stefan Schaal, A., Levine, S., & Brain, G. (2018) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2018.8462891 · Cited by 579 · see citing works
Self-supervised learning for medical image analysis using image context restoration
Chen, L., Bentley, P., Mori, K., Misawa, K., Fujiwara, M., & Rueckert, D. (2019) · Medical Image Analysis · doi:10.1016/j.media.2019.101539 · Cited by 575 · see citing works
Self-organizing neural network that discovers surfaces in random-dot stereograms
Becker, S., & Hinton, G. E. (1992) · Nature · doi:10.1038/355161a0 · Cited by 559 · see citing works
A transfer learning method with deep residual network for pediatric pneumonia diagnosis
Liang, G., & Zheng, L. (2019) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2019.06.023 · Cited by 501 · see citing works
Tian, Y., Krishnan, D., & Isola, P. (2019) · arXiv · doi:10.48550/arxiv.1906.05849 · Cited by 465 · see citing works
Contrastive Learning of Medical Visual Representations from Paired Images and TextPreprint
Zhang, Y., Jiang, H., Miura, Y., Manning, C. D., & Langlotz, C. P. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.00747 · Cited by 465 · see citing works
Sample-Efficient Deep Learning for COVID-19 Diagnosis Based on CT ScansPreprint
He, X., Yang, X., Zhang, S., Zhao, J., Zhang, Y., Xing, E., & Xie, P. (2020) · medRxiv · doi:10.1101/2020.04.13.20063941 · Cited by 434 · see citing works
CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison
Irvin, J., Rajpurkar, P., Ko, M., Yu, Y., Ciurea-Ilcus, S., Chute, C., Marklund, H., Haghgoo, B., Ball, R., Shpanskaya, K., et al. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.3301590 · Cited by 392 · see citing works
(2019) · neural information processing systems · Cited by 350 · see citing works
Semi-Supervised Medical Image Classification With Relation-Driven Self-Ensembling Model
Liu, Q., Yu, L., Luo, L., Dou, Q., & Heng, P. A. (2020) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2020.2995518 · Cited by 320 · see citing works
Med3D: Transfer Learning for 3D Medical Image AnalysisPreprint
Chen, S., Ma, K., & Zheng, Y. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.00625 · Cited by 318 · see citing works
What is being transferred in transfer learning?Preprint
Neyshabur, B., Sedghi, H., & Zhang, C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.11687 · Cited by 274 · see citing works
Towards a Better Understanding of Transfer Learning for Medical Imaging: A Case Study
Alzubaidi, L., Fadhel, M. A., Al-Shamma, O., Zhang, J., Santamaría, J., Duan, Y., & R. Oleiwi, S. (2020) · Applied Sciences · doi:10.3390/app10134523 · Cited by 258 · see citing works
Knowledge transfer for melanoma screening with deep learning
Menegola, A., Fornaciali, M., Pires, R., Bittencourt, F. V., Avila, S., & Junior, E. A. D. V. (2017) · IEEE International Symposium on Biomedical Imaging (ISBI) · doi:10.1109/isbi.2017.7950523 · Cited by 218 · see citing works
Self-Supervised Learning for Cardiac MR Image Segmentation by Anatomical Position Prediction
Bai, W., Chen, C., Tarroni, G., Duan, J., Guitton, F., Petersen, S. E., Guo, Y., Matthews, P. M., & Rueckert, D. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-32245-8_60 · Cited by 189 · see citing works
Rubik’s Cube+: A self-supervised feature learning framework for 3D medical image analysis
Zhu, J., Li, Y., Hu, Y., Ma, K., Zhou, S. K., & Zheng, Y. (2020) · Medical Image Analysis · doi:10.1016/j.media.2020.101746 · Cited by 162 · see citing works
FocalMix: Semi-Supervised Learning for 3D Medical Image Detection
Wang, D., Zhang, Y., Zhang, K., & Wang, L. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00401 · Cited by 157 · see citing works
Self-supervised Feature Learning for 3D Medical Images by Playing a Rubik’s Cube
Zhuang, X., Li, Y., Hu, Y., Ma, K., Yang, Y., & Zheng, Y. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-32251-9_46 · Cited by 153 · see citing works
Transfusion: Understanding Transfer Learning for Medical Imaging
Raghu, M., Zhang, C., Kleinberg, J. M., & Samy Bengio, A. (2019) · neural information processing systems · Cited by 146 · see citing works
Zhou, H.-Y., Yu, S., Bian, C., Hu, Y., Ma, K., & Zheng, Y. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-59710-8_39 · Cited by 100 · see citing works
Liu, J., Zhao, G., Fei, Y., Zhang, M., Wang, Y., & Yu, Y. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.01073 · Cited by 99 · see citing works
Improving Cytoarchitectonic Segmentation of Human Brain Areas with Self-supervised Siamese Networks
Spitzer, H., Kiwitz, K., Amunts, K., Stefan Harmeling, B., & Dickscheid, T. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-00931-1_76 · Cited by 77 · see citing works
MoCo-CXR: MoCo Pretraining Improves Representation and Transferability of Chest X-ray ModelsPreprint
Sowrirajan, H., Yang, J., Ng, A. Y., & Rajpurkar, P. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.05352 · Cited by 61 · see citing works
Self-Supervised Learning of Video-Induced Visual Invariances
Tschannen, M., Djolonga, J., Ritter, M., Mahendran, A., Houlsby, N., Gelly, S., & Lučić, M. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01382 · Cited by 59 · see citing works
Imbalance-Aware Self-supervised Learning for 3D Radiomic Representations
Li, H., Xue, F.-F., Chaitanya, K., Luo, S., Ezhov, I., Wiestler, B., Zhang, J., & Menze, B. H. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-87196-3_4 · Cited by 21 · see citing works
Joint Liver Lesion Segmentation and Classification via Transfer LearningPreprint
Heker, M., & Greenspan, H. K. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.12352 · Cited by 19 · see citing works
Xie, H., Shan, H., Cong, W., Zhang, X., Liu, S., Ning, R., & Wang, G. (2019) · Developments in X-Ray Tomography · doi:10.1117/12.2531198 · Cited by 18 · see citing works
Visualizing and Interpreting Feature Reuse of Pretrained CNNs for Histopathology
Graziani, M., Andrearczyk, V., & Müller, H. (2019) · Arrow - TU Dublin (Technological University Dublin) · Cited by 13 · see citing works
MoCo-Pretraining Improves Representations and Transferability of Chest X-ray ModelsPreprint
Sowrirajan, H., Yang, J. B., Ng, A. Y., & Rajpurkar, P. (2021) · arXiv (Cornell University) · Cited by 10 · see citing works
Transfer Learning by Adaptive Merging of Multiple Models
Geyer, R., Corinzia, L., & Wegmayr, V. (2018) · Cited by 5 · see citing works
Imbalance-Aware Self-Supervised Learning for 3D Radiomic RepresentationsPreprint
Li, H., Xue, F.-F., Chaitanya, K., Luo, S., Ezhov, I., Wiestler, B., Zhang, J., & Menze, B. H. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.04167 · Cited by 4 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Self-supervised learning in medicine and healthcare
Krishnan, R., Rajpurkar, P., & Topol, E. J. (2022) · Nature Biomedical Engineering · doi:10.1038/s41551-022-00914-1 · Cited by 605 · see citing works
Robust and data-efficient generalization of self-supervised machine learning for diagnostic imaging
Azizi, S., Culp, L., Freyberg, J., Mustafa, B., Baur, S., Kornblith, S., Chen, T., Tomasev, N., Mitrović, J., Strachan, P., et al. (2023) · Nature Biomedical Engineering · doi:10.1038/s41551-023-01049-7 · Cited by 250 · see citing works
Self-citationReview of multimodal machine learning approaches in healthcare
Krones, F., Marikkar, U., Parsons, G., Szmul, A., & Mahdi, A. (2025) · Information Fusion · doi:10.1016/j.inffus.2024.102690 · Cited by 235 · see citing works
RetCCL: Clustering-guided contrastive learning for whole-slide image retrieval
Wang, X., Du, Y., Yang, S., Zhang, J., Wang, M., Zhang, J., Yang, W., Huang, J., & Han, X. (2023) · Medical Image Analysis · doi:10.1016/j.media.2022.102645 · Cited by 218 · see citing works
Foundation Model for Cancer Imaging Biomarkers
Pai, S., Bontempi, D., Hadzic, I., Prudente, V., Sokač, M., Chaunzwa, T. L., Bernatz, S., Hosny, A., Mak, R. H., Birkbak, N. J., et al. (2024) · Nature Machine Intelligence · doi:10.1038/s42256-024-00807-9 · Cited by 199 · see citing works
Benchmarking Self-Supervised Learning on Diverse Pathology Datasets
Kang, M., Song, H., Park, S., Yoo, D., & Pereira, S. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.00326 · Cited by 187 · see citing works
Yan, R., Qu, L., Wei, Q., Huang, S.-C., Shen, L., Rubin, D. L., Xing, L., & Zhou, Y. (2023) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2022.3233574 · Cited by 183 · see citing works
A Multimodal Vision Foundation Model for Clinical Dermatology
Yan, S., Yu, Z., Primiero, C., Vico-Alonso, C., Wang, Z., Yang, L., Tschandl, P., Hu, M., Ju, L., Tan, G., et al. (2025) · Nature Medicine · doi:10.1038/s41591-025-03747-y · Cited by 117 · see citing works
“Their strong feature representations also enable data-efficient applications 26,27 , requiring fewer labeled samples, which is particularly crucial for medical domains in which expert-annotated data are often limited.”
Exploring scalable medical image encoders beyond text supervision
Pérez-García, F., Sharma, H., Bond-Taylor, S., Bouzid, K., Salvatelli, V., Ilse, M., Bannur, S., Castro, D. C., Schwaighofer, A., Lungren, M. P., et al. (2025) · Nature Machine Intelligence · doi:10.1038/s42256-024-00965-w · Cited by 115 · see citing works
Cross-modal autoencoder framework learns holistic representations of cardiovascular state
Radhakrishnan, A., Friedman, S. F., Khurshid, S., Ng, K., Batra, P., Lubitz, S. A., Philippakis, A. A., & Uhler, C. (2023) · Nature Communications · doi:10.1038/s41467-023-38125-0 · Cited by 82 · see citing works
Guha Roy, A., Ren, J., Azizi, S., Loh, A., Natarajan, V., Mustafa, B., Pawlowski, N., Freyberg, J., Liu, Y., Beaver, Z., et al. (2022) · Medical Image Analysis · doi:10.1016/j.media.2021.102274 · Cited by 77 · see citing works
Self-citationZambrano Chaves, J. M., Huang, S.-C., Xu, Y., Xu, H., Usuyama, N., Zhang, S., Wang, F., Xie, Y., Khademi, M., Yang, Z., et al. (2025) · Nature Communications · doi:10.1038/s41467-025-58344-x · Cited by 45 · see citing works
SimTriplet: Simple Triplet Representation Learning with a Single GPU
Liu, Q., Louis, P. C., Lu, Y., Jha, A., Zhao, M., Deng, R., Yao, T., Roland, J. T., Yang, H., Zhao, S., et al. (2021) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-87196-3_10 · Cited by 38 · see citing works
Pre-training in Medical Data: A Survey
Qiu, Y., Lin, F., Chen, W., & Xu, M. (2023) · Machine Intelligence Research · doi:10.1007/s11633-022-1382-8 · Cited by 30 · see citing works
Shen, Y., Li, H., Sun, C., Ji, H., Zhang, D., Hu, K., Tang, Y., Chen, Y., Wei, Z., & Lv, J. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01014-x · Cited by 21 · see citing works
Jiao, J., Sun, H., Huang, Y., Xia, M., Qiao, M., Ren, Y., Wang, Y., & Guo, Y. (2023) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2023.3278259 · Cited by 15 · see citing works
Beyond Medical Imaging - A Review of Multimodal Deep Learning in Radiology
Heiliger, L., Sekuboyina, A., Menze, B., Egger, J., & Kleesiek, J. (2022) · doi:10.36227/techrxiv.19103432 · Cited by 13 · see citing works
The role of multimodality in clinical disease diagnosis: advances, challenges, and opportunities
Nawaz, A., Edinat, A., Rana, M. R. R., Ali, T., Mustafa, G., Tahir, S., & Lee, S. W. (2026) · Frontiers in public health · doi:10.3389/fpubh.2026.1788454 · Cited by 3 · see citing works
- Authors
- Azizi, S., Mustafa, B., Ryan, F., Beaver, Z., Freyberg, J., Deaton, J., Loh, A., Karthikesalingam, A., Kornblith, S., Chen, T., Natarajan, V., & Norouzi, M.
- Affiliations
- Google (United States) (US)
- ORCID
- Venue
- 2021 IEEE/CVF International Conference on Computer Vision (ICCV)
- Publisher
- IEEE
- Published
- 2021-10
- Pages
- 3458-3468
- OpenAlex
- W3120430728
- Cited by
- 685
- References
- 54
- Citations by year
- 2026: 115 · 2025: 161 · 2024: 138 · 2023: 142 · 2022: 95 · 2021: 33 · 2020: 1
- FWCI
- 47.13
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- IEEE/CVF International Conference on Computer Vision (ICCV) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › COVID-19 diagnosis using AI
- Keywords
- self-supervised pre-training, medical image classification, chest X-ray classification, contrastive learning, domain-specific pre-training, distribution shift robustness, ImageNet pre-training
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
