CoChat
Sign Up
Cited by 250Domain Adaptation and Few-Shot LearningArtificial Intelligence in Healthcare and EducationMedical Image Segmentation Techniques

The DOI matches its Crossref record; 149 of the 149 references it cites were found in a registry or an index.

Robust and data-efficient generalization of self-supervised machine learning for diagnostic imaging

Azizi, Shekoofeh, Culp, Laura, Freyberg, Jan, Mustafa, Basil, Baur, Sebastien, Kornblith, Simon, Chen, Ting, Tomasev, Nenad, Mitrović, Jovana, Strachan, Patricia, Mahdavi, S. Sara, Wulczyn, Ellery, Babenko, Boris, Walker, Megan, Loh, Aaron, Chen, Po-Hsuan Cameron, Liu, Yuan, Bavishi, Pinal, McKinney, Scott Mayer, Winkens, Jim, Roy, Abhijit Guha, Beaver, Zach, Ryan, Fiona, Krogue, Justin, Etemadi, Mozziyar, Telang, Umesh, Liu, Yun, Peng, Lily, Corrado, Greg S., Webster, Dale R., Fleet, David, Hinton, Geoffrey, Houlsby, Neil, Karthikesalingam, Alan, Norouzi, Mohammad, Natarajan, Vivek·2023·Nature Biomedical Engineering
DOI 10.1038/s41551-023-01049-7
Open this paper in CoChat — free account

Abstract

Machine-learning models for medical tasks can match or surpass the performance of clinical experts. However, in settings differing from those of the training dataset, the performance of a model can deteriorate substantially. Here we report a representation-learning strategy for machine-learning models applied to medical-imaging tasks that mitigates such 'out of distribution' performance problem and that improves model robustness and training efficiency. The strategy, which we named REMEDIS (for 'Robust and Efficient Medical Imaging with Self-supervision'), combines large-scale supervised transfer learning on natural images and intermediate contrastive self-supervised learning on medical images and requires minimal task-specific customization. We show the utility of REMEDIS in a range of diagnostic-imaging tasks covering six imaging domains and 15 test datasets, and by simulating three realistic out-of-distribution scenarios. REMEDIS improved in-distribution diagnostic accuracies up to 11.5% with respect to strong supervised baseline models, and in out-of-distribution settings required only 1-33% of the data for retraining to match the performance of supervised models retrained using all available data. REMEDIS may accelerate the development lifecycle of machine-learning models for medical imaging.

Overview

Integrity signals

Registry

Crossref · 10.1038/s41551-023-01049-7

Cites

149 of 149 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

Show all 141 references

Research works printed but not linked: 51

  1. Zhu, X., Ghahramani, Z. & Lafferty, J. D. Semi-supervised learning using gaussian fields and harmonic functions. In Proc. 20th International Conference on Machine Learning (eds Fawcett, T. & Mishra, N.) 912–919 (AAAI Press, 2003).
  2. Sutton, R. S. Generalization in reinforcement learning: successful examples using sparse coarse coding. Adv. Neural Inf. Process. Syst. 8, 1038–1044 (1996).
  3. Baevski, A., Auli, M. & Mohamed, A. Effectiveness of self-supervised pre-training for speech recognition. Preprint at https://arxiv.org/abs/1911.03912 (2019). · arXiv
  4. Chen, T., Kornblith, S., Norouzi, M. & Hinton, G. A simple framework for contrastive learning of visual representations. In Proc. 37th International Conference on Machine Learning (eds Daumé, H. & Singh, A.) 1597–1607 (JMLR, 2020).
  5. Touvron, H. et al. Training data-efficient image transformers and distillation through attention. In Proc. 38th International Conference on Machine Learning (eds Meila, M. & Zhang, T.) 10347–10357 (PMLR, 2021).
  6. Liu, H. & Abbeel, P. Hybrid discriminative-generative training via contrastive learning. Preprint at https://arxiv.org/abs/2007.09070 (2020). · arXiv
  7. HaoChen, J. Z., Wei, C., Kumar, A. & Ma, T. Beyond separability: analyzing the linear transferability of contrastive representations to related subpopulations. Preprint at https://arxiv.org/abs/2204.02683 (2022). · arXiv
  8. Huh, M., Agrawal, P. & Efros, A. A. What makes ImageNet good for transfer learning? Preprint at https://arxiv.org/abs/1608.08614 (2016). · arXiv
  9. Raghu, M., Zhang, C., Kleinberg, J. & Bengio, S. Transfusion: understanding transfer learning for medical imaging. Adv. Neural Inf. Process. Syst. 33, 3347–3357 (2019).
  10. Hendrycks, D., Lee, K. & Mazeika, M. Using pre-training can improve model robustness and uncertainty. In Proc. 36th International Conference on Machine Learning (eds Chaudhuri, K. & Salakhutdinov, R.) 2712–2721 (PMLR, 2019).
  11. Sowrirajan, H., Yang, J., Ng, A. Y. & Rajpurkar, P. MoCo pretraining improves representation and transferability of chest X-ray models. In Proc. Fourth Conference on Medical Imaging with Deep Learning (eds Heinrich, M. et al.) 728–744 (PMLR, 2021).
  12. Graziani, M., Andrearczyk, V. & Müller, H. Visualizing and interpreting feature reuse of pretrained CNNs for histopathology. In Proc. IMVIP 2019: Irish Machine Vision and Image Processing (Technological University Dublin, 2019).
  13. Chen, T., Kornblith, S., Swersky, K., Norouzi, M. & Hinton, G. Big self-supervised models are strong semi-supervised learners. Adv. Neural Inf. Process. Syst. 33, 22243–22255 (2020).
  14. Neyshabur, B., Sedghi, H. & Zhang, C. What is being transferred in transfer learning? Adv. Neural Inf. Process. Syst. 33, 512–523 (2020).
  15. Ilse, M., Tomczak, J. & Welling, M. Attention-based deep multiple instance learning. In Proc. 35th International Conference on Machine Learning (eds Dy, J. & Krause, A.) 2127–2136 (PMLR, 2018).
  16. Chen, X., Fan, H., Girshick, R. & He, K. Improved baselines with momentum contrastive learning. Preprint at https://arxiv.org/abs/2003.04297 (2020). · arXiv
  17. Mitrovic, J., McWilliams, B., Walker, J., Buesing, L. & Blundell, C. Representation learning via invariant causal mechanisms. Preprint at https://arxiv.org/abs/2010.07922 (2020). · arXiv
  18. Zbontar, J., Jing, L., Misra, I., LeCun, Y. & Deny, S. Barlow twins: self-supervised learning via redundancy reduction. In Proc. 38th International Conference on Machine Learning (eds Meila, M. & Zhang, T.) 12310–12320 (PMLR, 2021).
  19. Gulrajani, I. & Lopez-Paz, D. In search of lost domain generalization. Preprint at https://arxiv.org/abs/2007.01434 (2020). · arXiv
  20. Vapnik, V. N. Statistical Learning Theory (Wiley-Interscience, 1998).
  21. Zhang, H., Cisse, M., Dauphin, Y. N. & Lopez-Paz, D. mixup: beyond empirical risk minimization. Preprint at https://arxiv.org/abs/1710.09412 (2017). · arXiv
  22. Ericsson, L., Gouk, H. & Hospedales, T. M. How well do self-supervised models transfer? In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition 5410–5419 (IEEE, 2021). · DOI
  23. Ciga, O., Martel, A. L. & Xu, T. Self-supervised contrastive learning for digital histopathology. Mach. Learn. 7, 100198 (2022).
  24. Taher, M. R. H., Haghighi, F., Gotway, M. B. & Liang, J. CAiD: context-aware instance discrimination for self-supervised learning in medical imaging. In Proc. 5th International Conference on Medical Imaging with Deep Learning (eds Konukoglu, E. et al.) 535–551 (PMLR, 2022).
  25. Taher, M. R. H., Haghighi, F., Feng, R., Gotway, M. B. & Liang, J. in Domain Adaptation and Representation Transfer, and Affordable Healthcare and AI for Resource Diverse Global Health (eds Albarqouni, S. et al.) 3–13 (Springer, 2021).
  26. Nguyen, T., Raghu, M. & Kornblith, S. Do wide and deep networks learn the same things? Uncovering how neural network representations vary with width and depth. Preprint at https://arxiv.org/abs/2010.15327 (2020). · arXiv
  27. Ioffe, S. & Szegedy, C. Batch normalization: accelerating deep network training by reducing internal covariate shift. In Proc. 32nd International Conference on Machine Learning (eds Bach, F. & Blei, D.) 448–456 (2015).
  28. Qiao, S., Wang, H., Liu, C., Shen, W. & Yuille, A. Micro-batch training with batch-channel normalization and weight standardization. Preprint at https://arxiv.org/abs/1903.10520 (2019). · arXiv
  29. You, Y., Gitman, I. & Ginsburg, B. Large batch training of convolutional networks. Preprint at https://arxiv.org/abs/1708.03888 (2017). · arXiv
  30. Tripuraneni, N., Jordan, M. I. & Jin, C. On the theory of transfer learning: the importance of task diversity. Adv. Neural Inf. Process. Syst. 33, 7852–7862 (2020).
  31. Du, S. S., Hu, W., Kakade, S. M., Lee, J. D. & Lei, Q. Few-shot learning via learning the representation, provably. Preprint at https://arxiv.org/abs/2002.09434 (2020). · arXiv
  32. Kingma, D. P. & Ba, J. Adam: a method for stochastic optimization. Preprint at https://arxiv.org/abs/1412.6980 (2014). · arXiv
  33. Loshchilov, I. & Hutter, F. Sgdr: stochastic gradient descent with warm restarts. Preprint at https://arxiv.org/abs/1608.03983 (2016). · arXiv
  34. Bengio, Y., Goodfellow, I. & Courville, A. Deep Learning (MIT Press, 2017).
  35. Liu, J., Hu, Z., Cui, P., Li, B. & Shen, Z. Heterogeneous risk minimization. In Proc. 38th International Conference on Machine Learning (eds Meila, M. & Zhang, T) 6804–6814 (PMLR, 2021).
  36. Robey, A., Pappas, G. J. & Hassani, H. Model-based domain generalization. Adv. Neural Inf. Process. Syst. 34, 20210–20229 (2021).
  37. Zhou, K., Liu, Z., Qiao, Y., Xiang, T. & Loy, C. C. Domain generalization: a survey. Preprint at https://arxiv.org/abs/2103.02503 (2021). · arXiv
  38. Kim, H. & Mnih, A. Disentangling by factorising. In Proc. 35th International Conference on Machine Learning (eds Dy, J. & Krause, A.) 2649–2658 (PMLR, 2018).
  39. Leeb, F. et al. Structure by architecture: disentangled representations without regularization. Preprint at https://arxiv.org/abs/2006.07796 (2020). · arXiv
  40. Träuble, F. et al. On disentangled representations learned from correlated data. In Proc. 38th International Conference on Machine Learning (eds Meila, M. & Zhang, T.) 10401–10412 (PMLR, 2021).
  41. Andreassen, A., Bahri, Y., Neyshabur, B. & Roelofs, R. The evolution of out-of-distribution robustness throughout fine-tuning. Preprint at https://arxiv.org/abs/2106.15831 (2021). · arXiv
  42. Ganin, Y. & Lempitsky, V. Unsupervised domain adaptation by backpropagation. In Proc. 32nd International Conference on Machine Learning (eds Bach, F. & Blei, D.) 1180–1189 (JMLR, 2015).
  43. Muandet, K., Balduzzi, D. & Schölkopf, B. Domain generalization via invariant feature representation. In Proc. 30th International Conference on Machine Learning (eds Dasgupta, S. & McAllester, D.) I-10–I-18 (2013).
  44. Heker, M. & Greenspan, H. Joint liver lesion segmentation and classification via transfer learning. Preprint at https://arxiv.org/abs/2004.12352 (2020). · arXiv
  45. Chen, S., Ma, K. & Zheng, Y. Med3D: transfer learning for 3D medical image analysis. Preprint at https://arxiv.org/abs/1904.00625 (2019). · arXiv
  46. Geyer, R., Corinzia, L. & Wegmayr, V. Transfer learning by adaptive merging of multiple models. In Proc. 2nd International Conference on Medical Imaging with Deep Learning (eds Cardoso, M. J. et al.) 185–196 (PMLR, 2019).
  47. van den Oord, A., Li, Y. & Vinyals, O. Representation learning with contrastive predictive coding. Preprint at https://arxiv.org/abs/1807.03748 (2018). · arXiv
  48. Bachman, P., Hjelm, R. D. & Buchwalter, W. Learning representations by maximizing mutual information across views. Adv. Neural Inf. Process. Syst. 15535–15545 (2019).
  49. Chaitanya, K., Erdil, E., Karani, N. & Konukoglu, E. Contrastive learning of global and local features for medical image segmentation with limited annotations. Adv. Neural Inf. Process. Syst. 33, 12546–12558 (2020).
  50. Zhang, Y., Jiang, H., Miura, Y., Manning, C. D. & Langlotz, C. P. Contrastive learning of medical visual representations from paired images and text. In Proc. 7th Machine Learning for Healthcare Conference (eds Lipton, Z. et al.) 2–25 (PMLR, 2020).
  51. Truong, T., Mohammadi, S. & Lenga, M. How transferable are self-supervised features in medical image classification tasks? In Proc. Machine Learning for Health (eds Roy, S. et al.) 54–74 (PMLR, 2021).

Cited by

Sorted by citations

Related

See all

Metadata

Authors
Azizi, S., Culp, L., Freyberg, J., Mustafa, B., Baur, S., Kornblith, S., Chen, T., Tomasev, N., Mitrović, J., Strachan, P., Mahdavi, S. S., Wulczyn, E., Babenko, B., Walker, M., Loh, A., Chen, P.-H. C., Liu, Y., Bavishi, P., McKinney, S. M., Winkens, J., Roy, A. G., Beaver, Z., Ryan, F., Krogue, J., Etemadi, M., Telang, U., Liu, Y., Peng, L., Corrado, G. S., Webster, D. R., Fleet, D., Hinton, G., Houlsby, N., Karthikesalingam, A., Norouzi, M., & Natarajan, V.
Affiliations
  • Google (United States) (US)
  • Google DeepMind (United Kingdom) (GB)
  • Georgia Institute of Technology (US)
  • Northwestern University (US)
Corresponding author
  • Azizi, Shekoofeh
  • Karthikesalingam, Alan
  • Natarajan, Vivek
ORCID
Venue
Nature Biomedical Engineering
Publisher
Springer Science and Business Media LLC
Published
2023-06-08
Volume
7
Issue
6
Pages
756-779
ISSN
2157-846X (ISSN-L)
DOI
10.1038/s41551-023-01049-7
OpenAlex
W4379878309
Cited by
250
References
141
Citations by year
2026: 68 · 2025: 92 · 2024: 77 · 2023: 12
FWCI
23.44
Type
journal article
Language
en
Open access
closed
Licence
Locations
  • Nature Biomedical Engineering (publishedVersion)
  • PubMed (publishedVersion)
Funders
Cancer Research UK
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Medical Image Segmentation Techniques
Keywords
self-supervised learning, representation learning, out-of-distribution generalization, model robustness, transfer learning, contrastive learning, diagnostic imaging, data efficiency
MeSH terms
Machine Learning, Supervised Machine Learning, Diagnostic Imaging

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review