CoChat
Sign Up
PreprintCited by 2,892Machine Learning and AlgorithmsDomain Adaptation and Few-Shot LearningMachine Learning and Data Classification

FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence

Open access
Sohn, Kihyuk, Berthelot, David, Li, Chun-Liang, Zhang, Zizhao, Carlini, Nicholas, Cubuk, Ekin D., Kurakin, Alex, Zhang, Han +1 more·2020·arXiv
DOI 10.48550/arxiv.2001.07685arXiv 2001.07685
Open this paper in CoChat — free accountOpen PDF

Abstract

Semi-supervised learning (SSL) provides an effective means of leveraging unlabeled data to improve a model's performance. In this paper, we demonstrate the power of a simple combination of two common SSL methods: consistency regularization and pseudo-labeling. Our algorithm, FixMatch, first generates pseudo-labels using the model's predictions on weakly-augmented unlabeled images. For a given image, the pseudo-label is only retained if the model produces a high-confidence prediction. The model is then trained to predict the pseudo-label when fed a strongly-augmented version of the same image. Despite its simplicity, we show that FixMatch achieves state-of-the-art performance across a variety of standard semi-supervised learning benchmarks, including 94.93% accuracy on CIFAR-10 with 250 labels and 88.61% accuracy with 40 -- just 4 labels per class. Since FixMatch bears many similarities to existing SSL methods that achieve worse performance, we carry out an extensive ablation study to tease apart the experimental factors that are most important to FixMatch's success. We make our code available at https://github.com/google-research/fixmatch.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2001.07685

Cites

3 of 3 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Semi-supervised learning approaches, using unlabelled data along with limited labelled instances, offer a practical balance, leveraging minimal supervision to improve performance significantly over fully unsupervised approaches.

    “Semi-supervised learning (SSL) approaches, which use unlabelled data along with some limited labelled instances, offer a practical balance, leveraging minimal supervision to improve performance significantly over fully unsupervised approaches [3,5].”

    From AnomalyMatch: Discovering Rare Objects of Interest with Semi-supervised and Active Learning (2025) · cites this paper as [5]

    The Introduction (p. 1) describes SSL as using unlabeled data to reduce the need for labels and obtain performance gains at low labeling cost. FixMatch also reports strong results with few labels (Abstract, p. 1). But the paper does not establish the claimed significant improvement over fully unsupervised approaches; its comparison to models trained without unlabeled data is a supervised baseline (B.8, p. 17). (LLM, conf=0.87)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 56

  1. [1][1] E. Arazo, D. Ortego, P. Albert, N. E. O'Connor, and K. McGuinness. Pseudo-labeling and confirmation bias in deep semi-supervised learning. arXiv preprint arXiv:1908.02983 , 2019. 5, 15 - · arXiv
  2. [2][2] P. Bachman, O. Alsharif, and D. Precup. Learning with pseudo-ensembles. In Advances in neural information processing systems , pages 3365-3373, 2014. 1, 3, 5 -
  3. [3][3] D. Berthelot, N. Carlini, E. D. Cubuk, A. Kurakin, K. Sohn, H. Zhang, and C. Raffel. Remixmatch: Semi-supervised learning with distribution matching and augmentation anchoring. In Eighth International Conference on Learning Representations , 2020. 1, 2, 3, 4, 5, 6, 7, 8, 18, 19, 20, 21 -
  4. [4][4] D. Berthelot, N. Carlini, I. Goodfellow, N. Papernot, A. Oliver, and C. A. Raffel. Mixmatch: A holistic approach to semi-supervised learning. In Advances in Neural Information Processing Systems , pages 5050-5060, 2019. 1, 2, 3, 5, 6, 7, 8, 14, 19 -
  5. [5][5] N. Carlini, Ú. Erlingsson, and N. Papernot. Distribution density, tails, and outliers in machine learning: Metrics and applications. arXiv preprint arXiv:1910.13427 , 2019. 7, 16 - · arXiv
  6. [6][6] O. Chapelle, B. Scholkopf, and A. Zien. Semi-Supervised Learning . MIT Press, 2006. 4 -
  7. [7][7] J. Chen and Q. Gu. Closing the generalization gap of adaptive gradient methods in training deep neural networks. arXiv preprint arXiv:1806.06763 , 2018. 15 - · arXiv
  8. [8][8] D. Choi, C. J. Shallue, Z. Nado, J. Lee, C. J. Maddison, and G. E. Dahl. On empirical comparisons of optimizers for deep learning. arXiv preprint arXiv:1910.05446 , 2019. 15 - · arXiv
  9. [9][9] A. Coates, A. Ng, and H. Lee. An analysis of single-layer networks in unsupervised feature learning. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics , 2011. 5 -
  10. [10][10] E. D. Cubuk, B. Zoph, D. Mane, V. Vasudevan, and Q. V. Le. Autoaugment: Learning augmentation strategies from data. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , June 2019. 4, 9, 20 -
  11. [11][11] E. D. Cubuk, B. Zoph, J. Shlens, and Q. V. Le. Randaugment: Practical automated data augmentation with a reduced search space. arXiv preprint arXiv:1909.13719 , 2019. 2, 4, 6, 7, 8, 18, 20 - · arXiv
  12. [12][12] Z. Dai, Z. Yang, F. Yang, W. W. Cohen, and R. R. Salakhutdinov. Good semi-supervised learning that requires a bad GAN. In Advances in Neural Information Processing Systems , 2017. 5 -
  13. [13][13] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A large-scale hierarchical image database. In IEEE Conference on Computer Vision and Pattern Recognition , 2009. 5 -
  14. [14][14] T. DeVries and G. W. Taylor. Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv:1708.04552 , 2017. 2, 4 - · arXiv
  15. [15][15] G. French, M. Mackiewicz, and M. Fisher. Self-ensembling for visual domain adaptation. In Sixth International Conference on Learning Representations , 2018. 4, 5 -
  16. [16][16] P. Goyal, P. Dollár, R. Girshick, P. Noordhuis, L. Wesolowski, A. Kyrola, A. Tulloch, Y. Jia, and K. He. Accurate, large minibatch sgd: Training imagenet in 1 hour. arXiv preprint arXiv:1706.02677 , 2017. 16 - · arXiv
  17. [17][17] Y. Grandvalet and Y. Bengio. Semi-supervised learning by entropy minimization. In Advances in neural information processing systems , 2005. 3, 5 -
  18. [18][18] D. Hendrycks and K. Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. arXiv preprint arXiv:1610.02136 , 2016. 15 - · arXiv
  19. [19][19] D. Hendrycks, N. Mu, E. D. Cubuk, B. Zoph, J. Gilmer, and B. Lakshminarayanan. Augmix: A simple data processing method to improve robustness and uncertainty. arXiv preprint arXiv:1912.02781 , 2019. 15 - · arXiv
  20. [20][20] J. Hestness, S. Narang, N. Ardalani, G. Diamos, H. Jun, H. Kianinejad, M. M. A. Patwary, Y. Yang, and Y. Zhou. Deep learning scaling is predictable, empirically. arXiv preprint arXiv:1712.00409 , 2017. 1 - · arXiv
  21. [21][21] R. Jozefowicz, O. Vinyals, M. Schuster, N. Shazeer, and Y. Wu. Exploring the limits of language modeling. arXiv preprint arXiv:1602.02410 , 2016. 1 - · arXiv
  22. [22][22] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In Third International Conference on Learning Representations , 2015. 4 -
  23. [24][24] S. Laine and T. Aila. Temporal ensembling for semi-supervised learning. In Fifth International Conference on Learning Representations , 2017. 1, 3, 5 -
  24. [25][25] D.-H. Lee. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In ICML Workshop on Challenges in Representation Learning , 2013. 1, 2, 3, 4, 5, 6 -
  25. [26][26] K. Lee, K. Lee, H. Lee, and J. Shin. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. In Advances in Neural Information Processing Systems , pages 7167-7177, 2018. 15 -
  26. [27][27] S. Liang, Y. Li, and R. Srikant. Enhancing the reliability of out-of-distribution image detection in neural networks. arXiv preprint arXiv:1706.02690 , 2017. 15 - · arXiv
  27. [28][28] I. Loshchilov and F. Hutter. SGDR: Stochastic gradient descent with warm restarts. In Fifth International Conference on Learning Representations , 2017. 4, 15 -
  28. [29][29] I. Loshchilov and F. Hutter. Decoupled weight decay regularization. In Sixth International Conference on Learning Representations , 2018. 15 -
  29. [30][30] D. Mahajan, R. Girshick, V. Ramanathan, K. He, M. Paluri, Y. Li, A. Bharambe, and L. van der Maaten. Exploring the limits of weakly supervised pretraining. In Proceedings of the European Conference on Computer Vision (ECCV) , 2018. 1 -
  30. [31][31] D. McClosky, E. Charniak, and M. Johnson. Effective self-training for parsing. In Proceedings of the main conference on human language technology conference of the North American Chapter of the Association of Computational Linguistics . Association for Computational Linguistics, 2006. 4 -
  31. [32][32] G. J. McLachlan. Iterative reclassification procedure for constructing an asymptotically optimal rule of allocation in discriminant analysis. Journal of the American Statistical Association , 70(350):365-369, 1975. 1, 3, 4 -
  32. [33][33] T. Miyato, S.-i. Maeda, S. Ishii, and M. Koyama. Virtual adversarial training: a regularization method for supervised and semi-supervised learning. IEEE transactions on pattern analysis and machine intelligence , 2018. 1, 3, 4, 5, 19 -
  33. [34][34] Y. E. Nesterov. A method of solving a convex programming problem with convergence rate o(kˆ2). Doklady Akademii Nauk , 269(3), 1983. 4 -
  34. [35][35] Y. Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, and A. Y. Ng. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning , 2011. 5 -
  35. [36][36] A. Oliver, A. Odena, C. A. Raffel, E. D. Cubuk, and I. Goodfellow. Realistic evaluation of deep semi-supervised learning algorithms. In Advances in Neural Information Processing Systems , pages 3235-3246, 2018. 2, 3, 5, 6, 9 -
  36. [37][37] B. Oreshkin, P. R. López, and A. Lacoste. Tadam: Task dependent adaptive metric for improved few-shot learning. In Advances in Neural Information Processing Systems , pages 721-731, 2018. 7 -
  37. [38][38] D. S. Park, W. Chan, Y. Zhang, C.-C. Chiu, B. Zoph, E. D. Cubuk, and Q. V. Le. Specaugment: A simple data augmentation method for automatic speech recognition. arXiv preprint arXiv:1904.08779 , 2019. 19 - · arXiv
  38. [39][39] H. Pham and Q. V. Le. Semi-supervised learning by coaching. Submitted to the 8th International Conference on Learning Representations , 2019. https://openreview.net/forum?id= rJe04p4YDB . 5 -
  39. [40][40] B. T. Polyak. Some methods of speeding up the convergence of iteration methods. USSR Computational Mathematics and Mathematical Physics , 4(5), 1964. 4 -
  40. [43][43] A. Rasmus, M. Berglund, M. Honkala, H. Valpola, and T. Raiko. Semi-supervised learning with ladder networks. In Advances in Neural Information Processing Systems , 2015. 5, 6 -
  41. [44][44] C. Rosenberg, M. Hebert, and H. Schneiderman. Semi-supervised self-training of object detection models. In Proceedings of the Seventh IEEE Workshops on Application of Computer Vision , 2005. 1, 4, 5 -
  42. [45][45] M. Sajjadi, M. Javanmardi, and T. Tasdizen. Mutual exclusivity loss for semi-supervised deep learning. In IEEE International Conference on Image Processing , 2016. 1, 3 -
  43. [46][46] M. Sajjadi, M. Javanmardi, and T. Tasdizen. Regularization with stochastic transformations and perturbations for deep semi-supervised learning. In Advances in Neural Information Processing Systems , 2016. 1, 3, 5 -
  44. [47][47] H. Scudder. Probability of error of some adaptive pattern-recognition machines. IEEE Transactions on Information Theory , 11(3), 1965. 1, 3, 4 -
  45. [49][49] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research , 15(1), 2014. 5 -
  46. [50][50] I. Sutskever, J. Martens, G. Dahl, and G. Hinton. On the importance of initialization and momentum in deep learning. In International conference on machine learning , 2013. 4 -
  47. [51][51] A. Tarvainen and H. Valpola. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. In Advances in neural information processing systems , 2017. 1, 3, 5, 6 -
  48. [52][52] V. Verma, A. Lamb, J. Kannala, Y. Bengio, and D. Lopez-Paz. Interpolation consistency training for semi-supervised learning. arXiv preprint arXiv:1903.03825 , 2019. 1 - · arXiv
  49. [53][53] A. C. Wilson, R. Roelofs, M. Stern, N. Srebro, and B. Recht. The marginal value of adaptive gradient methods in machine learning. In Advances in Neural Information Processing Systems , pages 4148-4158, 2017. 15 -
  50. [54][54] Q. Xie, Z. Dai, E. Hovy, M.-T. Luong, and Q. V. Le. Unsupervised data augmentation for consistency training. arXiv preprint arXiv:1904.12848 , 2019. 1, 2, 3, 4, 5, 6, 7, 8, 16, 20 - · arXiv
  51. [55][55] Q. Xie, M.-T. Luong, E. Hovy, and Q. V. Le. Self-training with noisy student improves imagenet classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10687-10698, 2020. 1, 4, 5 -
  52. [57][57] X. Zhai, A. Oliver, A. Kolesnikov, and L. Beyer. S4l: Self-supervised semi-supervised learning. In The IEEE International Conference on Computer Vision (ICCV) , October 2019. 7, 19 -
  53. [58][58] G. Zhang, C. Wang, B. Xu, and R. Grosse. Three mechanisms of weight decay regularization. arXiv preprint arXiv:1810.12281 , 2018. 15 - · arXiv
  54. [59][59] H. Zhang, M. Cisse, Y. N. Dauphin, and D. Lopez-Paz. MixUp: Beyond empirical risk minimization. arXiv preprint arXiv:1710.09412 , 2017. 4, 5, 19 - · arXiv
  55. [61][61] X. Zhu and A. B. Goldberg. Introduction to semi-supervised learning. Synthesis lectures on artificial intelligence and machine learning , 3(1), 2009. 4 -
  56. [62][62] Y. Zou, Z. Yu, B. Vijaya Kumar, and J. Wang. Unsupervised domain adaptation for semantic segmentation via class-balanced self-training. In Proceedings of the European Conference on Computer Vision (ECCV) , pages 289-305, 2018. 5

Cited by

Sorted by citations

Authors
Sohn, K., Berthelot, D., Li, C.-L., Zhang, Z., Carlini, N., Cubuk, E. D., Kurakin, A., Zhang, H., & Raffel, C.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2020
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2001.07685
arXiv
2001.07685
OpenAlex
W3001197829
Cited by
2,892
References
3
Citations by year
2026: 13 · 2025: 485 · 2024: 800 · 2023: 733 · 2022: 449 · 2021: 303 · 2020: 104 · 2019: 5
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (publishedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
Keywords
semi-supervised learning, FixMatch, consistency regularization, pseudo-labeling, confidence thresholding, unlabeled data, CIFAR-10

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review