CoChat
Sign Up
PreprintCited by 772Machine Learning and AlgorithmsDomain Adaptation and Few-Shot LearningMachine Learning and Data Classification

Realistic Evaluation of Deep Semi-Supervised Learning Algorithms

Open access
Oliver, Avital, Odena, Augustus, Raffel, Colin, Cubuk, Ekin D., Goodfellow, Ian J.·2018·arXiv
DOI 10.48550/arxiv.1804.09170arXiv 1804.09170
Open this paper in CoChat — free accountOpen PDF

Abstract

Semi-supervised learning (SSL) provides a powerful framework for leveraging unlabeled data when labels are limited or expensive to obtain. SSL algorithms based on deep neural networks have recently proven successful on standard benchmark tasks. However, we argue that these benchmarks fail to address many issues that these algorithms would face in real-world applications. After creating a unified reimplementation of various widely-used SSL techniques, we test them in a suite of experiments designed to address these issues. We find that the performance of simple baselines which do not use unlabeled data is often underreported, that SSL methods differ in sensitivity to the amount of labeled and unlabeled data, and that performance can degrade substantially when the unlabeled dataset contains out-of-class examples. To help guide SSL research towards real-world applicability, we make our unified reimplemention and evaluation platform publicly available.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1804.09170

Cites

5 of 5 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Performance gains in semi-supervised learning saturate quickly beyond a modest label count.

    “This effect is consistent with earlier findings in semi-supervised learning theory and empirical studies that suggest performance gains saturate quickly beyond a modest label count [30].”

    From AnomalyMatch: Discovering Rare Objects of Interest with Semi-supervised and Active Learning (2025) · cites this paper as [30]

    Section 4.5 (p. 7) says the performance of the tested semi-supervised techniques tends to converge as the number of labels grows. It does not establish that gains saturate quickly beyond a modest label count; its explicit finding that performance levels off concerns 80,000 *unlabeled* examples, not labels. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 46

  1. [1][1] Bachman, Philip, Alsharif, Ouais, and Precup, Doina. Learning with pseudo-ensembles. In Advances in Neural Information Processing Systems , 2014. -
  2. [2][2] Belkin, Mikhail and Niyogi, Partha. Laplacian eigenmaps and spectral techniques for embedding and clustering. In Advances in Neural Information Processing Systems , 2002. -
  3. [3][3] Belkin, Mikhail, Niyogi, Partha, and Sindhwani, Vikas. Manifold regularization: A geometric framework for learning from labeled and unlabeled examples. Journal of machine learning research , 7(Nov):2399-2434, 2006. -
  4. [4][4] Ben-David, Shai, Blitzer, John, Crammer, Koby, Kulesza, Alex, Pereira, Fernando, and Vaughan, Jennifer Wortman. A theory of learning from different domains. Machine learning Journal , 2010. -
  5. [5][5] Bengio, Yoshua, Delalleau, Olivier, and Le Roux, Nicolas. Label Propagation and Quadratic Criterion , chapter 11. MIT Press, 2006. -
  6. [6][6] Chapelle, Olivier, Scholkopf, Bernhard, and Zien, Alexander. Semi-Supervised Learning . MIT Press, 2006. -
  7. [7][7] Chrabaszcz, Patryk, Loshchilov, Ilya, and Hutter, Frank. A downsampled variant of ImageNet as an alternative to the CIFAR datasets. arXiv preprint arXiv:1707.08819 , 2017. - · arXiv
  8. [8][8] Coates, Adam and Ng, Andrew Y. The importance of encoding versus training with sparse coding and vector quantization. In International Conference on Machine Learning , 2011. -
  9. [9][9] Dai, Andrew M. and Le, Quoc V. Semi-supervised sequence learning. In Advances in Neural Information Processing Systems , 2015. -
  10. [10][10] Deng, Jia, Dong, Wei, Socher, Richard, Li, Li-Jia, Li, Kai, and Fei-Fei, Li. ImageNet: A large-scale hierarchical image database. In IEEE Conference on Computer Vision and Pattern Recognition , 2009. -
  11. [11][11] DeVries, Terrance and Taylor, Graham W. Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv:1708.04552 , 2017. - · arXiv
  12. [13][13] Fedus, William, Rosca, Mihaela, Lakshminarayanan, Balaji, Dai, Andrew M., Mohamed, Shakir, and Goodfellow, Ian. Many paths to equilibrium: GANs do not need to decrease a divergence at every step. International Conference on Learning Representations , 2018. -
  13. [15][15] Gammerman, Alexander, Vovk, Volodya, and Vapnik, Vladimir. Learning by transduction. In Proceedings of the Fourteenth Conference on Uncertainty in Artificial Intelligence , 1998. -
  14. [16][16] Ganin, Yaroslav, Ustinova, Evgeniya, Ajakan, Hana, Germain, Pascal, Larochelle, Hugo, Laviolette, François, Marchand, Mario, and Lempitsky, Victor. Domain-adversarial training of neural networks. The Journal of Machine Learning Research , 2016. -
  15. [17][17] Gastaldi, Xavier. Shake-shake regularization. Fifth International Conference on Learning Representations (Workshop Track) , 2017. -
  16. [18][18] Golovin, Daniel, Solnik, Benjamin, Moitra, Subhodeep, Kochanski, Greg, Karro, John, and Sculley, D. Google vizier: A service for black-box optimization. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , 2017. -
  17. [19][19] Goodfellow, Ian J., Courville, Aaron, and Bengio, Yoshua. Spike-and-slab sparse coding for unsupervised feature discovery. In NIPS Workshop on Challenges in Learning Hierarchical Models , 2011. -
  18. [20][20] Goodfellow, Ian J., Shlens, Jonathon, and Szegedy, Christian. Explaining and harnessing adversarial examples. In Third International Conference on Learning Representations , 2015. -
  19. [21][21] Grandvalet, Yves and Bengio, Yoshua. Semi-supervised learning by entropy minimization. In Advances in Neural Information Processing Systems , 2005. -
  20. [22][22] He, Kaiming, Zhang, Xiangyu, Ren, Shaoqing, and Sun, Jian. Identity mappings in deep residual networks. In European Conference on Computer Vision , 2016. -
  21. [23][23] Henderson, Peter, Islam, Riashat, Bachman, Philip, Pineau, Joelle, Precup, Doina, and Meger, David. Deep reinforcement learning that matters. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence , 2018. -
  22. [24][24] Hoeffding, Wassily. Probability inequalities for sums of bounded random variables. Journal of the American statistical association , 58(301):13-30, 1963. -
  23. [26][26] Joachims, Thorsten. Transductive inference for text classification using support vector machines. In International Conference on Machine Learning , 1999. -
  24. [27][27] Joachims, Thorsten. Transductive learning via spectral graph partitioning. In International Conference on Machine Learning , 2003. -
  25. [28][28] Ke, Rosemary Nan, Goyal, Anirudh, Lamb, Alex, Pineau, Joelle, Bengio, Samy, and Bengio, Yoshua (eds.). Reproducibility in Machine Learning Research , 2017. -
  26. [29][29] Kingma, Diederik P. and Ba, Jimmy. Adam: A method for stochastic optimization. In Second International Conference on Learning Representations , 2014. -
  27. [30][30] Kingma, Diederik P., Mohamed, Shakir, Rezende, Danilo Jimenez, and Welling, Max. Semisupervised learning with deep generative models. In Advances in Neural Information Processing Systems , 2014. -
  28. [32][32] Laine, Samuli and Aila, Timo. Temporal ensembling for semi-supervised learning. In Fifth International Conference on Learning Representations , 2017. -
  29. [33][33] Lasserre, Julia A., Bishop, Christopher M., and Minka, Thomas P. Principled hybrids of generative and discriminative models. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition , 2006. -
  30. [34][34] Lee, Dong-Hyun. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In ICML Workshop on Challenges in Representation Learning , 2013. -
  31. [35][35] Lucic, Mario, Kurach, Karol, Michalski, Marcin, Gelly, Sylvain, and Bousquet, Olivier. Are GANs created equal? A large-scale study. arXiv preprint arXiv:1711.10337 , 2017. - · arXiv
  32. [36][36] Maas, Andrew L., Hannun, Awni Y., and Ng, Andrew Y. Rectifier nonlinearities improve neural network acoustic models. In International Conference on Machine Learning , 2013. -
  33. [37][37] McLachlan, Geoffrey J. Iterative reclassification procedure for constructing an asymptotically optimal rule of allocation in discriminant analysis. Journal of the American Statistical Association , 70(350):365-369, 1975. -
  34. [38][38] Melis, Gábor, Dyer, Chris, and Blunsom, Phil. On the state of the art of evaluation in neural language models. In Proceedings of the 6th International Conference on Learning Representations , 2018. -
  35. [39][39] Miyato, Takeru, Maeda, Shin-ichi, Koyama, Masanori, and Ishii, Shin. Virtual adversarial training: a regularization method for supervised and semi-supervised learning. arXiv preprint arXiv:1704.03976 , 2017. - · arXiv
  36. [40][40] Netzer, Yuval, Wang, Tao, Coates, Adam, Bissacco, Alessandro, Wu, Bo, and Ng, Andrew Y. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning , 2011. -
  37. [41][41] Odena, Augustus. Semi-supervised learning with generative adversarial networks. arXiv preprint arXiv:1606.01583 , 2016. - · arXiv
  38. [43][43] Pu, Yunchen, Gan, Zhe, Henao, Ricardo, Yuan, Xin, Li, Chunyuan, Stevens, Andrew, and Carin, Lawrence. Variational autoencoder for deep learning of images, labels and captions. In Advances in Neural Information Processing Systems , 2016. -
  39. [44][44] Rosenberg, Chuck, Hebert, Martial, and Schneiderman, Henry. Semi-supervised self-training of object detection models. In Proceedings of the Seventh IEEE Workshops on Application of Computer Vision , 2005. -
  40. [45][45] Sajjadi, Mehdi, Javanmardi, Mehran, and Tasdizen, Tolga. Mutual exclusivity loss for semisupervised deep learning. In IEEE International Conference on Image Processing , 2016. -
  41. [46][46] Sajjadi, Mehdi, Javanmardi, Mehran, and Tasdizen, Tolga. Regularization with stochastic transformations and perturbations for deep semi-supervised learning. In Advances in Neural Information Processing Systems , 2016. -
  42. [47][47] Salakhutdinov, Ruslan and Hinton, Geoffrey E. Using deep belief nets to learn covariance kernels for Gaussian processes. In Advances in Neural Information Processing Systems , 2007. -
  43. [48][48] Salimans, Tim, Goodfellow, Ian, Zaremba, Wojciech, Cheung, Vicki, Radford, Alec, and Chen, Xi. Improved techniques for training GANs. In Advances in Neural Information Processing Systems , 2016. -
  44. [49][49] Szegedy, Christian, Zaremba, Wojciech, Sutskever, Ilya, Bruna, Joan, Erhan, Dumitru, Goodfellow, Ian, and Fergus, Rob. Intriguing properties of neural networks. In Second International Conference on Learning Representations , 2014. -
  45. [50][50] Tarvainen, Antti and Valpola, Harri. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. Advances in Neural Information Processing Systems , 2017. -
  46. [53][53] Zhu, Xiaojin, Ghahramani, Zoubin, and Lafferty, John D. Semi-supervised learning using gaussian fields and harmonic functions. In International Conference on Machine Learning , 2003.

Cited by

Sorted by citations

Authors
Oliver, A., Odena, A., Raffel, C., Cubuk, E. D., & Goodfellow, I. J.
Affiliations
Google (United States) (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2018
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1804.09170
arXiv
1804.09170
OpenAlex
W2794523151
Cited by
772
References
5
Citations by year
2026: 5 · 2025: 43 · 2024: 69 · 2023: 102 · 2022: 122 · 2021: 170 · 2020: 143 · 2019: 102 · 2018: 14 · 2017: 1
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
Keywords
semi-supervised learning, deep neural network, benchmark evaluation, unlabeled data, real-world applicability

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review