The DOI matches its Crossref record; 43 of the 43 references it cites were found in a registry or an index.
Curriculum Labeling: Revisiting Pseudo-Labeling for Semi-Supervised Learning
Open accessAbstract
In this paper we revisit the idea of pseudo-labeling in the context of semi-supervised learning where a learning algorithm has access to a small set of labeled samples and a large set of unlabeled samples. Pseudo-labeling works by applying pseudo-labels to samples in the unlabeled set by using a model trained on combination of the labeled samples and any previously pseudo-labeled samples, and iteratively repeating this process in a self-training cycle. Current methods seem to have abandoned this approach in favor of consistency regularization methods that train models under a combination of different styles of self-supervised losses on the unlabeled samples and standard supervised losses on the labeled samples. We empirically demonstrate that pseudo-labeling can in fact be competitive with the state-of-the-art, while being more resilient to out-of-distribution samples in the unlabeled set. We identify two key factors that allow pseudo-labeling to achieve such remarkable results (1) applying curriculum learning principles and (2) avoiding concept drift by restarting model parameters before each self-training cycle. We obtain 94.91% accuracy on CIFAR-10 using only 4,000 labeled samples, and 68.87% top-1 accuracy on Imagenet-ILSVRC using only 10% of the labeled samples.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380
Learning Multiple Layers of Features from Tiny Images
Krizhevsky, A. (2024) · doi:10.57702/zp44cu3g · Cited by 23,395
A survey on Image Data Augmentation for Deep Learning
Shorten, C., & Khoshgoftaar, T. M. (2019) · Journal of Big Data · doi:10.1186/s40537-019-0197-0 · Cited by 13,203
Zagoruyko, S., & Komodakis, N. (2016) · Procedings of the British Machine Vision Conference 2016 · doi:10.5244/c.30.87 · Cited by 7,244
mixup: Beyond Empirical Risk Minimization
Zhang, H., Cisse, M., Dauphin, Y. N., & Lopez-Paz, D. (2017) · arXiv · doi:10.48550/arxiv.1710.09412 · Cited by 5,831
Bengio, Y., Louradour, J., Collobert, R., & Weston, J. (2009) · Proceedings of the 26th Annual International Conference on Machine Learning · doi:10.1145/1553374.1553380 · Cited by 5,366 · cited 1×
“Prior work has shown that training with a curriculum improves performance in several machine learning tasks (Bengio et al. 2009; Hacohen and Weinshall 2019).”
BackgroundSGDR: Stochastic Gradient Descent with Warm Restarts
Loshchilov, I., & Hutter, F. (2016) · arXiv · doi:10.48550/arxiv.1608.03983 · Cited by 5,309
Semi-Supervised Learning Literature Survey
Zhu, X. (2005) · Minds at UW (University of Wisconsin) · Cited by 4,506
Tarvainen, A., & Valpola, H. (2017) · arXiv · doi:10.48550/arxiv.1703.01780 · Cited by 4,275
Reading digits in natural images with unsupervised feature learning
Netzer, Y. (2024) · doi:10.57702/e7wwo0zk · Cited by 4,149
Striving for Simplicity: The All Convolutional Net
Springenberg, J. T., Dosovitskiy, A., Brox, T., & Riedmiller, M. (2014) · arXiv · doi:10.48550/arxiv.1412.6806 · Cited by 3,447
RandAugment: Practical automated data augmentation with a reduced search space
Cubuk, E. D., Zoph, B., Shlens, J., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1909.13719 · Cited by 3,223
FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence
Sohn, K., Berthelot, D., Li, C.-L., Zhang, Z., Carlini, N., Cubuk, E. D., Kurakin, A., Zhang, H., & Raffel, C. (2020) · arXiv · doi:10.48550/arxiv.2001.07685 · Cited by 2,892
Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learning
Miyato, T., Maeda, S.-I., Koyama, M., & Ishii, S. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2858821 · Cited by 2,843
Best practices for convolutional neural networks applied to visual document analysis
Simard, P. Y., Steinkraus, D., & Platt, J. C. (2005) · Seventh International Conference on Document Analysis and Recognition, 2003. Proceedings. · doi:10.1109/icdar.2003.1227801 · Cited by 2,741
ImageNet: A large-scale hierarchical image database
Deng, J. H., Dong, W. F., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvprw.2009.5206848 · Cited by 2,714
The Effectiveness of Data Augmentation in Image Classification using Deep Learning
Perez, L., & Wang, J. (2017) · arXiv · doi:10.48550/arxiv.1712.04621 · Cited by 2,499
Revisiting Unreasonable Effectiveness of Data in Deep Learning Era
Sun, C., Shrivastava, A., Singh, S., & Gupta, A. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.97 · Cited by 2,484
Scheirer, W. J., de Rezende Rocha, A., Sapkota, A., & Boult, T. E. (2013) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2012.256 · Cited by 1,573
Averaging Weights Leads to Wider Optima and Better Generalization
Izmailov, P., Podoprikhin, D., Garipov, T., Vetrov, D., & Wilson, A. G. (2018) · arXiv · doi:10.48550/arxiv.1803.05407 · Cited by 1,409
Temporal Ensembling for Semi-Supervised Learning
Laine, S., & Aila, T. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1610.02242 · Cited by 1,395
Interpolation consistency training for semi-supervised learning
Verma, V., Kawaguchi, K., Lamb, A., Kannala, J., Solin, A., Bengio, Y., & Lopez-Paz, D. (2022) · Neural Networks · doi:10.1016/j.neunet.2021.10.008 · Cited by 881
Realistic Evaluation of Deep Semi-Supervised Learning Algorithms
Oliver, A., Odena, A., Raffel, C., Cubuk, E. D., & Goodfellow, I. J. (2018) · arXiv · doi:10.48550/arxiv.1804.09170 · Cited by 772
Semi-Supervised Classification by Low Density Separation
Chapelle, O., Zien, A., & Ghahramani, R. C. Z. (2005) · Cited by 765
S4L: Self-Supervised Semi-Supervised Learning
Zhai, X., Oliver, A., Kolesnikov, A., & Beyer, L. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00156 · Cited by 750
Label Propagation for Deep Semi-Supervised Learning
Iscen, A., Tolias, G., Avrithis, Y., & Chum, O. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00521 · Cited by 686
ReMixMatch: Semi-Supervised Learning with Distribution Matching and Augmentation AnchoringPreprint
Berthelot, D., Carlini, N., Cubuk, E. D., Kurakin, A., Sohn, K., Zhang, H., & Raffel, C. (2020) · arXiv (Cornell University) · Cited by 600
Regularization With Stochastic Transformations and Perturbations for Deep Semi-Supervised Learning
Sajjadi, M. S. M., Javanmardi, M., & Taşdizen, T. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.04586 · Cited by 532
Probability of error of some adaptive pattern-recognition machines
Scudder, H. (1965) · IEEE Transactions on Information Theory · doi:10.1109/tit.1965.1053799 · Cited by 518
Smooth Neighbors on Teacher Graphs for Semi-Supervised Learning
Luo, Y., Zhu, J., Li, M., Ren, Y., & Zhang, B. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00927 · Cited by 264
Transductive Semi-Supervised Deep Learning Using Min-Max Features
Shi, W., Gong, Y., Ding, C., Ma, Z., Tao, X., & Zheng, N. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01228-1_19 · Cited by 215
Zhu, X. (2011) · Encyclopedia of Machine Learning · doi:10.1007/978-0-387-30164-8_749 · Cited by 193
Learning to recognize patterns without a teacher
Fralick, S. C. (1967) · IEEE Transactions on Information Theory · doi:10.1109/tit.1967.1053952 · Cited by 172
McLachlan, G. J. (1975) · Journal of the American Statistical Association · doi:10.1080/01621459.1975.10479874 · Cited by 167
Unsupervised Data AugmentationPreprint
Xie, Q., Dai, Z., Hovy, E. H., Luong, M.-T., & Le, Q. V. (2019) · arXiv (Cornell University) · Cited by 166
Learning with a probabilistic teacher
Agrawala, A. K. (1970) · IEEE Transactions on Information Theory · doi:10.1109/tit.1970.1054472 · Cited by 139
Novelty Detection with Multivariate Extreme Value Statistics
Clifton, D. A., Hugueny, S., & Tarassenko, L. (2010) · Journal of Signal Processing Systems · doi:10.1007/s11265-010-0513-6 · Cited by 114
Pseudo-Labeling and Confirmation Bias in Deep Semi-Supervised Learning
Arazo, E., Ortego, D., Albert, P., O'Connor, N. E., & McGuinness, K. (2020) · 2020 International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn48605.2020.9207304 · Cited by 77
Adaptive Threshold Estimation via Extreme Value Theory
Broadwater, J. B., & Chellappa, R. (2009) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2009.2031285 · Cited by 65
Automated Novelty Detection in Industrial Systems
Clifton, D. A., Clifton, L., Bannister, P. R., & Tarassenko, L. (2008) · Studies in computational intelligence · doi:10.1007/978-3-540-78297-1_13 · Cited by 29
Semi-Supervised Learning by Label Gradient AlignmentPreprint
Jackson, J., & Schulman, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.02336 · Cited by 19
Cited by
ST++: Make Self-trainingWork Better for Semi-supervised Semantic Segmentation
Yang, L., Zhuo, W., Qi, L., Shi, Y., & Gao, Y. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00423 · Cited by 518
FlexMatch: Boosting Semi-Supervised Learning with Curriculum Pseudo Labeling
Zhang, B., Wang, Y., Hou, W., Wu, H., Wang, J., Okumura, M., & Shinozaki, T. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.08263 · Cited by 466
Soviany, P., Ionescu, R. T., Rota, P., & Sebe, N. (2022) · International Journal of Computer Vision · doi:10.1007/s11263-022-01611-x · Cited by 431
Metadata
- Authors
- Cascante-Bonilla, P., Tan, F., Qi, Y., & Ordonez, V.
- Venue
- Proceedings of the AAAI Conference on Artificial Intelligence
- Published
- 2021-05-18
- Volume
- 35
- Issue
- 8
- Pages
- 6912-6920
- ISSN
- 2374-3468
- 2159-5399
- OpenAlex
- W3174697615
- Cited by
- 281
- References
- 43
- FWCI
- 17.29
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.