The DOI matches its Crossref record; 33 of the 33 references it cites were found in a registry or an index.
Dual Student: Breaking the Limits of the Teacher in Semi-Supervised Learning
Open accessAbstract
Recently, consistency-based methods have achieved state-of-the-art results in semi-supervised learning (SSL). These methods always involve two roles, an explicit or implicit teacher model and a student model, and penalize predictions under different perturbations by a consistency constraint. However, the weights of these two roles are tightly coupled since the teacher is essentially an exponential moving average (EMA) of the student. In this work, we show that the coupled EMA teacher causes a performance bottleneck. To address this problem, we introduce Dual Student, which replaces the teacher with another student. We also define a novel concept, stable sample, following which a stabilization constraint is designed for our structure to be trainable. Further, we discuss two variants of our method, which produce even higher performance. Extensive experiments show that our method improves the classification performance significantly on several main SSL benchmarks. Specifically, it reduces the error rate of the 13-layer CNN from 16.84% to 12.39% on CIFAR-10 with 1k labels and from 34.10% to 31.56% on CIFAR-100 with 10k labels. In addition, our method also achieves a clear improvement in domain adaptation.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380
Squeeze-and-Excitation Networks
Hu, J., Shen, L., & Sun, G. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00745 · Cited by 35,667
(2014) · Cited by 33,419
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S., & Szegedy, C. (2015) · arXiv · doi:10.48550/arxiv.1502.03167 · Cited by 27,818
Distilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204
Aggregated Residual Transformations for Deep Neural Networks
Xie, S., Girshick, R., Dollar, P., Tu, Z., & He, K. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.634 · Cited by 12,465
Explaining and Harnessing Adversarial Examples
Goodfellow, I. J., Shlens, J., & Szegedy, C. (2014) · arXiv · doi:10.48550/arxiv.1412.6572 · Cited by 10,381
Combining labeled and unlabeled data with co-training
Blum, A., & Mitchell, T. (1998) · Proceedings of the eleventh annual conference on Computational learning theory · doi:10.1145/279943.279962 · Cited by 5,495
(2006) · The MIT Press eBooks · doi:10.7551/mitpress/9780262033589.001.0001 · Cited by 4,280
Tarvainen, A., & Valpola, H. (2017) · arXiv · doi:10.48550/arxiv.1703.01780 · Cited by 4,275
Reading digits in natural images with unsupervised feature learning
Netzer, Y. (2024) · doi:10.57702/e7wwo0zk · Cited by 4,149
Semi-Supervised Learning Literature Survey
Zhu, X. (2005) · Minds at UW (University of Wisconsin) · Cited by 3,844
Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learning
Miyato, T., Maeda, S.-I., Koyama, M., & Ishii, S. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2858821 · Cited by 2,843
Acceleration of Stochastic Approximation by Averaging
Polyak, B. T., & Juditsky, A. B. (1992) · SIAM Journal on Control and Optimization · doi:10.1137/0330046 · Cited by 1,827
Averaging Weights Leads to Wider Optima and Better Generalization
Izmailov, P., Podoprikhin, D., Garipov, T., Vetrov, D., & Wilson, A. G. (2018) · arXiv · doi:10.48550/arxiv.1803.05407 · Cited by 1,409
Temporal Ensembling for Semi-Supervised Learning
Laine, S., & Aila, T. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1610.02242 · Cited by 1,395
Creating artificial neural networks that generalize
Sietsma, J., & Dow, R. J. F. (1991) · Neural Networks · doi:10.1016/0893-6080(91)90033-2 · Cited by 597
Semi-Supervised Learning with Ladder Networks
Rasmus, A., Harri Valpola, B., Honkala, M., Berglund, M., & Raiko, T. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1507.02672 · Cited by 527
Semi-Supervised Learning with Generative Adversarial Networks
Odena, A. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.01583 · Cited by 509
Deep Co-Training for Semi-Supervised Image Recognition
Qiao, S., Shen, W., Zhang, Z., Wang, B., & Yuille, A. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01267-0_9 · Cited by 507
Batch Renormalization: Towards Reducing Minibatch Dependence in Batch-Normalized Models
Ioffe, S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1702.03275 · Cited by 367
Learning with Pseudo-Ensembles
Bachman, P., Alsharif, O., & Precup, D. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1412.4864 · Cited by 329
WebVision Database: Visual Learning and Understanding from Web Data
Li, W., Wang, L., Wei, L., Agustsson, E., & Gool, L. V. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1708.02862 · Cited by 278
There Are Many Consistent Explanations of Unlabeled Data: Why You Should AveragePreprint
Athiwaratkun, B., Finzi, M., Izmailov, P., & Wilson, A. G. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.05594 · Cited by 278
Smooth Neighbors on Teacher Graphs for Semi-Supervised Learning
Luo, Y., Zhu, J., Li, M., Ren, Y., & Zhang, B. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00927 · Cited by 264
Efficient Knowledge Distillation from an Ensemble of Teachers
Fukuda, T., Suzuki, M., Kurata, G., Thomas, S., Cui, J., & Ramabhadran, B. (2017) · Interspeech 2017 · doi:10.21437/interspeech.2017-614 · Cited by 243
Auxiliary Deep Generative Models
Maaløe, L., Sønderby, C. K., Sønderby, S. K., & Winther, O. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1602.05473 · Cited by 241
Tri-net for Semi-Supervised Deep Learning
Chen, D., Wang, W., Gao, W. D., & Zhou, Z. (2018) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2018/278 · Cited by 153
When semi-supervised learning meets ensemble learning
Zhou, Z. (2011) · Frontiers of Electrical and Electronic Engineering in China · doi:10.1007/s11460-011-0126-2 · Cited by 90
Realistic Evaluation of Semi-Supervised Learning Algorithms.
Oliver, A., Odena, A., Raffel, C., Cubuk, E. D., & Goodfellow, I. (2018) · International Conference on Learning Representations · Cited by 59
Adversarial Dropout for Supervised and Semi-Supervised Learning
Park, S., Park, J.-K., Shin, S., & Moon, I. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11634 · Cited by 56
Self-ensembling for domain adaptation.
French, G., Mackiewicz, M., & Fisher, M. H. (2017) · Cited by 32
Cited by
Semi-Supervised Semantic Segmentation with Cross Pseudo Supervision
Chen, X., Yuan, Y., Zeng, G., & Wang, J. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00264 · Cited by 1,209
A Survey on Deep Semi-Supervised Learning
Yang, X., Song, Z., King, I., & Xu, Z. (2023) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2022.3220219 · Cited by 923
Wang, L., & Yoon, K.-J. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3055564 · Cited by 752
Pham, H., Dai, Z., Xie, Q., & Le, Q. V. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01139 · Cited by 509
Ge, Y., Chen, D., & Li, H. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2001.01526 · Cited by 370
Metadata
- Authors
- Ke, Z., Wang, D., Yan, Q., Ren, J., & Lau, R.
- Venue
- 2019 IEEE/CVF International Conference on Computer Vision (ICCV)
- Publisher
- IEEE
- Published
- 2019-10
- Pages
- 6727-6735
- OpenAlex
- W2990231018
- Cited by
- 271
- References
- 33
- FWCI
- 10.22
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.